Search Images Maps Play YouTube News Gmail Drive More »
Sign in
Screen reader users: click this link for accessible mode. Accessible mode has the same essential features but works better with your reader.


  1. Advanced Patent Search
Publication numberUSRE34476 E
Publication typeGrant
Application numberUS 07/889,664
Publication dateDec 14, 1993
Filing dateMay 28, 1992
Priority dateMay 14, 1990
Fee statusPaid
Publication number07889664, 889664, US RE34476 E, US RE34476E, US-E-RE34476, USRE34476 E, USRE34476E
InventorsDonald D. Norwood
Original AssigneeNorwood Donald D
Export CitationBiBTeX, EndNote, RefMan
External Links: USPTO, USPTO Assignment, Espacenet
Hybrid information management system for handwriting and text
US RE34476 E
A computer system used with special input hardware for providing the benefits of computerization to handwritten as well as keyboard entered information to meet a wide spectrum of commonly-occurring, written information processing needs. The input hardware consists of a transparent digitizing tablet mounted atop a flat panel display screen such that a user can handwrite directly upon the display screen with an attached pen. Application program functions include appointment documentation and work scheduling, handwritten or text file creation and manipulation, author-to-typist liaison functions for document production, handwritten annotation to text, handwritten note creation from text, and text document creation from handwritten source material.
Previous page
Next page
I claim:
1. A computer information system for handwriting and text, comprising:
a) a computer including a non-volatile storage area and a display;
b) means, including a digitizing pen for indicating positional coordinate information of said pen as an image on said display when said pen is moved, and for initiating a plurality of commands to said computer based upon the position of said image on said display;
c) means for entering digital information into said computer;
d) means for entering handwritten and text information into said storage area and displaying said information on said display;
e) means for creating, in said storage area, handwriting files which accept handwritten information as well as text .[.and which can be calendar-sequenced upon user direction, and can have date indicia and positional format areas, said date indicia and said positional format areas being in a calendar format.].;
f) means for manipulating said handwritten and text information within and between said handwriting files in a plurality of different ways in accordance with said respective plurality of commands;
g) means for indexing, storing, retrieving, copying and deleting said handwriting files in said computer;
h) means for creating a program object image on said display, said program object image representing a visual element of a programming system, which is created by a command to said programming system, which appears as an image on said display, which possesses one of a plurality of programmer-controllable physical shapes or sizes, including that of a button object image, which can be moved to any location on said display, which can have program commands associated with it such that a program associated with said program object image can be activated by a user to perform specific program actions;
an improvement comprising:
i) means for allowing said user (1) to create, and place one or more of said program object images on an image of one of said handwriting files on said display, and (2) to associated one or more of said program object images with any specific section of said handwritten information; whereby labor savings or other beneficial actions produced by said user's activation of said program object image associated with said specified handwritten information can be derived repeatedly without repeated creation of said program object image or the commands that it automates;
j) means which can be activated by said user to move as a group only said specific sections of handwritten information and some or all of said program object images associated with said specified section of handwritten information to a specific location within or across said handwriting files which accept handwriting, such that the associations between said specific section of handwritten information and said program object images are preserved; whereby the substantial labor to recreate said linkages can be eliminated which otherwise would be necessary for said user to derive the benefit of said associations in another file location or in a different file; and
k) means for identifying precisely said specific section of handwritten information which may be carried out by one of a plurality of techniques, including encirclement of the desired material by the user with a line drawing cursor.
2. The system of claim .[.1.]. .Iadd.22.Iaddend., further including a means for uploading and downloading information between a portable computer and a non-portable computer.
3. The system of claim .[.1.]. .Iadd.22.Iaddend., further including means for jumping from .[.one of the following:.]. said handwriting file .[.and said calendar-sequenced file,.]. to another of the following: said handwriting files .[.and said calendar-sequenced files.]. for temporary processing, then returning to the departure position of said one file.
4. The system of claim .[.1.]. .Iadd.22.Iaddend., further including a graphic index of said handwriting files .[.and said calendar-sequenced files.]., said index being composed of elements which, when cursor selected, retrieve an associated file.
5. The system of claim .[.4.]. .Iadd.1 .Iaddend.further including means which can be activated by a single user command for moving multiple selected specified sections of said handwritten information and their respective associated program object images to a specific location within or across said handwriting files, such that the associations between said specific section of handwritten information and said respective program object images are preserved; whereby the substantial labor needed to recreate the multiple associations between numerous specified sections of handwritten information and their respective associated program objects may be eliminated.
6. The system of claim .[.4.]. .Iadd.1 .Iaddend.wherein said association of said specific section of handwritten information and said program object image is carried out by one of a plurality of programming techniques, including physical placement by said user of said program object image adjacent to said specific section of handwritten information; said physical placement being carried our by one of a plurality of known means, including acquiring said programming object image with the cursor-positioning device, dragging said program object image to the desired location and releasing it there.
7. The system of claim .[.4.]. .Iadd.1 .Iaddend.wherein said program object image appears as a button which can have program commands associated with it such that a program associated with said button can be activated by the user to perform specific program actions.
8. The system of claim .[.4.]. .Iadd.1 .Iaddend.wherein the activation of said programming object image is carried out by said user placing the cursor on said program object image and pressing a button on the cursor positioning device.
9. The system of claim .[.4.]. .Iadd.1 .Iaddend.wherein said program object image associated with said specific section of handwritten information appears as a button (104 in FIG. 5) which, when activated by said user retrieves a file, whereby a user can associate, with said specific section of handwritten information, a button which when activated retrieves a computer file containing information which can be handwritten or textual or both and which is related to the associated handwritten notation.
10. The system of claim .[.4.]. .Iadd.1 .Iaddend.wherein said means which can be activated by said user to move a plurality of said specific sections of handwritten information and any subset of said program object images respectively associated with said specific sections of handwritten information to specific locations within or across said .[.regular or calendar-sequenced.]. files which accept handwriting is further arranged to optionally modify the relative positioning at the target location of the items being moved upon command of said user; whereby said user can update handwritten notes and their associated benefit-producing buttons without fear of obliterating notes and objects on said target material.
11. The system of claim .[.4.]. .Iadd.1.Iaddend., further including a graphic index of said handwriting files .[.and said calendar-sequenced files.]., said index being composed of elements which, when cursor selected, retrieve an associated file.
12. The system of claim .[.4.]. .Iadd.1.Iaddend., further including means for uploading and downloading information from said computer to another computer.
13. The system of claim .[.4.]. .Iadd.1.Iaddend., further including means for jumping from one of .[.the following:.]. said handwriting file .[.and said calendar-sequenced file.]. to another of .[.the following:.]. said handwriting files .[.and said calendar-sequenced files.]. for temporary processing, then returning to the departure position of said one file.
14. The system of claim .[.4.]. .Iadd.1.Iaddend., further including means for associating a date with said handwriting files such that said file is automatically retrieved when said dates become contemporaneous or later than the current date.
15. The system of claim .[.4.]. .Iadd.1.Iaddend., further including means for linking a plurality of records containing handwriting in said handwriting files such that said plurality of records may be filed as single record in said handwriting files in said computer.
16. The system of claim .[.4.]. .Iadd.1.Iaddend., further including means for executing a program external to said system without quitting said system.
17. The system of claim .[.4.]. .Iadd.1.Iaddend., further including means for automatically placing current time and date information into one of said handwriting files.
18. The system of claim .[.4.]. .Iadd.1.Iaddend., further including means for searching through a plurality of said handwriting files to retrieve one of said handwriting files having a piece of embedded text identical to a sought-after search text.
19. The system of claim .[.4.]. .Iadd.1.Iaddend., wherein said means for identifying precisely said specific section of handwritten information is arranged to select all handwriting in a predefined fixed area on a particular file screen format which displays the handwriting. (FIG. 5).
20. The system of claim .[.4.]. .Iadd.1.Iaddend., wherein said program object image further combines the data entry and storage characteristics of a programming language field element which can accept user supplied data entered by a plurality of means including selection of letters and numbers from a set of alphanumeric images displayed on the screen, and which also evidences the action-triggering characteristics of a button object and which can further be characterized as a field-button; whereby activation of said program object image by said user performs a beneficial action which relies upon the user-supplied information provided to said program object image.
21. The system of claim 20, wherein said program object image is arranged to dial a telephone number supplied to the field-button object by said user when said computer is connected to a modem, whereby said user can associated with said specific section of handwritten information a field-button .[.(105 in FIG. 5).]. which when clicked on by said user can cause the number which has been entered into the field-button to be dialed.
22. In a computer information system for handwriting and text, comprising:
a) a computer including a non-volatile storage area and a display;
b) means, including a digitizing pen for indicating positional coordinate information of said pen as an image on said display when said pen is moved, and for initiating a plurality of commands to said computer based upon the position of said image on said display;
c) means for entering digital information into said computer;
d) means for entering handwritten and text information into said storage area and displaying said information on said display;
e) means for creating in said storage area, handwriting files which accept handwritten information as well as text;
f) means for manipulating said handwritten and text information within and between said files in a plurality of different ways in accordance with said respective plurality of commands; and
g) means for indexing, storing, retrieving, copying and deleting said files which accept handwritten information as well as and text in said computer; and
h) a keyboard and means for typing into one of said handwriting files;
the improvement comprising:
i) means for creating files which accept handwriting and text and which display a two-part format, one part comprising an area for handwriting, the other a scrolling text window which accepts information entered by means of said keyboard or displays previously created text, whereby a user can (1) handwrite notes in said handwriting area and (2) view text displayed in said scrolling text window; and
j) means for editing text in said text window with a plurality of commands, including at least cut, copy, paste, and wordwrap function commands.

This is a reissue of U.S. Pat. No. 5,063,600..Iaddend.


1. Field of Invention

This invention relates to computers, specifically to a computer information system which accepts handwritten as well as keyboard input.

2. Description of Prior Art

Keyboard Interface--an Impediment to Greater Realization of Computer Benefits

Personal computers possess the capability for storing, retrieving, and manipulating information more rapidly, accurately and efficiently than do corresponding manual methods. Yet despite their widespread implementation, with limited exceptions, their benefits have not significantly affected the bulk of the daily "mainstream" written information processing of most users.

The phrase "mainstream written information processing" is useful to describe written information processing associated with information items which occur frequently in the course of a normal day, and associated activities such as storage, retrieval, transmission, and manipulation. Thus, the mainstream written information commerce of a homemaker may be grocery lists, a list of things to do, reminders, miscellaneous notes, check writing and calendar annotations. That of a business employee may embrace a wide plurality of items, such as transmittals (memos) to others, messages, thoughts committed to a scratch pad, notes from meetings or telephone conversations, "to-do" lists, appointments, and a wide variety of materials to be typed by a secretary, such as letters, descriptions, reports, etc.

Significant gains in productivity have been achieved (1) with custom or specialized PC applications, and (2) in areas such as word processing, spreadsheet usage and desktop publishing. However, in comparison with the use of pencil and paper, the relative utilization of such applications among white-collar workers is relatively infrequent. Thus, a user may find a word processor to be the court of last resort when the secretary cannot type up an urgently needed communication, or use a spreadsheet program periodically for quarterly budget calculations. Even though there are a few high-frequency application areas, such as calendaring, for which PC software has been developed, in practice, users have generally resisted attempts to provide computerized assistance for the bulk of high-volume information commerce activities, such as those mentioned above. Thus, despite the large number of personal computers sold, little headway has been made in utilizing the computer to improve the efficiency of most user's mainstream everyday information processing. The truth of the observation that the potential computer benefits are not being realized for much of our daily business information processing is apparent, not only through close observation of business practice, but appears also to be borne out by economic statistics. It is widely accepted among contemporary economists that the productivity of white collar workers has not significantly increased over the past decade (1979-1989), despite the introduction into the business sector of the major portion of the 25 million or more personal computers which have been sold in the United States.

There would undoubtedly be many more computer users and more numerous applications targeted at common information processing tasks such as those mentioned above if a keyboard were not required to enter data. Many people don't know how to type effectively and, for any of a wide variety of reasons, don't want to learn. Typical examples include executives, housewives, older people, children and many professionals. The substantial learning requirement thus constitutes a significant drawback. In addition, the typical computer keyboard (known as the QWERTY keyboard) was purposely arranged by its inventors, Sholes et al, in an inefficient manner so as to purposely impede typing speed to prevent the crude keys of the time from jamming. The QWERTY keyboard itself thus deters novices from using it. While it is possible to type with one finger, users perceive little or no benefit from one-finger typing. Indeed, even proficient typists find that for many potential applications, such short notes, reminders, calendar annotations, expense record-keeping, personal data documentation or messages (i.e., items which constitute the bulk of their daily written information processing, the typing effort simply outweighs the advantages.

Even though typing can be faster than handwriting, the relatively precise positioning of the body, hands, and fingers required for typing frequently make it particularly inconvenient for short written work. Imagine the difficulty of trying to take typewritten notes while using a conventional telephone. Resistance toward computer usage among non-typists is further compounded by having to deal with a personal computer interface which, in the majority of cases, is characterized by novice users as "non-intuitive", "unfriendly", and "unnecessarily complex".

Another significant drawback to computer systems which do not admit handwritten data relates to the fact that their storage files generally may not contain handwritten data. Because most real-world business files contain some handwritten information, or at least handwritten annotations, a second set of manual storage files must be maintained. Significant potential benefits of computerization, such as instant, labor-less, retrieval and storage are thus not realized. Employment of a keyboard as the entry mechanism also limits the utility of very small portable computers because the restricted keyboard size makes data entry very inefficient. Thus, one could not effectively enter meeting notes on a current computer designed to fit into a vest pocket. There are other disadvantages associated with the keyboard entry mechanism. Different keyboards are required for cultures with different character sets. Also, the keyboard can not provide the communicative power of a drawing.

Generally, it seems preferable to carry out most written information processing using a pencil or pen and a variety of manual storage and retrieval devices. These include scratch pads, file folders and drawers, our all-important appointment book and "to do" lists, and, of course our desk top, heaped high with reports, papers, notes, reminders, at least one stack of magazines and other paraphernalia indicative of the rather chaotic way we carry out our information handling. Thus, in spite of the many potential benefits the personal computer can offer, we continue to conduct most of our information processing with paper, pencil, and other manual instruments.

Thus, it is clear that heretofore the requirement that information be entered via a keyboard represents a significant impediment to greater computer usage in terms of: 1) number of users; 2) number of beneficial applications; and 3) derivation of consequent productivity benefits.

Feasibility of Handwriting into the Computer

Conceptually, an interface which allowed users to handwrite information into the computer could alleviate these drawbacks. However, until recently, this idea was not greeted with much enthusiasm for several reasons. First, the required hardware accompaniments to enter and display such graphics were excessively costly or rather crude. Secondly, there seemed to be a general sentiment which suggested that information stored in a computer must be digitalized to be useful, and a practical system to digitize (i.e. recognize) handwriting (as opposed to printing) had not been developed, nor has it yet.

In the last several years, significant advances in hardware performance and dramatic reductions in costs have combined to make the physical entry of handwriting into a personal computer feasible and practical. It has become possible to combine a flat panel display of significantly high resolution with a high-resolution, transparent, digitizing tablet to develop what might be described as a screen tablet. Such a device represents a combined display and entry device about the size of a tablet of paper and approximately 2.5 cm. thick and may be placed on the desk or lap for use. The screen tablet has a pen which is attached by a wire to the digitizer and functions in a manner identical to the pen used with graphic tablets allowing the user to "write" or draw on the transparent digitizing surface, and have the writing show up on the flat panel display which underlies the digitizing surface. Such devices are available through several commercial sources including MicroTouch Inc. of Woburn, Mass. or Toishiba Advanced Systems Division of Irvine, Calif. However, because an appropriate information management system has not been available, their potential has not been fully exploited as an entry means to a system which could provide computerized assistance to the major share of a user's daily written information processing.

The feasibility of handwriting into the computer is made more apparent by the realization that it is not even necessary to have a keyboard to supply the limited amount of digitization needed to index graphical handwritten information. I have made an analysis of the items which constitute the bulk of our daily information processing. This analysis indicated that only a very minor part of the information (or an adjunct to it such as a file description) needed to be digitized to accomplish the desired information processing functions. For example, to send a note or memo to a colleague, only the electronic address of the intended receiver need be in digital format; the text or body of the information can remain in the graphic form in which it was entered and will be displayed by the recipient exactly as a handwritten note on his display. Additionally, the very minor amount of information which must be supplied in a digital format can be either selected from a list of menu items (such as the names of users on a network) or "keyed" in by touching graphic representations of "keys" pictured on the display. Thus, if a user wished to store handwritten notes taken during a telephone conversation, it can be accomplished by simply touching a sequence of graphic "keys" on the display to describe a desired file name, followed by another graphic button entitled "FILE".

Finally, development of handwriting interfaces has been facilitated by the appearance of software development systems which provide extensive capabilities for merging graphic and textual computer representations and offer the computer developer macro tools for manipulating graphics and text in what appears to be an almost seamless way. In summary, keyboards represent an impediment to realization of computer benefits and, they are unnecessary. It has become feasible, practical and highly desireable to handwrite directly into the computer.

Patents on Handwriting in Computers

Heretofore, a variety of approaches were made to apply handwriting in limited areas as opposed to the totality of the individual's daily written information processing. The following patents deal with handwriting editing (principally, reformatting to improve legibility or more remotely character recognition and combination display screen/graphic tablet technology. Fox et al, (in U.S. Pat. No. 4,727,588, Feb. 23, 1988) shows a system for handwriting duplication for multiple printed copies. Hardin Sr. et al, (in U.S. Pat. No. 4,817,034, Mar. 28, 1989) shows a system for document editing. Barker et al, (U.S. Pat. No. 4,815,029, Mar. 21, 1989) shows a system which allows manipulation of both graphic and textual images similar to text publishing programs now popular. None of these systems seek to apply handwriting computer technology to areas of high volume daily information processing where it could do the most good.

Handwriting Digitization Efforts

Interest in the use of handwriting in personal computers is definitely on the increase. The most notable area involves character recognition of hand printed characters. The erroneous belief apparently still persists that in order to be useful, information stored within a computer must be digitized. There are both significant research activities and commercial manifestations of this interest by parties at Linus Technologies, Reston, Va., Grid Computer Systems, Fremont, Calif., and at the IBM Research Laboratories in Yorktown Heights, N.Y. Although recognition of hand printed characters possesses the desirable property of being a very natural entry medium it has numerous drawbacks. It generally requires that the user pattern-train the computer to recognize the individual's printed characters. Without training, recognition errors are often excessive. This then, restricts entry to users who have "trained" the computer. In addition, entry errors often occur if the user varies the way the character is drawn. Finally, greater computer processing resources (CPU cycles) are required than with other entry mechanisms. In short, it is questionable whether much net benefit derives from the digitization of hand-printed characters through character recognition methods, particularly, when there exist other means for entering characters which possess fewer drawbacks, yet do not require a keyboard.

Handwriting Annotation to Text

A more closely related manifestation of handwriting computerization has been developed for personal computers by Wang Laboratories of Lowell, Mass. Wang's system, sold under the trademark Freestyle, consists of a graphic tablet, a plug-in expansion board and accompanying software. Its primary purpose is to enable a user to annotate captured screen images both with handwritten notes and voice messages. The system works by performing a screen capture of whatever is displayed, suspending operation of the current application, and invoking the Freestyle software. The user may then annotate the captured display either with handwritten graphics or with voice supplied through a modem connected telephone.

By using a special display image of a blank "piece of paper", a handwritten note may be created. The blank sheets also possess a very rudimentary capability to accept keyboard entered text (no editing except backspace erase). Such a composite file consisting of text (or other application data), voice and graphic elements may be stored, retrieved, or transmitted to other users. Similar such composite file objects may also be attached and detached to one another.

By means of a recording and playback feature, handwritten notes and voice annotation may be recorded, for example, on a spreadsheet document sent by another user. These "comments" may then be passed along to another user, or returned to the originator. Additional functions include the capability to organize iconic images representing the handwritten notes or annotated sheets by moving them around upon a "desktop" interface display. They may also be copied, printed, deleted, aggregated (or separated). While the Wang work represents a significant effort to introduce handwriting into the computer, it evidences substantial drawbacks and limitations.

The Wang system is narrowly focused on a business environment, most likely at the management level, as opposed to being a general purpose tool for simply any computer user. In particular, the system is marketed as a component of the Wang office computing environment. Its focus is further limited in that the functions are oriented more toward "annotation" of existing documents than to assumption of all the users handwritten information commerce. For example, Wang marketing information illustrates capabilities for applying a "handwritten" signature to a travel authorization form or annotating a budget spreadsheet received from a co-worker. In particular, the system effectively requires connection to a Local Area Network (LAN) (or have LAN server software installed on the computer) to file (store) substantial amounts of information. Each individual file possesses an icon on the desk-top interface. The available desktop space is reduced by the creation of each new document unless the documents are stacked. This is a disadvantageous solution because the "stacks" would have to be un-stacked to identify their member elements. Thus they are moved (stored) off the desktop in a "file cabinet" which constitutes a sub-directory on a LAN file server. Its business heritage is further evidenced by a feature which allows a user to see the history of a document's movement from among users. That is, the focus of the system is to track documents through their life cycle while preserving anecdotal additions. It has neither the tools nor capability to assume a user's entire handwritten information commerce.

The system is also designed strictly for one-time annotation or creation of an handwritten note. There is no capability for manipulating elements of handwritten material or for subsequent modification. Once the handwritten material is "saved", i.e., committed to file storage status, it cannot be modified, only erased totally. This prevents the system from being used with information in an appointment book or "to-do" list format where the capability for updating an information element is critical. This limitation upon the manipulation of handwritten information elements also effectively prevents the facile organization and combination of multiple handwritten information elements which may be contained as sub-elements of other handwritten information sources into a combined handwritten document. Such a capability is key to assembly of a report, outline or paper drawing on separately stored information sub-elements (ideas, facts and other). Because this kind of information assembly is a very prominent information processing activity, it represents a severe limitation of the system.

The inability of the Wang system to allow substantial revision of text also makes impossible a very desirable use of computerized handwritten information. That is the ability to revise textual material based upon handwritten notes committed directly on or around the text. For example, it is highly desirable to "edit" a typewritten letter or other text document by making handwritten notes in the margins or within the text itself such that the typist or originator can make changes to the text based upon the handwritten notes.

The Wang system also relies on a graphic tablet and a conventional monitor which presents a hand-to-eye coordination problem for handwriting. The display image of the handwriting is dislocated from the writing surface. Because the Wang system requires a keyboard, it is not suitable for use on tiny vest-pocket-size computers as miniaturized keyboards cannot effectively be used to enter and significant volume of written information. "Paging through" individually created notes on the Wang system is tedious because associated documents in a single pile which are not aggregated into a single file, must be un-stacked one by one. Also, there is no way to separate text and graphics. The Wang system also has no drawing tools. In short the Wang system is not a general purpose computerized handwriting system and possesses numerous limitations and drawbacks.


The drawbacks and limitations of prior-art systems can be summarized as follows. As the sole general purpose vehicle for data entry, the keyboard constrains the proliferation of computer benefits by limiting the numbers of users and the number of feasible applications. It also promotes duplication of file storage systems by requiring separate storage for handwritten items and requires the use of paper for handwritten annotations. Previous efforts to implement a partial handwriting interface, while, representing a step forward, manifest the drawbacks described above. Prior systems have concentrated upon the applying computer assistance to only relatively isolated pieces of the individuals information commerce. This approach does not attempt to accommodate the entire spectrum of individual information processing needs and thereby forgoes the opportunity for synergistic benefits which could complement specific application gains.

Systems developed heretofore have not been of a comprehensive nature and as such do not enjoy extensive positive feedback which magnifies benefits. By including capabilities for handling almost all types of information processing, extensive use of the system is encouraged, resulting in correspondingly more benefits. In addition some of the previous work to computerize daily information processing such as applications for appointment and calendar recording, reminder notations and short note documentation, have not represented workable solutions because the data entry effort is usually greater than the payoff.


Accordingly, the several objects of the present invention are to provide a computer system which: 1) allows both entry of handwritten data into the computer and operation of the computer by means of a pen; 2) may be alternately operated by means of a conventional keyboard; and 3) can manipulate handwritten data or textual information to accommodate an unrestricted spectrum of commonly-encountered daily information processing tasks.

Such a system overcomes the usage difficulties of keyboard systems thereby prompting the extension of computer benefits such as speed of information retrieval, storage, transmission, and manipulation to broad classes of new users. It also makes feasible many new uses. Virtually all of the vast amount of commonly-experienced, daily, written information processing currently carried out by hand, may now benefit from computer assistance. It further allows extension of computer benefits associated with full-sized keyboard systems to new miniaturized keyboard-less hardware such as tiny vest-pocket-sized computers. Specific benefits include significant labor savings, reduction in information processing throughput time, and cost savings associated with reductions in file storage requirements and paper usage. Advantages also include the following quality benefits: reductions in lost information and learning requirements; enhanced communication through use of drawings; and simultaneous recording of information on paper and in the computer. Further objects and advantages will become apparent from consideration of the ensuing description and drawings.


In the drawings, closely related figures have the same number but different alphabetic suffixes.

FIG. 1A shows a system diagram of a non-portable embodiment of a computer system employing a screen tablet, a computer and peripherals used in accordance with the present invention including a detailed view of the screen tablet.

FIG. 1B shows detail of a portable vest-pocket size embodiment of the screen tablet and a miniature computer used in accordance with an alternative embodiment of the present invention.

FIG. 2 shows a computer screen displaying a Main Menu of a software program used in accordance with the present invention.

FIG. 3 shows a computer screen displaying a file index of the software program.

FIG. 4 shows a computer screen displaying "to-do" items in an "Appointments/To-Do" module used in accordance with the present invention.

FIG. 5 shows a computer screen displaying "to-do" items allocated to Monday, a Sticky-Link button and a Tel-Assist button used in accordance with the present invention.

FIG. 6 shows a computer screen displaying a card of the Note Pad format as an expansion card in the "Appointments/To-Do" module.

FIG. 7 shows a computer screen displaying handwritten data recorded on a Note Pad card to be transmitted along with a handwritten explanatory note to a colleague.

FIG. 8 shows a computer screen displaying a handwritten letter to be transmitted to a typist included on the data card.

FIG. 9 shows a computer screen displaying a handwritten letter as source material (from FIG. 8) as it is received into the Secretary module and the draft typewritten letter.

FIG. 10 shows a computer screen displaying an Address Book with handwritten and typed entries used in accordance with the present invention.

FIG. 11 shows a computer screen displaying a correction card in the Secretary module.

FIG. 12 shows a computer screen displaying a scrolling text document with a handwritten annotation on the first page only.

FIG. 13 shows a computer screen displaying a text document displayed in a mode whereby handwritten annotations can be supplied throughout the document.

FIG. 14 shows a computer screen displaying a handwritten outline with a Sticky-Link button to supporting data cards for use in composing, organizing and typing a text document.

FIG. 15 shows a computer screen displaying a Letter Pad for taking notes from a text document used in accordance with the present invention.

FIG. 16 shows a computer screen displaying the Long Range Calendar format used in accordance with the present invention.

FIG. 17 shows a computer screen displaying a Check Book format used in accordance with the present invention.


18 screen tablet

19 cable connecting screen tablet and computer

20 flat panel display

21 computer and peripherals

22 transparent digitizing tablet

23 cable connecting digitizing tablet and computer

24 screen tablet pen

25 handwriting shown on the screen tablet

26 portable ancillary unit screen tablet

28 portable ancillary unit computer

30 cable to upload/download data between portable and main units

32 background of Main Menu

34 buttons

36 cursor

38 pen button

40 menu bar

41 Change Tool menu

42 icons for files

43 Select menu

44 previous card button

46 next card button

48 Secretary module button

50 ScriptBase module button

52 Index button

54 "Appointments/To-Do" button

56 Address Book button

58 Return button

60 Button keys

62 New blank card button

64 Cut Card button

66 Paste card button

68 Copy card button

70 Type button

72 Run button

74 Tel-Assist button

76 Go/Srch button

78 Remind button

82 Data stamp button

84 FILE button

86 file (index) button

88 Make Button button

90 Reset buttons

92 ltr (letter) button

94 Export button

96 Import button

98 Link button

100 Update button ("Appointments/To-Do" module)

102 Delete button--"Appointments/To-Do" module

104 Sticky-Link button

106 Tel-Assist telephone number

108 Find button

110 DB (datebook) Long Range Calendar button

112 Xpand (expansion) button

114 Update button--Secretary module

116 WP (word processor) button

118 "Appointments/To-Do" jump button

120 check box

122 handwritten letter

124 New check button

126 Account balance field

128 Reset dollar amount field

130 return

132 Tab button

134 Find check button

136 Date field

138 Payee and Memo

140 Amount button

142 Check number field

144 Amount field


A hybrid information system for handwriting and text in accordance with the present invention consists of three principal components: (1) a hardware device, or "screen tablet", shown in FIG. 1A, for entering and displaying graphic computer data, (2) programs for controlling and causing the computer to handle the entry of handwritten and textual information and manipulate same, and (3) a general purpose computer, also shown in FIG. 1A.


A screen tablet 18, a composite entry and display device, consists of two connected parts: (1) a flat panel display 20 (electroluminescent, LCD (Liquid Crystal Display), plasma, active matrix, etc.); and (2) a transparent digitizing tablet 22 which lies atop the display. A connected pen (or stylus) 24 accompanies tablet 22.

Tablet 22 functions exactly as a regular digitizing tablet but allows the user to see through it to view material shown on screen 20 beneath. Thus, the pen, in combination with the tablet's surface can sense the location of the pen tip on the tablet and transmit the coordinates of the pen to the computer which, using appropriate software (described below), displays the coordinate information as a cursor. Similarly, with the software, a user can write on the surface of tablet 22 so that the writing 25 shows on screen 20 at the tip of the pen. Tablet 22 is connected to a serial input port of a computer 21 by a cable 19. Display 20 is the same as a flat panel display of a laptop computer and is connected to a display interface card attached to the bus of computer 21 by a cable 23. A optional cable 30 may be used to connect a serial port of computer 21 to a serial port of another computer which serves as a portable ancillary unit. Screen tablet 18 is available through several commercial sources, including MicroTouch Inc. of Woburn, Mass., or Toishiba Advanced Systems Division of Irvine, Calif.

Tablet 18 is about the size of a regular scratch pad (21.6 cm by 28 cm) and is about 2.5 cm thick. It may be used on a desk or placed comfortably on the lap like a conventional clipboard. The device has a conventional leg support or strut (not shown) which may be extended from the back (in a manner similar to a desk top picture frame) such that tablet 18 can be positioned vertically in a free standing position, similar to the screen of a conventional computer display.

The following description of the features of tablet 18 assumes use of companion software which is part of the system and which is explained below. When a user draws lines on the surface of tablet 18 with a pen 24, they are instantly displayed on screen 20. In this particular application, the lines drawn generally will be handwriting. Simply put, in the mode of operation referred to as "writing" mode, the user handwrites on tablet 18 and the writing appears on the screen, just as it would, for example, on a scratch pad.

Alternately, the user may operate the screen tablet in a "choose" mode by depressing a side button 38 on pen 24. The side button is a standard feature of a screen tablet device. Depression of this button sends a signal to the computer which may be programmed to carry out specific actions upon receipt of the signal. In the choose mode the pen tip becomes a graphic cursor (or pointer) in the shape of a hand with an extended finger, as shown at 36 in FIG. 2. The choose mode is used to pick items from pull-down menus on the screen or to activate graphic buttons by touching the desired object button with the tip of the finger. The user can switch between write and choose modes with alternate depressions of button 38. Depressing button 38 twice quickly in succession turns the cursor into an eraser which erases screen material at the tip of the pen. This erase function is similar to that found in common paint programs.

In a "select" mode, pen 24 may be touched at a spot, then dragged to show a rectangle which can be moved around and thereby enclose selected material on the screen. The selected material (either handwriting or text), may then be manipulated (moved, copied, or deleted) through software options. In this fashion, the user can enter handwritten information into the computer simply by writing on tablet 18, execute operations by touching buttons, or select items from menus by touching elements of the item list. Tablet 18 also functions as a conventional flat panel computer display which echoes characters typed on a computer keyboard or displays screen output from conventional programs. A suitable computer is a personal computer such as the one sold under the trademark of Apple Macintosh II however, the system is not dependent upon the type of computer used.


The software component of this system consists of a set of application programs. These are interpretive routines which execute under the control of the HyperCard brand software development system from Apple Computer Co. The set of programs can also be configured as a stand-alone application program. The software generates computer-based analogies and/or substitutes for manual information processing appliances such as appointment books, scratch pads, etc. and processes such as letter writing, document annotation, and others. These computer appliances accept handwritten graphic inputs, and can generate digital character input by touch selections from screen tablet displays. They are the mechanisms through which computerization benefits, such as speed of storage, retrieval, transmission, manipulation, etc. are realized. Examples of these computer-based analogies include an appointment book, "to-do" lists, file cards, file folders, scratch pad, stick-on notes, and tickler files (reminder files). Newly developed substitutes for manual processes include a combination scratch pad/typing window and a document annotation appliance. A full listing of this set of application programs is provided in a microfiche Appendix. These programs can be run on the Macintosh II computer, with a screen tablet connected as shown in FIG. 1A or with a conventional graphic tablet and monitor.

The software analogies represent a combination of: (1) graphic replicas of their manual counterparts; (2) certain highly novel productivity features; (3) menu buttons similar to those one might see on a traditional PC application screen, and (4) miniaturized keyboard buttons for the alphabet and numbers 0-9, as well as miscellaneous symbols. Menu buttons may incorporate pull-down or pop-up windows. Menu buttons are selected to accomplish specific computer tasks, such as cut, paste, copy, file, jump, return, switch mode (writing/button selection). In the case of the button keys, they provide digital information for filing or other computer purposes.

In addition, the system incorporates program objects which may be created by user command. These program objects appear as images on the screen and may possess a variety of physical shapes and sizes including that of a button or a field. In some cases they may be moved about by the user. Program objects may themselves be programmed and the program associated with the object activated by the user to perform specific actions.

Furthermore, certain button program objects may possess qualities of both a field and a button. Such a program object may be characterized as a "field-button". Upon selection (activation) the appearance of a field-button changes to represent a field which accepts data from the user. Once the data has been entered, the button may resume its operational (action) role by acting on the data entered to accomplish a task such as dialing a telephone through an attached modem. The following buttons possess this field-button characteristic: Update 100, Find 108, File Index 86, Remind 78, Check Amount 140, and Tel-Assist 74.

In addition to the analogs of manual information processing, this system features several appliances which have no manual counterparts. One of these, shown in FIG. 12, is called the Import Pad and represents a means for bringing (importing) a computerized text document onto the screen such that it can be annotated with handwritten notes. The incoming document can be displayed in two ways. One method of display spreads the text across numerous pages such that each page offers ample room for handwritten annotation to any text element. The other method, shown in FIG. 13, brings the entire document into a text window on a single page such that the user can scroll through the material in the window to read the entire document. In this mode, as shown in FIG. 12, the user may not annotate throughout the body of the text, but may write notations above and aside the text window. This latter mode is useful for sending a document with an accompanying handwritten note, while the former is best applied for annotation throughout textual documents.

Another element which has no manual analog represents a combination scratch pad for handwriting and a word processor, as shown in FIG. 14. Two-thirds of the screen represents a scratch pad for writing ideas, thoughts, etc. that might be incorporated into a text document. The latter may be created in a scrollable typing window which occupies the other one-third of the screen. Another extremely productive application for this combination writing/typing device, involves its use as a means for taking handwritten notes while reading a textual document which has been imported into the computer. FIG. 15 illustrates this use. Thus, for example, a text document which may have been scanned in or received from other computer users on a network, or retrieved from other storage, may be called up into the text window. The user may scroll the text and take notes on the handwriting window portion of the screen while reading. These notes may be stored on consecutive pages or the handwriting area may also be scrolled to accommodate the length of the notes. Upon conclusion, the notes may be easily filed. A module for use by a typist to translate handwritten material into typewritten text represents a third non-analogous module.


The information presented here, under Operation, and in the microfiche Appendix is sufficient to enable construction of the system by a worker knowledgeable in use of the HyperCard software development system. Employment of HyperCard terminology where relevant will assist in realizing this objective. The primary software entities consist of a set of files ("stacks" in HyperCard vernacular), each possessing capabilities and attributes unique to its function. These file structures contain data records ("cards" in the language of HyperCard) as well as application programs. As shown in FIG. 2, a stack contains one or more display format templates (backgrounds) 32 which contain "buttons" 34 for navigation within the software or for initiating feature functions. It should be noted that the term "button" in this specification shall refer to a generalized functional element which initiates a sequence of commands upon being actuated rather than to any specific button shape. The templates may also contain fields for data entry. Execution of a button's function is accomplished by placing the pen atop the button and pushing down on the tip. By means of a feature of the software, included in each module, the pen mode may be changed between write, choose, and erase modes as described above.

The screen templates (backgrounds) also contain a menu bar 40 displaying names of submenus which expand upon selection to display a choice list. With the exception of Change Tool and Select, these menus are permanent fixtures of the HyperCard system. Numerous of the choices offered by these menus have been accommodated in a way more suitable to the system's requirements by customized buttons, and thus are not used (although they are operational). The following menu features are utilized. File Menu: Compact Stack (delete excess space in files), Protect Stack (set a confidential stack access code), Delete Stack, and printing functions; Edit Menu: Undo, and the Cut, Copy, Paste and Clear functions for graphics. The two following menus were added as shown in FIG. 3: Change Tool 41 (select button/field selection tool, pencil tool, or eraser tool); and Select 43 (choose graphic selection tool mode).

Specifically, the functions and the objects of these modules are:

(1) A planning and scheduling module ("Appointments/To-Do") which: (a) automates certain work task planning and scheduling operations; (b) functions as a center for organization and control of work tasks and their associated supporting information files; (c) creates linkages between work task items and supporting information files to automate retrieval of the latter.

(2) A note creation module (Note Pad) which accommodates both handwriting and text which features automatic file recall based upon dates.

(3) A module (Import Pad) which allows handwritten annotation of text.

(4) A module (Secretary) which automates aspects of the document production process between author and secretary involving handwritten source material and handwritten annotation of draft text.

(5) A module (Letter Pad) which (a) provides computer assistance to automate aspects of the formal document creation process which involve handwriting; and (b) provides a means for handwriting notes into the computer while reading a scrolling text document.

The system consists of a main (stack) entitled "ScriptBase" which includes the following display templates (backgrounds): the Main Menu as shown in FIG. 2, the Index as shown in FIG. 3, the Note Pad as shown in FIG. 6, the Import Pad as shown in FIG. 12, the Letter Pad as shown in FIG. 14 and Reminders (the Reminders background is simply a full card field for storing reminder dates and their associated file card names. Other stacks are: Appointments/To-Do as shown in FIG. 4, Long-Range Calendar as shown in FIG. 16, Address Book as shown in FIG. 10, and Secretary as shown in FIG. 9. The structure of the main module will be described first.

As shown in FIG. 2, the Main Menu displays selection choices represented by buttons 34. These buttons, when touched by the pen, take the user to the stack or background with the desired functions. The Index, shown in FIG. 3, represents a card displaying icons 42 for stacks which may be applications or data files. New data icons are automatically created by a stack program when the user decides to file information (handwritten or text) for which there is no existing file name. Upon creation, the user may move the icon and fix it in the desired location.


The Note Pad, illustrated in FIG. 6, constitutes the primary entry format (card background) for handwritten information, although text may also be typed into this format. The lower right-hand edge contains a set of navigational buttons 44-58. Placing the pen stop any one of these buttons will "jump" the user to any of the following software destinations: previous card 44, next card 46, Secretary 48, ScriptBase Main Menu 50, Index 52, "Appointments/To-Do" 54, Address Book 56, and Return 58 (to location preceding last jump). The accompanying software routines save the current location and branch to the target. A Return button on the target enables an instant jump return. Of key importance is a set of button keys 60 which, upon touch, provide the computer with digital characters, numbers, or symbols for file names, times, dates, telephone numbers or other uses. These function in a manner subordinate to other buttons by supplying digital data to a field opened by the primary button. This is accomplished in the program by using the name of the target button. Functions for creating new blank cards, or cutting, pasting, or copying them are accomplished with the buttons of corresponding names: New 62, Cut 64, Paste 66. Copy 68. A "Type" button 70 allows the user to switch between typing and handwriting. The Run button 72, lets the user exit the current program to run another program.

The button named "Tel-Assist" 74, shown in FIG. 5, represents a button which turns into a field to accommodate a telephone number entry; can be moved to any location, and when fixed, returns to a button state. The program associated with this "data" button then allows the user to "dial" the imbedded number by touching it with the pen. This is accomplished programmatically by creating a new data field, entering the number either from an Address book selection or by button keys, then locking the field such that it can function as a button and execute a routine that dials the number through an attached modem.

A Go/Srch button 76 provides a mechanism for searching through a stack of cards (an information file) for a specific card heading. A Remind button 78 allows the user to attach a specific date to a data card. Upon start up, the computer selectively displays data cards associated with a date equal to the current date. A "D" (date) button 82 upon touch provides a date/time stamp which can be placed anywhere on the data card. A FILE button 84 works in conjunction with data supplied to a "file (index)" button 86. Touching the file (index) button changes it into a field to accommodate entry of a file name. Subsequent selection of the file button will file the card under the given file (index) name (if valid). If the name does not exist, the user may choose to create a new file with the name given, or jump to the index to select the name of an existing file. The button routine then proceeds to file the card, creating a new index entry if necessary. "MB" is the acronym for a "make button" operation. Button 88 associated with this function lets the user create and place a button on a data card which, when actuated, jumps the user to another data card. Three buttons R, F, and #90 are provided to reset the Reminder, File (index), and Telephone number fields, if necessary. A Ltr button 92 copies handwriting committed to the lower one-third of this screen to the Secretary module for preparation into a typewritten document.

The second module of the ScriptBase stack is the Import Pad shown in FIG. 12. The Import Pad is used to make handwritten annotations upon a text document in either of two ways. If the user desires to make a handwritten note only on the first page of a document, a scrolling text mode is used wherein the document is called up in a scrolling window. The display format provides ample room for a handwritten note, while allowing the user to scroll through an entire document on a single screen as shown in FIG. 12. Such a function accommodates the need, for example, to send a copy of a document with a brief note attached. Alternately, the user can opt for a mode which spreads the imported document across numerous cards illustrated in FIG. 13 such that there is ample room for multiple handwritten annotations throughout the body of the text. This mode accommodates the need for more thorough editorial treatment throughout the material. An Export button 94 on this module allows the user to export text which has been imported and subsequently modified. The button entitled "Import" 96 provides the mechanism for importing an ASCII text document. A Link button 98 is provided to facilitate the "filing" of a set of Import cards as a single operation. The filing operation will be performed upon all cards for which the link function has been turned on.

The last module of the ScriptBase stack, entitled Letter Pad, (shown in FIG. 14) provides a platform for creating text documents based upon handwritten notes. Similar to the way individuals supplement the use of a word processor by jotting down handwritten notes, this template may be used to guide the composition of typewritten text. Handwritten notes may be entered upon the upper two-thirds of the screen, while text may be typed into a scrolling window on the lower third. The handwritting area may be entirely cleared with a Command-Delete sequence or items may be selectively deleted using the eraser or the graphic selection tools.

The Letter Pad is ideal for taking handwritten notes on a text document as illustrated in FIG. 15. The document may be scrolled in the text window and handwritten notes recorded on the upper portion of the screen. A button, not shown in the prototype, may be activated to copy a screenfull of handwritten notes to a designated file and clear the handwriting area in preparation for additional notes.


While the ScriptBase module serves as the primary handwriting entry appliance, the "Appointments/To-Do" module illustrated in FIG. 5 represents the focal point for organization of information, particularly when used in a business setting. This tool is modeled after the ubiquitous "appointment book" carried by virtually all business people and other employed professionals. It combines the appointment documentation function with the "to-do" list planning function.

The objects of the "Appointments/To-Do" module are the following: (1) Automate certain of the planning and scheduling operations of the appointment book and "to-do" list functions (updating and allocation of items over time and retrieval and storage of long range calendar items) (2) Function as an information organization and control center for information which supports "to-do" items (creation/retrieval, revision, transmittal, storage--directly from the task item). (3) Dial telephone numbers (through computer and modem) associated with "to-do" list task items. The "to-do" list is used by its creators as a planning vehicle for accomplishing work over a variety of time frames; long range, this month, this week, today. As time proceeds from the long range perspective to the immediate, "to-do" items get allocated into a daily time slot framework.

The "to-do" list thus represents not only the center point for work efforts, but constitutes the heart of one's information processing. In particular, the bulk of this written information commerce proceeds from initiation in the to "to-do" list. This entity is thus the most appropriate focal point for organizing daily information processing. By utilizing a computerized format for the "to-do" list documentation we focus computerization on the heart of our daily written information commerce and proceed downward through subordinate information processing activities. This scheme maximizes the amount of information processing handled with computer assistance and hence maximizes the benefits of computerization.

The computerized "Appointments/To-Do" module closely resembles its manual cousin, thus it allows simultaneous entry of information into a manual appointment book and the computerized version. This is possible with most digitizing tablet technologies. Thus the user can record entries into a manual appointment book and the computerized counterpart without dual effort. Upon program start up, the computer automatically creates new "Appointments/To-Do" pages for all dates between the last page date and the next thirty days (thirty days from the current date). It then returns to the current date. Planning for item beyond the thirty day detailed planning range is accomplished using the Long Range Calendar (illustrated in FIG. 16) which provides a more abbreviated documentation format across 365 days. Handwritten items entered in the Long Range Calendar may be brought into the "Appointments/To-Do" module with computer assistance.

The "Appointments/To-Do" page for Saturday is nominally reserved for documenting weekend information, and that for Sunday for a cumulative listing of "to-do" items to be undertaken the following week as shown in FIG. 4. However, each page follows the conventional format and may be used accordingly. The benefits of this use convention will be further explained under Operation. There are several new operating buttons in this module which are shown in FIGS. 4 and 5. An Update button 100 and a Delete button 102 work in conjunction with the checkboxes 120 for each line of the "to-do" list. Update button 100 operates on handwritten "to-do" items with corresponding checkboxes selected by updating them to a specified date or by default to the next day. Similarly, Delete button 102 removes selected item. These functions operate not only upon handwritten notes but also upon "Sticky-Link" buttons 104 buttons or telephone numbers 106 associated with the "to-do" items. The significant value of such a function can be better appreciated in the context of the scenario described under Operation.

The Update button thus automates the process of allocating "to-do" items and their information support appendages across time. Because jumps into the "Appointments/To-Do" module always bring the user to the current date, a Find button 108 may be used to jump to a specific (non-current) date. If the desired date is more than 30 days beyond the current date, the user can jump to the Long Range Calendar which provides a one year time span. DB represents an acronym for DateBook, another name for the Long Range Calendar. DB button 110 jumps the user to the Long Range Calendar. Xpand button 112 performs the simple, but highly important function of introducing a blank note card of the ScriptBase format as the next page of the "Appointments/To-Do" book. This instant scratch pad may be used very expeditiously in prosecution of the days "to-do" items as described under Operation.


The function of the Secretary module, illustrated in FIG. 9, is to assist in the transformation of handwritten material into typewritten format. A Ltr button 92 on the ScriptBase Note Pad, shown in FIG. 6, is used to transmit (generally over a network) handwritten material from a Note Pad card of the Secretary module which is similar to that of the Letter Pad with minor modifications. The Secretary types the handwritten information into the typing window in a fashion identical to that employed in the Letter Pad. When the typing is complete, the author may access the Secretary module and apply handwritten corrections and/or annotations as shown in FIG. 11. This is accomplished by marking the desired portion of scrolling text which creates a dynamic correction card following the Typing screen. Annotations may then be applied upon the correction card to the portion of text in question. Multiple such cards may be created. To correct the text, the secretary simply makes the corrections in the correction card typing window and touches Update button 114 which updates the original text. Because the Secretary module possesses only rudimentary word processing functions, the secretary may desire to finish the document by exporting it into the computer's primary word processing program by touching WP button 116.


The address book module, illustrated in FIG. 10, represents a piece of code which not only provides traditional address book functions but also functions as a telephone dialer. This module as well as the Long Range Calendar module represent code developed by Apple Computer for distribution with the HyperCard system which was modified for the prototype.


FIG. 17 shows a module which enables the user to employ the system for bank checking account documentation. This capability is especially useful when employed with a vest-pocket-sized portable embodiment of the system as described under Alternate Embodiments and Variations. This feature draws upon a capability of digitizing tablet technology which makes it possible to use the pen to write on a piece of paper laid atop the screen while simultaneously writing into the computer. A blank check may be placed on the screen tablet surface, aligned with the corresponding check image as shown in FIG. 17, fixed in place by any of a variety of means, and handwritten. The button keys or character recognition software may be employed to enter digital numbers into the amount field.

The following operating buttons and numeric fields are provided. A New button 124 creates a new computer check image displaying a sequential check number in a numeric field 142. An amount button 140, (shown offset in FIG. 17 to reveal the underlying Amount field 144) allows the user to enter digital numbers by means of the button keys into the Amount field 144 (which is revealed when the Amount button is pushed). The Amount button also causes the current date to be displayed in a Date field 136. A current balance field 126 is also displayed on the computer check. A Find check button 134 is used in conjunction with the Tab button 132 to search for a check of a given date. A reset button 128 resets the amount field to blank and a Return button 130 is used to exit the module.


By allowing virtually all of an individual's written information processing to be computerized, computer benefits such as speed of storage, retrieval, transmission and reliability are extended to a level heretofore not achieved. This also overcomes major disadvantages of prior art which necessitates maintenance of manual files for handwritten items (in addition to computer files) and labor expenditures associated with their retrieval, transmittal, and storage. By allowing elimination of manual files, it promises large savings in labor and physical storage costs. This device also overcomes a key drawback associated with keyboard use which makes short data items (e.g. messages, notes, reminders, etc.) unsuitable for computerization because the data entry burden exceeds the payoff. When installed on a tiny vest-pocket size computer, this system obviates the need for a keyboard. Miniaturized keyboards on such machines have prevented their use for entry of extensive textual information. It thus overcomes a significant drawback of present vest-pocket computers and expands their utility to enter larger volumes of textual data such as meeting notes, appointments, reminders, etc. which may be subsequently downloaded to a fixed computer. This represents a major extension of computer benefits into a heretofore untapped arena.

Coupled to these extensions of applications are substantial quality benefits. These include instantaneous storage and retrieval, reduction of "lost" information (a drawback of paper systems), and a significant reduction in the learning requirements (a drawback of keyboard systems). A user need learn almost nothing to be able to enter, store, retrieve and transmit information. The elimination of the keyboard entry barrier promotes the entry of whole new classes of potential users. Accordingly, computerization benefits are extended to a broader population. Another advantage derives from the capability for writing with an ink-filled stylus onto a piece of paper and into the computer at the same time. Further advantage of this copy feature relates to the frequent need for a copy of a "signed" document. Handwritten entry overcomes character set dependence, another disadvantage of keyboard based computers. It also offers the ability to draw pictures, a communication advantage not generally available from keyboard dependent systems.

The experimental scenario described under Operation showed labor time savings in the order of 30 percent and throughput improvement of 100 percent. Cost savings from reduction in paper use and elimination of physical file storage needs can be very significant. The system eliminates the need for making paper copies of documents which are to receive handwritten annotations such as draft versions or editorial commentary. Paper usage for forms incorporating hand-entered data along with multiple associated copies may also be eliminated. In summary, the advantages of the present system include significant cost reductions, productivity and quality gains. Furthermore, it offers the potential for extensive expansion of the benefits of computerization to broad new applications and large classes of non-users.

It may be noted that the extent of benefits which result from such a comprehensive system were not expected for a number of reasons. (1) Prior to the recent advent of a coincident display and writing interface, handwriting entry suffered from hand-to-eye coordination difficulties caused by the separation of the writing surface and the location in which the writing was observed (the display). (2) Lower resolution screens made handwriting difficult to read. (3) A wide-spread misconception appeared to exist which suggested that only data which was digitized could be usefully manipulated in a computer. (4) The magnitude of potential computer benefits available through assumption by the computer of the large bulk of an individual's handwritten information commerce had not been calculated or appreciated. It thus represents an unexpected result. This may be due, in part, to the fact that handwriting is generally considered more laborious than typing. The time-saving and throughput benefits of such a hybrid system cumulate to assume significant amounts, in particular, because of the synergistic effects of the system which can handle virtually any type of handwritten or keyboard-entered information processing; a result of the " system" nature of the device. For example, an "to-do" note which represents an uncompleted work item can instantly be moved to the next day but may retain linkages to supporting information files. Thus the unexpected liaison between hypertext type programming appliances and handwriting has significantly increased the computing benefits. Because of its breadth, the comprehensive hybrid system serves as the user's primary interface. As such, the naturalness of the pen and pad interface overcomes a long felt need for computer interface improvement.


Numerous advantages derive from specific features of the system as programmed which are detailed under Operation. Among other things, such features include: the ability to automatically move (e.g. update) handwritten appointment notes or "to-do" list items; hypertext features which attach to handwritten notes which travel with these notes wherever they go; jump out and pop return capabilities to allow instantaneous context interruptions for performance of alternate functions; handwritten annotation of both text and other handwriting; automatic creation of file indexes; automatic recall of handwritten reminders; complete secretarial functions; and a combined text and handwriting facility which allows a user to a) take handwritten notes while scrolling through a text document, or b) set down and organize handwritten notes which can be used to guide the typing of a textual document, much as one would make notes on a scratch pad preliminary to typing up a document.

Two features with special importance are "Sticky-link" buttons and telephone number dialer buttons. Sticky-link buttons represent software buttons which can be created on the fly by the user to link information stored in different files. By "pushing" a sticky-link button, the user can jump from the file which incorporates the button to the file which represents the link target. The user may thus instantly access subordinate information structures which are physically disconnected but informationally related. The truly nice feature of these link buttons is that they travel with the informational element of which they are a part. Thus, if a copy of an information item which embodies a link button is transmitted, the receiver inherits the link button capability. This concept of traveling operational buttons is further exemplified in a feature which allows the user to embed phone numbers as buttons in an informational element. By "clicking" the pen button on the phone number, the computer can actually dial the party whose number is embedded in the button through a modem attached to the computer. These buttons are also created on the fly by the user and travel with their host information. The "to do" list portion of the appointment book allows automatic updating of handwritten notes and accompanying sticky links and/or phone number buttons without the normal cut and paste rigmarole.


The operation of this system is best described by relating a typical use scenario. Unless otherwise noted, the subject information is handwritten. On a given day, the first time a user starts the program, a screen dialogue box asks if the computer should check for Reminder notes associated with a date equal to the current data. A "yes" response results in the display of a handwritten note reminding the user of an important birthday two weeks hence, along with some gift ideas.


After all outstanding reminders have been displayed, the computer inquires if the user would next like to view the Long-Range Calendar, illustrated in FIG. 16. This appliance represents an extension (abbreviated in detail) of the user's computerized appointment book across an entire year. It provides a place for recording appointments or other important time dependent information which are more than 30 days future from the current date. Assuming again a "yes" response, the user may jump to the card displaying long-range calendar information for the current week as shown in FIG. 16. If desired, the user can easily move information pertinent to the current day's activities for pasting onto the current day's activity schedule in the "Appointments/To-Do" module.


Touching AP jump button 118 in the margin takes the user, instantly, to the current day in the "Appointments/To-Do" module, as shown in FIG. 5. Assume that it's Monday and that the first order of business is to plan the week's activities. As illustrated in FIG. 4, the user can list things to do on the left side, then allocate them to various days of the week (or any date). By checking (touching) box 120 (shown in FIG. 4) to the left of the handwritten item, then touching Update 100, the user may indicate the date (or day of the week) to which the item should be posted. Mechanically, the sequence unfolds as follows: the Update button turns into a field; a date or day of the week is entered by touching the appropriate button keys; and the tab button key is touched to lock the field and carry out movement of the handwritten graphics to the target date. The user is assisted by a help message appearing at the bottom of the screen. The sequence is, of course, carried out much more quickly than corresponding manual operations. Items not completed on the target day can be moved forward to the next day almost instantaneously with two quick touches.

INFORMATION PROCESSING FOR A TYPICAL WORK ITEM Incorporating a Telephone Number into the Work Item

FIG. 5 shows that two items "to contact Terry Hansen of NEC" and "make reservations for Comdex" have been allocated to Monday's list of things to do. To begin work on the first item, the user touches Tel-Assist button 74. It opens into a field, and at the same time inquires (through a dialogue box) whether the user wishes to enter the telephone number directly or get it from the address book. Using the same methodology as with the Update button, the user may enter the data into the telephone field. (Alternately, the program jumps to the Address Book where the user selects the number.) This time however, clicking the Tab key turns the data field into a moveable button/field (which incorporated the telephone number) and, which the user may drag next to the handwritten entry by using the pen as a mouse. Subsequent touches on the newly created phone number button directs the computer to dial the number through an attached modem. Assume, Mr. Hansen is not in, but it is discovered that he will return at 2 PM. The user can jot a brief reminder next to 2 PM on the appointment side. At 2 PM, we simply touch the number to re-initiate the call.

Recording of Work Item Supporting Information

While waiting for Mr. Hansen's secretary to transfer the call, the user can touch Xpand 112 button to call up a scratch pad as illustrated in FIG. 6 so handwritten notes can be taken during the conversation. As a part of these notes, the user will get Mr. Hansen's address and the phone number of a subordinate contact he suggests. While the address is entered in handwritten form, the user will enter the number digitally using the Tel-Assist button because it's easily done with the button keys and will thus be operational.

Sending a Memo which Includes the Work Item Supporting Information

Finishing the conversation, the user decides to assign a subordinate, Bill W., to pursue this activity. To do this the conventional, manual way, the user would likely get out a memo and laboriously transcribe the collected information (since it is desirable to keep a tickler copy). Or perhaps, it may be easier to walk to the copy machine; make a copy and add a note of explanation. In either case, the process is time consuming and the transit of the memo through the company mail system, not as rapid as one would like. With the present system, the note of explanation can be added at the bottom as shown in FIG. 7 and a copy sent electronically to Bill by touching Copy and then filing the copy to Bills mail box by touching file (index), File, and selecting the Bill W icon from the index. Not only has the user saved a good deal of time and drastically improved information throughput, but he has also conveyed an operational attribute which saves Bill time. He too can dial the enclosed number simply by touching it.

Creation of a Letter

Because this was a rather important contact, the user would like to confirm the substance of the conversation by writing a letter to the individual contacted. Again, it's useful to compare the traditional, manual process with the methodology enabled by this system. A conventional approach for would entail handwriting the letter (or possibly dictating it), and hand delivering or mailing it to the secretary. The latter would type a draft, print it, and mail, or hand deliver it. The author would, in turn, make handwritten corrections and/or changes and return it to the secretary. After final typing, the letter would wend its way again back to the author for signature. A good deal of the labor and almost all of the transit delay is eliminated by being able to transmit handwritten information through the computer as follows.

Using the bottom one-third of the Note Pad screen as shown in FIG. 8, a user may handwrite material 122 which will be automatically transmitted to the secretary module by touching the Ltr button. Of course, more than one card may be used to transmit longer material. FIG. 9 shows the handwritten material in the Secretary module following transmission. The secretary will type the material into a scrolling typing window shown at the bottom of the screen. To begin, the secretary may access the author's address book as shown in FIG. 10 for the recipients address. Noting that the author had made a handwritten entry, the secretary may formalize it by typing it, then copy and paste the address to the letter in preparation. The digitizing of the address is done only once, and by the secretary as opposed to the user.

Correction of Letter Draft

The author may then access the Secretary screen and make handwritten changes directly on the source document. To do this, the author scrolls through the text to a point where a handwritten annotation is desired. At this point, the visible section of text is first marked by selecting a reference word in the upper left hand corner and touching Return. This makes a copy of the desired text which the author then annotates with handwritten comments as shown in FIG. 11. The secretary will subsequently view the handwritten correction cards; make the indicated typing changes directly on the card; and simply touch the Update button to transfer the changes to the original. The Secretary module possesses only rudimentary word processing capabilities; thus, the secretary may wish to transfer the letter into the regular word processor for finalization by touching the WP button. As another nice feature of this system, it may be noted that given a printer of sufficient resolution, additional time and throughput benefits may be garnered by having the author sign the letter electronically.

Filing Supporting Information, Linking it to the Work Item, and Updating the Work Item

Before leaving the "Appointments/To-Do" module, the user would like to file the note documenting this activity and update the appointment book to follow up on task progress in a couple of weeks. The note card illustrated in FIG. 6 may be filed by entering a file name into "file (index)" button/field 86 by touching the file (index) button and then desired button keys) or it may be entered by using File 84 and the index icons. After the data is filed, the user is returned to the current day's appointment page. Touching the MB (Make Button) creates a "Sticky-Link" button 104. After it is created the button is dragged to the "Terry Hansen" entry area and fixed in location by pressing the pen side button twice and clicking on the Tab button key. Then, by touching the check box to the left of the entry, touching Update and supplying a target date via the button keys, the entire entry including the telephone number and Sticky-Link button is moved to the desired date. An option is available, as a part of the updating process to take a quick peek at the target card to confirm that the information being updated will not overlay any existing information on the target card. When using the "to-do" list for that target date, the user need only touch the Sticky Link button to recall the notes and touch the number to re-establish a telephone contact. The time saving potential of this feature for individuals who perform repeated telephone contact work, such as real estate agents do, is significant.

Annotating and Sending a Document

Next, assume that the user would like to send the boss a copy of the letter with a short explanatory note attached. This would normally be handled by making an extra printout or copy, applying a stick-on note and mailing it. With the present system, labor, paper cost, and transit time can be eliminated by using the Import Pad, as shown in FIG. 12. With two button touches ("SB" on the margin of the "Appointments/To-Do" page and then "Import Pad" on the Main Menu), the user jumps to the Import Pad. Assume at this point that the user is interrupted by an overseas phone call. The company's French representative is calling from an airport phone to respond to an urgent and important technical question. His flight leaves in a couple minutes. Under conventional circumstances, the call recipient would likely rummage through papers or try to find the physical file holding the handwritten details of the query, wasting, of course, valuable and expensive time. With the present system the user is able to interrupt current work; jump to the index; then, instantly, to the required material; upon which the answer may be recorded such that it may then be sent electronically to the source of the query. The problem has been solved; and with a single touch on the Return button, the user is back at the Import Pad.

To bring the letter in question on the screen, the user touches the Import button, selects the letter's location (by selection within a dialogue box or by providing a button key entry of the file name); selects its type (text or data); and selects a format (single card scrolling window or multi-card). To the right of the scrolling text window, the user supplies the desired note as shown in FIG. 12, and mails it to the boss simply by touching "file (index)", File and selecting the Bosses' icon on the index. The Letter Pad also accommodates another frequently undertaken information processing activity: that of supplying handwritten annotations in the margin of some textual material which is then filed or transmitted. For example, a user might enter a magazine article into the computer by using a digital scanner; jot a note on the margin; and send it to a colleague. The ease of making one or more copies represents an added benefit.

Annotation Throughout a Document

Another highly cost-beneficial use of the Import Pad arises when it is desired to provide handwritten comments throughout a document. The document may be brought up onto the screen as above, except that the format specification may provide that the document be spread out across multiple cards as illustrated in FIG. 13. This format provides plenty of room for annotations throughout the text as opposed to a note only at the beginning. When this procedure is applied in the circulation of a document to multiple individuals, the magnitude of the time and transit benefits can be readily appreciated. With traditional manual methods, multiple copies of the physical document would have to be made and transmitted, annotated, returned, requiring much greater labor and throughput time.

Creating a Typewritten Document

Another frequently undertaken information processing area deals with the creation of written reports. If a user possesses at least rudimentary typing skill, this does represent an area where the keyboard interface can be of value. The keyboard is however, not usually the only writing instrument involved. It is common to preface the entry of text with handwritten notes on a scratch pad: a list of ideas, topics, numeric annotations indicating order of topic entry, or perhaps even an outline. Another prefatory activity involves the physical collection of source data. Frequently this is gathered from a wide range of sources including scratch pad notes buried somewhere, various file folders, magazine articles or other printed materials, etc. It is instructive to see how this process also can be beneficially accomplished with the present system.

The appliance conceived to perform such a function is named a "Letter Pad". As shown in FIG. 14, the upper two-thirds of the Letter Pad is reserved for handwriting while the bottom one-third features a scrolling typing window. The handwriting portion can be used in a number of ways. As a topic list, entries can be ordered (by moving the graphic entries around) into an outline for typing expansion detail into the window beneath. Alternately, a list of major headings may be copied to successive new cards for further expansion and subsequent typing into their respective windows. The typing in each window can subsequently be combined into a single text. Perhaps the most beneficial use involves writing names of supporting data sources and applying Sticky Link buttons. These buttons, upon touch, instantly call up the supporting data Note Pad cards (such as that shown in FIG. 6) upon which the source data was originally entered. By clicking on the Type button on these source cards, the user may type, from the handwritten notes and directly on the source cards, then move the typewritten material back to the original Letter Pad window. This process represents a truly productive methodology for document production, which avoids the physical handling of multiple manual files and physically disparate data sources.

Taking Notes from a Scrolling Document

Another significant usage suggests that an alternate name for this appliance would be "Read-Write Pad". In this usage, roles are reversed and the "typing" window becomes simply a "reading" window into which any document can be displayed by touching the Import button. The entire text may be read by scrolling the document and handwritten notes taken by the user. When the handwriting area becomes filled, a button may be touched to move the handwritten content to a blank card and clear the area to accommodate more notes. This "note-taking" feature significantly enhances movement towards fully computerizing our information processing.

Banking Check Subsystem

A user may employ the check writing module to reduce the labor associated with this chore and reduce the likelihood of committing arithmetic errors. The check writing module is accessed by selecting the checking icon from the index which brings the user to the stack which contains images of the previously written checks. The number of checks kept in this stack is limited only by system file size constraints.

Upon entry into the stack, the last written check is displayed. An account balance field shows the users current balance. Touching the New button, produces an image of the next sequentially numbered blank check and displays the Amount button. The user touches the Amount button and enters the desired figures by means of the button keys. The Amount button program also supplies the current date to the computer check. Depression of the pen side button stores the amount figure and updates the user's balance field. A blank check may then be placed atop the screen tablet surface, aligned with the corresponding check image as shown in FIG. 17, fixed in place by any of a variety of means and handwritten. The date, payee, script dollar amount, memo, and signature may then be handwritten on the physical check. These are recorded simultaneously into the computer check. Thus, a duplicate image of the actual physical check is archived in the computer. These images may be periodically copied or moved to a floppy disk for long term storage. Of course, the user may easily page back and forth among check written using the arrow buttons.


While the modules ("Appointments/To-Do", Note Pad, Secretary, Import, and Letter Pad) were designed to work in concert to meet the broad spectrum of needs of an individual's commonly-experienced daily written information processing each module represents a significant stand-alone device. A number of benefits of each module are enumerated and/or summarized below.

The "Appointments/To-Do" module provides the ability to organize and carry out work from a list of "to-do" items. This device's capabilities for retrieval and storage of updated files which contain handwritten information offer substantial and easily identifiable labor savings over corresponding manual methods. These savings are further extended by automatic telephone dialing. Savings realized by not having to physically retrieve and restore file folders are often multiplied in situations such as the following: when there are multiple files associated with a single handwritten work item; when multiple telephone calls are made in attempting to reach a desired party; when work items are repeated periodically or simply when a work item is updated for future activity or because it was not completed. Numerous jobs exist (e.g. stock brokers, real estate sales) which require frequent periodic telephone contact and updating of a client's file. Application in such situations would result in very substantial labor savings. In addition, daily calendars (of the type generally carried about with the individual) frequently do not span an entire year and thus entail maintenance of a seperate long range calendar. This requires transposition of items and prevents access to long range information when the user is distant from the yearly calendar. The combination long range and short range calendar of this system overcomes these disadvantages. In addition, entries may be made by a secretary and the users portable version automatically updated.

Extensive labor saving may also be realized through use of the Secretary module as an independent device. Its employment eliminates the following time consuming manual steps: physical transmittal of handwritten source copy; printing of one or more draft copies for editing/correction; return transmittal of the hand edited draft material; and copying of source and final documents for certain purposes. As compared with internal mail systems, electronic transmittal of handwritten source and annotated material results in a dramatic increase in throughput--a significant quality benefit.

The Note Pad module also offers significant benefits as an independent entity. Essentially it represents a vehicle for providing the benefits of computerization to the myriad number of information processing activities which heretofore have been undertaken with pencil and paper because the associated computer keyboard entry labor exceeded the perceived benefits. This application spectrum is also broad and includes incidental notes, reminders, telephone messages, and lists of various types. More substantial items are also embraced such as meeting notes, memos, handwritten materials to be formalized, and handwritten data bases which may be constructed in hierarchical fashion with Sticky-Link buttons linking constituent elements. In addition to the direct labor savings associated with computerization of such applications, substantial quality benefits are present. Instantaneous information retrieval and reduction in lost information are prominent examples. The ability to transmit information along with hypertext features such as Sticky-Link buttons and telephone dialing is also valuable.

There are many occasions when it is desirable to transmit a document with a brief handwritten note attached. More and more documents are being stored in computer format. It is also becoming increasingly easy to read into the computer typewritten documents and periodical material with hand-held and flat-bed scanners. With greatly increasing frequency, such text is being transmitted by computer. It is awkward, somewhat more difficult, and for some (e.g., non-typists), undesirable to append a typewritten note to such documents. The Import module makes this annotation process both less laborious, less difficult, and more natural by allowing transmission of handwritten notes on a document. For example, a magazine article of interest to a colleague may be quickly scanned with a hand scanner, annotated, and sent off in less than a minute. The capability of the Import module system to enable users to make handwritten annotations throughout a document offers very significant labor savings for frequently occurring situations in which documents are circulated for editing and/or comment.

Substantial benefits may also be associated with independent use of the Letter Module. This device possesses two modes of operation. In the "write to type" mode, the user uses handwritten notes committed on one portion of the display as a guide for entering typing material into a typing window. This is especially useful for organizing and typing a document, especially a complex one, or one with multiple data sources. In addition Sticky-Link buttons may be invoked to link handwritten subject outline items to data files of supporting data. As an electronic information organizing tool, use of this device is far less time consuming than retrieval and layout of multiple physical source materials which incorporate handwritten information. Such labor savings are leveraged considerably for intermittent work on complex projects where source materials must be repeatedly retrieved and put away. In its alternate "read to write" mode of operation, the user may read material in the scrolling text window and take handwritten notes on the remaining part of the screen. This provides the benefits of computerization for handwritten notes. Thus a user might access various computer data bases; make notes onto multiple cards; and later organize (manipulate) them into a cohesive whole; which might also be formalized using the "write to type" mode discussed above.

The capabilities of the checkbook module overcomes the following disadvantages of current manual checkbook devices: (1) duplication of labor to create a record of transaction information to a separate check register, or alternately, reliance upon carbonless carbon paper duplicates which are frequently illegible; and (2) likelihood of arithmetic errors associated with manual balance calculation. This capability is especially useful when employed with a vest-pocket-sized portable embodiment of the system as described under Alternate Embodiments and Variations.

Alternate Embodiments and Variations

The system may be configured to run on a portable vest-pocket sized computer as shown in FIG. 1B for entry of handwritten information remote from the users primary computer. Such a device consists of a miniaturized personal computer 28 with a correspondingly small combination display-entry screen 26 similar in function to that described above as the primary entry device. Physically, this ancillary unit looks like a vest-pocket appointment book with dimensions of approximately 16.5 cm by 8.8 cm by 1.3 cm and consists of two sides connected by a seam along one long dimension and folds to the size above, just as does an appointment book. One side 28 of the ancillary unit houses a micro computer such as that currently under development by Chips & Technologies of San Jose, Ca. The opposite facing side 26 provides a screen tablet of size 16.5 cm by 8.8 cm which is used to capture handwritten information and send it to the CPU for storage.

It should be noted that the virtual screen size can be substantially larger. That is, through graphic software the graphic screen can be scrolled up, down, left or right. An input/output port is included such that the information captured remotely may be downloaded by means of a cable 30 to the primary computer to update existing files or create new ones. Alternately, information such as appointments, things to do, may be copied from the primary computer to the portable unit to be retrieved by the user when remote from the primary computer. This development represents a significant advance for portable computing. A substantial amount of the information that an individual would potentially desire to store on their computer, is be gathered at locations remote from the individual's computer. Examples include meeting notes, appointments, ideas, drafts of letters, etc. Heretofore, computers which were small enough to be carried in a coat pocket were not suitable for entry of written information of any substantial volume e.g. meeting notes. This system overcomes this significant drawback.

Many other extensions, variations, and embellishments are possible. A few of these ramifications are discussed below. Under the heading of hardware related variations the following may be envisioned. A touch screen may be built to work with a cordless stylus (pen). A flat panel display can be built which includes electronics to sense the location of the probe (This has been experimentally demonstrated with an electroluminescent panel.) A special area on the screen may be included which accelerates the movement of the cursor and is useful for cross-screen cursor movement. This acceleration feature may also be triggered via a side button on the pen. An obvious variation is the use of the software on a conventional computer using a graphic tablet for input. Such a usage suffers from somewhat of a hand-to-eye coordination problem. Incorpoation of a hand-held digital scanner would be particularly useful with a portable unit for example doing library research. The user could scan in passages from books and combine handwritten note with the textual input. It may be noted that school children who do not type could benefit from use of the system to carry out handwritten homework assignments by being able to erase and manipulate handwritten work. A CD-ROM unit loaded with an encyclopedia could further encourage use of the system by school children.

A somewhat more technical extension envisions multiple interconnected screen tablets which form a large desktop surface perhaps three feet long and two feet wide. This arrangement would allow multiple documents or information appliances to be displayed and worked with in a fashion closely analogous to current manual environments but with the benefits of computerization. Thus, separate document or pages, scratch pad, appointment book, etc may all be displayed simultaneously in a truly electronic desktop. Such an embodiment would be particularly useful for editing multi-page materials.

Numerous variations and improvements in the software may also be anticipated. Prominent among these would be incorporation of the button-keys into a pull-down or tear-off menu format to leave more room on the screen for handwriting or text. Other of the buttons and combination button fields may be candidates for similar treatment. Certain functional buttons may be combined (e.g. file (index) and File). Update buttons may be added to the appointment side of the "Appointments/To-Do" module. In the embodiment currently programmed, it is necessary to manually change from writing mode to browse mode by depressing the pen button in order to click on a button not on the menu bar. This switching could be automated as it is for the menu bar.

Computer assistance for transferring Long Range Calendar items to the "Appointments/To-Do" module may be implemented. Relating to the automatic updating of handwritten items and associated buttons and/or fields, an algorithm may be developed to make sure that if there were graphics or other items in the target area, that the item being moved can be shifted to a clear area to avoid overlaying existing items. As discussed under the heading of "Updating the work item" an option may be selected to allow the user to view the target area to ascertain if the update will overwrite any existing items. The following paragraph presents an example of a possible programming methodology for moving update items to a clear area after exercising the option to check the update target card. The discussion assumes that the relevant material in the microfiche appendix (Computer Program Listings for implementing the functions described) has been comprehended.

To move the source handwriting, buttons, and or fields to a clear target line, the user may select a checkbox having a clear target area. The coordinates of this selected area which are thus appended to the update buffer area "selectCoords" may be moved from the last set to the second set of check box coordinates in the buffer. To move the handwriting, the first set of selectCoords in the buffer is used to drag a rectangle around the source handwriting. Then, a programmed "click" at the first coordinate pair of that set (which are the first and second items of selectCoords) will select the enclosed handwriting. Finally, the captured material may be dragged from the first coordinate pair of the first set (items 1 and 2 of selectCoords) to the first coordinate pair of the second set--the new target (items 5 and 6 of selectCoords). To move a button or field, set the location of the source item to new coordinates where the new x coordinate is the same as the source location (derived with a "get location" command) and the new y coordinate is the first y coordinate of the second set (item 6 of selectCoords) plus 9.

The Tel-Assist button may automatically retrieve an information file associated with the party whose number is being dialed. A live pull-down calendar might be useful for quickly selecting target dates for updating "Appointments/To-Do" item. Especially useful would be scrolling graphics which would allow the user to create handwritten notes of virtually any size on a single card. A zoom feature which could reduce the size of note cards would also be highly useful for organizing and/or manipulating data from multiple cards on the same screen. Character recognition software may also be of value under certain circumstances, although data input rates are not as rapid aa with button keys. Security software may be added to offer selected protection of data. The an obvious extension to the reminder function would involve having the program pull up all reminder notes for dates which have passed as well as those for the current date. This may be done by modifying the code to compare the date on each reminder line with the current date and pull those cards having dates less than or equal to the current date. This would make it easier to examine reminder notes associated with dates in which the system was not used.

Finally, incorporation of a number of additional software modules (applications) can be very productive. Inclusion of these functions would encourage use of the system by a wider population. A project management module which, of course, featured handwritten input, would be very valuable, particularly for managers and executives (a class known to be not skilled in keyboard entry). An offshoot of the project management module which would be of interest to the public at large would involve personal goal management. Similarly, addition of a spreadsheet module would extend the system's utility to more users. This is an addition which is particularly sympathetic with the hardware. Numbers may be enter into spreadsheet cells less laboriously with the screen tablet hardware than with conventional alternates because selection of the target cell and entry of a number may be combined into a single operation. Keyboards and mice require two separate operations. Character recognition software may be employed for number entry or a miniature button-key number pad may be popped up next to the target cell. Use of the graphic manipulation capabilities of the system for writing and revising handwritten letters has not been mentioned but is obviously possible. Also, employment of the system for telephone messages, perhaps with a special card format is anticipated to be a high volume usage. An Expenses record keeping format would be useful, particularly for the portable embodiment. An obvious extension to the check writing module would be a check register to display the following check information: check (or deposit) number, date, payee and memo, and amount. Additionally, character recognition software may be used to digitize the handwritten entry of the check dollar amount in the numeric dollar field. This would obviate the need to enter this number with the button keys into the computer check image. An algorithm to smooth handwritten characters may also be employed to improve the ease with which it is read, particularly on displays of lower resolution.


Accordingly, the reader will see that I have provided a system which works with a screen tablet and a computer to provide a mechanism for easy entry of handwritten information into the computer. In this way the benefits of computing become available to (1) individuals who do not wish to use a keyboard to enter data, and (2) for applications in which the drawbacks of keyboard usage exceed the payoff. Benefits of computerization include great enhancement in the speed and reliability with which a user's information can be retrieved, stored, transmitted and manipulated as well as very significant reductions in labor associated with such information processing activities. It allows elimination of a large proportion of a user's manual files promising lower physical file storage costs. Because handwritten annotation or entries may be applied to computerized documents, the need for paper document objects is greatly reduced resulting not only in cost savings, but also in a reduction in usage of a scarce national resource.

Such a system can extend the benefits of computing, not only to large numbers of new users, but also to broad classes of new uses. Its comprehensive scope brings the benefits of computerized assistance to virtually all of an individual's commonly encountered information processing needs. This system also expands the applicability of portable computing to heretofore untapped areas. Specifically, incorporation of such a handwriting information management system into miniaturized portable computers overcomes the limitations of keyboard entry which seriously limits the rate at which data can be entered and hence their usefulness. Computerization of high frequency information processing activities which have heretofore proved resistant to automation, such as taking notes a meeting, now become possible. Finally, the naturalness of the pen and pad interface reflects an important improvement in the man-machine interface. In summary, the advantages of the present system include significant cost reductions, productivity and quality gains.

While the above description contains many specificities, these should not be construed as limitations on the invention, but rather as exemplification of preferred embodiments thereof. Many other variations are possible. Accordingly, the scope of the invention should be determined not by the embodiments described, but by the appended claims and their legal equivalents.

Patent Citations
Cited PatentFiling datePublication dateApplicantTitle
US4727588 *Oct 9, 1986Feb 23, 1988International Business Machines CorporationSystem for automatic adjustment and editing of handwritten text image
US4815029 *Sep 23, 1985Mar 21, 1989International Business Machines Corp.In-line dynamic editor for mixed object documents
US4817034 *Feb 11, 1986Mar 28, 1989E.S.P. Systems, Inc.Computerized handwriting duplication system
US4912640 *Aug 21, 1984Mar 27, 1990Sharp Kabushiki KaishaElectronic type memo device
US4924075 *Sep 29, 1988May 8, 1990Kabushiki Kaisha ToshibaSmart IC card
USRE32655 *Apr 1, 1985Apr 26, 1988Kyocera CorporationElectronic calendar and diary
Non-Patent Citations
1Grid Systems, sales brochures: "Introducing GridPad", 1989, Technical Specifications, 1989.
2 *Grid Systems, sales brochures: Introducing GridPad , 1989, Technical Specifications, 1989.
3IBM, research report: "The Paperlike Interface", Feb. 3, 1989, Wolf, Rhyne and Ellozy.
4 *IBM, research report: The Paperlike Interface , Feb. 3, 1989, Wolf, Rhyne and Ellozy.
5Linus Technologies, sales brochures, technology paper: "Linus Write-Top", undated; Forms Write/Arrows, 1989; Handwritten Data into a Computer, Jun. 1989.
6 *Linus Technologies, sales brochures, technology paper: Linus Write Top , undated; Forms Write/Arrows, 1989; Handwritten Data into a Computer, Jun. 1989.
7Toshiba America, sales brochure: "Toshiba Pen PC", 1989.
8 *Toshiba America, sales brochure: Toshiba Pen PC , 1989.
9Wang Laboratories Inc., various sales brochures: "Freestyle Personal Computing", Oct. 1988; The Wang Freestyle System, Oct. 1988; Wang Freestyle/LAN Office Interface, Apr. 1989.
10 *Wang Laboratories Inc., various sales brochures: Freestyle Personal Computing , Oct. 1988; The Wang Freestyle System, Oct. 1988; Wang Freestyle/LAN Office Interface, Apr. 1989.
Referenced by
Citing PatentFiling datePublication dateApplicantTitle
US5327342 *Oct 14, 1993Jul 5, 1994Roy Prannoy LMethod and apparatus for generating personalized handwriting
US5446882 *Oct 2, 1992Aug 29, 1995Apple Computer, Inc.Interface for a computerized database having card and list views
US5479596 *May 27, 1993Dec 26, 1995Apple Computer, Inc.Method and apparatus for formatting paragraphs
US5530795 *Feb 15, 1994Jun 25, 1996Wan; Hong K.Computer conferencing
US5544358 *May 26, 1995Aug 6, 1996Apple Computer, Inc.Interface for a computerized database having card and list views
US5546538 *Dec 14, 1993Aug 13, 1996Intel CorporationSystem for processing handwriting written by user of portable computer by server or processing by the computer when the computer no longer communicate with server
US5556339 *Jun 27, 1995Sep 17, 1996Cohen; Justin R.Computer picture toy for infants and very young children
US5566248 *Jan 24, 1996Oct 15, 1996Apple Computer, Inc.Method and apparatus for a recognition editor and routine interface for a computer system
US5603658 *Jun 27, 1995Feb 18, 1997Cohen; Justin R.Computer play toy for infants and very young children
US5606712 *Mar 17, 1995Feb 25, 1997Casio Computer Co., Ltd.Information managing apparatus capable of utilizing related information in different function modes
US5644735 *Apr 19, 1995Jul 1, 1997Apple Computer, Inc.Method and apparatus for providing implicit computer-implemented assistance
US5655118 *Mar 21, 1994Aug 5, 1997Bell Communications Research, Inc.Methods and apparatus for managing information on activities of an enterprise
US5664013 *Jun 7, 1995Sep 2, 1997Compaq Computer CorporationEmbedded notepad system for a cellular telephone
US5671438 *Jun 7, 1995Sep 23, 1997Apple Computer, Inc.Method and apparatus for formatting paragraphs
US5732221 *Mar 14, 1995Mar 24, 1998Documation, Inc.Electronic documentation system for generating written reports
US5734882 *Jun 5, 1995Mar 31, 1998Panasonic Technologies, Inc.Pictographic bitmap naming of files in pen-based computer systems
US5734915 *Oct 31, 1994Mar 31, 1998Eastman Kodak CompanyMethod and apparatus for composing digital medical imagery
US5737740 *Oct 25, 1995Apr 7, 1998NumonicsApparatus and method for processing electronic documents
US5748808 *Dec 6, 1994May 5, 1998Yashima Electric Co., Ltd.Image reproducing method and apparatus capable of storing and reproducing handwriting
US5778404 *Aug 7, 1995Jul 7, 1998Apple Computer, Inc.String inserter for pen-based computer systems and method for providing same
US5821881 *Aug 9, 1996Oct 13, 1998International Business Machines CorporationData input by means of a keyboard and a touch-sensitive graphical data input field
US5828783 *Oct 21, 1997Oct 27, 1998Fujitsu LimitedApparatus and method for input-processing hand-written data
US5864339 *Jul 28, 1994Jan 26, 1999Hewlett-Packard CompanyFreehand input information management device and method
US5873095 *Aug 12, 1996Feb 16, 1999Electronic Data Systems CorporationSystem and method for maintaining current status of employees in a work force
US5893126 *Aug 12, 1996Apr 6, 1999Intel CorporationMethod and apparatus for annotating a computer document incorporating sound
US6168440Aug 28, 1998Jan 2, 2001National Computer Systems, Inc.Multiple test item scoring system and method
US6183260Aug 28, 1998Feb 6, 2001National Computer Systems, Inc.Method and system for preventing bias in test answer scoring
US6183261Aug 28, 1998Feb 6, 2001National Computer Systems, Inc.Collaborative and quality control scoring system and method
US6193521Aug 28, 1998Feb 27, 2001National Computer Systems, Inc.System for providing feedback to test resolvers
US6215901 *Mar 7, 1997Apr 10, 2001Mark H. SchwartzPen based computer handwriting instruction
US6256009Feb 24, 1999Jul 3, 2001Microsoft CorporationMethod for automatically and intelligently scrolling handwritten input
US6256399Sep 27, 1999Jul 3, 2001Ncs Pearson, Inc.Method of distribution of digitized materials and control of scoring for open-ended assessments
US6337698 *Nov 20, 1998Jan 8, 2002Microsoft CorporationPen-based interface for a notepad computer
US6359572Sep 3, 1998Mar 19, 2002Microsoft CorporationDynamic keyboard
US6389434Apr 9, 1998May 14, 2002Aurigin Systems, Inc.System, method, and computer program product for creating subnotes linked to portions of data objects after entering an annotation mode
US6459779 *Sep 4, 1998Oct 1, 2002Ameritech CorporationSystem and method for creating a billing record with a called party's name
US6466683May 21, 2001Oct 15, 2002Ncs Pearson, Inc.System and method of distribution of digitized materials and control of scoring for open-ended assessments
US6552719 *Jan 7, 1999Apr 22, 2003Microsoft CorporationSystem and method for automatically switching between writing and text input modes
US6558166Sep 12, 2000May 6, 2003Ncs Pearson, Inc.Multiple data item scoring system and method
US6633746Nov 16, 1998Oct 14, 2003Sbc Properties, L.P.Pager with a touch-sensitive display screen and method for transmitting a message therefrom
US6642458Jun 19, 2002Nov 4, 2003Motorola, Inc.Touch screen device and method for co-extensively presenting text characters and rendering ink in a common area of a user interface
US6671684 *Sep 1, 2000Dec 30, 2003Ricoh Co., Ltd.Method and apparatus for simultaneous highlighting of a physical version of a document and an electronic version of a document
US6674426 *Mar 8, 2001Jan 6, 2004Oregon Health & Science UniversityAugmenting and not replacing paper based work practice via multi-modal interaction
US6675133Mar 5, 2001Jan 6, 2004Ncs Pearsons, Inc.Pre-data-collection applications test processing system
US6741749Jan 24, 2001May 25, 2004Advanced Digital Systems, Inc.System, device, computer program product, and method for representing a plurality of electronic ink data points
US6749435Apr 29, 2003Jun 15, 2004Ncs Pearson, Inc.Collaborative and quality control scoring system and method
US6751351Mar 5, 2001Jun 15, 2004Nsc Pearson, Inc.Test question response verification system
US6779178Sep 16, 1999Aug 17, 2004Signature Mail. Com, LlcSystem and method for personalizing electronic mail messages
US6798907Jan 24, 2001Sep 28, 2004Advanced Digital Systems, Inc.System, computer software product and method for transmitting and processing handwritten data
US6810232Mar 5, 2001Oct 26, 2004Ncs Pearson, Inc.Test processing workflow tracking system
US6816274 *May 23, 2000Nov 9, 2004Silverbrook Research Pty LtdMethod and system for composition and delivery of electronic mail
US6819315Feb 8, 2002Nov 16, 2004Microsoft CorporationSoft input panel system and method
US6820111Dec 7, 1999Nov 16, 2004Microsoft CorporationComputer user interface architecture that saves a user's non-linear navigation history and intelligently maintains that history
US6825956 *Nov 12, 2002Nov 30, 2004Silverbrook Research Pty LtdMethod and system for composition and delivery of electronic mail with coded marks
US6826551May 10, 2000Nov 30, 2004Advanced Digital Systems, Inc.System, computer software program product, and method for producing a contextual electronic message from an input to a pen-enabled computing system
US6862105 *Nov 12, 2002Mar 1, 2005Silverbrook Research Pty LtdMethod and system for composition and delivery of electronic mail using processing sensor
US6868147Aug 15, 2002Mar 15, 2005Sbc Properties, L.P.System and method for creating a billing record with a called party's name
US6877137Dec 7, 1999Apr 5, 2005Rose Blush Software LlcSystem, method and computer program product for mediating notes and note sub-notes linked or otherwise associated with stored or networked web pages
US6918772Oct 20, 2003Jul 19, 2005Ncs Pearson, Inc.Categorized data item reporting system and method
US6957233Dec 7, 1999Oct 18, 2005Microsoft CorporationMethod and apparatus for capturing and rendering annotations for non-modifiable electronic content
US6961482Mar 5, 2001Nov 1, 2005Ncs Pearson, Inc.System for archiving electronic images of test question responses
US6970828 *Dec 19, 2000Nov 29, 2005Xerox CorporationMethod for mixed human and computer-supported distributed scheduling
US6976847Oct 20, 2003Dec 20, 2005Ncs Pearsons, Inc.System for providing feedback to evaluators
US6992687Dec 7, 1999Jan 31, 2006Microsoft CorporationBookmarking and placemarking a displayed document in a computer system
US7009738Nov 12, 2002Mar 7, 2006Silverbrook Research Pty LtdMethod and system for composition and delivery of electronic mail using sensor with identifier
US7013029 *Jun 29, 2001Mar 14, 2006Intel CorporationIncorporating handwritten notations into an electronic document
US7028267Dec 7, 1999Apr 11, 2006Microsoft CorporationMethod and apparatus for capturing and rendering text annotations for non-modifiable electronic content
US7032187Dec 11, 2001Apr 18, 2006Microsoft CorporationPen-based interface for a notepad computer
US7044743Oct 20, 2003May 16, 2006Ncs Pearson, Inc.Dynamic on-line scoring guide and method
US7054464Oct 15, 2002May 30, 2006Ncs Pearson, Inc.System and method of distribution of digitized materials and control of scoring for open-ended assessments
US7091959Mar 31, 2000Aug 15, 2006Advanced Digital Systems, Inc.System, computer program product, computing device, and associated methods for form identification and information manipulation
US7120299Dec 28, 2001Oct 10, 2006Intel CorporationRecognizing commands written onto a medium
US7139762 *Feb 27, 2001Nov 21, 2006Microsoft CorporationSystem and method for filtering database records
US7149292Jun 29, 2004Dec 12, 2006Sbc Holdings Properties, L.P.System and method for creating a billing record with a called party's name
US7185274Dec 7, 1999Feb 27, 2007Microsoft CorporationComputer user interface architecture wherein users interact with both content and user interface by activating links
US7234108 *Jun 29, 2000Jun 19, 2007Microsoft CorporationInk thickness rendering for electronic annotations
US7243299Apr 21, 2000Jul 10, 2007Microsoft CorporationMethods and apparatus for displaying multiple contexts in electronic documents
US7260781Jun 13, 2003Aug 21, 2007Microsoft CorporationSystem, method and user interface for active reading of electronic content
US7337389Dec 7, 1999Feb 26, 2008Microsoft CorporationSystem and method for annotating an electronic document independently of its content
US7382482Feb 17, 2005Jun 3, 2008Silverbrook Research Pty LtdMethod and apparatus for allowing a user to send an electronic mail message
US7388685Sep 20, 2004Jun 17, 2008Silverbrook Research Pty LtdMethod and system for composition and transmission of an electronic mail message via an interactive document
US7411582Nov 15, 2004Aug 12, 2008Microsoft CorporationSoft input panel system and method
US7421155Apr 1, 2005Sep 2, 2008Exbiblio B.V.Archive of text captures from rendered documents
US7437023Aug 18, 2005Oct 14, 2008Exbiblio B.V.Methods, systems and computer program products for data gathering in a digital and hard copy document environment
US7458014Dec 7, 1999Nov 25, 2008Microsoft CorporationComputer user interface architecture wherein both content and user interface are composed of documents with links
US7466440 *Aug 12, 2005Dec 16, 2008Silverbrook Research Pty LtdDevice and method for communication
US7486824Aug 16, 2004Feb 3, 2009Ads Software Mgmt. L.L.C.System, computer software product and method for transmitting and processing handwritten data
US7496829Sep 15, 2004Feb 24, 2009Microsoft CorporationMethod and apparatus for displaying multiple contexts in electronic documents
US7496830Jun 25, 2004Feb 24, 2009Microsoft CorporationComputer user interface architecture that saves a user's non-linear navigation history and intelligently maintains that history
US7496856Sep 26, 2005Feb 24, 2009Microsoft CorporationMethod and apparatus for capturing and rendering text annotations for non-modifiable electronic content
US7508926Oct 19, 2006Mar 24, 2009At&T Intellectual Property 1, L.P.Method and system for creating a billing record with a called party's name
US7559037Jan 27, 2006Jul 7, 2009Microsoft CorporationPen-based interface for a notepad computer
US7568168Sep 14, 2005Jul 28, 2009Microsoft CorporationMethod and apparatus for capturing and rendering text annotations for non-modifiable electronic content
US7593605Apr 1, 2005Sep 22, 2009Exbiblio B.V.Data capture from rendered documents using handheld device
US7594187Jun 21, 2004Sep 22, 2009Microsoft CorporationBookmarking and placemarking a displayed document in a computer system
US7596269Apr 1, 2005Sep 29, 2009Exbiblio B.V.Triggering actions in response to optically or acoustically capturing keywords from a rendered document
US7599580Apr 1, 2005Oct 6, 2009Exbiblio B.V.Capturing text from rendered documents using supplemental information
US7599844Apr 1, 2005Oct 6, 2009Exbiblio B.V.Content access with handheld document data capture devices
US7606741Apr 1, 2005Oct 20, 2009Exbibuo B.V.Information gathering system and method
US7639876Jan 14, 2005Dec 29, 2009Advanced Digital Systems, Inc.System and method for associating handwritten information with one or more objects
US7702624Apr 19, 2005Apr 20, 2010Exbiblio, B.V.Processing techniques for visual capture data from a rendered document
US7703047 *May 5, 2005Apr 20, 2010Microsoft CorporationPen-based interface for a notepad computer
US7706611Aug 23, 2005Apr 27, 2010Exbiblio B.V.Method and system for character recognition
US7707039Dec 3, 2004Apr 27, 2010Exbiblio B.V.Automatic modification of web pages
US7720286May 25, 2005May 18, 2010Advanced Digital Systems, Inc.System and method for associating handwritten information with one or more objects via discontinuous regions of a printed pattern
US7730391Nov 24, 2006Jun 1, 2010Microsoft CorporationInk thickness rendering for electronic annotations
US7742953Apr 1, 2005Jun 22, 2010Exbiblio B.V.Adding information or functionality to a rendered document via association with an electronic counterpart
US7769611Nov 3, 2000Aug 3, 2010International Business Machines CorporationSystem and method for automating travel agent operations
US7777729Jun 27, 2006Aug 17, 2010Clary Gregory JSystem and method for editing handwritten data
US7812860Sep 27, 2005Oct 12, 2010Exbiblio B.V.Handheld device for capturing text from both a document printed on paper and a document displayed on a dynamic display device
US7812987Oct 12, 2010Silverbrook Research Pty LtdSystem for composition and transmission of an e-mail message
US7818215May 17, 2005Oct 19, 2010Exbiblio, B.V.Processing techniques for text capture from a rendered document
US7825897May 5, 2005Nov 2, 2010Microsoft CorporationPen-based interface for a notepad computer
US7831912Apr 1, 2005Nov 9, 2010Exbiblio B. V.Publishing techniques for adding value to a rendered document
US7855718Jan 3, 2007Dec 21, 2010Apple Inc.Multi-touch input discrimination
US7859699Jun 1, 2007Dec 28, 2010Silverbrook Research Pty LtdMethod of generating E-mail message
US7869655Feb 2, 2009Jan 11, 2011Ads Software Mgmt. L.L.C.System, computer software product and method for transmitting and processing handwritten data
US7876310Jan 3, 2007Jan 25, 2011Apple Inc.Far-field input identification
US7877111Jan 25, 2011Silverbrook Research Pty LtdMobile device for enabling interaction with a printed email document
US7929170Nov 23, 2008Apr 19, 2011Silverbrook Research Pty LtdSystem for electronic transmission of handwritten information
US7949728Aug 31, 2006May 24, 2011Rose Blush Software LlcSystem, method, and computer program product for managing and analyzing intellectual property (IP) related transactions
US7990556Feb 28, 2006Aug 2, 2011Google Inc.Association of a portable scanner with input/output and storage devices
US8005720Aug 18, 2005Aug 23, 2011Google Inc.Applying scanned information to identify content
US8019648Apr 1, 2005Sep 13, 2011Google Inc.Search engines and systems with handheld document data capture devices
US8024652Apr 10, 2007Sep 20, 2011Microsoft CorporationTechniques to associate information between application programs
US8081849Feb 6, 2007Dec 20, 2011Google Inc.Portable scanning and memory device
US8115748Jun 18, 2010Feb 14, 2012Ads Software Mgmt. L.L.C.Electronically capturing handwritten data
US8130203May 31, 2007Mar 6, 2012Apple Inc.Multi-touch input discrimination
US8179563Sep 29, 2010May 15, 2012Google Inc.Portable scanning device
US8193908Jul 16, 2003Jun 5, 2012At&T Intellectual Property I, LpPager with a touch-sensitive display screen and method for transmitting a message therefrom
US8196041Aug 21, 2007Jun 5, 2012International Business Machines CorporationMethod and system for processing information relating to active regions of a page of physical document
US8214387Apr 1, 2005Jul 3, 2012Google Inc.Document enhancement system and method
US8243041Jan 18, 2012Aug 14, 2012Apple Inc.Multi-touch input discrimination
US8261094Aug 19, 2010Sep 4, 2012Google Inc.Secure data gathering from rendered documents
US8269727Jan 3, 2007Sep 18, 2012Apple Inc.Irregular input identification
US8297979May 27, 2005Oct 30, 2012Mattel, Inc.Electronic learning device with a graphic user interface for interactive writing
US8346620Sep 28, 2010Jan 1, 2013Google Inc.Automatic modification of web pages
US8358289Jan 22, 2013Silverbrook Research Pty LtdController arrangement for an optical sensing pen with an infrared data association (IRDA) interface
US8384684Dec 10, 2010Feb 26, 2013Apple Inc.Multi-touch input discrimination
US8418055Feb 18, 2010Apr 9, 2013Google Inc.Identifying a document by performing spectral analysis on the contents of the document
US8442331Aug 18, 2009May 14, 2013Google Inc.Capturing text from rendered documents using supplemental information
US8447066Mar 12, 2010May 21, 2013Google Inc.Performing actions based on capturing information from rendered documents, such as documents under copyright
US8489624Jan 29, 2010Jul 16, 2013Google, Inc.Processing techniques for text capture from a rendered document
US8505090Feb 20, 2012Aug 6, 2013Google Inc.Archive of text captures from rendered documents
US8515816Apr 1, 2005Aug 20, 2013Google Inc.Aggregate analysis of text captures performed by multiple users from rendered documents
US8531425Jul 27, 2012Sep 10, 2013Apple Inc.Multi-touch input discrimination
US8542210Feb 15, 2013Sep 24, 2013Apple Inc.Multi-touch input discrimination
US8555198Jul 8, 2009Oct 8, 2013Microsoft CorporationAnnotations for electronic content
US8600196Jul 6, 2010Dec 3, 2013Google Inc.Optical scanners, such as hand-held optical scanners
US8620083Oct 5, 2011Dec 31, 2013Google Inc.Method and system for character recognition
US8627197Jul 1, 2011Jan 7, 2014Microsoft CorporationSystem and method for annotating an electronic document independently of its content
US8638363Feb 18, 2010Jan 28, 2014Google Inc.Automatically capturing information, such as capturing information using a document-aware device
US8713418Apr 12, 2005Apr 29, 2014Google Inc.Adding value to a rendered document
US8781228Sep 13, 2012Jul 15, 2014Google Inc.Triggering actions in response to optically or acoustically capturing keywords from a rendered document
US8791921Aug 19, 2013Jul 29, 2014Apple Inc.Multi-touch input discrimination
US8799099Sep 13, 2012Aug 5, 2014Google Inc.Processing techniques for text capture from a rendered document
US8831365Mar 11, 2013Sep 9, 2014Google Inc.Capturing text from rendered documents using supplement information
US8874504Mar 22, 2010Oct 28, 2014Google Inc.Processing techniques for visual capture data from a rendered document
US8892495Jan 8, 2013Nov 18, 2014Blanding Hovenweep, LlcAdaptive pattern recognition based controller apparatus and method and human-interface therefore
US8953886Aug 8, 2013Feb 10, 2015Google Inc.Method and system for character recognition
US8990235Mar 12, 2010Mar 24, 2015Google Inc.Automatically providing content associated with captured information, such as information captured in real-time
US9008447Apr 1, 2005Apr 14, 2015Google Inc.Method and system for character recognition
US9024906Jul 28, 2014May 5, 2015Apple Inc.Multi-touch input discrimination
US9030699Aug 13, 2013May 12, 2015Google Inc.Association of a portable scanner with input/output and storage devices
US9075779Apr 22, 2013Jul 7, 2015Google Inc.Performing actions based on capturing information from rendered documents, such as documents under copyright
US9081799Dec 6, 2010Jul 14, 2015Google Inc.Using gestalt information to identify locations in printed information
US9092545Aug 31, 2006Jul 28, 2015Rose Blush Software LlcIntellectual property asset manager (IPAM) for context processing of data objects
US9116890Jun 11, 2014Aug 25, 2015Google Inc.Triggering actions in response to optically or acoustically capturing keywords from a rendered document
US9134947 *Mar 31, 2009Sep 15, 2015Ricoh Company, Ltd.Annotating digital files of a host computer using a peripheral device
US9143638Apr 29, 2013Sep 22, 2015Google Inc.Data capture from rendered documents using handheld device
US20020077878 *Dec 19, 2000Jun 20, 2002Xerox CorporationMethod for mixed human and computer-supported distributed scheduling
US20030004991 *Jun 29, 2001Jan 2, 2003Keskar Dhananjay V.Correlating handwritten annotations to a document
US20050013422 *Jun 29, 2004Jan 20, 2005Wardin William KonradSystem and method for creating a billing record with a called party's name
US20050157334 *Feb 17, 2005Jul 21, 2005Kia SilverbrookMethod and apparatus for allowing a user to send an electronic mail message
US20050198592 *May 5, 2005Sep 8, 2005Microsoft CorporationPen-based interface for a notepad computer
US20050200636 *May 9, 2005Sep 15, 2005Silverbrook Research Pty LtdMobile device for enabling interaction with a printed email document
US20050204301 *May 5, 2005Sep 15, 2005Microsoft CorporationPen-based interface for a notepad computer
US20050204305 *May 5, 2005Sep 15, 2005Microsoft CorporationPen-based interface for a notepad computer
US20050270576 *Aug 12, 2005Dec 8, 2005Silverbrook Research Pty LtdDevice and method for communication
US20060136840 *Jan 27, 2006Jun 22, 2006Microsoft CorporationPen-based interface for a notepad computer
US20120113011 *Mar 20, 2009May 10, 2012Genqing WuIme text entry assistance
WO1997015891A1 *Oct 24, 1996May 1, 1997Numonics CorpApparatus and method for processing electronic documents
U.S. Classification382/186, 400/83, 400/61, 382/187, 345/173, 715/839, 708/110, 708/107, 715/781, 715/251, 715/268
International ClassificationG06F3/033
Cooperative ClassificationG06F3/033, G06F3/04883
European ClassificationG06F3/0488G, G06F3/033
Legal Events
Jun 13, 1995REMIMaintenance fee reminder mailed
Aug 8, 1995FPAYFee payment
Year of fee payment: 4
Aug 8, 1995SULPSurcharge for late payment
May 17, 1999FPAYFee payment
Year of fee payment: 8
May 17, 1999SULPSurcharge for late payment
Nov 5, 2002FPAYFee payment
Year of fee payment: 12