|Publication number||US5963966 A|
|Application number||US 08/746,450|
|Publication date||Oct 5, 1999|
|Filing date||Nov 8, 1996|
|Priority date||Nov 8, 1995|
|Publication number||08746450, 746450, US 5963966 A, US 5963966A, US-A-5963966, US5963966 A, US5963966A|
|Inventors||Brian T. Mitchell, Margaret J. Whalen|
|Original Assignee||Cybernet Systems Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (6), Referenced by (257), Classifications (9), Legal Events (6)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This application claims priority of U.S. provisional application Ser. No. 60/006,372, filed Nov. 8, 1995, the entire contents of which are incorporated herein by reference.
The present invention relates generally to document processing, and, in particular, to automated document conversion to digital form for broadcast over networks, including the World Wide Web, and/or transfer media such as disks or CD-ROM.
Two major components of document conversion are page decomposition and text recognition (OCR). Page decomposition identifies the overall layout of a document page, whereas OCR identifies the ASCII components found within the page components.
A page decomposition or segmentation module accepts an input document page, and processes it into its constituent parts, including text, tables, references, procedural data, and graphics. Because page decomposition occurs first in the processing chain, it is one of the most important modules of any automatic document understanding system. It is very difficult to recover from any errors that occur at this stage of processing, and as such it is important that a very reliable page decomposition module be developed.
Most page segmentation methods can be classified into three broad categories: bottom-up, top-down and hybrid. The bottom-up strategies usually begin with the connected components of the image and merge them into larger and larger regions. The components are merged into words, words into lines, lines into columns, etc., until the entire page is completely assembled. In the top-down approaches, the page is first split into blocks, and these blocks are identified and subdivided appropriately, often using projection profiles. The hybrid methods combine aspects of both the top-down and bottom-up approaches. The page can be roughly segmented by a sequence of horizontal and vertical projections and then connectivity analysis is used to complete the segmentation.
Existing document conversion techniques include: 1) OCR (optical character recognition) document conversion, 2) form-based OCR, and 3) combined image and OCR systems. The first technique understands and translates the document into its individual components. The second utilizes spatial and content constraints of document forms to increase OCR reliability. The third leaves the document in its image form, but also utilizes OCRed text to provide indexing into the image database.
An OCR document conversion system which translates document images into ASCII and graphics components is illustrated in FIG. 1. The approach begins with page decomposition, wherein each page is segmented into graphic and text regions. Algorithms to process these regions are incorporated into the system. Graphical operations include extracting text from within graphics and raster-to-vector conversion. OCR is computed for the text regions, as well as any text located within the graphical regions. An integration step combines the results of the graphical and textual processes into a final electronic format.
The ASCII and graphics format obtained in this system is the only record of the document which is stored, as no images of the original document are kept. Since any errors introduced during page decomposition are propagated forward in the system, it is important that the graphical and textual regions of the page be correctly identified. Thus, the automatic regions generated must be manually checked or the page must be manually separated into regions. It is also important that the text be correctly recognized to ensure that the document is correctly captured. This requires that the OCR results be manually scanned and any errors corrected. Similarly, the text within graphics must be correctly extracted and recognized, again requiring manual checking. This manual checking can involve large cost which is often prohibitive for many applications.
Form-based OCR systems use the spatial and content constraints found in document forms to increase OCR conversion performance. According to this approach, a "form" may be defined as a document containing data written in fields that are spatially stable on the document. All form processing systems require the creation of a template showing the system what a particular form looks like and where to find the fields to read.
Form identification decides which master form was used in a given image, and passes that information to the form removal functions. Form removal strips all standard data from the scanned forms, including lines, instructions and examples, leaving only the information entered into the form by the applicant. The space necessary to store images is reduced by allowing users to save stripped forms and later add back the master form data before displaying or printing.
Full function form processors, in addition to machine print, also read hand printed characters and optical marks such as checkboxes. Form processing systems offer a broad array of tools, some standardized and some custom, which exploit information specific to a particular form type for error detection or correction. Restrictions can be applied to fields to increase recognition accuracy. For example, the field masking tool permits individual characters within a field to be recognized exclusively as either an alpha character or a numeric character. Thus, a social security number field can be required to contain only nine numeric characters and possibly two dashes. Segmentation of the field is then limited to 9 or 11 characters and recognition is limited to 10 digits and 1 dash. Additional tools include external table lookups, where field contents can be compared to a table of possible responses and the best match selected; checks of digit computations, where errors are detected by performing a mathematical operation on a field's contents and comparing the result to a predetermined total contained in the same field; and range limits, where the number in the field is checked to determine if it is within a valid range.
Unlike generic OCR systems, which only allow the error tolerance level to be set globally, form processing systems permit error tolerance levels (confidence levels) to be set on a field level. Thus, important fields, where field importance differs based on the form, can have their confidence levels set to a threshold requiring higher OCR accuracy.
Combined OCR and image systems utilize both data sources to provide robust retrieval, as illustrated in FIG. 2. The approach does not convert the document page directly to ASCII and graphics format, but rather saves a bit-mapped image of the document page and devises indexing schemes to retrieve specific pages. When a page is retrieved operations can then be performed on the areas of interest. For example, OCR can be computed on selected areas, or graphics can be extracted.
Once the text regions of a document have been identified, the characters within the regions need to be recognized. There are many different approaches to character recognition, but they can be generally grouped into two main categories: template-based methods and feature-based methods. Template methods maintain a collection of sample letters and identify a component in question by finding the closest-matching template. Feature methods, on the other hand, try to break the component into a collection of "features" by identifying where strokes join or curve significantly.
The classic template solutions compare each component to a collection of models representing all possible letters in all possible fonts. Thus, templates must be created for each of the different fonts. Contrarily, feature based recognition algorithms need not be tuned to individual typefaces, because they are based on finding characteristic features of each letter. For example, regardless of the typeface, a lowercase "t" consists of a strong vertical stroke crossed with a horizontal stroke. Thus, the feature based methods attempt to find this essence of the letter.
Each of the OCR techniques has its benefits and shortcomings. Combining the various methods in a voting scheme can overcome the limitations of each of the individual methods. In a voting scheme, the results of each of the OCR modules are passed to a decision module to determine a final recognition result. Since the decision module has knowledge about each of the OCR modules, it can determine the best possible answer.
The decision module can keep track of the character results and which OCR methods presented the correct response to the decision module. For example, if three methods report that the input character is a "B" and one method decides the character is an "8", the decision module will likely choose "B" as the best result. Further, the module that made the mistake will be noted for the next time. This adaptive learning approach allows the system to learn from its mistakes.
It is important to note that voting systems perform best when the hypotheses from the OCR systems are of high accuracy. When a text region is degraded and difficult to read, there is usually much disagreement among the recognizers, which is difficult for a voting system to resolve.
Each year, the Information Science Research Institute (ISRI) at the University of Nevada, Las Vegas (UNLV) conducts a test of the performance of various OCR systems, many of which are commercially available. Although recently tested OCR systems do not quite reach 100%, current recognition rates are impressive and improvement is ongoing. Achieving the last few percent is always the most difficult part, but OCR developers are steadily increasing their performance. With the incorporation of a voting scheme, the recognition rates increase even more.
If the OCR generated text is to be used in a text retrieval application, the percentage of words correctly recognized by the OCR system is of considerable interest. In a text retrieval system, the documents are retrieved from a database by matching search terms with words in the document. Thus, the word accuracy of the OCR-generated text is very important. Common words, such as "and," "of," "the," etc., usually provided no retrieval value in an indexing system. These words are termed stop words, and all other words are termed non-stop words. It is the recognition rate of these non-stop words that is of greatest importance to text retrieval applications.
If OCR is to be used as a conversion process to input technical manuals into ASCII form, manual checking and correcting of the OCR of the text will be necessary. Assuming an OCR character accuracy rate of 99%, a page with 4000 characters would result in 40 character errors per page. The issue becomes the cost of this manual correction versus the effectiveness of OCR, i.e. is it cheaper to correct the OCRed version or simply retype it?
To answer this question, we conducted a test using the OmniPage Professional OCR product to determine the time needed to correct an OCRed document versus the time needed to retype the document. It was assumed that the documents can be scanned and OCRed in a batch mode with the results saved to a file for future manual correction. Thus, only the actual labor costs are measured, not any time spent scanning and recognizing the document.
Seven pages from various documents were scanned and OCRed. The pages chosen for this test were quite simple, but included different fonts, and bold and italic characters. They contained single columns of text, no graphics and very few underlined sentences, since underlines tended to present a problem to the OmniPage recognizer. A bibliography page was also included in the set to introduce digits (from dates and page numbers) and proper nouns (author's names) which cannot be automatically corrected by dictionary look-ups.
OmniPage offers a method to check its OCR results. Any characters that the system has a difficult time recognizing are highlighted and the original image of the word in the context of the original page is presented to the user for possible correction. This method does not flag all OCR errors and presents numerous correct characters for viewing. Thus, this process was not incorporated in our timing test. The person correcting the text did not use this feature of the OmniPage system, but was allowed to use Microsoft Word spell checker to flag possible misspellings for correction.
Each OCRed page was manually corrected, and the correction time recorded. It took approximately 56 minutes to correct the seven pages. Assuming a typist can type 50 wpm, the time to retype these pages is 74 minutes or about 62 minutes at 60 wpm. From these numbers, it appears that OCRing the documents may be slightly more beneficial. However, a closer review of the manual corrections is needed.
The typical OCR errors include character omissions, additions and substitutions, bold and italic typeface errors, and incorrect spacing. Most of the missed errors were words recognized as bold typeface which were not bold in the original documents. The bibliography page (page five in the tables above) proved to be quite a challenge for the OCR system with fifty detected errors. The page was included because of its intermingling of digits and characters, and its inclusion of proper names and acronyms. This type of text must be carefully reviewed for errors. It is not like regular paragraphs where the corrector can simply read the flow of the sentences to check it. Dates, page numbers, and author's names must be carefully checked. Indeed, although the time required for retyping and manually correcting the pages in our test set were similar, the manual correction stage still left many errors uncorrected. Depending on the accuracy required, each page may need to be corrected by more than one person, thus doubling the time of manual correction.
The results of this experiment confirmed that OCR technology cannot be used to convert documents (either automatically or semi-automatically) in a cost-effective manner. More cost-effective methods are desperately needed, however, to convert existing large-scale, paper-document data bases into electronic form. Within the U.S. Government community, for example, reauthoring technical manuals into hypertext format costs between $200 and $1500 per page.
The use of hypertext documents has proven as a costeffective tool for supporting military equipment maintenance through the Department of Defense (DoD) Computer-aided Acquisition and Logistic Support (CALS) program. In this program, a hypertext format (IETM) was used for storing textual, graphical, audio, or video data in a revisable database. The IETM form enables the electronic data user to locate information easily, and to present it faster, more comprehensibly, more specifically matched to the configuration, and in a form that requires much less storage than paper. Power troubleshooting procedures not possible with paper Technical Manuals are possible using the computational capability of the IETM Display Device.
At the center of the IETM concept is the Interactive Electronic Technical Manual DataBase (IETMDB). This data structure is constructed from composite nodes which form the basic units of information within the IETMDB. These nodes are comprised of primitives, relationships to other pieces of information, and context attributes. The primitives include text, tables, graphics, and dialogs. The IETMDB is "format-free" in that it does not contain presentation information. As such, it does not impose structural requirements on the actual Data Base Management System (DBMS) methodology in use.
In summary, a hypertext-based approach to document conversion has potential for large-scale projects. However, in order to serve a greater technical and digital library community, existing hypertext approaches will need to be extended, to include more general encoding, revising, and distribution capabilities applicable to electronic technical data and documents.
Broadly, the present invention provides a method for converting paper documents into a hypertext-based format so that they can be accessed through networks such as the Internet or on media such as disk or CD-ROM. The method eliminates many of the errors normally associated with existing document conversion in a very cost effective manner, thus providing a low-cost, high-performance solution for converting existing paper-based databases into a form that can be accessed through the information highway.
More particularly, the method generalizes the concept of form-based recognition while extending the concept of document retrieval to include document structure knowledge, thereby providing the advantages found in both form-based recognition (utilization of document structure knowledge) and image-based information retrieval (robustness).
In one embodiment, the process utilizes the SGML format as a primary translation target to leverage ongoing work in developing this data format standard and tools. SGML is also the required markup language for the CALS text files as established by DoD standard MIL-M-28001, and thus, this design can be used to convert technical documents to this military standard. The invention also enables direct translation into HTML format (a subset of SGML), and thus provides a mechanism for translating documents into a format that can be accessed through the Internet using current browsers like Mosaic or Netscape. Accordingly, the method can be used to create electronic documents for a rapidly increasing Internet user population, which is currently estimated to be growing at a rate of over 70,000 new users each month.
FIG. 1 is a diagram which represents a classic OCR approach to document conversion;
FIG. 2 is a diagram which represents a combined image and OCRed text approach to document conversion;
FIG. 3 is an overview of a system design according to the invention;
FIG. 4 is a screen display of an HTML, image-based document;
FIG. 5 is a diagram illustrating an HTML generator according to the invention;
FIG. 6 is a HTML page for the system of FIG. 5;
FIG. 7 is a screen display of Mosaic page generated according to the invention;
FIG. 8 is a file map for the pages generated according to FIGS. 6 and 7;
FIG. 9 is a section of text which shows words associated with hyperlinks stored as reverse video;
FIG. 10 is a shell program to perform search and display results;
FIG. 11 show results of search for the word "vector";
FIG. 12 is a scan of text at 300 dpi;
FIG. 13 is a scan of text at 72 dpi;
FIG. 14 is a screen capture of a 72 dpi image; and
FIG. 15 is a stripe decomposition with 3 layers and 3 stripes.
The present invention resides in a generalized OCR engine which extends techniques found in form-based OCR and combined-image-text OCR technologies. An overview of the method corresponding applications is illustrated in FIG. 3, and includes two primary components: 1)translation from paper to electronic form (SGML, HTML, text), and 2) a user application for reading these electronic documents. As also seen in the figure, the translation component is built from a generalized OCR engine and an authoring environment.
The generalized OCR engine according to this invention combines desirable features found in the form-based recognition systems and image-retrieval systems to produce a solution that is both cost-effective (in its solution) and elegant (in its generality) by generalizing the concept of form-based recognition to include both logical (document structure) with physical (spatial location).
The authoring environment provides a mechanism not only for controlling the document conversion process overall, but for adding to documents, once converted. As it is inevitable that the converted documents will need to be modified or extended, the complete document conversion process may be embedded within a multimedia editor which combines imaging, word processing, and multimedia tools in support of document extension. Conversion is only the first step. Active use of the resulting electronic documents is essential for sustaining value for many of the targeted databases. This approach is essential for maximizing the overall value of the document conversion system.
The user application (browser) component allows the user to access the resultant document using several different potential user applications, depending on the form of the electronic document. One example is an Internet Browser. By translating the document into HTML format (or an extended version of HTML), Internet users may easily access the translated documents regardless of geography. This is a powerful idea, in that it eliminates existing boundaries which limit the use of the knowledge contained within many existing paper-based databases.
Prototype HTML document Format Description
To demonstrate the potential of creating Internet documents, a prototype HTML document based on both images and OCRed text was implemented. A representative document, scanned from its paper form into an HTML document, is viewable at the following World Wide Web address: http://www.cybernet.com/˜ganz/ietm-- index.html using an HTML browser such as Mosaic, MS Explorer or Netscape. A simple search routine has also been implemented, which allows the document to be searched using the dirty OCRed text. A screen shot of this prototype document is illustrated in FIG. 4. Note that the Table Of Contents found within this figure contains items that are hyperlinked (using document structure knowledge) to various document images that can be brought into the viewer. Also note that the slider bar can be used to move sequentially through the document in a manner similar to reading a paper document from start to finish.
In developing this prototype, a demonstration document conversion method was also developed wherein the pages of the paper document were scanned and OCRed. Any errors in the automatic zone detection were corrected, thus all text and graphics were correctly separated and labeled. However, no effort was made to correct the resulting OCR, thus the text database was uncorrected or "dirty". FIG. 5 provides an overview of the process used in developing this prototype.
As illustrated in the figure by dashed lines, the parsing of the sample paper document into a table of contents and list of figures was done manually, although this could also be automated. An HTML page was created with hyperlinks to connect each item in the Table of Contents and List of Figures with its corresponding HTML document page. A document understanding algorithm using document structure knowledge is preferably used to parse the text and create the Table of Contents, List of Figures and appropriate hyperlinks.
Method for HTML Coding of Pages
The method described encodes the document by forming an HTML (or SGML) page for each page in the source paper document. Each HTML page (typically a separate file, but multiple pages can be accommodated in a single file) contains an image of a document page and hyperlinks to four other pages, including the index page containing the Table of Contents, the section page containing the beginning of the section, the previous page of the document, and the next page of the document (links to other pages or HTML documents can also be inserted as needed for special formats). FIG. 6 shows the resulting HTML page.
The display generated by the HTML page on a Mosaic browser is illustrated in FIG. 7. As seen in the figure, the words between <TITLE> and </TITLE> in the HTML page appear in the title section of the Mosaic browser. The <BODY> section of the HTML page creates the four buttons Index! Up! Previous! and Next! seen in the Mosaic display. It also establishes the hyperlinks between the buttons and the specific proposal page to be viewed when a button is selected. For convenience, these buttons appear at both the top and bottom of the displayed page.
The <BODY> section of the HTML page also identifies the page image to be displayed when this HTML page is accessed. The <IMG> tag provides the location of the image, as well as its height and width. The browser uses the height and width information to place a properly-sized bounding box upon encountering the IMG tag and continues laying out the document text, with no performance delay to go discover the width and height of the image over the network.
Also included in the <IMG> tag is the ISMAP specification. This attribute indicates that the image is active and clicking inside the image may cause hyperlinks to be accessed. The ISMAP tag requires that a map file for the image be included. When a user clicks on the image, the coordinates of the click are passed to a gateway program image map and the appropriate map file is accessed to determine the hyperlink. FIG. 8 shows the map file for page 6 of the prototype document. Clicking on the appropriate page image within the rectangle (89, 258, 199, 269), which corresponds to the reverse video area containing the words "(U.S.P.S. Zip Codes)," accesses page 18 of the prototype. Clicking anywhere else within the image accesses the no operation shell, no-- op.sh, and no change will be made.
The images of each of the 25 pages in the prototype document are stored in the Graphics Interchange Format (GIF) format. Although some browsers support multiple image formats, there are three formats that are always viewable, GIF images, X-Bitmaps and X-Pixelmaps. Both the X-Bitmap and X-Pixelmap formats store the image data as ASCII text formatted as Standard C character string arrays, and, as such, are an inefficient way of storing large images. Thus, GIF is currently the most common image format in World Wide Web applications.
The GIF format can store black-and-white, grayscale or color images, with a limit of 256 colors per image. The image data in GIF format is always compressed using the Lempel-Ziv-Welch (LZW) compression scheme. Thus, the images are stored in compressed format, and algorithms read the compressed GIF files without an intermediate step of having to uncompress the entire image.
Method for Creating Image Pages
To create the binary images of document pages, each page is scanned. In this example data scanning was done at 300 dots-per-inch, although higher or lower resolutions are possible. Since the sample document pages contain large white borders that do not contribute any information, the images were cropped to remove the borders and then rescaled to 8.0×10.35 at 300 dpi resolution. The resulting sample images are approximately 910 KBytes in size, which would be much too large to send across a telephone network in a timely manner (it would be acceptable over T1 networks or better and as stored data on a CDROM).
Since most monitors can only display 72 dpi, it is best if only 72 dpi images are sent across the network. Otherwise, data that cannot be displayed is being wastefully sent across the network. A downsampling routine was needed to rescale the images from 300 dpi down to 72 dpi in a manner that preserves the strokes of the characters in the image. The JBIG compression method, described in detail in a subsequent section, with its progressive coding and sophisticated resolution reduction algorithm was used to optimally downsample the images to 72 dpi. Finally, these 72 dpi JBIG images were converted to GIF format for a final file size of about 20 KBytes.
To establish hyperlinks within the document images, the HTML attribute ISMAP is used. This features allows images to be made fully active. When the user clicks inside the image, the coordinates of the click are sent to an image map program. An image map database file corresponding to the image relates the region selected to a specific hyperlink. The hyperlink is then accessed.
Since the text images are stored in binary format, words or phrases which have hyperlinks are converted to reverse video, as shown in FIG. 9, to make them distinguishable to the user. If the user clicks anywhere within the reverse video area, the hyperlink will be activated. Clicks outside the reverse video areas will cause no action.
Method for Text Searching within Documents
A search strategy may be implemented in the method as follows. When a search string is entered, the uncorrected OCR of the document pages is searched. The entire line of text containing the matching string is retrieved. Next, an HTML page is formatted to contain the page on which the string matched, as well as the full text line containing the string. Additionally, hyperlinks are established that allow the user to click on the page number and have the document page retrieved. FIG. 10 shows the shell program that is used to perform the search and display the results.
In the sample implementation, each document page was scanned and OCRed using the OmniPage professional software package. No attempt was made to correct the OCR and just the text recognition for each page was retained. No graphics were saved. These text files served as the database for the search and retrieval algorithms.
The UNIX "grep" command was used to implement the search routine for the demonstration. "Grep" searches files for a pattern and prints all lines that contain that pattern. The results of the "grep" were sent to a formatting program which creates a hypertext page with the results. FIG. 11 shows the results of a search for the string "vector" in the prototype document. Clicking on either the highlighted "page 13" or "page 16" causes the respective pages to be retrieved for viewing.
Data Compression and Display
The tradeoff between display quality and document image size is a major design consideration for image-based HTML documents. Documents stored as 8-bit grayscale images provide adequate image quality when displayed on a standard computer screen, but take too long to transmit across Internet connections. Documents stored as compressed 1-bit images can be easily transmitted across the Internet, but lack sufficient display quality.
The remainder of this section describes the issues associated with this tradeoff, and proposes a quality solution to this problem.
In operation, it was observed that the GIF images found in the conventional HTML standard were not ideally suited for document images. Document images compressed under this format were difficult to read unless stored as large gray-scale images, and transmitting such images across the network is very time consuming.
There are three factors involved in determining document image size: dots per inch (dpi), bits per pixel (bpp), and compression technique. The challenge is to tradeoff these factors to obtain minimal storage and sufficient quality.
Added to this problem is the factor that many computer monitors are low resolution, and thus require low dpi display of document images. If document images are stored at 300 dpi and are displayed on a 72 dpi monitor, much of the document data is discarded when the 300 dpi image is mapped to the 72 dpi display. FIG. 12 shows a portion of a 300 dpi, 1 bpp image. The character seen in this figure are crisp and legible.
When the image is downsampled to 72 dpi, much of the image quality is lost, as illustrated in FIG. 13. As seen in this figure, this downsampling can produce such effects as broken characters, touching characters, eliminated character features, and merged characters. In some cases, this process can create ambiguous character representations.
When such downsampled images are displayed on a computer screen the human eye tends to overlook many of these deficiencies. However, not all of these problems are overlooked, and the resulting display appears to be less than optimal as illustrated in FIG. 14. The appearance of such low resolution images can be somewhat improved by storing more than 1 bpp, since shades of gray can be used to ease the jagged character features. However, this comes at a high data storage and transmission cost.
The third factor, data compression, plays an equally important role in this problem. Lossless compression techniques assure that no important data will be lost, but do not significantly reduce the amount of data required to store document images. Lossy compression techniques are more aggressive in reducing the amount of storage required, but can eliminate important data.
One compression standard deals with many of the aspects associated with this issue. The International Organization for Standardization/International Electro-technical Commission (ISO/IEC), in collaboration with the International Telegraph and Telephone Consultative Committee (CCITT), defined an image compression standard for lossless image coding of bi-level images (ISO/IEC 11544:1993) . The JBIG (Joint Bi-level Image Experts Group) standard defines a method of compressing two-tone or black/white images in a bit-preserving manner, wherein decoded images are digitally identical to the originally encoded image.
The JBIG standard can be parameterized for progressive coding. Thus, it is possible to transmit a low resolution image first, followed by resolution enhancement data. When decoding an image that has been progressively encoded, a low-resolution rendition of the original is made available first with subsequent doublings of resolution as more data is decoded. The progressive encoding mode utilizes a very sophisticated resolution reduction algorithm, PRES (progressive reduction standard), which offers the highest quality low resolution versions.
The progressive coding feature of JBIG is advantageous when an image is used by output devices with widely differing resolution capabilities. For example, when an image is displayed on a low resolution monitor (72 dpi), only that information in the compressed image required for reconstruction to the resolution of the display is transmitted and decoded. Then, if a higher resolution is needed for, say, printing to a 300 dpi printer, additional compressed data is transmitted and built upon the already transmitted data to obtain the higher resolution image for the printer.
Progressive coding is a way to send an image gradually to a receiver instead of all at once. During sending, more and more detail is sent and the receiver can build the image from low to high detail. JBIG uses discrete steps of detail by successively doubling the resolution. The sender computes a number of resolution layers for the image, d, and transmits these starting at the lowest resolution, d1. Resolution reduction uses pixels in the high resolution layer and some already computed low resolution pixels as an index into a lookup table. The contents of this table can be specified by the user.
Compatibility between progressive and sequential coding is achieved by dividing an image into stripes. Each stripe is a horizontal bar with a user definable height. Each stripe is separately coded and transmitted, and the user can define in which order stripes, resolutions and bit planes (if more than one) are intermixed in the coded data. A progressive coded image can be decoded sequentially by decoding each stripe, beginning with the one at the top of the image, to its full resolution, and then proceeding to the next stripe. Progressive decoding can be done by decoding only a specific resolution layer.
FIG. 15 shows an image decomposed into three stripes, s, and three resolution layers, d. Each stripe s at each resolution d is coded into a subfile Cs,d. The JBIG file to describe the total image is a concatenation of header information and the Cs,d subfiles. Four ways of concatenating the stripe codings are defined in Table 1. Decoders work naturally from low resolution up, and so prefer the first two orderings of the table.
TABLE 1______________________________________POSSIBLE DATA ORDERINGSHi to Low Seq. Order______________________________________0 0 0, 1, 2 3, 4, 5 6, 7, 80 1 0, 3, 6 1, 4, 7 2, 5, 81 0 6, 7, 8 3, 4, 5 0, 1, 21 1 6, 3, 0 7, 4, 1 8, 5, 2______________________________________
After dividing an image into bit planes, resolution layers and stripes, eventually a number of small bi-level bitmaps are left to compress. Compression is done using a Q-coder which codes bi-level pixels as symbols using the probability of occurrence of these symbols in a certain context. JBIG defines two kinds of context, one for the lowest resolution layer (the base layer), and one for all other layers (differential layers). Differential layer contexts contain pixels in the layer to be coded, and in the corresponding lower resolution layer.
The probability distribution of white and black pixels can be different for each combination of pixel values in a context. In an all white context, the probability of coding a white pixel will be much greater than that of coding a black pixel. The Q-coder assigns, just like a Huffman coder, more bits to less probable symbols, and so achieves compression. The Q-coder can, unlike a Huffman coder, assign one output code bit to more than one input symbol, and thus is able to compress bi-level pixels without explicit clustering, as would be necessary using a Huffman coder. Maximum compression is achieved when all probabilities (one set to each combination of pixel values in the context) follow the probabilities of the pixels. The Q-coder therefore continuously adapts these probabilities to the symbols it sees.
Since JBIG supports multiple bit planes, it is possible to effectively use the JBIG standard for the lossless coding of grayscale and color images as well. Images with eight or fewer bits/pixel compress well with the JBIG method, with more than eight bits/pixel other compression algorithms are more effective. In a study of international standards for lossless still image compression, JBIG compression was up to 29 percent superior to lossless JPEG compression for images with up to six bits/pixel. It was also found that JBIG had a 1.1 to 1.5 times better compression ratio on typical scanned documents, compared to G4 fax compression which had been the best compression algorithm for scanned documents available prior to JBIG.
Cost Analysis vs. Existing Document Coding Methods
Existing options for document conversion can be categorized into four classes: OCR, Image Database with Keywords, Image Database with OCR, and Structured Image Database with OCR. Each of these approaches to document conversion has its advantages and disadvantages. A summary of these is provided in Table 2.
The OCR approach is the most labor expensive. It requires complete conversion to electronic format. As indicated by our experiments, current OCR technology has only achieved a level to where it is cost competitive with manual reentry. This is very costly, but also produces the best final product. It uses the least amount of disk space, supports the best retrieval, and has optimal display quality. It just costs a lot to convert documents to this format.
The Image Database with Keywords approach is the next most labor expensive. This approach utilizes document images and document keywords. The keywords are manually obtained, and this requires extensive labor. Furthermore, the quality of the keywords is highly dependent on the expertise of the individual selecting the keywords. As such, the selection of keywords requires significant labor from a highly skilled individual. Both document images and ASCII keywords are stored under this approach, and thus, this method requires significantly more storage space than the OCR approach. Document retrieval is limited to the keywords associated with each document, and thus is only as comprehensive as the keywords. Display quality is limited to image quality. The conversion accuracy is optimal, since the content of the original document is preserved in the document image.
TABLE 2__________________________________________________________________________DOCUMENT CONVERSION SUMMARYConversion Retrieval Display ConversionLabor Disk Space Accuracy Quality Accuracy__________________________________________________________________________OCR Very ASCII Text Text Based Optimal Limited to expensive OCT qualityImage Keyword Document Limited to Limited to ContentDatabase entry can images and keywords image preservedwith be costly ASCII quality in imageKeywords KeywordsImage Lowest Document Limited to Limited to ContentDatabase cost images and dirty OCR image preservedwith OCR ASCII text quality in imageStructured Document Document Dirty OCR Limited to ContentImage DB parsing images and and image preservedwith OCR cost ASCII text document quality in image structure__________________________________________________________________________
The Image Database with OCR approach is the most labor inexpensive. This approach utilizes document images and uncorrected (or dirty) OCR. Since no corrections are made to the OCR, this approach minimizes conversion labor costs. Both document images and ASCII text are stored under this approach, and thus, this method requires more storage space than the Image Database with Keywords approach. The uncorrected OCR is used to retrieve images. This approach relies on the fact that the English language has many redundancies, and thus, a few OCR mistakes can be overlooked in the retrieval process. Display quality is limited to image quality. The conversion accuracy is optimal, since the content of the original document is preserved in the document image.
The Structured Image Database with OCR approach requires slightly more labor cost than the inexpensive Image Database with OCR approach. This approach utilizes document images, document structure knowledge, and uncorrected OCR. Both document images and ASCII text are stored under this approach, and thus, this method requires storage space that is similar to that of the Image Database with OCR approach. The uncorrected OCR is used to retrieve images, and again relies on English language redundancies for retrieval accuracy. Document structure can also be used in the retrieval process to focus query retrievals. Display quality is limited to image quality. The conversion accuracy is optimal, since the content of the original document is preserved in the document image.
Labor cost is the major price factor involved in each of these conversion approaches. For any large scale conversion task the labor costs associated with the OCR approach are simply prohibitive. Similarly, the labor cost associated with the Image Database with Keywords approach is also very expensive. Although not as extreme as the OCR labor costs, the selection of keywords involves manually categorizing the document contents. This requires a high degree of skill, and thus, is not inexpensive. The labor costs associated with the Image Database with OCR approach are minimal. Documents are simply scanned to create the document image database and OCRed to create the corresponding uncorrected ASCII text. Both can be highly automated. The labor costs associated with the Structured Image Database with OCR approach is slightly higher, since it involves manually monitoring and correcting the document parsing process. Simple tools can be developed to minimize the labor required for this process.
This invention employs the Structured Image Database with OCR approach. The prohibitive labor costs associated with the OCR and Image Database with Keyword approaches eliminated these approaches from further consideration. The costs associated with the Image Database with OCR and the Structured Image Database approaches were most comparable. The Structured Image Database with OCR approach was selected over the Image DB with OCR approach, however, because the structure knowledge was viewed as necessary to support simple network interfaces, valuable for retrieval performance, and relatively inexpensive.
The approach utilizes both document images and OCRed text. It is robust in that only document images are viewed (the quality of these images can be improved through the development of text specific image downsampling routines). Furthermore, the document is fully hypertext linked, and, as such, it is easy to navigate. Additionally, the search using the "dirty" OCRed text provides significant flexibility for finding information within the document (which can be further improved by registering the dirty ASCII text to image zones).
An efficient encoding method, including means for document searching, hyper-link indexing, and HTML coding has been described which provides a cost-effective means for many organizations to make data, which currently exists in only paper form, available to the rapidly growing number of computer and Internet users. Because the conversion method enables direct translation into HTML format, it can be used to create electronic documents that can be viewed by current Internet browsers like Mosaic or Netscape as well as disk and CDROM viewers (conventional word processor programs and edit/ viewer utilities).
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US5528740 *||Feb 25, 1993||Jun 18, 1996||Document Technologies, Inc.||Conversion of higher resolution images for display on a lower-resolution display device|
|US5608874 *||Apr 8, 1995||Mar 4, 1997||Autoentry Online, Inc.||System and method for automatic data file format translation and transmission having advanced features|
|US5708826 *||Dec 29, 1995||Jan 13, 1998||Fujitsu Limited||Apparatus and method for converting presentation data|
|US5721827 *||Oct 2, 1996||Feb 24, 1998||James Logan||System for electrically distributing personalized information|
|US5740425 *||Sep 26, 1995||Apr 14, 1998||Povilus; David S.||Data structure and method for publishing electronic and printed product catalogs|
|US5745360 *||Aug 14, 1995||Apr 28, 1998||International Business Machines Corp.||Dynamic hypertext link converter system and process|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US6185585 *||Dec 16, 1997||Feb 6, 2001||Corporate Media Partners||System and method for distribution and navigation of internet content|
|US6332039 *||Mar 27, 1998||Dec 18, 2001||Matsushita Graphic Communication Systems, Inc.||Structured document preparation apparatus and structured document preparation method|
|US6336124 *||Jul 7, 1999||Jan 1, 2002||Bcl Computers, Inc.||Conversion data representing a document to other formats for manipulation and display|
|US6338033||Apr 20, 1999||Jan 8, 2002||Alis Technologies, Inc.||System and method for network-based teletranslation from one natural language to another|
|US6400845 *||Apr 23, 1999||Jun 4, 2002||Computer Services, Inc.||System and method for data extraction from digital images|
|US6446036 *||Apr 20, 1999||Sep 3, 2002||Alis Technologies, Inc.||System and method for enhancing document translatability|
|US6537324 *||Feb 13, 1998||Mar 25, 2003||Ricoh Company, Ltd.||Generating and storing a link correlation table in hypertext documents at the time of storage|
|US6567079 *||Sep 13, 1999||May 20, 2003||Carnegie Mellon University||Portable computer system with ergonomic input device|
|US6620205||Jan 23, 2001||Sep 16, 2003||Coporate Media Partners D/B/A Americast||System and method for distribution and navigation of internet content|
|US6651060||Nov 1, 2000||Nov 18, 2003||Mediconnect.Net, Inc.||Methods and systems for retrieval and digitization of records|
|US6678738 *||Jun 19, 1998||Jan 13, 2004||International Business Machines Corporation||Web server providing html pages embedded with non-HTML views|
|US6697165 *||Dec 29, 1997||Feb 24, 2004||Canon Kabushiki Kaisha||Information processing system and method therefor|
|US6701376||Jun 19, 1998||Mar 2, 2004||International Business Machines Corporation||Web server enabling browser access to HTML and Non-HTML documents|
|US6732332||Oct 23, 2000||May 4, 2004||Und Aerospace Foundation||Automated web site creation system|
|US6741986||Nov 9, 2001||May 25, 2004||Ingenuity Systems, Inc.||Method and system for performing information extraction and quality control for a knowledgebase|
|US6742161 *||Mar 7, 2000||May 25, 2004||Scansoft, Inc.||Distributed computing document recognition and processing|
|US6772160 *||Dec 8, 2000||Aug 3, 2004||Ingenuity Systems, Inc.||Techniques for facilitating information acquisition and storage|
|US6938051 *||Feb 26, 1999||Aug 30, 2005||Verizon Services Corp.||Method, storage medium and system for electronically viewing multi-page document while preserving appearance of printed pages|
|US6950101 *||Dec 23, 2002||Sep 27, 2005||Research Investment Network, Inc.||Computer implemented method and system for transmitting graphical images from server to client at user selectable resolution|
|US7051271 *||May 31, 2000||May 23, 2006||Fuji Xerox Co., Ltd.||Method, system and article of manufacture for linking a video to a scanned document|
|US7062710 *||Jan 16, 2002||Jun 13, 2006||Vmp Mortgage Solutions, Inc.||File transforming output engine having a decomposer and multiple writers|
|US7082475||Jun 19, 1998||Jul 25, 2006||International Business Machines Corporation||Web server with automated workflow|
|US7099861 *||Jun 7, 2001||Aug 29, 2006||Ccr Inc.||System and method for facilitating internet search by providing web document layout image|
|US7117271||Jan 31, 2001||Oct 3, 2006||Internatioal Business Machines Corporation||Web server enabling direct scheduling and calendaring capability|
|US7124363||May 29, 2002||Oct 17, 2006||Pioneer Corporation||System and method for disclosing design information|
|US7194679 *||Oct 20, 1998||Mar 20, 2007||International Business Machines Corporation||Web-based file review system utilizing source and comment files|
|US7200271 *||Oct 4, 2001||Apr 3, 2007||The Boeing Company||Method, computer program product, and system for performing automated text recognition and text search within a graphic file|
|US7209246 *||Oct 27, 1998||Apr 24, 2007||Canon Kabushiki Kaisha||Information processing apparatus and method for processing read image data|
|US7233698 *||Dec 13, 2002||Jun 19, 2007||The Boeing Company||Apparatus and methods for converting raster illustrated parts images into intelligent vector-layered files|
|US7237185||Sep 13, 1999||Jun 26, 2007||Corporate Media Partners||System and method for distributing internet content in the form of an album|
|US7305612 *||Mar 31, 2003||Dec 4, 2007||Siemens Corporate Research, Inc.||Systems and methods for automatic form segmentation for raster-based passive electronic documents|
|US7418323 *||Jan 26, 2005||Aug 26, 2008||Tag One, Inc.||Method and system for aircraft data and portfolio management|
|US7418653 *||Nov 17, 2000||Aug 26, 2008||Olive Software Inc.||System and method for data publication through web pages|
|US7437351 *||Sep 30, 2003||Oct 14, 2008||Google Inc.||Method for searching media|
|US7512253 *||May 17, 2004||Mar 31, 2009||Chieko Taketomi||Method of digitizing graphic information|
|US7512712 *||Feb 26, 2003||Mar 31, 2009||International Business Machines Corporation||Method for translation of documents|
|US7529988||Sep 2, 2003||May 5, 2009||Advanced Micro Devices, Inc.||Storage of descriptive information in user defined fields of failure bitmaps in integrated circuit technology development|
|US7532738 *||Feb 17, 2005||May 12, 2009||Oki Electric Industry Co., Ltd.||Print medium quality adjustment system, inspection watermark medium output device for outputting watermark medium to undergo inspection, watermark quality inspection device, adjusted watermark medium output device, print medium quality adjustment method and inspection watermark medium to undergo inspection|
|US7561742 *||Sep 22, 2006||Jul 14, 2009||The Boeing Company||Apparatus and methods for converting raster illustrated parts images into intelligent vector-layered files|
|US7577683||Feb 2, 2004||Aug 18, 2009||Ingenuity Systems, Inc.||Methods for the construction and maintenance of a knowledge representation system|
|US7600183 *||Jun 2, 2003||Oct 6, 2009||Olive Software Inc.||System and method for data publication through web pages|
|US7603620 *||Dec 20, 2004||Oct 13, 2009||Ricoh Co., Ltd.||Creating visualizations of documents|
|US7610309 *||Aug 25, 2005||Oct 27, 2009||Verizon Corporate Services Group Inc.||Method and system for electronically viewing multi-page documents while preserving appearance of printed pages|
|US7650339 *||Jun 8, 2004||Jan 19, 2010||Ingenuity Systems, Inc.||Techniques for facilitating information acquisition and storage|
|US7668372||Aug 26, 2004||Feb 23, 2010||Open Text Corporation||Method and system for collecting data from a plurality of machine readable documents|
|US7712017||Feb 24, 2006||May 4, 2010||Fuji Xerox Co., Ltd.||Method, system and article of manufacture for linking a video to a scanned document|
|US7756819 *||Aug 17, 2005||Jul 13, 2010||Agency For Defense Development||Integrated authoring system for electronic technical manual and paper technical manual using logistics support analysis data and authoring method therefor|
|US7761789||Jan 13, 2006||Jul 20, 2010||Ricoh Company, Ltd.||Methods for computing a navigation path|
|US7788579||Mar 6, 2006||Aug 31, 2010||Ricoh Co., Ltd.||Automated document layout design|
|US7792362||Aug 16, 2007||Sep 7, 2010||Ricoh Co., Ltd.||Reformatting documents using document analysis information|
|US7849398 *||Apr 26, 2007||Dec 7, 2010||Xerox Corporation||Decision criteria for automated form population|
|US7853866 *||Jun 13, 2006||Dec 14, 2010||Canon Kabushiki Kaisha||Apparatus, method and system for document conversion, apparatuses for document processing and information processing, and storage media that store programs for realizing the apparatuses|
|US7929773 *||Jun 27, 2007||Apr 19, 2011||Compal Electronics, Inc.||Method for recognizing characters|
|US7937409 *||May 20, 2008||May 3, 2011||Adobe Systems Incorporated||Retrieving documents transitively linked to an initial document|
|US7986843||Nov 29, 2006||Jul 26, 2011||Google Inc.||Digital image archiving and retrieval in a mobile device system|
|US7990556||Feb 28, 2006||Aug 2, 2011||Google Inc.||Association of a portable scanner with input/output and storage devices|
|US8005720||Aug 18, 2005||Aug 23, 2011||Google Inc.||Applying scanned information to identify content|
|US8005843||May 20, 2008||Aug 23, 2011||Adobe Systems Incorporated||Retrieving documents transitively linked to an initial document|
|US8019648||Apr 1, 2005||Sep 13, 2011||Google Inc.||Search engines and systems with handheld document data capture devices|
|US8064700||Mar 10, 2010||Nov 22, 2011||Google Inc.||Method and system for character recognition|
|US8081849||Feb 6, 2007||Dec 20, 2011||Google Inc.||Portable scanning and memory device|
|US8103702 *||Aug 25, 2008||Jan 24, 2012||Ricoh Company, Ltd.||Information processing device, electronic manual managing method, and electronic manual managing program|
|US8146156||Sep 2, 2008||Mar 27, 2012||Google Inc.||Archive of text captures from rendered documents|
|US8179563||Sep 29, 2010||May 15, 2012||Google Inc.||Portable scanning device|
|US8201086 *||Jan 18, 2007||Jun 12, 2012||International Business Machines Corporation||Spellchecking electronic documents|
|US8214387||Apr 1, 2005||Jul 3, 2012||Google Inc.||Document enhancement system and method|
|US8261094||Aug 19, 2010||Sep 4, 2012||Google Inc.||Secure data gathering from rendered documents|
|US8270721||Jun 30, 2009||Sep 18, 2012||Open Text S.A.||Method and system for acquiring data from machine-readable documents|
|US8332744||Sep 28, 2007||Dec 11, 2012||International Business Machines Corporation||Translation of documents|
|US8346620||Sep 28, 2010||Jan 1, 2013||Google Inc.||Automatic modification of web pages|
|US8373880||Dec 30, 2008||Feb 12, 2013||Industrial Technology Research Institute||Technical documents capturing and patents analysis system and method|
|US8392353||Jul 8, 2009||Mar 5, 2013||Ingenuity Systems Inc.||Computerized knowledge representation system with flexible user entry fields|
|US8418055||Feb 18, 2010||Apr 9, 2013||Google Inc.||Identifying a document by performing spectral analysis on the contents of the document|
|US8442331||Aug 18, 2009||May 14, 2013||Google Inc.||Capturing text from rendered documents using supplemental information|
|US8447066||Mar 12, 2010||May 21, 2013||Google Inc.||Performing actions based on capturing information from rendered documents, such as documents under copyright|
|US8447111||Feb 21, 2011||May 21, 2013||Google Inc.||Triggering actions in response to optically or acoustically capturing keywords from a rendered document|
|US8447144||Aug 18, 2009||May 21, 2013||Google Inc.||Data capture from rendered documents using handheld device|
|US8452088||Nov 15, 2000||May 28, 2013||Stmicroelectronics S.R.L.||Content-based digital-image classification method|
|US8453045 *||Sep 8, 2010||May 28, 2013||Canon Kabushiki Kaisha||Apparatus, method and system for document conversion, apparatuses for document processing and information processing, and storage media that store programs for realizing the apparatuses|
|US8488173||Jul 14, 2011||Jul 16, 2013||Mphj Technology Investments, Llc||Distributed computer architecture and process for document management|
|US8489334||Feb 3, 2003||Jul 16, 2013||Ingenuity Systems, Inc.||Drug discovery methods|
|US8489624||Jan 29, 2010||Jul 16, 2013||Google, Inc.||Processing techniques for text capture from a rendered document|
|US8505090||Feb 20, 2012||Aug 6, 2013||Google Inc.||Archive of text captures from rendered documents|
|US8515816||Apr 1, 2005||Aug 20, 2013||Google Inc.||Aggregate analysis of text captures performed by multiple users from rendered documents|
|US8583637||Mar 21, 2007||Nov 12, 2013||Ricoh Co., Ltd.||Coarse-to-fine navigation through paginated documents retrieved by a text search engine|
|US8584042||Mar 21, 2007||Nov 12, 2013||Ricoh Co., Ltd.||Methods for scanning, printing, and copying multimedia thumbnails|
|US8600196||Jul 6, 2010||Dec 3, 2013||Google Inc.||Optical scanners, such as hand-held optical scanners|
|US8619147||Oct 6, 2010||Dec 31, 2013||Google Inc.||Handheld device for capturing text from both a document printed on paper and a document displayed on a dynamic display device|
|US8619287||Aug 17, 2009||Dec 31, 2013||Google Inc.||System and method for information gathering utilizing form identifiers|
|US8620083||Oct 5, 2011||Dec 31, 2013||Google Inc.||Method and system for character recognition|
|US8620114||Jul 12, 2011||Dec 31, 2013||Google Inc.||Digital image archiving and retrieval in a mobile device system|
|US8620760||Oct 11, 2010||Dec 31, 2013||Google Inc.||Methods and systems for initiating application processes by data capture from rendered documents|
|US8621349||Oct 5, 2010||Dec 31, 2013||Google Inc.||Publishing techniques for adding value to a rendered document|
|US8631012 *||Sep 29, 2006||Jan 14, 2014||A9.Com, Inc.||Method and system for identifying and displaying images in response to search queries|
|US8638363||Feb 18, 2010||Jan 28, 2014||Google Inc.||Automatically capturing information, such as capturing information using a document-aware device|
|US8640179||Dec 27, 2011||Jan 28, 2014||Network-1 Security Solutions, Inc.||Method for using extracted features from an electronic work|
|US8650170 *||Jun 22, 2011||Feb 11, 2014||Verisign, Inc.||Systems and methods for inter-object pattern matching|
|US8656441||Mar 14, 2013||Feb 18, 2014||Network-1 Technologies, Inc.||System for using extracted features from an electronic work|
|US8781228||Sep 13, 2012||Jul 15, 2014||Google Inc.||Triggering actions in response to optically or acoustically capturing keywords from a rendered document|
|US8782726||Mar 14, 2013||Jul 15, 2014||Network-1 Technologies, Inc.||Method for taking action based on a request related to an electronic media work|
|US8793073||Aug 1, 2003||Jul 29, 2014||Ingenuity Systems, Inc.||Drug discovery methods|
|US8793162||May 5, 2010||Jul 29, 2014||Google Inc.||Adding information or functionality to a rendered document via association with an electronic counterpart|
|US8799099||Sep 13, 2012||Aug 5, 2014||Google Inc.||Processing techniques for text capture from a rendered document|
|US8799303||Oct 13, 2010||Aug 5, 2014||Google Inc.||Establishing an interactive environment for rendered documents|
|US8799401 *||Jul 8, 2004||Aug 5, 2014||Amazon Technologies, Inc.||System and method for providing supplemental information relevant to selected content in media|
|US8799772 *||Jul 18, 2005||Aug 5, 2014||International Business Machines Corporation||System and method for gathering, indexing, and supplying publicly available data charts|
|US8812969||Mar 21, 2007||Aug 19, 2014||Ricoh Co., Ltd.||Methods for authoring and interacting with multimedia representations of documents|
|US8831365||Mar 11, 2013||Sep 9, 2014||Google Inc.||Capturing text from rendered documents using supplement information|
|US8874504||Mar 22, 2010||Oct 28, 2014||Google Inc.||Processing techniques for visual capture data from a rendered document|
|US8892495||Jan 8, 2013||Nov 18, 2014||Blanding Hovenweep, Llc||Adaptive pattern recognition based controller apparatus and method and human-interface therefore|
|US8897579||Oct 9, 2013||Nov 25, 2014||Google Inc.||Digital image archiving and retrieval|
|US8903759||Sep 21, 2010||Dec 2, 2014||Google Inc.||Determining actions involving captured information and electronic content associated with rendered documents|
|US8904464||Mar 13, 2013||Dec 2, 2014||Network-1 Technologies, Inc.||Method for tagging an electronic media work to perform an action|
|US8904465||Mar 14, 2013||Dec 2, 2014||Network-1 Technologies, Inc.||System for taking action based on a request related to an electronic media work|
|US8943402||Dec 29, 2010||Jan 27, 2015||Banbury Technologies Llc||System and method for distribution and navigation of internet content|
|US8953886||Aug 8, 2013||Feb 10, 2015||Google Inc.||Method and system for character recognition|
|US8984393 *||Jan 26, 2007||Mar 17, 2015||Aatrix Software, Inc.||Method and apparatus for creating and filing forms|
|US8990235||Mar 12, 2010||Mar 24, 2015||Google Inc.||Automatically providing content associated with captured information, such as information captured in real-time|
|US9015159||Sep 14, 2012||Apr 21, 2015||Google Inc.||Method for searching media|
|US9030699||Aug 13, 2013||May 12, 2015||Google Inc.||Association of a portable scanner with input/output and storage devices|
|US9069731 *||Dec 29, 2010||Jun 30, 2015||Olive Software Inc.||System and method for providing online versions of print-medium publications|
|US9075779||Apr 22, 2013||Jul 7, 2015||Google Inc.||Performing actions based on capturing information from rendered documents, such as documents under copyright|
|US9081799||Dec 6, 2010||Jul 14, 2015||Google Inc.||Using gestalt information to identify locations in printed information|
|US9092414||Feb 7, 2008||Jul 28, 2015||Adobe Systems Incorporated||Using document templates to assemble a collection of documents|
|US9116890||Jun 11, 2014||Aug 25, 2015||Google Inc.||Triggering actions in response to optically or acoustically capturing keywords from a rendered document|
|US9143638||Apr 29, 2013||Sep 22, 2015||Google Inc.||Data capture from rendered documents using handheld device|
|US9256885||Mar 13, 2013||Feb 9, 2016||Network-1 Technologies, Inc.||Method for linking an electronic media work to perform an action|
|US9268852||Sep 13, 2012||Feb 23, 2016||Google Inc.||Search engines and systems with handheld document data capture devices|
|US9275051||Nov 7, 2012||Mar 1, 2016||Google Inc.||Automatic modification of web pages|
|US9282359||Mar 14, 2013||Mar 8, 2016||Network-1 Technologies, Inc.||Method for taking action with respect to an electronic media work|
|US9323784||Dec 9, 2010||Apr 26, 2016||Google Inc.||Image search using text-based elements within the contents of images|
|US9348820||Mar 15, 2013||May 24, 2016||Network-1 Technologies, Inc.||System and method for taking action with respect to an electronic media work and logging event information related thereto|
|US9355072 *||Sep 7, 2012||May 31, 2016||Google Inc.||Recursive embedding by URL parameterization|
|US9454764||Oct 14, 2010||Sep 27, 2016||Google Inc.||Contextual dynamic advertising based upon captured rendered text|
|US9514102||Sep 27, 2013||Dec 6, 2016||Google Inc.||User interface for presentation of a document|
|US9514134||Jul 15, 2015||Dec 6, 2016||Google Inc.|
|US9514408||Jan 18, 2013||Dec 6, 2016||Ingenuity Systems, Inc.||Constructing and maintaining a computerized knowledge representation system using fact templates|
|US9529870||Dec 28, 2015||Dec 27, 2016||Network-1 Technologies, Inc.||Methods for linking an electronic media work to perform an action|
|US9535563||Nov 12, 2013||Jan 3, 2017||Blanding Hovenweep, Llc||Internet appliance system and method|
|US9536141 *||Jun 29, 2012||Jan 3, 2017||Palo Alto Research Center Incorporated||System and method for forms recognition by synthesizing corrected localization of data fields|
|US9536253||Dec 28, 2015||Jan 3, 2017||Network-1 Technologies, Inc.||Methods for linking an electronic media work to perform an action|
|US9538216||Dec 28, 2015||Jan 3, 2017||Network-1 Technologies, Inc.||System for taking action with respect to a media work|
|US9544663||Dec 28, 2015||Jan 10, 2017||Network-1 Technologies, Inc.||System for taking action with respect to a media work|
|US9558190||Dec 28, 2015||Jan 31, 2017||Network-1 Technologies, Inc.||System and method for taking action with respect to an electronic media work|
|US9582291 *||Sep 24, 2009||Feb 28, 2017||International Business Machines Corporation||Selecting a mapping that minimizes conversion costs|
|US9633013||Mar 22, 2016||Apr 25, 2017||Google Inc.|
|US9684676||Sep 8, 2008||Jun 20, 2017||Google Inc.||Method for searching media|
|US20010005848 *||Jan 31, 2001||Jun 28, 2001||Paul Haverstock||Web server enabling direct scheduling and calendaring capability|
|US20010032241 *||Apr 12, 2001||Oct 18, 2001||Alvise Braga Illa||Platform for handling digital contents coming from heterogeneous sources|
|US20010051985 *||Jun 19, 1998||Dec 13, 2001||Paul Haverstock||Web server providing html pages embedded with non-html views|
|US20010056418 *||Jun 7, 2001||Dec 27, 2001||Youn Seok Ho||System and method for facilitating internet search by providing web document layout image|
|US20020035619 *||Aug 2, 2001||Mar 21, 2002||Dougherty Carter D.||Apparatus and method for producing contextually marked-up electronic content|
|US20020038357 *||Jun 19, 1998||Mar 28, 2002||Paul Haverstock||Web server with automated workflow|
|US20020095440 *||Jun 12, 2001||Jul 18, 2002||Martin Burgess||Data format conversion|
|US20020107883 *||Feb 8, 2001||Aug 8, 2002||Ofer Schneid||Distributed visual communications content development method and system|
|US20020188635 *||Mar 7, 2002||Dec 12, 2002||Larson Stephen C.||System and method for incorporation of print-ready advertisement in digital newspaper editions|
|US20020191848 *||Oct 4, 2001||Dec 19, 2002||The Boeing Company||Method, computer program product, and system for performing automated text recognition and text search within a graphic file|
|US20030014383 *||Dec 8, 2000||Jan 16, 2003||Ingenuity Systems, Inc.||Techniques for facilitating information acquisition and storage|
|US20030051213 *||Sep 6, 2002||Mar 13, 2003||Hidetsugu Mitsui||Document passing system, document updating controller, client device, document management server, method of controlling updating of document, recording medium and program|
|US20030072299 *||Dec 23, 2002||Apr 17, 2003||Hunt William J.||Method and system for transmitting graphical images|
|US20030115284 *||Dec 18, 2001||Jun 19, 2003||Henry Steven G.||Method and apparatus for accessing network data associated with a document|
|US20030172347 *||Jan 16, 2002||Sep 11, 2003||Gunn Paul John||File transforming output engine having a decomposer and multiple writers|
|US20030182453 *||Feb 26, 2003||Sep 25, 2003||International Business Machines Corporation||Computer, server and method for translation of documents|
|US20030200507 *||Jun 2, 2003||Oct 23, 2003||Olive Software, Inc.||System and method for data publication through web pages|
|US20030234967 *||Jun 19, 2003||Dec 25, 2003||Rodriguez Eric D.||Interactive document capture and processing software|
|US20040002886 *||Jun 27, 2002||Jan 1, 2004||Dickerson William M.||System and method for processing a service order|
|US20040012601 *||Jul 18, 2002||Jan 22, 2004||Sang Henry W.||Method and system for displaying a first image as a second image|
|US20040021790 *||May 30, 2003||Feb 5, 2004||Soichiro Iga||Method of and system for processing image information on writing surface including hand-written information|
|US20040044743 *||May 11, 2001||Mar 4, 2004||Craig Monell||Method and apparatus for hyperlinked graphics tool|
|US20040114801 *||Dec 13, 2002||Jun 17, 2004||Boose Molly L.||Apparatus and methods for converting raster illustrated parts images into intelligent vector-layered files|
|US20040122811 *||Sep 30, 2003||Jun 24, 2004||Google, Inc.||Method for searching media|
|US20040151377 *||Feb 4, 2003||Aug 5, 2004||Boose Molly L.||Apparatus and methods for converting network drawings from raster format to vector format|
|US20040194035 *||Mar 31, 2003||Sep 30, 2004||Amit Chakraborty||Systems and methods for automatic form segmentation for raster-based passive electronic documents|
|US20040199875 *||Apr 3, 2003||Oct 7, 2004||Samson Jason Kyle||Method for hosting analog written materials in a networkable digital library|
|US20040220815 *||Aug 20, 2001||Nov 4, 2004||Johanne Belanger||Apparatus and method for the compilation, assembly, and distribution of product documentation and associated information|
|US20040220969 *||Feb 2, 2004||Nov 4, 2004||Ingenuity Systems, Inc., A Delaware Corporation||Methods for the construction and maintenance of a knowledge representation system|
|US20040236740 *||Mar 16, 2004||Nov 25, 2004||Ingenuity Systems, Inc.||Method and system for performing information extraction and quality control for a knowledgebase|
|US20040237029 *||May 22, 2003||Nov 25, 2004||Medicke John A.||Methods, systems and computer program products for incorporating spreadsheet formulas of multi-dimensional cube data into a multi-dimentional cube|
|US20050013468 *||May 17, 2004||Jan 20, 2005||Chieko Taketomi||Method of digitizing graphic information|
|US20050044071 *||Jun 8, 2004||Feb 24, 2005||Ingenuity Systems, Inc.||Techniques for facilitating information acquisition and storage|
|US20050050452 *||Aug 27, 2003||Mar 3, 2005||Weitzel Wade D.||Systems and methods for generating an electronically publishable document|
|US20050055347 *||Mar 16, 2004||Mar 10, 2005||Ingenuity Systems, Inc.||Method and system for performing information extraction and quality control for a knowledgebase|
|US20050065893 *||Sep 19, 2003||Mar 24, 2005||The Alliance Group Of Texas||System and Method for Commingled Remittance Payment Processing|
|US20050154760 *||Jan 12, 2004||Jul 14, 2005||International Business Machines Corporation||Capturing portions of an electronic document|
|US20050165524 *||Jan 26, 2005||Jul 28, 2005||Tag One, Inc.||Method and system for aircraft data and portfolio management|
|US20050256868 *||Mar 17, 2005||Nov 17, 2005||Shelton Michael J||Document search system|
|US20050289448 *||Jul 18, 2005||Dec 29, 2005||International Business Machines Corporation||System and method for gathering, indexing, and supplying publicly available data charts|
|US20060036368 *||Feb 3, 2003||Feb 16, 2006||Ingenuity Systems, Inc.||Drug discovery methods|
|US20060045355 *||Aug 16, 2005||Mar 2, 2006||Kyocera Corporation||Mobile terminal, and computer controlling method and program for use in the same|
|US20060047669 *||Aug 26, 2005||Mar 2, 2006||Durrence Hugh D||System and method for document and electronic file management|
|US20060075327 *||Sep 29, 2004||Apr 6, 2006||Joe Sriver||User interface for presentation of a document|
|US20060116977 *||Aug 25, 2005||Jun 1, 2006||Burger Robert T||Method and system for electronically viewing multi-page documents while preserving appearance of printed pages|
|US20060136803 *||Dec 20, 2004||Jun 22, 2006||Berna Erol||Creating visualizations of documents|
|US20060143567 *||Feb 24, 2006||Jun 29, 2006||Fuji Xerox Co., Ltd.||Method, system and article of manufacture for linking a video to a scanned document|
|US20060149770 *||Aug 17, 2005||Jul 6, 2006||Agency For Defence Development||Integrated authoring system for electronic technical manual and paper technical manual using logistics support analysis data and authoring method therefor|
|US20060282760 *||Jun 13, 2006||Dec 14, 2006||Canon Kabushiki Kaisha||Apparatus, method and system for document conversion, apparatuses for document processing and information processing, and storage media that store programs for realizing the apparatuses|
|US20070033208 *||Oct 9, 2006||Feb 8, 2007||Srikrishna Talluri||Method and system for archiving and retrieving bibliography information and reference material|
|US20070064013 *||Sep 22, 2006||Mar 22, 2007||Boose Molly L|
|US20070065011 *||Aug 26, 2004||Mar 22, 2007||Matthias Schiehlen||Method and system for collecting data from a plurality of machine readable documents|
|US20070124665 *||Jan 26, 2007||May 31, 2007||Aatrix Software, Inc.||Method and apparatus for creating and filing forms|
|US20070178473 *||Aug 1, 2003||Aug 2, 2007||Chen Richard O||Drug discovery methods|
|US20070201719 *||Feb 17, 2005||Aug 30, 2007||Takashi Hashimoto||Print Medium Quality Adjustment System, Inspection Watermark Medium Output Device For Outputting Watermark Medium To Undergo Inspection, Watermark Quality Inspection Device, Adjusted Watermark Medium Output Device, Print Medium Quality Adjustment Method And Inspection Watermark Medium To Undergo Inspection|
|US20070201768 *||Aug 26, 2004||Aug 30, 2007||Matthias Schiehlen||Method And System For Acquiring Data From Machine-Readable Documents|
|US20070208996 *||Mar 6, 2006||Sep 6, 2007||Kathrin Berkner||Automated document layout design|
|US20080016454 *||Jun 28, 2007||Jan 17, 2008||Kyocera Mita Corporation||Data input-output device|
|US20080028305 *||Sep 28, 2007||Jan 31, 2008||Hyonsu Kim||Computer, server and method for translation of documents|
|US20080033819 *||Jul 27, 2007||Feb 7, 2008||Ingenuity Systems, Inc.||Genomics based targeted advertising|
|US20080037873 *||Aug 16, 2007||Feb 14, 2008||Kathrin Berkner||Reformatting documents using document analysis information|
|US20080082497 *||Sep 29, 2006||Apr 3, 2008||Leblang Jonathan A||Method and system for identifying and displaying images in response to search queries|
|US20080126415 *||Nov 29, 2006||May 29, 2008||Google Inc.||Digital Image Archiving and Retrieval in a Mobile Device System|
|US20080137955 *||Jun 27, 2007||Jun 12, 2008||Compal Electronics, Inc.||Method for recognizing characters|
|US20080141152 *||Jun 19, 2007||Jun 12, 2008||Shenzhen Futaihong Precision Industrial Co.,Ltd.||System for managing electronic documents for products|
|US20080154099 *||Nov 6, 2007||Jun 26, 2008||Saskatchewan Telecommunications||Health monitoring system and method|
|US20080162602 *||Dec 28, 2006||Jul 3, 2008||Google Inc.||Document archiving system|
|US20080162603 *||Aug 29, 2007||Jul 3, 2008||Google Inc.||Document archiving system|
|US20080178076 *||Jan 18, 2007||Jul 24, 2008||Barry Alan Kritt||Method and apparatus for spellchecking electronic documents|
|US20080222091 *||May 20, 2008||Sep 11, 2008||Adobe Systems Incorporated||Retrieving Documents Transitively Linked to an Initial Document|
|US20080235207 *||Mar 21, 2007||Sep 25, 2008||Kathrin Berkner||Coarse-to-fine navigation through paginated documents retrieved by a text search engine|
|US20080235585 *||Mar 21, 2007||Sep 25, 2008||Ricoh Co., Ltd.||Methods for authoring and interacting with multimedia representations of documents|
|US20080252912 *||May 20, 2008||Oct 16, 2008||Adobe Systems Incorporated||Retrieving Documents Transitively Linked To An Initial Document|
|US20080267505 *||Apr 26, 2007||Oct 30, 2008||Xerox Corporation||Decision criteria for automated form population|
|US20090063573 *||Aug 25, 2008||Mar 5, 2009||Takemoto Ryo||Information processing device, electronic manual managing method, and electronic manual managing program|
|US20090244556 *||Dec 30, 2008||Oct 1, 2009||Chen-Kun Chen||Technical Documents Capturing And Patents Analysis System And Method|
|US20100010957 *||Jul 8, 2009||Jan 14, 2010||Ingenuity Systems, Inc., A Delaware Corporation||Methods for the Construction and Maintenance of a Computerized Knowledge Representation System|
|US20100094888 *||Jun 30, 2009||Apr 15, 2010||Matthias Schiehlen||Method and system for acquiring data from machine-readable documents|
|US20100107059 *||Sep 24, 2009||Apr 29, 2010||International Business Machines Corporation||Generation apparatus, program, and generation method|
|US20100329567 *||Sep 8, 2010||Dec 30, 2010||Canon Kabushiki Kaisha||Apparatus, method and system for document converstion, apparatuses for document processing and information processing, and storage media that store programs for realizing the apparatuses|
|US20110154177 *||Dec 29, 2010||Jun 23, 2011||Sequeira William J||System and method for distribution and navigation of internet content|
|US20110161806 *||Dec 29, 2010||Jun 30, 2011||Olive Software Inc.||System and method for providing online versions of print-medium publications|
|US20110191286 *||Feb 16, 2011||Aug 4, 2011||Cho Raymond J||Method And System For Performing Information Extraction And Quality Control For A Knowledge Base|
|US20110258182 *||Jan 14, 2011||Oct 20, 2011||Singh Vartika||Systems and methods for automatically extracting data from electronic document page including multiple copies of a form|
|US20120041941 *||Jul 20, 2011||Feb 16, 2012||Google Inc.||Search Engines and Systems with Handheld Document Data Capture Devices|
|US20120330994 *||Jun 22, 2011||Dec 27, 2012||Verisign, Inc.||Systems and Methods for Inter-Object Pattern Matching|
|US20140006917 *||Jun 29, 2012||Jan 2, 2014||Palo Alto Research Center Incorporated||System and method for forms recognition by synthesizing corrected localization of data fields|
|US20140195904 *||Oct 23, 2013||Jul 10, 2014||Chao-Chin Chang||Technical documents capturing and patents analysis system and method|
|US20150199304 *||Sep 7, 2012||Jul 16, 2015||Christopher Richard Uhlik||Recursive Embedding By URL Parameterization|
|USRE45952 *||Jan 9, 2014||Mar 29, 2016||Google Inc.||Method for searching media|
|CN1996989B||Jan 1, 2006||Jun 23, 2010||腾讯科技（深圳）有限公司||A personal page system and its realization method|
|DE10345526A1 *||Sep 30, 2003||May 25, 2005||Océ Document Technologies GmbH||Verfahren und System zum Erfassen von Daten aus maschinell lesbaren Dokumenten|
|EP1148432A2 *||Apr 4, 2001||Oct 24, 2001||TXT e-solutions S.P.A.||Platform for handling digital contents coming from heterogeneous sources|
|EP1148432A3 *||Apr 4, 2001||Feb 2, 2005||TXT e-solutions S.P.A.||Platform for handling digital contents coming from heterogeneous sources|
|EP1227416A2 *||Jan 16, 2002||Jul 31, 2002||The PenPic Corporation Limited||Data format conversion|
|EP1227416A3 *||Jan 16, 2002||Dec 15, 2004||The PenPic Corporation Limited||Data format conversion|
|EP1271381A2 *||May 29, 2002||Jan 2, 2003||Pioneer Corporation||System and method for disclosing design information|
|EP1271381A3 *||May 29, 2002||May 12, 2004||Pioneer Corporation||System and method for disclosing design information|
|EP2105851A2||Mar 20, 2009||Sep 30, 2009||Industrial Technology Research Institut||Capturing system and method for technical documents and patent analysis system and method|
|WO2001057786A1 *||Feb 1, 2001||Aug 9, 2001||Scansoft, Inc.||Automatic conversion of static documents into dynamic documents|
|WO2001067292A1 *||Mar 1, 2001||Sep 13, 2001||Scansoft, Inc.||Distributed computing document recognition and processing|
|WO2001088760A2 *||May 11, 2001||Nov 22, 2001||Biocarta, Inc.||Method and apparatus for hyperlinked graphics tool|
|WO2001088760A3 *||May 11, 2001||Dec 31, 2003||Biocarta Inc||Method and apparatus for hyperlinked graphics tool|
|WO2001098948A2 *||Jun 15, 2001||Dec 27, 2001||Olive Software Inc.||System and method for data publication through web pages|
|WO2001098948A3 *||Jun 15, 2001||Aug 29, 2002||Olive Software Inc||System and method for data publication through web pages|
|WO2002041170A2 *||Nov 16, 2001||May 23, 2002||Interlegis, Inc.||System and method of managing documents|
|WO2002041170A3 *||Nov 16, 2001||Aug 14, 2003||Interlegis Inc||System and method of managing documents|
|WO2003048958A1 *||Nov 15, 2002||Jun 12, 2003||Controldocs Operating Corp.||A process for storing electronic documents on an internet-accessible document storage system|
|WO2005114538A2 *||Jan 27, 2005||Dec 1, 2005||Tag One, Inc.||Method and system for aircraft data and portfolio management|
|WO2005114538A3 *||Jan 27, 2005||Sep 21, 2006||Tag One Inc||Method and system for aircraft data and portfolio management|
|U.S. Classification||715/236, 715/277, 707/E17.013|
|International Classification||G06K9/20, G06F17/30|
|Cooperative Classification||G06F17/30855, G06K9/00442|
|European Classification||G06F17/30V5H, G06K9/00L|
|Mar 10, 2003||FPAY||Fee payment|
Year of fee payment: 4
|Feb 8, 2007||FPAY||Fee payment|
Year of fee payment: 8
|Feb 2, 2010||AS||Assignment|
Owner name: PERIPHERAL VISION, LLC,CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:CYBERNET SYSTEMS CORPORATION;REEL/FRAME:023882/0516
Effective date: 20091208
|Oct 27, 2010||AS||Assignment|
Owner name: GOOGLE INC., CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PERIPHERAL VISION, LLC;REEL/FRAME:025204/0112
Effective date: 20100225
|Nov 30, 2010||AS||Assignment|
Owner name: CYBERNET SYSTEMS, INC., MICHIGAN
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MITCHELL, BRIAN T.;WHALEN, MARGARET J.;REEL/FRAME:025402/0298
Effective date: 19951103
|Apr 5, 2011||FPAY||Fee payment|
Year of fee payment: 12