CA1209274A - Context redundancy text compression - Google Patents
Context redundancy text compressionInfo
- Publication number
- CA1209274A CA1209274A CA000461046A CA461046A CA1209274A CA 1209274 A CA1209274 A CA 1209274A CA 000461046 A CA000461046 A CA 000461046A CA 461046 A CA461046 A CA 461046A CA 1209274 A CA1209274 A CA 1209274A
- Authority
- CA
- Canada
- Prior art keywords
- text
- character
- characters
- context
- code
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Document Processing Apparatus (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
- Memory System (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
ABSTRACT
A generalized method and apparatus for compression and decompression oftextual information. Compression is performed by reading in succession each character and an associated context of a text, selecting for each character and associated context a corresponding code character, and providing the resulting code characters as the compressed form of the text. Decompression is a symmetric, inverse operation wherein the contexts associated with the code characters are read from a known decompressed portion of text. The context of a character is a group of text characters associated with a text character and containing a fixed number of characters. Text and code characters and contexts are related through a table containing a relative alphabet for the contexts of the language of the text. Each relative alphabet contains the text characters occurring in association with a corresponding context, and text characters within a given relative alphabet are identified by code characters assigned from set of code characters shared by the relative alphabets.
A generalized method and apparatus for compression and decompression oftextual information. Compression is performed by reading in succession each character and an associated context of a text, selecting for each character and associated context a corresponding code character, and providing the resulting code characters as the compressed form of the text. Decompression is a symmetric, inverse operation wherein the contexts associated with the code characters are read from a known decompressed portion of text. The context of a character is a group of text characters associated with a text character and containing a fixed number of characters. Text and code characters and contexts are related through a table containing a relative alphabet for the contexts of the language of the text. Each relative alphabet contains the text characters occurring in association with a corresponding context, and text characters within a given relative alphabet are identified by code characters assigned from set of code characters shared by the relative alphabets.
Description
$274 f,' A~3-00 CCNTEXT ~EDUNDANCY TEXT CGMPRESSICN
BACKGROUND QF THE INVENTION
1. Field of the Invention The present invention relates to data and word processing and communications systems and, more particularly, to a method and apparatus ~or compressing textual information for storage or transmission. In this context, textual information is defined as any information which is represented by a structured order of symbols or characters selected from a de~ined set, or alphabet, of symbols or characters. Common examples of textual information may include documents such as letters, reports and manuscripts written, for example, in English, German or French, business and accounting records, scientific data, and graphic displays comprised of arrangements of graphic symbols.
BACKGROUND QF THE INVENTION
1. Field of the Invention The present invention relates to data and word processing and communications systems and, more particularly, to a method and apparatus ~or compressing textual information for storage or transmission. In this context, textual information is defined as any information which is represented by a structured order of symbols or characters selected from a de~ined set, or alphabet, of symbols or characters. Common examples of textual information may include documents such as letters, reports and manuscripts written, for example, in English, German or French, business and accounting records, scientific data, and graphic displays comprised of arrangements of graphic symbols.
2. Prior Art A recurring problem in data processing and communications systems is that of storing, processing and communicating ever increasing volumes of information. The information handling requirements of such systems increases at least as rapidly, and often more rapidly, than does the capacity of available memories and data links. In addition, there are often physical or economic limits upon the memory or communications capability that may be provided with or added to a particular system.
3~25~79~
, ~
As a result, other methods than increasing memory or data link capacity have been developed to enable systems to handle increased volurnes of information. One such method is referred to as data compression, wherein information comrnunicated into a system by a user of the system is transformed by the system into a more compact, or reduced, form for storage or transmission. The information may subsequently be transformed, or decompressed, from its reduced form to its original form to be communicated to the user.
Typically, the language in which information is communicated between a system and a user of the system contains a significant degree of redundancy. That is, the language in which information is expressed contains more information than is required to completely and accurately represent the actual information. ~ common example occurs in word processing wherein information, that is, text, is communicated between the user and system in the English language, including punctuation and format characters such as periods, commas, spaces, tabs and line returns. Text compression is possible because of such redundancy and essentially transforms a user language text into a more compact form by deleting the redundant information from the user language version of the text.
Te~t compression methods of the prior art have been based upon distributional redundancy, that is, the nonlinearity in frequency of use or occurrence of certain characters, character combinations, and words in particular user languages. For example, in the English language the characters 'e' and ispace' occur more frequently than 'y' or 'z', and certain letter pairs, or digraphs, such as 'th' and 'es', and certain words, such as 'the', 'of', and 'and', occur frequently.
12(~27~
( ., ~
Prior art schemes have used this distributional redundancy to achieve compression by assigning variable length code words, or characters, to represent the frequently appearing characters, character combinations and words in particular languages. That is, the most frequently appearing characters, character combinations and words are assigned short code characters. Less common character combinations and words are, depending upon frequency of occurrence, assigned longer code characters or are 'spelled out' as sequences of the more frequently occurring characters, character combinations and words.
The actual compression and decompression of text in data and word processing and communications systems is generally implemented through the use of 'look-up' tables relating the frequently occurring characters, character combinations and words to the corresponding assigned code characters. The compression and decompression tables are generated separately from the actual cornpression/decompression operation and typically require a thorough, detailed linguistic analysis of very large volumes of text in the user language. It should be noted that while it is possible to assign a code character to each possible word and character in a particular language, the resulting code characters and tables become so large as to require more memory space than would be saved by text compression.
Distributional redundancy methods of text compression are very dependent upon the linguistic characteristics of the individual languages in which the original texts are created, particularly with regard to larger linguistic units, such as character combinations and words. For example, English, German, French, Russian, Italian and the Scandanavian languages 74~ ' .~ P
all have distinctly different linguistic characteristics, require different methods of analysis, and result in very different compression and decompression tables. As such, the compression schemes of the prior art have required a detailed linguistic analysis of of very large volumes of text in each separate user language in order to generate compression/decompression tables.
Because of the linguistic dependency of distributional redundancy methods, in particular with regard to the larger linguistic units, it is difficult to develope a completely general purpose method for analyzing distributional redundancy for a broad range of languages. Moreover, and for the same reasons, the compression/decompression tables for a particular language may depend upon the particular 'dialect' of text to be operated upon; for example, the linguistic characteristics for business, scientific and literary text may differ sufficiently to require separate tables for each application.
Further, because such methods use linguistic units and code words of differing sizes, compression/decompression requires relatively sophisticated programs with complex parsing capabilities and corresponding increases in processing capabilities and times and program memory space.
For the same reason, the compression and decompression operations may not be symmetric9 that is, may require separate tables and the execution of different routines with, again, increased processing and memory requirements. Finally, and for the same reasons, such methods are not suitable for continuous, in-line text processing or communication as the text must be processed as a series of small 'batch' operations, where the size of the batches are determined by the sizes of the linguistic units and code words.
7~
It is an object of the present invention to provide an improved method and apparatus for text compression.
According to a first broad aspect of the invention, there is provided in an information processing system, a method for compressing textual information, comprising the steps of:
reading in succession each character and an associated context of a text, selecting, for each said text character and associated context, a corresponding code character, and providing said code characters to comprise said text in compressed form.
According to a second broad aspect of ~he invention, there is provided in an information processing system, means for com-pressing textual information, comprising:
means for reading in succession each charaGter and an associated context of a text, means responsive to said reading means for selecting, for each said text character and associated context, a correspon-ding code character and providing said code characters to comprise said text in compressed form.
-7~
The text compression method and apparatus of the present invention operates upon the basis of context redundancy, that is, the nonuniform distribution of the contexts in which individual characters occur in texts, or, equivalently, the nonuniform distribution of occurrence of characters in a given set of contexts. Contexts, as defined herein, comprise uniform sized groups of characters preceding or following the individual characters comprising a text.
The method and apparatus described herein operates with abstract groupings and relationships between text characters and is not based upon any form of linguistic analysis nor upon any form of linguistic characteristic or element of any language. The method and apparatus is thereby general and is equally applicable for any language or 'dialect' of a language.
The method and apparatus operates with uniform sized groups of text and code characters and is thereby less complex and requires less processing time and capability than do other methods and is thereby faster in execution. The method is symmetric in execution, requires only a single, combined compression/decompression table, and is suitable for on-line, continuous processes~
Part of the information defining the relationships between code and text characters resides in the contexts, that is, in the text itself, in both its compressed and noncompressed forms. Because of this, fewer unique code characters, and thus shorter code characters, are required to unambiguously represent compressed text. The resulting compressed texts may thereby be represented by fewer oits of information.
$2~4 Other objects, advantages and features of the present invention will be understood by those of ordinary skill in the art after referring to the following detailed description of a pre-ferred embodiment and drawings, wherein:
FIGURE 1 is an isometric view of a computer system incorporating the present invention;
FIGURES 2A and 2B are a block diagram of the computer system of Figure l; and, FIGURE 3 is a diagrammatic representation of the software structure of the computer system of Figures 1, 2~ and 2B.
The following description will first briefly present the general structure and operation of ~he hardware and software of a computer system capable of performing data and word processing and communications operations and incorporating the present invention. The structure and operation of such a system will be well understood, as presented herein, by one of ordinary skill in the art and further details of the s~ructure and operation of such a system are presented in our Canadian Patent Application S.N. 440,241, filed Nov. 2, 1933. Having presen~ed a context, that is, apparatus, in which to practice the method of the present invention, the contex~ redundancy text compression method of the present invention will then be described in detail.
~2~ 7~
1. Computer System Structure and Operation Referring to Fig. 1, an isometric view of the computer systern is shown.
System lû includes a Display 12~ a Keyboard 14 and a Central Processing Unit tCPU) 16. Display 12 and Keyboard 14 are the primary means by which information, for example, text, is communicated between the system and a user. CPU 16, which is connected to Display 12 and Keyboard 14 by cables which are not shown~ includes a memory for storing programs and data and a general purpose arithmetic and logic unit (ALU)~ CPU 16 may further include disc drives for storing programs and data and interfaces to peripheral devices, such a printers, disc drives and telecommunications devices. System 10 may be comprised, for example, of a "Professional Computer" available from Wang Laboratories, Inc., Lowell, MA 018510 A. Hardware Structure and Operation a. ALU and Busses Referring to Figs. 2~ and ZB, a block diagram of System 10 is shown. System ln's ALU in CPU 16 is comprised of a Main Processor ~Main P) 18 and a Co-Processor (CO-P) 20. Main P 18 and CO-P 20 may, for example, respectively be a 16 bit Intel 8086 ALU and an Intel 8087 numerics processor extension. Main P 18 and CO-P 20 perform all arithmetic and logic operations for CPU 16, including addressing, memory references, and control of Input/Output (I/O) operations.
* trade mark ~2~274 Main P 18 and CO-P 20 communicate through Local Bus 22 and with the remainder of CPU 16, Display 12, Keyboard 14 and all peripheral devices through Address Bus 24, Data Bus 26 and Control Bus 28. ~he interface between Main P 18 and CO-P 20 and Busses 24, 26 and 28 is through Address Latch 30, Data Transceiver 32 and Bus Control 3~.
b. Internal Control Logic Referring next to CPU 16's internal control logic, associated with Main P
18 and CO-P 20 are System Clock 36 and Wait State Logic 38. System Clock 36 is the source o~ all clock timing signals for C~U 16. Wai-t State Logic 38 essentially monitors the operations of CPU 1~ and issues control signals, to System Clock 36 and to other elements of CPU 16 through Control Bus 28, to prevent conflicts in CPU 1~ operations.
Other of CPU 16's cnntrol elements include Counter and Timer Chip (CTC) 40 and Programmable Interrupt Chip ~PIC) 42. CTC 4C may, for example, be an Intel 8253 and PIC an Intel 8~5A. Non-Maskable Interrupt Logic 44 operates in conjunction with PIC 42 to handle interrupt conditions which must be handled immediately, that is, which cannot be masked for later action. Such interrupt conditions include parity and I/O errors.
CPU 16 is capable of performing both mapped memory references and Direct Memory Access (DMA) operations between CPU 16's memories, described below, and, ~or example, System lO's peripheral devices. DMA operations are controlled by DMA Bus Acquisition Logic 46 and DMA Control (DMA) 48. DMA
Acquisition Logic 46 monitors the operation of System 10, in particular memory and bus operations, and issues DMA grant signals on Control Bus 28 - ~ * trade mark -9~
7~ ( when DMA operations may be executed. DMA 4~ detects requests for DMA
operations, for example, from peripheral devices, informs DMA Bus Acquisition Logic 46 of the presence of such requests, and controls DMA
operations when DMA Bus Acquisition Logic 46 grants access to CPU 16's busses.
c. Memory CPU 16's primary internal memory is 128 KB Random Access Memory (RAM) 50, which is used to store operating system and applications programs and data, such as text, to be operated upon. The operating system program may, for example, be comprised of the commercially available Micro Soft Disk Operating System (MSDOS, a tradename) from Micro Soft Corporation and may include the Basic Input and Output System (BIOS). MSDOS essentially controls the executive internal operations of System 10 while BIOS
comprises programs controlling the interface between System 10, Display 12 and Keyboard 14 and a wide range of peripheral devices.
Where necessary, the capacity of 128 KB RAM SO may be augmented by the addition of 512 KB RAM 52. 512 KB RAM 52 is connected from Address Bus 24, Data Bus 26 and Control Bus 28 in parallel with 128 KB RA~l 50 and operates in parallel with and effectively as a part of 128 KB RAM 50.
Erasable Programmable Read Only Memory (EPRCM) 54 stores and provides programs used to load the operating system and application programs described above from diskettes in Disc Drive 56 and into 128 KB RAM 50 and 512 KB RAM 52.
* trade mark ~ $~7~ ( As indicated in Flg.2A~ Disc Drive 56 is connected from Address Bus 24, Data Bus 26 and Control Bus 28 and is controlled by Floppy Controller 58.
In addition to storing and providing operating system and applications programs, Disc Drive 56 may be used as additional memory capacity augmenting 128 KB RAM 50 and 512 KB RAM 52 and may be used to store and load data, such as text to be operated upon. In this regard, Disc Drive 56 may be used as an V 0 device, for example, to trans~er text or data from one system to another on diskette.
The capacity of Disc Drive 56 may be augmented by the addition of Winchester Hard Disc Drive 60 and Disc Controller 62, which are connected from Address Bus 28, Address Bus 30 and Control Bus 32 in parallel with Disc Drive 56 and RAMs 50 and 52. Hard Disc Drive 56 rnay be used as an extension to RAMs 50 and 52, for storing programs and data to be operated upon.
d. I/0 Devices As previously described, Display 12 and Keyboard 14 are the primary I/0 2û means for communication between Systern 10 and a user. Display 12 is a conventional CRT display connected to Address Bus 28, Data Bus 30 and Control Bus 32 through Monitor Control 13. Monitor Control 13 may, ~or example, be a Nippon Electric Corporation uPD 7220 Graphic Data Controller. Keyboard 14 is a conventional keyboard having an internal microprocessor, for example a Zilog Z80~ for controlling keyboard operation and data/control communications between Keyboard 14 and system Busses 28, 30 and 32 through Universal Asynchronous Receiver/Transmitter (UART) 15.
-~ * trade mark -11-~` 12~7~ !
Other I/O devices include Printers 54 and 66. Printer 6~ may, for example, be a conventional daisy wheel or dot matrix type printer~ Printer 66 may, for example, be a thermographic printer for graphics printing, such as a Microplot 80 available from Gulton Industries, Inc. of East Greenwich, RI
02818.
Finally, communication between System 10 and other systems or devices is provided through Telecommunications Controller (TC) 68. rc 68 is a microprocessor controlled interface device, incorporating for example a Zilog Corporation Z80 microprocessor and serial I/û chip, for communication between system busses 28, 70 and 32 and communications modems or external devices, such as memories and displays.
It should be noted that the System 10 described above is exemplary and that the text compression method described further below may be implemented, for example, on a variety of system architectures. The method may, for example, be implemented on a processor based upon*Zilog Z80, ~otorola MC68000, Intel 8080, 8086 or 80186, or*National Semiconductor ~S16000 microprocessors, upon a minicamputer or mainframe computer, or ~0 upon a word processor or communications processor.
Having described the structure and operation of System lO's hardware, the software structure and operation of System 10 will be described next below.
B. Software Structure and Operation Referring to Fig. ~9 a diagrammic representation of System lO's software structure is shown. As indicated therein7 the system software structure is * trade mark -12-:~2~$~:7~ ( a multilayered structure fGr transforming user actions into specific, detailed operations by the system hardware. The software structure includes elements for interfacing with the user at the upper layers and with the system hardware elements at the lower layers. The middle layers essentially transform user commands and requests into instructions directing the operations of the hardware.
Beginning at the user interface level, routines 70, 72, 74 and 76 comprises the system's application programs and determine and direct the specific applications operations performed by the system, for example, text or graphics editing or data processing.
User Interface Routines 70 includes all routines for interacting with the user by means of, for example, Display 12, Keyboard 14 and the system discs and printers. While User Interface Routines 70 are oriented towards the user, Application Function Interface Routines 72 interface between the user action operations as performed by User Interface Routines 70 and the software ~hich performs the actual applications operations, such as text editing. Essentially, Application Function Interface Routines 72 transform the results of the user interface operations into instructions and commands for directing the applications operations specified by the user.
Applications Routines 74 and 76 are the actual applications programs, that is, the programs directing the system to perform, for example, text or graphics editing operations or data processing operations. Applications Routines 74 and 76 receive direction from Application Function Interface Routines 72 regarding what operations have been requested by the user and provide instructions to Operating System 78 as to what system operations must be performed to execute the requested operations.
$~:7 I
As previously described, Operating System 78 may be comprised of MSDOS andincludes all routines necessary ~or executive control of the system.
Operating System 78 in turn interfaces with Hardware Interface System 80, which is comprised of BIOS, previously described. Essentially, Operating System 78 determines what detailed operations must be performed by the system to carry out the operations requested by the user and provides corresponding instructions to Hardware Interface System 80. Hardware Interface System 80 in turn responds by providing detailed commands to the various elements, previously described, comprising System lO's hardware structure.
As previously described, text compression allows text information to be stored within a system, for example, in memory or upon disc, or to be communicated between s~stems or devices in a more compact form. In the system described above, for example, text entered through the keyboard by a user, or loaded from a disc or through a communications llnk, could be compressed to be stored in RAM or upon disc or to be transmitted through a communications link. The text could subsequently be decompressed to be displayed to a user, for example, through the displa~ or on a printer~
Essentially, text compression and decompression may be performed whenever text information is communicated between a system element and a source or recipient using a language containing a significant degree of redundancy, for example~ EnglishO
As such, the text compression method described below would most probably be implemented in such a system as described above at the applications program level, corresponding to Applications Routines 74, 76 level of Fig.
3. The method could, however, also be implemented at the Application ~z~ 74t Function Interface 72 level, depending upon the degree of processingcapability present in, for example, the display and keyboard. Alternately, the method could be implemented at the Operating System 78 or Hard~are Interface System 80 level if the implementation was to be a basic feature of system operation. Depending upon the implementation, the compression/
decompression tables described below may reside in RAM or in ROMs.
Having described above the context, or apparatus, in which the method of the present invention may be practiced, the context redundancy text compression method of the present invention will be described next below.
2. Context Redundancy Text Compression As previously described, text compression depends upon the occurrence of information redundancy in the language of a text to be compressed. The prior art schemes described have relied upon distributional redundancy, that is, the non uniformity in frequency of occurrence of certain characters, character combinations and words in the original text language.
The text compression method of the present invention is based upon context redundancy, that is, the statistically uneven occurrence of certain characters in sequential association, or context, with other characters.
For example, the character 'u' most frequently follows the character 'q', 's' most frequently follows 'es' and a 'blank', or 'space', most frequently follows the characters 'est'.
~z~?2~
.
In this method, 'context' is defined as the sequence of characters, taken in order, preceding or following a particular character and is referred to, respectively, as preceding context or following context. A context need not be located immediately adjacent to associated character, although adjacent contexts ~re used in the following examples. Preceding and following contexts are defined with reference to the sequence in which the characters of a particular language are normally written. In English, for example, preceding context are the characters appearing to the left of a particular characters and following context are those appearing to the right of the character. In other languages, for example, Hebrew or Chinese, preceding context may appear to the right of or above a particular character and following context to the left or below~ In the example shown just above, 'q' is the left context of 'u', 'es' that of 's' and 'est' that of 'blank'.
While the text compression method described herein ~ay be implemented with equal facility for bnth preceding and following context, preceding context is preferred as most suitable for 'on line' compression and decompression.
Following context compression and decompression requires 'batch' processing7 that is, the storing and operating upon of sequential, overlapping groups of characters. Following context compression may, however, be advantageous in certain circumstances or for certain languages.
The 'order' of the context used in compressing and decompressing text is defined herein as the number of characters in the context and will preferably be constant in a particular implementation of the method. In the example shown above, 'q' is the first order context of 'u', 'es' the second order context of 's' and 'est' the third order context of 'blank'.
$;~74~
To further define the above terms, let a particular language be expressedin an alphabet A comprised of the characters xi where i = 1 to u, ' 1' X2 xu_l~ xu. Assuming that the language is normally written left to right, if the sequence of characters xnxn ix3x2xlxi appears in the text, then the sequence of characters xnxn ix3x2x1 is the n order preceding, or left, context of xi. It should be noted that the context of any character xi does not include the character xi itself.
The present method may be implemented ~or any value of order n and examples for the orders of n=l and n=2 are presented below. Higher values of order may~ however, be preferable.
For a given xi and a given order n, the set of all possible unique nth order preceding contexts is designated as PCn(xi) and may contain up to u~ members since there are u possible unique characters xi and n possible positions for each unique character. The set of all possible unique nth order preceding contexts for a given language is designated PCn and may contain up to un+l members since xi can be any of u unique characters.
The text compression method of the present invention requires the use of 'look-up' table for compression and decompression. Although it appears from the above that such tables would be very large, in practice the tables are of acceptable size~ As will be described further below, the present method and the construction of tables to implement the present ~21~ 7~
method are not dependent upon the linguistic characteristics of the text languages. The linguistic characteristics do, however, effec-t the size of the tables because certain contexts are rare or do not appear in particular languages, thereby in practice limiting the sizes of the tables. For exampleg 'wc' rarely appears as a prior context of 's' in English, and thus may not appear in an English language compression table, but may appear in, for example, a Polish language table. The compression method and the means by which the tables are generated are, however, the same in both English and Polish.
The compression/decompression tables of the present invention may be generated heuristically or by linguistic analysis. In the more preferred method, however9 the tables are generated through context analysis of a suitable body oF text, for example, dictionaries, thesauruses, manuscripts or documents.
The preferred method of table generation is a mechanical rather than linguistic analysis. First, for a given order of context n, the sample body of text is scanned character by character in sequence in the direction normally written and, for each character xi, each occurrence of each prior context PCn(xi) occurring is tabulated. The analysis will provide a table of frequency of occurrence of each PCn(xi) for each xi and essentially maps contextual redundancy into a corresponding distributional redundancy of contexts for each character in the alphabet.
This process may be readily accomplished by generating a tabular array using xi and PCn(xi) as indices to the cells thereof and appropriately incrementing the cell contents as the characters xi of the sample text are scanned.
27~
Having generated the above described table, a final table is generated by sorting, for each PCn(xi), each xi which appears in a particular context. That is, the final table will contain, for each particular context, all characters appearing in that context. The characters will generally be sorted, for each context, in order of frequency of occurrence. It should be noted that this final table may be generated directly, rather than as a derivation from the first table, by scanning successive contexts, shifting by one character at a time, and tabulating the occurring contexts and the characters appearing in each such context.
The final table then allows a 'relative alphabet' to be defined for each context, the relative alphabet for a particular context being comprised of all characters which occur in that context in order of frequency of occurrence. As will be described further below, this final table defines the relationships between all possible noncompressed text characters and compressed text code characters and is used for both compression and decompression of text.
Referring to Table 1, an illustrative example of a final compression~
decompression table for a prior context of order n=2 is shown. The sample body of text used to generate Table 1 was comprised of general office correspondence. As Table 1 is intended solely for illustrative purposes, only a few of the occurring contexts are shown and for each context shown only the eight most frequently occurring text characters are shown.
~21~7~ /
-RANK CODE CHARACTER
D A o B T E
e_ a t w s c i p o o_ t m a c p i h d t w t a i o c y p a p c s d g b f r e x n d a 1 v s q ee t d m k n te r d _ n m c g ae 1 nul nul nul nul nul nul nul ie s n v d t f r re _ s a 1 d e c n ne _ s r w d x t e se _d n *M r s le _ a x s c *M t _o f u n r t p w v eo v p g r n u *M nul oo k n d m p t to r m *M d n t p io n r u g d nul nul ro d m n v f c p g no w t n u so o m r n f 1 t lo s o w p g n _ t co n m p u v r s d ot h _ a i e , _ *M
oa c s r d t nul nul nul or d e k t m s w _t h o e i a w r u ta i n k b 1 t g al 1 i s *M m t 1 b a s t p f c E
s p u e o y i t a sa u 1 t y n m b g es s _ t a e ~2~274 The various two character contexts are listed as the left, vertical index of Table l and the text characters appearing in each context are listed in the rows to the right of the corresponding contexts. For example, the text characters, or relative alphabet occurring for the context 'col are, in order of frequency o~ occurrence, 'n', 'm', 'p', 'u', 'v', 'rl, 'sl and 'dl. It should be noted that the text character l_l is used to represent a space, or blank, I*MI represents an end-of-line character, and 'nul represents an entry with a zero frequency count.
The rank numbers appearing as the horizontal index at the top of Table l comprise, for each context, the symbols, or code characters, used to represent the various text characters of each of the relative alphabets.
Thus, the code character for text character Iml in context Icol is '2l.
An examination of compression/decompression Table l shows that, for example, code character l2' also represents text character Id' in contexts 'ee', 'te', 'se' and 'or' and text character Ir' in contexts 'to' and 'io'. Similarly, the code character '5' represents text character 'g' in contexts 'a ', 'io' and 'lo' and text character 'd' in contexts 're', 'ne' and '~o'. In further example, code characters 'll' represent 'ss' if the prior context is 'ie' but represent 'th' if the prior context is 'o_' It is apparent from the above that the same code character may represent different text characters and that a given text character may be represented by different code characters. That is, that there is not a unique, one to one relationship between code characters and text - - \
~LZ~%74 characters in themselves. The relationship between a code character and a text character depends upon the particular context in which the code and text character appear.
This illustrates property of context redundancy compression which will be further illustrated by example below, that a code character is not required to contain, in itself, the total amount of information required to uniquely define a particular text character. Part of the information defining the relationships between code and text characters resides in the text itself, that is, in the contexts. Because of this, fewer unique code characters, and thus shor~er code characters, are required to unambiguously represent compressed text. The resulting compressed texts may thereby be represented by fewer bits of information. Thus the text itself, in both the compressed and noncompressed forms contains, in addition to the textual information9 a part of the information required to transform between the compressed and noncompressed forms.
Because of the context dependent relationship between text and code characters, a prior context, for example, 'nuls' or 'blanks', is either assumed or imposed for the first n characters of text. This 'header context' provides a known context for compressing or decompressing the first character or characters of text, the following characters being compressed or decompressed in the context of the text itself.
To illustrate the use of the method and table of the present invention for compression and decompression of text, assume that the phrase 'total salt' appears in the text and that the prior context of the first 't' in 'total' is 'e_' 9 as in (th)e_. The compression and subsequent decompression of ~L;2Q~7~
this phrase is illustrated with Tables 2 and 3 presented below. Table 2 illustrates the compression process and Table 3 the decompression process.
CGMPRESSIGN OF 'total salt' Text Character Prior Context Code Character t e_ 2 o - t 2 t to 7 a ot 3 1 ta 5 _ al s 1 3 a _s 8 1 sa 2 t al 7 To compress the sample text, each text character and its associated context are read in turn and, in the preferred embodiment, in the sequence in which the text is normally written. In the present example, the reading will be ~rom left to right. Since the present example uses a 2nd order context, that is, n=2, the context ~or each character is comprised of the two characters immediately preceding the character. Having determined the t~o character prior context of a particular character, that context is then used as an index into the compression/decompression table to locate ~ 2~274.
the relative alphabet for that context. The character read in association with the context is then located in the relative alphabet and the corresponding code character located in the rank index at the top oF the table. This process is continued, a character at a time, until the compression operation is completed.
For example, the prior context of the first 't' in 'total' is 'e '. 'e_' is used as an index into the table to locate the corresponding relative alphabet, which appears as the second relative alphabet from the top. The relative alphabet is then scanned to locate text character 'tl, the second entry frqm the left, and the corresponding code character is read from the top of the table as '2'. The next text character is the 'o' of 'total' and the context of 'o' is '_t', so that the code character for 'o' is again '2'. The third text character is the second 't' of 'total', the prior context of this 't' is 'to' and the code character is determined to be '7'.
This process continues to the end of the text and the cnmpressed form of 'total salt' is found to be '2273513~27'.
The reverse operation is performed, using the same compression/
decompression table, to decompress compressed text. In this case, the 'text' to be read is the compressed text and the 'characterst read, one at a time and in sequence as written, are the code characters of the compressed text. The prior context of each code character is comprised of the preceding decoded characters o~ the restored text and are determined as the text is decompressed. The contexts so determined are again used as indices into the table to select the appropriate relative alphabets and the code characters are used as indices to determine the appropriate text -24~
~Z~27~
characters within the selected relative alphabets. The known 'header context' used in the compression operation is again used as the beginning context in decompressing the first character or characters of the compressed text.
To illustrate the decompression operation using the same example as above, assume that the text to be decompressed is comprised of the code character sequence '2273513827' and that the preceding decompressed text has been determined to again be 'e_'. The decompression operation is then illustrated in Table 3 below.
TABL~ 3 DECOMPRESSI~N CF '2273513827' . . .
Prior Context Code Character Text Character e 2 t t 2 o to 7 t ot 3 a ta S
al 3 s s 8 a sa 2 al 7 t Assuming that the preceding compressed text has been decompressed to the start of the sequence '2273513827', the prior context of the first '2' in the sequence is known from the previously decompressed portion of the text to be 'e '. 'e_' is then used as an index into the table to ~etermine the reiative alphabet of the first '2' of the sequence and, for this character, is the second relative alphabet from the top of the table. The first '2' of the sequence is then used as an index into the table to determine that the corresponding text character of that relative alphabet is 't'. The prior context of the second l2t in the sequence is then kno~n ~$~74 to be ' t' and, using ' t' and '2' as indices into the table, thetext charactercorresponding to the second '2' is determined to be '0'. The third code characte.r is '7' with a now known prior context of 'to' and the text character can be determined to be 't', and so on.
In a yet further implementation, the compression/decom-pression described above may be used recursively, that is, the compressed texts may themselves be compressed usin~ the same method as described above. Recursive compression requires the generation of compression/decompression tables in the same manner as described above, but using the compressed forms of the texts to determine and tabulate the context redundancy of the compressed te~tual matexial. The compression of text then becomes a two step process, a first step to compress the oriyinal text and a second step to compress the compressed text, as does the decompres-sion process. The two step, or recursive, compression/decompres-sion may, as in a single step process, be performed 'on-line', that is, as a continuous process.
To reiterate certain features of the above described method of text compression, the present method operates upon the basis of context redundancy, that is, the nonuniform distribution of the contexts in which individual characters occur. Contexts, as defined herein, comprise uniform ~2~$~7~
sized groups of characters preceding or following the individual characters comprising a text.
Although the compression/decompression tables used in the method are effected by the linguistic characteristics of particular languages, the method is not based upon any form of linguistic analysis nor upon any form of linguistic characteristic or element of any language. The method is thereby completely general and is equally applicable for any language or 'dialect' of a language.
1~
Because the method operates with uniform sized groups of text and code characters~ the method is less complex and requires less processing time and capability than do other methods and is thereby faster in execution.
For the same reasons, the method is symmetric in execution, requires only a single compression/decompression table, and is suitable for on-line~
continuous processes.
Part of the inform~tion defining the relationships between code and text characters resides in the contexts, that is, in the text itself, in both its compressed and noncompressed forms. Because of this, fewer unique code characters, and thus shorter code characters, are required to unambiguously represent compressed text. The resulting compressed texts may thereby be represented by fewer bits of information.
It will be apparent to those of ordinary skill in the art that the present invention may be embodied in yet other specific forms without departing from the spirit or essential characteristics thereof. Thus, the present embodiments are to be considered in all respects as illustrative and \
restrictive, the scope of the invention being indicated by the appendedclaims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.
1~
, ~
As a result, other methods than increasing memory or data link capacity have been developed to enable systems to handle increased volurnes of information. One such method is referred to as data compression, wherein information comrnunicated into a system by a user of the system is transformed by the system into a more compact, or reduced, form for storage or transmission. The information may subsequently be transformed, or decompressed, from its reduced form to its original form to be communicated to the user.
Typically, the language in which information is communicated between a system and a user of the system contains a significant degree of redundancy. That is, the language in which information is expressed contains more information than is required to completely and accurately represent the actual information. ~ common example occurs in word processing wherein information, that is, text, is communicated between the user and system in the English language, including punctuation and format characters such as periods, commas, spaces, tabs and line returns. Text compression is possible because of such redundancy and essentially transforms a user language text into a more compact form by deleting the redundant information from the user language version of the text.
Te~t compression methods of the prior art have been based upon distributional redundancy, that is, the nonlinearity in frequency of use or occurrence of certain characters, character combinations, and words in particular user languages. For example, in the English language the characters 'e' and ispace' occur more frequently than 'y' or 'z', and certain letter pairs, or digraphs, such as 'th' and 'es', and certain words, such as 'the', 'of', and 'and', occur frequently.
12(~27~
( ., ~
Prior art schemes have used this distributional redundancy to achieve compression by assigning variable length code words, or characters, to represent the frequently appearing characters, character combinations and words in particular languages. That is, the most frequently appearing characters, character combinations and words are assigned short code characters. Less common character combinations and words are, depending upon frequency of occurrence, assigned longer code characters or are 'spelled out' as sequences of the more frequently occurring characters, character combinations and words.
The actual compression and decompression of text in data and word processing and communications systems is generally implemented through the use of 'look-up' tables relating the frequently occurring characters, character combinations and words to the corresponding assigned code characters. The compression and decompression tables are generated separately from the actual cornpression/decompression operation and typically require a thorough, detailed linguistic analysis of very large volumes of text in the user language. It should be noted that while it is possible to assign a code character to each possible word and character in a particular language, the resulting code characters and tables become so large as to require more memory space than would be saved by text compression.
Distributional redundancy methods of text compression are very dependent upon the linguistic characteristics of the individual languages in which the original texts are created, particularly with regard to larger linguistic units, such as character combinations and words. For example, English, German, French, Russian, Italian and the Scandanavian languages 74~ ' .~ P
all have distinctly different linguistic characteristics, require different methods of analysis, and result in very different compression and decompression tables. As such, the compression schemes of the prior art have required a detailed linguistic analysis of of very large volumes of text in each separate user language in order to generate compression/decompression tables.
Because of the linguistic dependency of distributional redundancy methods, in particular with regard to the larger linguistic units, it is difficult to develope a completely general purpose method for analyzing distributional redundancy for a broad range of languages. Moreover, and for the same reasons, the compression/decompression tables for a particular language may depend upon the particular 'dialect' of text to be operated upon; for example, the linguistic characteristics for business, scientific and literary text may differ sufficiently to require separate tables for each application.
Further, because such methods use linguistic units and code words of differing sizes, compression/decompression requires relatively sophisticated programs with complex parsing capabilities and corresponding increases in processing capabilities and times and program memory space.
For the same reason, the compression and decompression operations may not be symmetric9 that is, may require separate tables and the execution of different routines with, again, increased processing and memory requirements. Finally, and for the same reasons, such methods are not suitable for continuous, in-line text processing or communication as the text must be processed as a series of small 'batch' operations, where the size of the batches are determined by the sizes of the linguistic units and code words.
7~
It is an object of the present invention to provide an improved method and apparatus for text compression.
According to a first broad aspect of the invention, there is provided in an information processing system, a method for compressing textual information, comprising the steps of:
reading in succession each character and an associated context of a text, selecting, for each said text character and associated context, a corresponding code character, and providing said code characters to comprise said text in compressed form.
According to a second broad aspect of ~he invention, there is provided in an information processing system, means for com-pressing textual information, comprising:
means for reading in succession each charaGter and an associated context of a text, means responsive to said reading means for selecting, for each said text character and associated context, a correspon-ding code character and providing said code characters to comprise said text in compressed form.
-7~
The text compression method and apparatus of the present invention operates upon the basis of context redundancy, that is, the nonuniform distribution of the contexts in which individual characters occur in texts, or, equivalently, the nonuniform distribution of occurrence of characters in a given set of contexts. Contexts, as defined herein, comprise uniform sized groups of characters preceding or following the individual characters comprising a text.
The method and apparatus described herein operates with abstract groupings and relationships between text characters and is not based upon any form of linguistic analysis nor upon any form of linguistic characteristic or element of any language. The method and apparatus is thereby general and is equally applicable for any language or 'dialect' of a language.
The method and apparatus operates with uniform sized groups of text and code characters and is thereby less complex and requires less processing time and capability than do other methods and is thereby faster in execution. The method is symmetric in execution, requires only a single, combined compression/decompression table, and is suitable for on-line, continuous processes~
Part of the information defining the relationships between code and text characters resides in the contexts, that is, in the text itself, in both its compressed and noncompressed forms. Because of this, fewer unique code characters, and thus shorter code characters, are required to unambiguously represent compressed text. The resulting compressed texts may thereby be represented by fewer oits of information.
$2~4 Other objects, advantages and features of the present invention will be understood by those of ordinary skill in the art after referring to the following detailed description of a pre-ferred embodiment and drawings, wherein:
FIGURE 1 is an isometric view of a computer system incorporating the present invention;
FIGURES 2A and 2B are a block diagram of the computer system of Figure l; and, FIGURE 3 is a diagrammatic representation of the software structure of the computer system of Figures 1, 2~ and 2B.
The following description will first briefly present the general structure and operation of ~he hardware and software of a computer system capable of performing data and word processing and communications operations and incorporating the present invention. The structure and operation of such a system will be well understood, as presented herein, by one of ordinary skill in the art and further details of the s~ructure and operation of such a system are presented in our Canadian Patent Application S.N. 440,241, filed Nov. 2, 1933. Having presen~ed a context, that is, apparatus, in which to practice the method of the present invention, the contex~ redundancy text compression method of the present invention will then be described in detail.
~2~ 7~
1. Computer System Structure and Operation Referring to Fig. 1, an isometric view of the computer systern is shown.
System lû includes a Display 12~ a Keyboard 14 and a Central Processing Unit tCPU) 16. Display 12 and Keyboard 14 are the primary means by which information, for example, text, is communicated between the system and a user. CPU 16, which is connected to Display 12 and Keyboard 14 by cables which are not shown~ includes a memory for storing programs and data and a general purpose arithmetic and logic unit (ALU)~ CPU 16 may further include disc drives for storing programs and data and interfaces to peripheral devices, such a printers, disc drives and telecommunications devices. System 10 may be comprised, for example, of a "Professional Computer" available from Wang Laboratories, Inc., Lowell, MA 018510 A. Hardware Structure and Operation a. ALU and Busses Referring to Figs. 2~ and ZB, a block diagram of System 10 is shown. System ln's ALU in CPU 16 is comprised of a Main Processor ~Main P) 18 and a Co-Processor (CO-P) 20. Main P 18 and CO-P 20 may, for example, respectively be a 16 bit Intel 8086 ALU and an Intel 8087 numerics processor extension. Main P 18 and CO-P 20 perform all arithmetic and logic operations for CPU 16, including addressing, memory references, and control of Input/Output (I/O) operations.
* trade mark ~2~274 Main P 18 and CO-P 20 communicate through Local Bus 22 and with the remainder of CPU 16, Display 12, Keyboard 14 and all peripheral devices through Address Bus 24, Data Bus 26 and Control Bus 28. ~he interface between Main P 18 and CO-P 20 and Busses 24, 26 and 28 is through Address Latch 30, Data Transceiver 32 and Bus Control 3~.
b. Internal Control Logic Referring next to CPU 16's internal control logic, associated with Main P
18 and CO-P 20 are System Clock 36 and Wait State Logic 38. System Clock 36 is the source o~ all clock timing signals for C~U 16. Wai-t State Logic 38 essentially monitors the operations of CPU 1~ and issues control signals, to System Clock 36 and to other elements of CPU 16 through Control Bus 28, to prevent conflicts in CPU 1~ operations.
Other of CPU 16's cnntrol elements include Counter and Timer Chip (CTC) 40 and Programmable Interrupt Chip ~PIC) 42. CTC 4C may, for example, be an Intel 8253 and PIC an Intel 8~5A. Non-Maskable Interrupt Logic 44 operates in conjunction with PIC 42 to handle interrupt conditions which must be handled immediately, that is, which cannot be masked for later action. Such interrupt conditions include parity and I/O errors.
CPU 16 is capable of performing both mapped memory references and Direct Memory Access (DMA) operations between CPU 16's memories, described below, and, ~or example, System lO's peripheral devices. DMA operations are controlled by DMA Bus Acquisition Logic 46 and DMA Control (DMA) 48. DMA
Acquisition Logic 46 monitors the operation of System 10, in particular memory and bus operations, and issues DMA grant signals on Control Bus 28 - ~ * trade mark -9~
7~ ( when DMA operations may be executed. DMA 4~ detects requests for DMA
operations, for example, from peripheral devices, informs DMA Bus Acquisition Logic 46 of the presence of such requests, and controls DMA
operations when DMA Bus Acquisition Logic 46 grants access to CPU 16's busses.
c. Memory CPU 16's primary internal memory is 128 KB Random Access Memory (RAM) 50, which is used to store operating system and applications programs and data, such as text, to be operated upon. The operating system program may, for example, be comprised of the commercially available Micro Soft Disk Operating System (MSDOS, a tradename) from Micro Soft Corporation and may include the Basic Input and Output System (BIOS). MSDOS essentially controls the executive internal operations of System 10 while BIOS
comprises programs controlling the interface between System 10, Display 12 and Keyboard 14 and a wide range of peripheral devices.
Where necessary, the capacity of 128 KB RAM SO may be augmented by the addition of 512 KB RAM 52. 512 KB RAM 52 is connected from Address Bus 24, Data Bus 26 and Control Bus 28 in parallel with 128 KB RA~l 50 and operates in parallel with and effectively as a part of 128 KB RAM 50.
Erasable Programmable Read Only Memory (EPRCM) 54 stores and provides programs used to load the operating system and application programs described above from diskettes in Disc Drive 56 and into 128 KB RAM 50 and 512 KB RAM 52.
* trade mark ~ $~7~ ( As indicated in Flg.2A~ Disc Drive 56 is connected from Address Bus 24, Data Bus 26 and Control Bus 28 and is controlled by Floppy Controller 58.
In addition to storing and providing operating system and applications programs, Disc Drive 56 may be used as additional memory capacity augmenting 128 KB RAM 50 and 512 KB RAM 52 and may be used to store and load data, such as text to be operated upon. In this regard, Disc Drive 56 may be used as an V 0 device, for example, to trans~er text or data from one system to another on diskette.
The capacity of Disc Drive 56 may be augmented by the addition of Winchester Hard Disc Drive 60 and Disc Controller 62, which are connected from Address Bus 28, Address Bus 30 and Control Bus 32 in parallel with Disc Drive 56 and RAMs 50 and 52. Hard Disc Drive 56 rnay be used as an extension to RAMs 50 and 52, for storing programs and data to be operated upon.
d. I/0 Devices As previously described, Display 12 and Keyboard 14 are the primary I/0 2û means for communication between Systern 10 and a user. Display 12 is a conventional CRT display connected to Address Bus 28, Data Bus 30 and Control Bus 32 through Monitor Control 13. Monitor Control 13 may, ~or example, be a Nippon Electric Corporation uPD 7220 Graphic Data Controller. Keyboard 14 is a conventional keyboard having an internal microprocessor, for example a Zilog Z80~ for controlling keyboard operation and data/control communications between Keyboard 14 and system Busses 28, 30 and 32 through Universal Asynchronous Receiver/Transmitter (UART) 15.
-~ * trade mark -11-~` 12~7~ !
Other I/O devices include Printers 54 and 66. Printer 6~ may, for example, be a conventional daisy wheel or dot matrix type printer~ Printer 66 may, for example, be a thermographic printer for graphics printing, such as a Microplot 80 available from Gulton Industries, Inc. of East Greenwich, RI
02818.
Finally, communication between System 10 and other systems or devices is provided through Telecommunications Controller (TC) 68. rc 68 is a microprocessor controlled interface device, incorporating for example a Zilog Corporation Z80 microprocessor and serial I/û chip, for communication between system busses 28, 70 and 32 and communications modems or external devices, such as memories and displays.
It should be noted that the System 10 described above is exemplary and that the text compression method described further below may be implemented, for example, on a variety of system architectures. The method may, for example, be implemented on a processor based upon*Zilog Z80, ~otorola MC68000, Intel 8080, 8086 or 80186, or*National Semiconductor ~S16000 microprocessors, upon a minicamputer or mainframe computer, or ~0 upon a word processor or communications processor.
Having described the structure and operation of System lO's hardware, the software structure and operation of System 10 will be described next below.
B. Software Structure and Operation Referring to Fig. ~9 a diagrammic representation of System lO's software structure is shown. As indicated therein7 the system software structure is * trade mark -12-:~2~$~:7~ ( a multilayered structure fGr transforming user actions into specific, detailed operations by the system hardware. The software structure includes elements for interfacing with the user at the upper layers and with the system hardware elements at the lower layers. The middle layers essentially transform user commands and requests into instructions directing the operations of the hardware.
Beginning at the user interface level, routines 70, 72, 74 and 76 comprises the system's application programs and determine and direct the specific applications operations performed by the system, for example, text or graphics editing or data processing.
User Interface Routines 70 includes all routines for interacting with the user by means of, for example, Display 12, Keyboard 14 and the system discs and printers. While User Interface Routines 70 are oriented towards the user, Application Function Interface Routines 72 interface between the user action operations as performed by User Interface Routines 70 and the software ~hich performs the actual applications operations, such as text editing. Essentially, Application Function Interface Routines 72 transform the results of the user interface operations into instructions and commands for directing the applications operations specified by the user.
Applications Routines 74 and 76 are the actual applications programs, that is, the programs directing the system to perform, for example, text or graphics editing operations or data processing operations. Applications Routines 74 and 76 receive direction from Application Function Interface Routines 72 regarding what operations have been requested by the user and provide instructions to Operating System 78 as to what system operations must be performed to execute the requested operations.
$~:7 I
As previously described, Operating System 78 may be comprised of MSDOS andincludes all routines necessary ~or executive control of the system.
Operating System 78 in turn interfaces with Hardware Interface System 80, which is comprised of BIOS, previously described. Essentially, Operating System 78 determines what detailed operations must be performed by the system to carry out the operations requested by the user and provides corresponding instructions to Hardware Interface System 80. Hardware Interface System 80 in turn responds by providing detailed commands to the various elements, previously described, comprising System lO's hardware structure.
As previously described, text compression allows text information to be stored within a system, for example, in memory or upon disc, or to be communicated between s~stems or devices in a more compact form. In the system described above, for example, text entered through the keyboard by a user, or loaded from a disc or through a communications llnk, could be compressed to be stored in RAM or upon disc or to be transmitted through a communications link. The text could subsequently be decompressed to be displayed to a user, for example, through the displa~ or on a printer~
Essentially, text compression and decompression may be performed whenever text information is communicated between a system element and a source or recipient using a language containing a significant degree of redundancy, for example~ EnglishO
As such, the text compression method described below would most probably be implemented in such a system as described above at the applications program level, corresponding to Applications Routines 74, 76 level of Fig.
3. The method could, however, also be implemented at the Application ~z~ 74t Function Interface 72 level, depending upon the degree of processingcapability present in, for example, the display and keyboard. Alternately, the method could be implemented at the Operating System 78 or Hard~are Interface System 80 level if the implementation was to be a basic feature of system operation. Depending upon the implementation, the compression/
decompression tables described below may reside in RAM or in ROMs.
Having described above the context, or apparatus, in which the method of the present invention may be practiced, the context redundancy text compression method of the present invention will be described next below.
2. Context Redundancy Text Compression As previously described, text compression depends upon the occurrence of information redundancy in the language of a text to be compressed. The prior art schemes described have relied upon distributional redundancy, that is, the non uniformity in frequency of occurrence of certain characters, character combinations and words in the original text language.
The text compression method of the present invention is based upon context redundancy, that is, the statistically uneven occurrence of certain characters in sequential association, or context, with other characters.
For example, the character 'u' most frequently follows the character 'q', 's' most frequently follows 'es' and a 'blank', or 'space', most frequently follows the characters 'est'.
~z~?2~
.
In this method, 'context' is defined as the sequence of characters, taken in order, preceding or following a particular character and is referred to, respectively, as preceding context or following context. A context need not be located immediately adjacent to associated character, although adjacent contexts ~re used in the following examples. Preceding and following contexts are defined with reference to the sequence in which the characters of a particular language are normally written. In English, for example, preceding context are the characters appearing to the left of a particular characters and following context are those appearing to the right of the character. In other languages, for example, Hebrew or Chinese, preceding context may appear to the right of or above a particular character and following context to the left or below~ In the example shown just above, 'q' is the left context of 'u', 'es' that of 's' and 'est' that of 'blank'.
While the text compression method described herein ~ay be implemented with equal facility for bnth preceding and following context, preceding context is preferred as most suitable for 'on line' compression and decompression.
Following context compression and decompression requires 'batch' processing7 that is, the storing and operating upon of sequential, overlapping groups of characters. Following context compression may, however, be advantageous in certain circumstances or for certain languages.
The 'order' of the context used in compressing and decompressing text is defined herein as the number of characters in the context and will preferably be constant in a particular implementation of the method. In the example shown above, 'q' is the first order context of 'u', 'es' the second order context of 's' and 'est' the third order context of 'blank'.
$;~74~
To further define the above terms, let a particular language be expressedin an alphabet A comprised of the characters xi where i = 1 to u, ' 1' X2 xu_l~ xu. Assuming that the language is normally written left to right, if the sequence of characters xnxn ix3x2xlxi appears in the text, then the sequence of characters xnxn ix3x2x1 is the n order preceding, or left, context of xi. It should be noted that the context of any character xi does not include the character xi itself.
The present method may be implemented ~or any value of order n and examples for the orders of n=l and n=2 are presented below. Higher values of order may~ however, be preferable.
For a given xi and a given order n, the set of all possible unique nth order preceding contexts is designated as PCn(xi) and may contain up to u~ members since there are u possible unique characters xi and n possible positions for each unique character. The set of all possible unique nth order preceding contexts for a given language is designated PCn and may contain up to un+l members since xi can be any of u unique characters.
The text compression method of the present invention requires the use of 'look-up' table for compression and decompression. Although it appears from the above that such tables would be very large, in practice the tables are of acceptable size~ As will be described further below, the present method and the construction of tables to implement the present ~21~ 7~
method are not dependent upon the linguistic characteristics of the text languages. The linguistic characteristics do, however, effec-t the size of the tables because certain contexts are rare or do not appear in particular languages, thereby in practice limiting the sizes of the tables. For exampleg 'wc' rarely appears as a prior context of 's' in English, and thus may not appear in an English language compression table, but may appear in, for example, a Polish language table. The compression method and the means by which the tables are generated are, however, the same in both English and Polish.
The compression/decompression tables of the present invention may be generated heuristically or by linguistic analysis. In the more preferred method, however9 the tables are generated through context analysis of a suitable body oF text, for example, dictionaries, thesauruses, manuscripts or documents.
The preferred method of table generation is a mechanical rather than linguistic analysis. First, for a given order of context n, the sample body of text is scanned character by character in sequence in the direction normally written and, for each character xi, each occurrence of each prior context PCn(xi) occurring is tabulated. The analysis will provide a table of frequency of occurrence of each PCn(xi) for each xi and essentially maps contextual redundancy into a corresponding distributional redundancy of contexts for each character in the alphabet.
This process may be readily accomplished by generating a tabular array using xi and PCn(xi) as indices to the cells thereof and appropriately incrementing the cell contents as the characters xi of the sample text are scanned.
27~
Having generated the above described table, a final table is generated by sorting, for each PCn(xi), each xi which appears in a particular context. That is, the final table will contain, for each particular context, all characters appearing in that context. The characters will generally be sorted, for each context, in order of frequency of occurrence. It should be noted that this final table may be generated directly, rather than as a derivation from the first table, by scanning successive contexts, shifting by one character at a time, and tabulating the occurring contexts and the characters appearing in each such context.
The final table then allows a 'relative alphabet' to be defined for each context, the relative alphabet for a particular context being comprised of all characters which occur in that context in order of frequency of occurrence. As will be described further below, this final table defines the relationships between all possible noncompressed text characters and compressed text code characters and is used for both compression and decompression of text.
Referring to Table 1, an illustrative example of a final compression~
decompression table for a prior context of order n=2 is shown. The sample body of text used to generate Table 1 was comprised of general office correspondence. As Table 1 is intended solely for illustrative purposes, only a few of the occurring contexts are shown and for each context shown only the eight most frequently occurring text characters are shown.
~21~7~ /
-RANK CODE CHARACTER
D A o B T E
e_ a t w s c i p o o_ t m a c p i h d t w t a i o c y p a p c s d g b f r e x n d a 1 v s q ee t d m k n te r d _ n m c g ae 1 nul nul nul nul nul nul nul ie s n v d t f r re _ s a 1 d e c n ne _ s r w d x t e se _d n *M r s le _ a x s c *M t _o f u n r t p w v eo v p g r n u *M nul oo k n d m p t to r m *M d n t p io n r u g d nul nul ro d m n v f c p g no w t n u so o m r n f 1 t lo s o w p g n _ t co n m p u v r s d ot h _ a i e , _ *M
oa c s r d t nul nul nul or d e k t m s w _t h o e i a w r u ta i n k b 1 t g al 1 i s *M m t 1 b a s t p f c E
s p u e o y i t a sa u 1 t y n m b g es s _ t a e ~2~274 The various two character contexts are listed as the left, vertical index of Table l and the text characters appearing in each context are listed in the rows to the right of the corresponding contexts. For example, the text characters, or relative alphabet occurring for the context 'col are, in order of frequency o~ occurrence, 'n', 'm', 'p', 'u', 'v', 'rl, 'sl and 'dl. It should be noted that the text character l_l is used to represent a space, or blank, I*MI represents an end-of-line character, and 'nul represents an entry with a zero frequency count.
The rank numbers appearing as the horizontal index at the top of Table l comprise, for each context, the symbols, or code characters, used to represent the various text characters of each of the relative alphabets.
Thus, the code character for text character Iml in context Icol is '2l.
An examination of compression/decompression Table l shows that, for example, code character l2' also represents text character Id' in contexts 'ee', 'te', 'se' and 'or' and text character Ir' in contexts 'to' and 'io'. Similarly, the code character '5' represents text character 'g' in contexts 'a ', 'io' and 'lo' and text character 'd' in contexts 're', 'ne' and '~o'. In further example, code characters 'll' represent 'ss' if the prior context is 'ie' but represent 'th' if the prior context is 'o_' It is apparent from the above that the same code character may represent different text characters and that a given text character may be represented by different code characters. That is, that there is not a unique, one to one relationship between code characters and text - - \
~LZ~%74 characters in themselves. The relationship between a code character and a text character depends upon the particular context in which the code and text character appear.
This illustrates property of context redundancy compression which will be further illustrated by example below, that a code character is not required to contain, in itself, the total amount of information required to uniquely define a particular text character. Part of the information defining the relationships between code and text characters resides in the text itself, that is, in the contexts. Because of this, fewer unique code characters, and thus shor~er code characters, are required to unambiguously represent compressed text. The resulting compressed texts may thereby be represented by fewer bits of information. Thus the text itself, in both the compressed and noncompressed forms contains, in addition to the textual information9 a part of the information required to transform between the compressed and noncompressed forms.
Because of the context dependent relationship between text and code characters, a prior context, for example, 'nuls' or 'blanks', is either assumed or imposed for the first n characters of text. This 'header context' provides a known context for compressing or decompressing the first character or characters of text, the following characters being compressed or decompressed in the context of the text itself.
To illustrate the use of the method and table of the present invention for compression and decompression of text, assume that the phrase 'total salt' appears in the text and that the prior context of the first 't' in 'total' is 'e_' 9 as in (th)e_. The compression and subsequent decompression of ~L;2Q~7~
this phrase is illustrated with Tables 2 and 3 presented below. Table 2 illustrates the compression process and Table 3 the decompression process.
CGMPRESSIGN OF 'total salt' Text Character Prior Context Code Character t e_ 2 o - t 2 t to 7 a ot 3 1 ta 5 _ al s 1 3 a _s 8 1 sa 2 t al 7 To compress the sample text, each text character and its associated context are read in turn and, in the preferred embodiment, in the sequence in which the text is normally written. In the present example, the reading will be ~rom left to right. Since the present example uses a 2nd order context, that is, n=2, the context ~or each character is comprised of the two characters immediately preceding the character. Having determined the t~o character prior context of a particular character, that context is then used as an index into the compression/decompression table to locate ~ 2~274.
the relative alphabet for that context. The character read in association with the context is then located in the relative alphabet and the corresponding code character located in the rank index at the top oF the table. This process is continued, a character at a time, until the compression operation is completed.
For example, the prior context of the first 't' in 'total' is 'e '. 'e_' is used as an index into the table to locate the corresponding relative alphabet, which appears as the second relative alphabet from the top. The relative alphabet is then scanned to locate text character 'tl, the second entry frqm the left, and the corresponding code character is read from the top of the table as '2'. The next text character is the 'o' of 'total' and the context of 'o' is '_t', so that the code character for 'o' is again '2'. The third text character is the second 't' of 'total', the prior context of this 't' is 'to' and the code character is determined to be '7'.
This process continues to the end of the text and the cnmpressed form of 'total salt' is found to be '2273513~27'.
The reverse operation is performed, using the same compression/
decompression table, to decompress compressed text. In this case, the 'text' to be read is the compressed text and the 'characterst read, one at a time and in sequence as written, are the code characters of the compressed text. The prior context of each code character is comprised of the preceding decoded characters o~ the restored text and are determined as the text is decompressed. The contexts so determined are again used as indices into the table to select the appropriate relative alphabets and the code characters are used as indices to determine the appropriate text -24~
~Z~27~
characters within the selected relative alphabets. The known 'header context' used in the compression operation is again used as the beginning context in decompressing the first character or characters of the compressed text.
To illustrate the decompression operation using the same example as above, assume that the text to be decompressed is comprised of the code character sequence '2273513827' and that the preceding decompressed text has been determined to again be 'e_'. The decompression operation is then illustrated in Table 3 below.
TABL~ 3 DECOMPRESSI~N CF '2273513827' . . .
Prior Context Code Character Text Character e 2 t t 2 o to 7 t ot 3 a ta S
al 3 s s 8 a sa 2 al 7 t Assuming that the preceding compressed text has been decompressed to the start of the sequence '2273513827', the prior context of the first '2' in the sequence is known from the previously decompressed portion of the text to be 'e '. 'e_' is then used as an index into the table to ~etermine the reiative alphabet of the first '2' of the sequence and, for this character, is the second relative alphabet from the top of the table. The first '2' of the sequence is then used as an index into the table to determine that the corresponding text character of that relative alphabet is 't'. The prior context of the second l2t in the sequence is then kno~n ~$~74 to be ' t' and, using ' t' and '2' as indices into the table, thetext charactercorresponding to the second '2' is determined to be '0'. The third code characte.r is '7' with a now known prior context of 'to' and the text character can be determined to be 't', and so on.
In a yet further implementation, the compression/decom-pression described above may be used recursively, that is, the compressed texts may themselves be compressed usin~ the same method as described above. Recursive compression requires the generation of compression/decompression tables in the same manner as described above, but using the compressed forms of the texts to determine and tabulate the context redundancy of the compressed te~tual matexial. The compression of text then becomes a two step process, a first step to compress the oriyinal text and a second step to compress the compressed text, as does the decompres-sion process. The two step, or recursive, compression/decompres-sion may, as in a single step process, be performed 'on-line', that is, as a continuous process.
To reiterate certain features of the above described method of text compression, the present method operates upon the basis of context redundancy, that is, the nonuniform distribution of the contexts in which individual characters occur. Contexts, as defined herein, comprise uniform ~2~$~7~
sized groups of characters preceding or following the individual characters comprising a text.
Although the compression/decompression tables used in the method are effected by the linguistic characteristics of particular languages, the method is not based upon any form of linguistic analysis nor upon any form of linguistic characteristic or element of any language. The method is thereby completely general and is equally applicable for any language or 'dialect' of a language.
1~
Because the method operates with uniform sized groups of text and code characters~ the method is less complex and requires less processing time and capability than do other methods and is thereby faster in execution.
For the same reasons, the method is symmetric in execution, requires only a single compression/decompression table, and is suitable for on-line~
continuous processes.
Part of the inform~tion defining the relationships between code and text characters resides in the contexts, that is, in the text itself, in both its compressed and noncompressed forms. Because of this, fewer unique code characters, and thus shorter code characters, are required to unambiguously represent compressed text. The resulting compressed texts may thereby be represented by fewer bits of information.
It will be apparent to those of ordinary skill in the art that the present invention may be embodied in yet other specific forms without departing from the spirit or essential characteristics thereof. Thus, the present embodiments are to be considered in all respects as illustrative and \
restrictive, the scope of the invention being indicated by the appendedclaims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.
1~
Claims (28)
1) In an information processing system, a method for compressing textual information, comprising the steps of:
reading in succession each character and an associated context of a text, selecting, for each said text character and associated context, a corresponding code character, and providing said code characters to comprise said text in compressed form.
reading in succession each character and an associated context of a text, selecting, for each said text character and associated context, a corresponding code character, and providing said code characters to comprise said text in compressed form.
2) In an information processing system, a method for decompressing context compressed textual information, comprising the steps of:
reading in succession each code character of a compressed text, reading, for each said code character, an associated context from a known decompressed portion of said text, selecting, for each said code character and associated context, a corresponding text character, and providing said text characters to comprise said text in decompressed form.
reading in succession each code character of a compressed text, reading, for each said code character, an associated context from a known decompressed portion of said text, selecting, for each said code character and associated context, a corresponding text character, and providing said text characters to comprise said text in decompressed form.
3) The method of claim 1, wherein each said context comprises:
a group of text characters associated with said text character.
a group of text characters associated with said text character.
4) The method of claim 3, wherein each said context containes a fixed number of said text characters.
5) The method of claim 1, further comprising the initial step of generating a table relating said text and code characters and said contexts, comprising the steps of:
reading each said text character and associated context of a representative sample of text, generating, for each said context occuring in said sample text, a relative alphabet comprising said text characters occurring in association with said each said context, and associating a corresponding code character with each said text character of each said relative alphabet.
reading each said text character and associated context of a representative sample of text, generating, for each said context occuring in said sample text, a relative alphabet comprising said text characters occurring in association with said each said context, and associating a corresponding code character with each said text character of each said relative alphabet.
6) The method of claim 5, wherein:
each of said code characters are selected from a set of code characters shared by said relative alphabets.
each of said code characters are selected from a set of code characters shared by said relative alphabets.
7) In an information processing system, a method for compressing textual information, comprising the steps of:
reading each text character and associated context of a representative sample of text, generating, for each said context occuring in said sample text, a relative alphabet comprising said text characters occurring in association with said each said context, and associating a corresponding code character with each said text character of each said relative alphabet, reading in succession each character and an associated context of a text, selecting, for each said text character and associated context, a corresponding code character, and providing said code characters to comprise said text in compressed form.
reading each text character and associated context of a representative sample of text, generating, for each said context occuring in said sample text, a relative alphabet comprising said text characters occurring in association with said each said context, and associating a corresponding code character with each said text character of each said relative alphabet, reading in succession each character and an associated context of a text, selecting, for each said text character and associated context, a corresponding code character, and providing said code characters to comprise said text in compressed form.
8) The method of claim 2, wherein each said context comprises:
a group of text characters associated with said text character.
a group of text characters associated with said text character.
9) The method of claim 8, wherein each said context containes a fixed number of said text characters.
10) The method of claim 2, further comprising the initial step of generating a table relating said text and code characters and said contexts, comprising the steps of:
reading each said text character and associated context of a representative sample of text, for each said context occuring in said sample text, generating a relative alphabet comprising said text characters occurring in association with said each said context, and associating a corresponding code character with each said text character of each said relative alphabet.
reading each said text character and associated context of a representative sample of text, for each said context occuring in said sample text, generating a relative alphabet comprising said text characters occurring in association with said each said context, and associating a corresponding code character with each said text character of each said relative alphabet.
11) The method of claim 10, wherein:
each of said code characters are selected from a set of code characters shared by said relative alphabets.
each of said code characters are selected from a set of code characters shared by said relative alphabets.
12) In an information processing system, a method for decompressing context compressed textual information, comprising the steps of:
reading each said text character and associated context of a representative sample of text, generating, for each said context occuring in said sample text, a relative alphabet comprising said text characters occurring in association with said each said context, and associating a corresponding code character with each said text character of each said relative alphabet, reading in succession each code character of a compressed text, reading, for each said code character, an associated context from a known decompressed portion of said text, selecting a corresponding text character for each said code character and associated context, and providing said text characters to comprise said text in decompressed form.
reading each said text character and associated context of a representative sample of text, generating, for each said context occuring in said sample text, a relative alphabet comprising said text characters occurring in association with said each said context, and associating a corresponding code character with each said text character of each said relative alphabet, reading in succession each code character of a compressed text, reading, for each said code character, an associated context from a known decompressed portion of said text, selecting a corresponding text character for each said code character and associated context, and providing said text characters to comprise said text in decompressed form.
13) In an information processing system, a method for storing and retreiving textual information, comprising the steps of:
reading in succession each character and an associated context of a text, selecting, for each said text character and associated context, a corresponding code character, and storing said code characters to comprise said text in compressed form, reading in succession each code character of said stored compressed text, reading, for each said code character, an associated context from a known decompressed portion of said text, selecting a corresponding text character for each said code character and associated context, and providing said text characters to comprise said text in decompressed form.
reading in succession each character and an associated context of a text, selecting, for each said text character and associated context, a corresponding code character, and storing said code characters to comprise said text in compressed form, reading in succession each code character of said stored compressed text, reading, for each said code character, an associated context from a known decompressed portion of said text, selecting a corresponding text character for each said code character and associated context, and providing said text characters to comprise said text in decompressed form.
14) In an information processing system, a method for communicating textual information, comprising the steps of:
reading in succession each character and an associated context of a text, selecting, for each said text character and associated context, a corresponding code character, said code characters comprising said text in compressed form, and communicating said code characters, reading in succession each said communicated code character, reading, for each said code character, an associated context from a known decompressed portion of said text, selecting a corresponding text character for each said code character and associated context, and providing said text characters to comprise said text in decompressed form.
reading in succession each character and an associated context of a text, selecting, for each said text character and associated context, a corresponding code character, said code characters comprising said text in compressed form, and communicating said code characters, reading in succession each said communicated code character, reading, for each said code character, an associated context from a known decompressed portion of said text, selecting a corresponding text character for each said code character and associated context, and providing said text characters to comprise said text in decompressed form.
15) In an information processing system, means for compressing textual information, comprising:
means for reading in succession each character and an associated context of a text, means responsive to said reading means for selecting, for each said text character and associated context, a corresponding code character and providing said code characters to comprise said text in compressed form.
means for reading in succession each character and an associated context of a text, means responsive to said reading means for selecting, for each said text character and associated context, a corresponding code character and providing said code characters to comprise said text in compressed form.
16) In an information processing system, means for decompressing context compressed textual information, comprising:
means for reading in succession each code character of a compressed text, means responsive to said code character reading means for reading, for each said code character, an associated context from a known decompressed portion of said text, means responsive to said code character and context reading means for selecting, for each said code character and associated context, a corresponding text character and providing said text characters to comprise said text in decompressed form.
means for reading in succession each code character of a compressed text, means responsive to said code character reading means for reading, for each said code character, an associated context from a known decompressed portion of said text, means responsive to said code character and context reading means for selecting, for each said code character and associated context, a corresponding text character and providing said text characters to comprise said text in decompressed form.
17) The compressing means of claim 15, wherein each said context comprises:
a group of text characters associated with said text character.
a group of text characters associated with said text character.
18) The compressing means of claim 17, wherein each said context containes a fixed number of said text characters.
19) The compressing means of claim 15, further comprising:
means for storing a table relating said text and code characters and said contexts and responsive to said selecting means for providing said corresponding code characters, said table including, for each context of the language of said text, an associated relative alphabet containing said text characters occurring in association with said contexts of said language, and wherein each of said text characters of each of said relative alphabets is associated with a said corresponding code character.
means for storing a table relating said text and code characters and said contexts and responsive to said selecting means for providing said corresponding code characters, said table including, for each context of the language of said text, an associated relative alphabet containing said text characters occurring in association with said contexts of said language, and wherein each of said text characters of each of said relative alphabets is associated with a said corresponding code character.
20) The compressing means of claim 19, wherein:
each of said code characters are selected from a set of code characters shared by said relative alphabets.
each of said code characters are selected from a set of code characters shared by said relative alphabets.
21) The decompressing means of claim 16, wherein each said context comprises:
a group of text characters associated with said text character.
a group of text characters associated with said text character.
22) The decompressing means of claim 21, wherein each said context containes a fixed number of said text characters.
23) The decompressing means of claim 16, further comprising:
means for storing a table relating said text and code characters and said contexts and responsive to said selecting means for providing said corresponding text characters, said table including, for each context of the language of said text, an associated relative alphabet containing said text characters occurring in association with said contexts of said language, and wherein each of said text characters of each of said relative alphabets is associated with a said corresponding code character.
means for storing a table relating said text and code characters and said contexts and responsive to said selecting means for providing said corresponding text characters, said table including, for each context of the language of said text, an associated relative alphabet containing said text characters occurring in association with said contexts of said language, and wherein each of said text characters of each of said relative alphabets is associated with a said corresponding code character.
24) The decompressing means of claim 23, wherein:
each of said code characters are selected from a set of code characters shared by said relative alphabets.
each of said code characters are selected from a set of code characters shared by said relative alphabets.
25) In an information processing system, means for storing and retreiving textual information, comprising:
compressing means comprising means for reading in succession each character and an associated context of a text, and means responsive to said reading means for selecting for each said text character and associated context a corresponding code character, means for storing said code characters to comprise said text in compressed form, and decompressing means comprising means for reading in succession from said storing means each code character of said compressed text, means responsive to said code character reading means for reading, for each said code character, an associated context from a known decompressed portion of said text, and means responsive to said code character and context reading means for selecting, for each said code character and associated context, a corresponding text character and providing said text characters to comprise said text in decompressed form.
compressing means comprising means for reading in succession each character and an associated context of a text, and means responsive to said reading means for selecting for each said text character and associated context a corresponding code character, means for storing said code characters to comprise said text in compressed form, and decompressing means comprising means for reading in succession from said storing means each code character of said compressed text, means responsive to said code character reading means for reading, for each said code character, an associated context from a known decompressed portion of said text, and means responsive to said code character and context reading means for selecting, for each said code character and associated context, a corresponding text character and providing said text characters to comprise said text in decompressed form.
26) The storing and retreiving means of claim 25, further comprising:
means for storing a table relating said text and code characters and said contexts and responsive to said code and text character selecting means for providing said corresponding code and text characters, said table including, for each context of the language of said text, an associated relative alphabet containing said text characters occurring in association with said contexts of said language, and wherein each of said text characters of each of said relative alphabets is associated with a said corresponding code character.
means for storing a table relating said text and code characters and said contexts and responsive to said code and text character selecting means for providing said corresponding code and text characters, said table including, for each context of the language of said text, an associated relative alphabet containing said text characters occurring in association with said contexts of said language, and wherein each of said text characters of each of said relative alphabets is associated with a said corresponding code character.
27) In an information processing system, means for communicating textual information, comprising:
means for compressing a text, comprising means for reading in succession each character and an associated context of a text, and means responsive to said reading means for selecting for each said text character and associated context a corresponding code character, said code characters comprising said text in compressed form, means for communicating said code characters, and means for decompressing said compressed text, comprising means for reading in succession each said communicated code character, means responsive to said code character reading means for reading, for each said code character, an associated context from a known decompressed portion of said text, and means responsive to said code character and context reading means for selecting, for each said code character and associated context, a corresponding text character and providing said text characters to comprise said text in decompressed form.
means for compressing a text, comprising means for reading in succession each character and an associated context of a text, and means responsive to said reading means for selecting for each said text character and associated context a corresponding code character, said code characters comprising said text in compressed form, means for communicating said code characters, and means for decompressing said compressed text, comprising means for reading in succession each said communicated code character, means responsive to said code character reading means for reading, for each said code character, an associated context from a known decompressed portion of said text, and means responsive to said code character and context reading means for selecting, for each said code character and associated context, a corresponding text character and providing said text characters to comprise said text in decompressed form.
28) The communicating means of claim 27, wherein:
said compressing means further comprises means for storing a table relating said text and code characters and said contexts and responsive to said code character selecting means for providing said corresponding code characters, said table including, for each context of the language of said text, an associated relative alphabet containing said text characters occurring in association with said contexts of said language, and wherein each of said text characters of each of said relative alphabets is associated with a said corresponding code character, and said decompressing means further comprises means for storing a table relating said text and code characters and said contexts and responsive to said text character selecting means for providing said corresponding text characters, said table including, for each context of the language of said text, an associated relative alphabet containing said text characters occurring in association with said contexts of said language, and wherein each of said text characters of each of said relative alphabets is associated with a said corresponding code character.
said compressing means further comprises means for storing a table relating said text and code characters and said contexts and responsive to said code character selecting means for providing said corresponding code characters, said table including, for each context of the language of said text, an associated relative alphabet containing said text characters occurring in association with said contexts of said language, and wherein each of said text characters of each of said relative alphabets is associated with a said corresponding code character, and said decompressing means further comprises means for storing a table relating said text and code characters and said contexts and responsive to said text character selecting means for providing said corresponding text characters, said table including, for each context of the language of said text, an associated relative alphabet containing said text characters occurring in association with said contexts of said language, and wherein each of said text characters of each of said relative alphabets is associated with a said corresponding code character.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US523,739 | 1983-08-16 | ||
| US06/523,739 US4672679A (en) | 1983-08-16 | 1983-08-16 | Context redundancy text compression |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA1209274A true CA1209274A (en) | 1986-08-05 |
Family
ID=24086269
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA000461046A Expired CA1209274A (en) | 1983-08-16 | 1984-08-15 | Context redundancy text compression |
Country Status (6)
| Country | Link |
|---|---|
| US (1) | US4672679A (en) |
| EP (1) | EP0134543B1 (en) |
| JP (4) | JP2581661B2 (en) |
| AU (1) | AU574702B2 (en) |
| CA (1) | CA1209274A (en) |
| DE (1) | DE3482065D1 (en) |
Families Citing this family (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPS59205874A (en) * | 1983-05-10 | 1984-11-21 | Canon Inc | Picture data compressor |
| CA1266524A (en) * | 1983-08-30 | 1990-03-06 | Shinobu Arimoto | Image processing system |
| JPS60254871A (en) * | 1984-05-31 | 1985-12-16 | Kokusai Denshin Denwa Co Ltd <Kdd> | Encoding method of facsimile image |
| GB8618093D0 (en) * | 1986-07-24 | 1986-09-03 | Serif Software Ltd | Data compression |
| US5590317A (en) * | 1992-05-27 | 1996-12-31 | Hitachi, Ltd. | Document information compression and retrieval system and document information registration and retrieval method |
| JPH0764214A (en) * | 1993-08-31 | 1995-03-10 | Fuji Photo Film Co Ltd | Photographic printing information printer and photographic printer |
| US6460036B1 (en) * | 1994-11-29 | 2002-10-01 | Pinpoint Incorporated | System and method for providing customized electronic newspapers and target advertisements |
| US5758257A (en) | 1994-11-29 | 1998-05-26 | Herz; Frederick | System and method for scheduling broadcast of and access to video programs and other data using customer profiles |
| JP3238854B2 (en) * | 1995-02-21 | 2001-12-17 | 富士通株式会社 | Data compression method and data compression device, and data decompression method and data decompression device |
| US5951623A (en) * | 1996-08-06 | 1999-09-14 | Reynar; Jeffrey C. | Lempel- Ziv data compression technique utilizing a dictionary pre-filled with frequent letter combinations, words and/or phrases |
| US5999949A (en) * | 1997-03-14 | 1999-12-07 | Crandall; Gary E. | Text file compression system utilizing word terminators |
| US7630986B1 (en) * | 1999-10-27 | 2009-12-08 | Pinpoint, Incorporated | Secure data interchange |
| US8705810B2 (en) * | 2007-12-28 | 2014-04-22 | Intel Corporation | Detecting and indexing characters of videos by NCuts and page ranking |
| US8484170B2 (en) * | 2011-09-19 | 2013-07-09 | International Business Machines Corporation | Scalable deduplication system with small blocks |
| JP6645013B2 (en) * | 2015-01-30 | 2020-02-12 | 富士通株式会社 | Encoding program, encoding method, encoding device, and decompression method |
| US10838922B2 (en) | 2017-03-31 | 2020-11-17 | International Business Machines Corporation | Data compression by using cognitive created dictionaries |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR1543777A (en) * | 1966-12-23 | 1900-01-01 | Ibm | Identifying characters by using context |
| US3717851A (en) * | 1971-03-03 | 1973-02-20 | Ibm | Processing of compacted data |
| US4058795A (en) * | 1972-10-03 | 1977-11-15 | International Business Machines Corporation | Method and apparatus for context-aided recognition |
| DE2748859C3 (en) * | 1977-10-31 | 1980-09-04 | Siemens Ag, 1000 Berlin Und 8000 Muenchen | Circuit arrangement for storing a text |
| AU6700181A (en) * | 1979-12-12 | 1981-07-06 | Key Tronic Corp. | Document reader |
| US4383307A (en) * | 1981-05-04 | 1983-05-10 | Software Concepts, Inc. | Spelling error detector apparatus and methods |
| DE3278850D1 (en) * | 1981-11-09 | 1988-09-01 | Ibm | Data translation apparatus translating between raw and compression encoded data forms |
| US4597057A (en) * | 1981-12-31 | 1986-06-24 | System Development Corporation | System for compressed storage of 8-bit ASCII bytes using coded strings of 4 bit nibbles |
-
1983
- 1983-08-16 US US06/523,739 patent/US4672679A/en not_active Expired - Lifetime
-
1984
- 1984-08-03 DE DE8484109259T patent/DE3482065D1/en not_active Expired - Fee Related
- 1984-08-03 EP EP84109259A patent/EP0134543B1/en not_active Expired - Lifetime
- 1984-08-09 AU AU31743/84A patent/AU574702B2/en not_active Ceased
- 1984-08-15 CA CA000461046A patent/CA1209274A/en not_active Expired
- 1984-08-16 JP JP59170985A patent/JP2581661B2/en not_active Expired - Lifetime
-
1992
- 1992-02-27 JP JP4041600A patent/JP2575259B2/en not_active Expired - Lifetime
- 1992-02-27 JP JP4041591A patent/JP2568343B2/en not_active Expired - Lifetime
- 1992-02-27 JP JP4041592A patent/JP2568344B2/en not_active Expired - Lifetime
Also Published As
| Publication number | Publication date |
|---|---|
| AU3174384A (en) | 1985-02-21 |
| JP2575259B2 (en) | 1997-01-22 |
| EP0134543A2 (en) | 1985-03-20 |
| JPS60101669A (en) | 1985-06-05 |
| JP2568343B2 (en) | 1997-01-08 |
| JP2568344B2 (en) | 1997-01-08 |
| JPH05197760A (en) | 1993-08-06 |
| JPH05197720A (en) | 1993-08-06 |
| DE3482065D1 (en) | 1990-05-31 |
| JP2581661B2 (en) | 1997-02-12 |
| EP0134543B1 (en) | 1990-04-25 |
| AU574702B2 (en) | 1988-07-14 |
| JPH05197721A (en) | 1993-08-06 |
| EP0134543A3 (en) | 1985-10-30 |
| US4672679A (en) | 1987-06-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA1209274A (en) | Context redundancy text compression | |
| EP0052725B1 (en) | Method of reducing the print element changes in a text processing system | |
| US4091270A (en) | Electronic calculator with optical input means | |
| US4650349A (en) | Speed typing apparatus and method | |
| US4425626A (en) | Apparatus for translation of character codes for application to a data processing system | |
| US5329598A (en) | Method and apparatus for analyzing character strings | |
| US4152697A (en) | Parallel run-length decoder | |
| US4523331A (en) | Automated image input, storage and output system | |
| CN111340207B (en) | Floating point number conversion method and device | |
| JP3327650B2 (en) | Recording device | |
| EP0313207A2 (en) | Method and apparatus for inputting characters | |
| CN112926314A (en) | Document repeatability identification method and device, electronic equipment and storage medium | |
| Freeman | Computers and classification systems | |
| WO1994027374A1 (en) | Method and apparatus for efficient compression of data having redundant characteristics | |
| Smith | Developments in SPECOL—a retrieval language for the non-programmer | |
| US20020118877A1 (en) | Methods and apparatus for associating character codes with optimized character codes | |
| Robinson et al. | A cathode-ray-tube plotting system for the Control Data 3600 Computer | |
| SU555395A1 (en) | Input device | |
| Forrester | The Design and Implementation of a Simple Incremental Assembler on the Hewlett Packard 2100A Computer | |
| JPH02287739A (en) | Memory access method | |
| Inouye et al. | Technological Tools | |
| Agenbroad | Personal Names and the Chinese Character Code for Information Interchange, Vol. 1:(CCCII/1)-Adequacy and Implications | |
| Bayne | Sequential Simplex Optimization, by FH Walters, LR Parker, Jr., SL Morgan and SN Deming: CRC Press, Inc., Boca Raton, FL, USA, 1991, 325 pages, price US $49.95, ISBN 0-8493-5894-9 | |
| JPS60252947A (en) | Code converter | |
| Tidball et al. | Educational Time-Sharing on a Minicomputer. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| MKEX | Expiry |