CA2227383A1

CA2227383A1 - Method and apparatus for automated search and retrieval processing

Info

Publication number: CA2227383A1
Application number: CA002227383A
Authority: CA
Inventors: Alwin B. Carus; Michael Wiesner; Ateeque R. Haque; Keith Boone
Original assignee: Alwin B. Carus; Michael Wiesner; Ateeque R. Haque; Keith Boone; Inso Corporation; Lernout & Hauspie Speech Products N.V.; Vantage Technology Holdings
Current assignee: Vantage Technology Holdings LLC
Priority date: 1995-07-19
Filing date: 1996-07-19
Publication date: 1997-02-06
Also published as: EP0971294A2; US5794177A; EP0839357A1; WO1997004405A1; US5890103A

Abstract

This invention provides a method and apparatus for automated search and retrieval processing that includes a tokenizer, a noun phrase analyzer, and a morphological analyzer. The tokenizer includes a parser that extracts characters from the stream of text, and identifying element for identifying a token formed of characters in the stream of text that include lexical matter, and a filter for assigning tags to those tokens requiring further linguistic analysis. The tokenizer, in a single pass through the stream of text, determines the further linguistic processing suitable to each particular token contained in the stream of text. The noun phrase analyzer annotates tokens with tags identifying characteristics of the tokens and contextually analyzes each token. During processing, the noun phrase analyzer can also disambiguate individual token characteristics and identify agreement between tokens.
Themorphological analyzer organizes, utilizes, analyzes, and generates morphological data related to the tokens. In particular, the morphological analyzer locates a stored lexical expression representative of a candidate token found in a stream of natural language text, identifies a paradigm for the candidate token based upon the stored lexical expression, and applies transforms contained within the identified paradigm to the candidate token.

Description

CA 02227383 l998-Ol-l9 W O 97~0440~ PCTrUS96/12018 METHOD ~ND APPARATUS FOR AUTOMATED
SEARCH AND RETR~EVAL PROCESSING

Back~round of the Invention The present invention relates to automated language analysis systems, and relates to such systems embodied in the computer for receiving digitally encoded text composed in a natural language. In particular, it relates to systems for tok~ni7:in~ aLnd analyzing lexical matter ~ound in a stream of natural language text. In another aspect, the invention pertains to a noun-phrase system for identifying noun phrases contained in natural l~n~n~e text, wh;le other aspects of the invention concern systems for incorporating morphological analysis and generation of natural language text.
Alltom~tecl language analysis systems embedded in a computer typically include a lexicon module and a processing module. The lexicon module is a 'Idictionary'' or database cont~ining words and se]nan~ic knowledge related to each word. The processing module includes a pluralit,v of analysis modules which operate upon the input text and the lexicon module in order to process the text and generate a computer understandable semantic representation of the natural language text. Automated natural language analysis systems design~(l in this manner provide for an efficient language analyzer capable of achieving great benefits in performing tasks such as inforrnation retrieval.
Typically the processing of natural language text begins with the processing module fetc3~ing a continuous stream of electronic text from the input buffer. The processing module then decomposes the stream of natural language text into individual words, sentences, and messages. For instance, individual words can be identified by joining together a string of adjacent character codes between two consecutive occurrences of a white space code (i.e. a space, tab, or carriage return). These individual words identified by the processor are actually just "tokens" that may be found as entries in the lexicon module. This first stage of processing by the processing module is referred to as tokeni7~tion and the processor module at this stage is referred to as a tokenizer.
Following the tokeni7~tion phase, the entire incoming stream of natural language text may be su~jected to further higher level linguistic procç~ing. For in~t~n- e, the entire incoming stream of text might be parsed into sentences having the subiect, the main verb, the direct and indirect objects (if any) prepositional phrases, relative clauses, adverbials, etc., identified for each sentence iIl the stream of incoming natural language text.
lEokeni~ers current]y used in the art encounter problems regarding selective storagç and process;ng of i~formation found in the stream of text. In particular, prior art tokeni7Pr~ store and ~r~cess all white space delimited characters (i.e. "tokens") found in the strea~n of text. Bllt it is not desirable, from an information processing standpoint, to process and store numbers, hyphens, and other forms of punctuation that are characterized as "tokens"

W O 97/0440~ PCT/US96/12018

-2--by the prior art tokenizers. Rather, it is preferable to design a tokenizer that identifies as tokens only those character strings forming words that are relevant to inforrnation procescing Prior art tokenizers have the additional drawback that each token extracted from the stream of text must be processed by each higher level linguistic processor in the 5 automated language analysis system. For instance, each token must be processed by a noun phrase analysis module to tletPrrnine whether the token is part of a noun phrase. This system results in an extensive amount of unnecessary higher level linguistic processing on in~lop~iate tokens.
Other prior art systems have been developed for the automatic recognition of 10 syntactic information contained within a natural stream of text, as well as systems providing grammatical analysis of digitally encoded natural }anguage text. Additional prior systems contain sentence ana~ysis techniques for forming noun phrases from words present in the encoded text. These prior noun phrase identifying techniques assign ranktng~ to words within a stream of text based upon the probability of any individual word type being found 15 within a noun phraset and these techniques form noun phrases by analyzing the ranks of individual words wi~in the strearn of text.
One drawback of prior systems concerns the inflexibility of these systems and their inability to be effective with multiple languages. In particular, prior techniques use a combination of hard-coded rules and tables that can not be easily changed for use with 20 different ~anguages.
Another drawback to prior systems concerns the inaccuracy in for~ning noun phrases. The inaccuracies in prior systems result from the failure to disambiguate ambiguous words that have mu~tiple part-o~-speech tags. The prior systems also fail to consider the agreement rules relating to words found within a noun phrase. Moreover, earlier automated 25 textual analysis systems failed to adequately address the contP~ l setting of each word within a noun phrase.
A~itional studies in the field of information processing have involved work in the field of lexical morphological analysis. Lexical morphology involves the study and description of wor~ forrnation in a language, and in particular e~oph~ s the ~min~tion of 30 inflections, derivations, and compound and cliticized words. Inflectional morphology refers to the study of the alternations of the form of words by adding affixes or by çhsm~ing the form of a base in order to indicate grammatical features such as number, gender, case, person, mood, or voice ~e.g., the inflected forms of book: book, book's, books, and books r).
Derivational morphology refers to the study of the processes by which words are formed 35 from ~ tin~ words or bases by adding or removing affixes (e.g., singer from sing) or by ch~nging the shape of the word or base (e.g., song from sing). Compounding refers to the process by which words are formed from two or more elements which are themselves words or speci~l combining forms of words (e.g., the German ~ersicherungsgesellschaft [insurance company] con~i~ting of Vers~cherung + s + Gesellschaft). Cliticizing refers to the process by ~ 3, , ~

which special words or particles which have no independent accent are combined with stressed content words (e.g., the French l 'cole consists of the preposed enclitic le [the]
and the word cole [schoor~).
Many text processing systems utilize crude affix stripping methods called 5 "stemmers" to morphologically analyze natural language text. Other more sophisticated, linguistically based morphological systems reduce all word forms to the same constant length character string, which is itself not necessarily a word. This "stem" portion of the word remains invariant during the morphological analysis. For example, a sophisticated morphological system might strip oi~f the varying suffix letters to map every word to the 10 longest common prefix character string. Thus, all the forms of arrive (i.e., arrive, arrives, arrived, and arriving) are stripped back to the longest common character string, arriv (without an e). Note that this procedure does not map forms of arrive back to - arrive because the e character fails to appear in arriving. These same algorithms convert all inflected forms of swim to sw because this is the longest common substring. Both 15 st~mming and more refined moIphological analysis systems, however, have proven difficult to implement because of the special mech~ni~m~ required to deal with irregular morphological p~tt~rn~
Often an exception dictionary is provided to deal with irregularities in inflection and derivation, but as a result of the number of entries in this exception 20 dictionary it can become large and cumbersome. One ~lt~rn~tive to using a large exception dictionary involves farming a system having a smaller, yet incomplete,exception dictionary. Although this alternative is not as cumbersome, the incomplete data structure rapidly forms inaccurate representations of the natural language text under consideration. These two ~ltt~rn~tive lexicons exemplify the problems involved in prior 25 art systems, i.e., the difficulty in using the lexicons and the inaccuracies within the lexicon. Accordingly, many in the field have concluded that current st~mmin~
procedures cannot significantly improve coverage of the st~mming algorithm without re~llcin~ their accuracy.
Another drawback of these prior systems is their inability to generate all 30 the variant forms from a given stem. Traditional stemmin~ algorithms can be used for finding stems, but not for generating inflections or derivations. ~urthermore, these techni~ues are not linguistically general and require different algorithms and particular exception dictionaries for each natural language.

~M.Nn~ S~l~~S
15M\002PC~SU135HEI~T DOC

-3a-Fagen et al., European Patent Application EP-A-394633, is understood to disclose a tokenizer that identifies tGkens as a function of the classification of characters in the input. The characters in the input are assigned to one of three classifications by comparing the characters in the input string to a character table. In Fagen, the three 5 broad classification of characters include: 1) characters which are always delimiters; 2) characters which are never delimiters (e.g. (comma, braces, ampersand); and 3) characters which may conditionally be delimiters depending on the context in which they are used. Following the isolation of the tokens based on the character classification system the tokens are further processed by m~t~hing for specific prefixes and suffixes.
Clearly, there is a need in the art for an information processing system that overcomes the problems noted above. In particular, there exists a need in the art for a tokenizer capable of more advanced processing that reduces the overall amount of data being processed by higher level linguistic processors and increases the overall system throughput. Other needs include an information processing system that analyzes natural 15 language text in a manner that improves the precision and recall of information retrieval systems.
Accordingly, an object of the invention is to provide an improved tokenizer that identifies a selected group of tokens appropriate for higher level linguistic processing.

9 S~

ISM\002PC~SU~SHEET WC ~ -CA 02227383 l998-Ol-l9 W O 97/04405 PCT~US96/12018 --4-Another object of the invention is to provide a contextual analysis system which identifies noun phrases by looking at a window of words surrounding each extracted word. Other objects of the invention include providing a morphological analysis and generation system that irnproves efficiency, increases recall, and increases the precision of index pre-proc~?c~ing, searcih pre-proces~inp;, and search expansion techniques.
Other general and specific objects of the invention will be a~p~icl.l and evident from the accompanying drawings and the following description.

Summar~r of the Invention 1~ The invention provides a system which enables people to enhance tlhe quality of their writing and to use information more effectively. The tokenizer, the noun-phrase analyzer, and the morphological aLnalyzer and generator are powerful software tools that hardware and software m~nllf~ctllrers can integrate into applications to help end-users find and rekieve information quickly and easily in multiple languages. The invention achieves its objectives by providing a linguistically intelligent approach to index pre-proce~ing, search pre-proGes~in~, and search expansion that increases both recall (i.e., the ratio of relevant items retrieved to the total nurnber of relevant items) and precision (i.e., the ratio of relevarlt items retrieved to the total number of retrieved items) in automated search and retrieval processes.
For example, the inventive tokenizer disclosed herein increases the throughput of the overall natural language processing system by filtering the tokens prior to higher-level lingulstic aI;Lalysis. The tokenizer manages to achieve this increased throughput across multiple languages and during a single pass through the incoming stream of text.Furthermore, the invention provides a noun phrase analyzer that identifies the form and function of words and phrases in the stream of natural language text and converts them to a~ r~pl;ate forms for in~lf xing In particular, the invention can distinguish between noun phrases such as ~'Emergency Broadcast System" and the individual words "emergency", "broadcast", and "system", thereby ensuring that the index entries more accurately reflect the conten,t.
Moreover, the invention provides a morphological analyzer that identifies the form and fonnati~n of words in the source of text, including inflectional and derivational analysis and gener~tion. This allows a ~l~t~h~e query to be easily expanded to include morphologically related terrns. Additionally, the invention can provide inflectional and derivational analysis and generation to other text-based applications such as dictionaries, thesauruses, and lexicons for spelling eorrectors and machine-translation systems.
The t~keni7~r disclosecL herein operates under a new paradigm for iclentifying inforrnation i~n a stream of natural language text. In particular, the tokenizer views the incoming stream of natural language texts as con~i~tin~ of alt~rn~ting lexical and non-lexical matter. Lexical matter is ~}roadly defined as information that can be found in a lexicon or W O 97tO4405 PCT~US96/12018 dictionary, and that is relevant for information retrieval processes. The tokenizer of the invention does not view the incoming stream of text as merely cont,7ining words separated by white space.
This new tokenization paradigm allows the invention to associate the attributes of lexical matter found in a token and the attributes of non-lexical matter following the token w~th the token. The combined attributes of the lexical and non-lexical matter associated with any particular token are referred to as the parameters of the particular token.
These parameters of the tokens forming the stream of natural language text are processed by the language analysis system, thereby providing for increased ef~lciency, throughput, amd ~0 accuracy in the langllage analysis system.
The o7Ojects of the invention are filrther achieved based upon the discovery that linguistic infiorms~tion is encoded at many dirr~ t levels in the natural language streann of text. The invention accordingly provides for a tokenizer that filters the tokens during the tokenizS7tion process and uses the filtered information to guide and constrain further linguistic amalysis. For instance~ the tokenizer filters the tokens to select those tokens that require, or are candidates for, higher ~evel linguistic processing. Thus, the tokenizer advantageously selects a group of tokens for a particular higher level linguistic procçeein{~, rather than subjecting all tokens to the particular higher level linguistic proce~sin~, as cornmonly found in the prior art.
The tokenizer, in accordance with the invention, comprises a parsing element for extracting lexical and non-lexical characters from the stream of ~.7igiti7P 1 text, am identifying element for iidentifying a set of tokens, and a filter element for selecting a candidate t~ken from the set of tokens. The tokenizer operates such that the filter selects out those candidate tokens suitable for additional linguistic processing from the stream of natural 2~ language text.
The filter elemPnt found in the tokenizer can include a character analyzer for selecting a ~s7n~7~te token from various tokens found in the stream of text. The ch,7rs7~ter analyzer operates by co~ a, . .Ig a ,selected character in the stream of text witll entries in a character table~ and by associating a first tag with a first token located proximal to the selected character, when the selected character has an equivalent entry in the character table.
~Jnder an SlltPrn~t;ve approach, the filter element found in the tokenizer includes a contextual processor for selecting a candidate token from various tokens found in the stream of text.
~ The cont~xt7~l processor selects the candidate token as a function of the lexical and non-lexical cl~aracters surroutnding a character in the stream of text.
Both the character analyzer and the contextlls 1 analyzer operate effectively under many lsanguages. For instance, the character analyzer and the contextual analyzer operate in F:n~ h, French, Catalan, Spanish, Italian, Portuguese, German, Danish, Norwegism, Swedish~ ~utch, Finis;h, Russian, and Czech. The rules governing the character W O 97~044Q~ PCT~US96/12018 analyzer and the contextual analyzer are extremely accurate across many languages and accordingly are language independent.
A particularly advantageous feature of the tokenizer is its ability to achieve filtering operations during a single scan of the stream of text. The tokenizer achieves this performance based in part upon the lexical paradigm adopted in this invention, and in part due to the language sensitivity in the design of the tokenizer. In particular, the rules and structure goverr~ing the tokenizer provide sufficient information to determine the appropriate additional linguistic processing without requiring additional scans through the stream of text.
Other aspects of the tokenizer provide for an associative processing element that associates a tag with the selected candidate token. The associated tag is used to identify additional linguistic processes applicable to the selected candidate token. The applicable processes can be stored in a memory element that is located using the tag. Additionally, the tokenizer can include an additional processing element that associates a plurality of tags with a plurality of selected candidate tokens, each of the plurality of tags identifying additional 1~ linguistic processing suitable for the respective c~n~litl~tç tokens. Typically, the plurality of tags is formed as a function of a selected c~nc1i~1~te token. For example, based upon a particular character in the strearn of text, a token including the particular character and the surrounding tokens could be i~lçntifie~l as potential candidate for additional noun phrase analysls.
An additional feature of the tokenizer is the inclusion of a processor that modifies the candidate token. The candidate token may be modified based upon the tag or based UpOIl the additional linguistic processing associated with the candidate token through t~e tag. This modifying processor can include processing modules that either: split tokens, strip tokçns of particular characters, ignore characters in the token or surround non-lexical 2~ matter, or merge tokens in the strearn of text.
According to a further aspect of the invention, the tokenizer stores and retrieves data from a memory element that is either associated with the tokenizer within the confines of the larger linguistic analysis system or that is a functional sub-element within the tok~ni7~r itself. The data stored and retrieved by the tokenizer can include digital signals representative of the stream of natural language text and digital signals repres~?nt~tive of the parameters of each token.
Token parameters stored in the memory element by the tokenizer can include:
flags identEfying the number of lexical characters and non-lexical characters forrning a token;
flags identil~ying the lQcation of an output signal generated by said tokenizer; flags identifying

3~ the number of lexical characters forrning a token; and flags identifying the lexical and non-lexical attri~utes of a token. The lexical attributes can include intçrn~l character attributes of the token, speci~ processing for the token, end of sentence attributes of the token, and noun p~rase attributes of the token. The non-lexical attributes can include white space attributes, single new line attributes, and multiple new line attributes. These token pararneters and W O 97/044U5 PCTnJS96/12018 attributes advantageously aid in identifying additional linguistic processing suitable to a selected candidate token.
The invention further comprises a method for tokenizing natural language text in order to achieve higher throug]hput, efficiency, and accuracy. In particular, the tokeni~in~;
5 method includes the steps of extracting lexical and non-lexical characters from the stream of text, identifying a set of tokens, and selecting a candidate token from the set of tokens. The method operates such that the candidate token selected is suitable for additional linguistic proces~in~
The candidate token can be selected in accordance with the invention by l 0 c~ aLillg a selected character in the parsed stream of text with entries in a character table.
When a selected character in the 1ext m~t(~hPs an entry in the character table, a first tag identifying additional linguistic processing is associated with the token located proximal to the selected character in the text. Alternatively, the cAn(litlAte token can be selected based upon a contextual analysis. For instance, the lexical and non-lexical characters surrounding a l5 selected character in the stream of text to ~ t~rmine whether a token located proxirnal to the selected character is suitable for additional linguistic proce~in~.
Fur~her in accordance with the invention, the tokeni7ing method can further include associating a tag with those selected candidate tokens suited for additional linguistic proc~s~in~ The tag typically identifies the additional linguistic processing suited for the 20 selected candidate token. In addition, a plurality of tags can be associated with a plurality of tokens as a function of a candidate token being selected for additional linguistic processing Under another aspect of the invention, the tok~ni7ing method comprises the step of modifying a selected candidate token. The selected candidate token is modified based upon the additional linguistic processing det~nnined as suitable for the candidate token and 25 icllPntified by the tag associated with the selected candidate token. Additional features of the invention include a modifying step that either splits the candidate token into multiple tokens, strips a ch~rA~ttr from the candidate token, ignores a non-lexical character surrounding the c~nrli-1At~ token, or merges the cantli-l~t~ token with another token.
Another embodiment of the invention provides for a noun phrase analyzer that 30 extracts a sequence of token words from the natural language text, stores the sequence of token words in a memory element, ~leterrninec a part-of-speech tag and gr:~nnm~ features for each token word, and identifies tokens which can participate in the construction of noun phrases ~y contç~t~ ly analyzing each of the tokens The contextl-A1 analysis can include in~pectin~ the part-of-speech tags and the grammatical features of each token in a window of ex~acted tokens In accordance with the noun phrase analyzer embodiment of the invention, the system forms a noun phrase from a stream of natural language words by extracting a sequence of toker}s from the stream, storing the sequence of tokens in a memory element, ~let~rrnining a par~-of-speech tag amd grammatical features for each token, identifying tokens W O 97/0440~ PCTAUS96/12018 whiGh can participate in the construction of noun phrase by inspecting the part-of-speech tags of successive tokens, and iteratively checking agreement between elements of the noun phrase found within the stream of text. Further in accordance with the invention, the system iclentifi~s a word contained within the noun phrase as the end of the noun phrase when the word in question does not agree with earlier words contained within the noun phrase.
Further features of this invention check agreement between parts of the noun phrase by monitoring person, number, gender, and case agreement between the parts of the noun phrase, monitoring agreement in these categories between the parts of the noun phrase.
Further aspects of the invention provide for a system that extracts a sequence of tokens from the stream of natural language text, stores the sequence of tokens, deterrnines at least one part-of-speech tag for each token, disambiguates the part-of-speech tags of a token having multiple part-of-speech tags by inspecting a window of sequential tokens surroun~ing the ambiguous word, and identifies the parts of a noun phrase by inspecting the part-of-speech tags of sllccec~ive extracted tokens.
S Another aspect of this invention provides for a system capable of promoting at least one of the secondary part-of-speech tags of an ambiguous token to the primary part-of-speech tag as a function of a window of sequential tokens surrounding the ambiguous token.
The in~entio~} also provides a rule-based approach for replacing the primary part-of-speech tag with a generated primary part-of-speech tag, wherein the generated tag is formed as a ~nction of the window of sequential tokens cont~inin~ the ambiguous token.
Additional aspects of the invention provide methods and apparatus for ~l~L~I ", illing the part-of-speech tags associated with each token. In one embodiment of this aspect ofthe invention, the system provides for a first addressable table col,l~i,,;,,~ a list of lexical expressions with each lexical expression being associated with at least one part-of-speech tag. The extracted words can be located within the first addressable table and thereby become associated with at least one part-of-speech tag. In an alternate embodiment, the invention provides for a second addressable table con1~inin~ a list of stored suffixes with each stored suffLx being associated with at least one part-of-speech tag. The last three char~cters of an extracted word can be referenced against one of the suffixes contained in the 3Q second addressa~le tablie and thereby become associated with at least one part-of-speech tag.
The invention further provides for a step of associating a default part-of-speech tag of "noun"
with an extracted token.
A third embodiment of the invention provides for a uni~ue system of or~ni~i~n~, u~tlizing, and analyzing morphological data associated with a candidate word obtained from a stream of natural language text. The invention includes a processor for analyzing the strealm of text and for manipulating digital signals representative of moIphological pattern, and a memory element for storing digital signals. The digital signals representing morphological transforms are stored within a memory element and are oLgiq~li7f--l CA 02227383 1998-01-l9 W O 97~W40~ PCTAUS96/12018 as a list of paradigms, wherein each paradigm contains a grouping of one or more of morpholiogical trans~orms.
Each morphological transform in the paradigm can include a first character string that is stripped from the candidate word and a second string that is added to the 5 character word to morphologically transform the candidate word. Each morphological transforrn in the paradigm can further include baseform part-of-speech tags and the part-of-speech tag of the morphologically transforrned c~n~ te word. These part-of-speech tags aid in identifving al~pr~liate morphological transforms contained within a particular p~r~fligm for application to the candidate word. The morphological analysis system of the invention 10 filrther provides for a processor capable of stripping character strings and adding character strings to candidate words to fonn baseforms of variable length.
The morphological embodiment of the invention provides an addressable memory element having a first addressable table for storing a list of lexical expressions and having a second addressable table for storing a list of parA~ligm~, each paradigm having one 15 ~r more morpho~ogical transforms associated with particular morphological patterns. The lexical expressions stored in the first addressable table of the first memory element can be associated with one or more paradigms listed in the second addressable table.
Further aspects of the invention provide for a data processor having various processing modules. For example, the data processor can include a processing element for 20 mAtrhin~ a morphological transform in an identified paradigm with the candidate word, a proc~sin~ element for stripping a character string from the candidate word to form an interme~ te baseform, and a processJing element for adding a character string to the interm~liAte baseform in accordance with an identified morphological transform.
In accordance witll further aspects of the invention, the morphological system 25 provides for identifying a paradigm stored in the memory element equivalent to a candidate word found in a stream of natural language text, m~t-~hing a morphological pattern in the identified paradigm with the candidate word, and morphologically kansforrning the candidate word by stripping a first character string from the candidate word and adding a second character string to the cAn~ te ~,vord. The morphological system can also identify a 30 pAr7~ m representative of a candlidate word found in natural language text by locating a first lexical ~ ion in the first addressable table equivalent to the c~n~ lAte word and by identifying a paradigm as a function of the located first lexical expression. The association ~et~reelln the first and the second addressable tables allows the identified paradigm to be representative of the ~andidate word.
~ 3~ Further ~a~ules of the invention include identifying a part-of-speech tag of the candidate word and m~tçhing a morphological pattern in the identified paradigm with the c~nfli~te wo~d when the morphoilogical pattern has a part-of-speech tag equivalent to the part-of-speech tag associated with the candidate word. Additional embo-limentc of the invention include forming an intermediate baseform by stripping a first ch~r~çtçr s~ng from CA 02227383 1998-01-l9 W O 97/044Q5 PCT~US96/12018 the candidate word such that the intermediate baseform varies in length as a result of the particular morphological pattern contained within an identified paradigm.
The morphological system can additionally provide for the use of portm~n paradigms in the second addressable table. The portm~nt~ par~-ligm~, in comparison to 5 other paratligm~, do not necessarily contain inflectional kansforms. Rather, the portln~nte~ll paradigms can contain the locations of a plurality of paradigms. The portm~nte~ paradigm acts as a br~nching point to other paradigms that contain morphological patterns and morphological transforms. The system thus provides skuctures and method steps for identifying a plurality of paradigms associated with a lexical expression.
In addition, the portm:~nt~-l paradigms can include the location of noun par~ligm~, verb par:~ligmc7 and adjective/adverb par~ m~ Accordingly, matching an iate morphological paradigm with a candidate word can entail additional steps, which in turn increase the accuracy of morphological transforms. For instance, the matching step can re~uire that the baseform part-of-speech tag associated with a particular morphological pattern match the part-of-speech of the portm~nte~l~ paradigm currently under consideration.
Further aspects of the invention include systems for morphologically transfo~ning a candidate word by altering character strings located at any position within the candiidate word. For example, the invention kansforrns digital signals representative of a c~n~ tP word by either altering affixes attached to the front, middle, or end of the word (e.g.7 prefixcs, infixes, or suffixes). The invention can accomrnodate the various locations of affixes by using its unique skip and add algorithrn.

Brief DescTiption of the Drawin~s FIGURE 1 is a block diagram of a programmable multilingual text processor according to the present invention;
FIGURE 2 illustrates a group of data structures formed by the processor of FI~. I according to one practice of the invention;
FIGURE 3 shows a word data table utilized by the processor of FIG. 1;
FIGUl~E 4A illustrates a part-of-speech combination table referenced by the 3n worddatatabIeofFIG.3;
FIGURE 4B illustrates a suffix table for referencing entries in the part-of-speech combination table of FIG. 4A;
FIG~JRE 4C illustrates a morphological pattern file referenced by the word data table of lFIG. 3;
FIGURE S illustrates possible associations between the tables of FIG. 3, FIG. 4A, a~d FIG. 4B;
FIGURE 6 is a detailed block diagram of a noun-phrase analyzer contained within the text processor of FIG. l;
FIGURES 7A-7C show flow charts for the tokenizer illllctr~tecl in FIG. l;

. CA 02227383 1998-01-19 W O g7/04405 PCT~US96/12018 FIGURE 8 is a flow chart for the processor shown in FIG. 6;
FIGURE 9 is a representative table of rules for the disambiguator shown in FIG. 6;
FIGU~E 10 illustrates pseudocode for the agreement checker of FIG. 6;
FTGURE 11 contains pseudocode for the noun-phrase truncator of FIG. 6;
FIGURE 12 illustrates an example of noun-phrase analysis in accordance with the invention;
FIGURE 13 contains pseudocode for the morphological analyzer of FIG. 1;
FIGURE 14 is a flow chart for the uninflection (inflection reduction) module of FIG. l;
FIGURE 15 is a flow chart for the inflection expansion module of ~FIG. 1;
FIGURE 16 is a flow chart for the underivation (derivation reduction) module of FIG. l;
FIGURE 17 is a fl ow chart for the derivation expansion module of FIG. l; and FIGURE~ 18 is a detailed block diagram of the tokenizer shown in FIG. 1.

I~eta~led Description of the Drawin~s FIGURE 1 illuskates a mllltilingual text processor 10 in accordance with the invention. The text processor 10 includes a digital computer 12, an external memory 14, a source of text 16, a keyboard 18, a display 20, an application program interface 1 1, a tokenizer ~, a morphological ana]yzer/generator 2, and a noun-phrase analyzer 13. Digital computer 12 includes a memory element 22, an input/output controller 26, and a programmable processor 3û.
Many of the elements of the multilingual text processor 10 can be selected 2~ from ar~y of numerous commercially available devices. For example, digital COlllpUl~;l 12 can be a UNIQ 486/33 MHz personal colll~ul~l, ext~rn~l memory 14 can be a high speed non-volatile storage device, such as a SCSI hard drive, integral memory 22 can be 16MB of RAM, keyboard 1!8 can be a standard co~ uLel keyboard, and display 20 can be a video monitor. In operation, keyboard 18 and display 20 provide structural elements for interfacing with a user of the multilingual text processor 10. In particular, keyboard 18 inputs user typed c~mm~nfl~ and display 20 outputs for viewing signal generated by the text processor 10.
The Fxtern~l memory 14 is coupled with the digital computer 12, preferably ~ through the Input/Output Controller 26. Data stored in the F~ctern~l Memory 14 caII be do~nln2fle~ to memory element 22, and data stored in the memory 22 can be - 3~ correspondingly uploaded to the external memory 14. The external memory 14 can contain ~rarious ta~les utilized by the digital computer 12 to analyze a noun phrase or to perform morphological analys~s.
The source of text 16 can be another application program, a keyboard, a co~un~cations liink~ or a data storage device. In either case, the source of text generates and W O 97/04405 PCT~US96/12018 outputs to the digital computer 12 a stream of natural language text. Alternatively, the digital computer 12 may receive as an input from the source of text 16 sentences of encoded text with sentence boundary markers inserted. Sentence splitting per se is known in the art, and is disclosed in Kucera et al., U.S. Pat. No. 4,773,009, entitled Method and Apparatus for Text S Analysis. Preferably, the stream of natural language text with identified sentence boundaries enters the digital computer 12 at the Input/Output controller 26.
The Input/Output controller 26 organizes and controls the flow of data between the digital computer 12 and external accessories, such as external memory 14, keyboard 18, display 20, and the source of text 16. Input/Output controllers are known in the 10 art, and frequently are an integral part of standard digital computers sold in the market today.
Application Program ~nterface 11 includes a set of closely related functions, data types, and operations used in interfacing the co~ ul~;;ll2 with the noun-phrase analyzer 13 and with the morphological analyzer/generator 2. In particular, the application program interface 11 compri~es four functional elements: App Block, Database Block, Word Block, 15 and Buffer Block. The App Block initiates an application instance, assigns an identification number to it, and passes user processing options to the Noun-phrase Analyzer 13, the morphological analyzer/generator 2, and the tokenizer 1. The Database Block initi~li7.~s a ~l~t~k:l~e that provides linguistic information about a language. Word Block performs operations on individual words obtained from source text 16, and Buffer Block performs 20 operations on an entire buffer of text obtained from source text 16. Each of the functional elements, i.e., App, D~t~b~e, Word, and Buffer, contained in interface 11 have associated data structures used to pass information to the noun-phrase analyzer 13, the morphological analyzerJgenerator ~, and the tokenizer 1, before proces~in~; The functional elements, i.e., App, D~h~e7 Word, and Buffer, contained in interf~e 11 also include data structures to ~5 return information from the Application Program Interface 11 after processing by the tokenizer 1, the morphological analyzer 2, and the noun phrase analyzer 13.
l[ he four main functional elements contained in interface 1 1 perforrn operations on data structures formed by the application program interface 11. Memory for these functional elements and their associated databases is supplied by the digital computer 30 12 through ~e utilization of memory in internal memory element 22 and in external memory element 14.
In operation, App Block is the first functional block called. App Block initiates a sessi~n in the noun-phrase analyzer 13, the morphological analyzer/generator 2, or the tokenizer 1, ~n~ assigns a number to the session that uniquely identifies the session. The 35 identifying number is used to track the allocated memory and execution status and to ~ltnm~tically free the memory once the session ends. App Block can start a session to process a single word or an entire buffer of text. In particular, App Block preferably processes one ~Jord at a time when the morphological analyzer/generator 2 is called and App - W O 97/04405 PCT~US96/12018 Block preferably processes an entire buffer of text when noun-phrase analyzer 13 or the tokenizer 1 is cal~ed Next, ~)atabase block is accessed in order to initialize a language data~ase.
The language databases provide linguistic information for processing text in a particular language and are used by the noun-phrase analyzer 13 and the morphological analyzer~generator 2. Multiple languages can be processed during any particular session if multiple calls to the dRt~ba~e block are made during the session.
After initi~li7ing a session by calling App Block and initi~li7ing a database bycalling Database block, either Word Block or Buffer Block is called, depending on whether a larger amount o~text is being processed or one word at a time is being handled. The digital computer 12 fills an input buffer in the app}ication program interface 11 with data from the source text 16, and then calls either Word Block or Buffer Block to begin processing of the text by analyzer 13, morphological analyzer/generator 2, or tokenizer 1. Following the call, noun-phrase analyzer, the morphological analyzer, or the tokenizer scans the input buffer, and creates a str~am of tokens in the output buffer and an array that correlates the input and output buffers.
FIGU~E 1 further illustrates a morphological analyzer/generator 2 that includes an inflection module 4, aLn llninflection (inflection reduction) module 5, a derivation expansion moduLe 6, and an underivation (derivation reduction) module 7. The inflection module 4 and the uninflection module 5 contain structural features that allow the morphological analyzer/generator 2 to produce all inflected forms of a word given its baseform and to produce all baseforms of a word given an inflection. The derivation expansion module 6 and the underivation module 7 contain features that allow themorphological analyzer/generator 2 to produce all derivatives of a word given its derivational baseform and to produce a derivational baseform of a word given a derivation.
FIGURE 2 illustrates one potential operation of multilingual processor l 0. In particular, FIG. 2 shows an input buffer 15, a token list 17, and an output buffer 19. The source of text 16 supplies a strearn of natural language text to input/output controller 26 that in turn routes the text to processor 30. Processor 30 supplies the application program interface 1 l with the stream of text, and places the text in the input buffer 15. Processor 30 initiates operation of the noun-phrase analyzer 13 by making the calls to the interface 11, as described ~o~re.
Noun-phrase analyzer 13 operates upon the text contained in input buffer 15 and g,~ dl~S ~nd places in the in~;erface 1 1 the token list 17 and the output buffer 19. Token list 17 is atl array of tokens that describes the relationship between the input and output data.
Token list 17 contains a token 21 for each output word 23. Each token 21 links an input word 25 w~tilh its corresponding ou;tput word 23 by pointing to both the input word 25 and the output word 23. ln ~ddition to linking the input and output, each token describes the words they identify. For exarnple, each token 21 can point to a memory address storing inforrnation CA 02227383 l998-Ol-l9 W O 97/04405 PCT~US96/12018 -14-regarding the particular token. Inforrnation associated with each particular token can include, for example, the part-of-speech of the token, the capitalization code of the token, the noise-word status of the token, and whether the token is a member of a noun phrase.
In operation, computer 12 obtains a buffer of text from source of text 16, 5 relevant language databases from either the external memory 14 or the internal memory 22, and user selected operations from keyboard 18. Computer 12 then outputs to interface 11 a buffer of text 1 ~, an empty output buffer 19, and the specific operations to be performed on the buffer of text. Noun-phrase analyzer 13 then performs the specified operations on the buffer of text 15 and places the generated output into the output buffer l 9 and places the token list 17 that correlates the input buffer oftext 15 with the output buffer 19 into the application program interface 11.

THE WORD DATA TABLE
FIGURE 3 illustrates a word data table 31 used in conjunction with the 15 multilingual text processor 10. Word data table 31 includes digital codings representative of a list of expressions labeled Exp. N 1 through Exp. Nm. The word data table acts as a dictionary of expressions, wherein each expression contains a pointer to an entry, such as the representati~re entry 33. Various word data tables exist, each being representative of either different languages, dialects, technical language fields, or any subgroup of lexical expressions 20 that can be processed by text processor 30.
The word data table 31 can be an addressable table, such as an 11 byte RAM
table stored in a portion of either the external memory 14 or in the memory 12. Each representative entry 33 in the word data table describes the characteristics of one or more words. In partic~lar, entry 33 contains a column, labeled item 35, that describes a particular 25 characteristic of a word. Entry 33 also contains a column, labeled item 37, that identifies wl}ich bytes, out of a possible 32-byte prefix position, identify a particular characteristic of the word. For example, particular bytes in the 32-byte prefix position can contain bytes represcntative of a particular word char~t~ tic, such as the capitalization code of word, or particular l~its in the 32-byte prefix position can contain bytes that point to a portion of 30 memory in either memory element 22 or memory element 14 that include information pertai~ng to a particular characteristic of the word, such as the parts-o~-speech of a word.
Characteristics of a word stored in representative entry 33 include the part-of-spee~h combination index of a word, and the grammatical features of the word. In particular the part-of-speech combination index of a word is identified by the labeled field 44 in FI~. 3, 3~ while the grammatical features of the word are identified by the labeled fields 32, 34, 36, 38, 40, 4~, 46, 48, 50, 52, 54, 56, 58, and 60 in FIG. 3. Additional grammatical features of a word include the word length, the language code, whether the word is an abbreviation, and whether the word is a contraction. Although not shown in FIG. 3, addresses to these additior~a~ gra}r~natical features of a word can be stored in a representative entry 33. For W O 97/04405 PCT~US96/12018 example, positions 12-13 in the 32-byte prefix location can identify the word length, positions 1-2 in the 32-byte prefix location can identify the language code; position 19 can indicate whether the word is an abbreviation, and position 20 can indicate whether the word is a contraction. The "l~rt;llt;d implementation is for the byte values in the 32-byte prefix to S be encoded in a compressed forrn The Capcode Field 32 identifies the capitalization of the word. For example, Capcode Field 32 can store a binary number representative of the capitalization characteristics of the word, such as: "000" can represent all lowercase letters; "OOI " can represent initial letter uppercase; ''010" can represent all uppercase letters; "011 " can 10 represent the use of a capitalization map (mixed capitalization); " 100" can represent no capit~ tion, unless the word is located at the beginning of a sentence, and "101 " can represent that capitalization is not applicable.
The Dialect Field 34 is used to identify words properly spelled in one dialect, but i~ op~ly spelled in another dialect. A common example of this behavior can be 15 demon~t~d using the American term color and its British cow~ t colour. This field is generally accessed during the decoding process to filter words based on the dialect of the word.
Ihe Has Mandatory Hyphen Field 36 stores inforrnation about words which change spelling when hyphen~te~l at the ends of lines. In Germanic languages, the spelling of 20 a word may change if it is hyphenated. This information can be encoded for both the hyphP~t~l and unhyphenated forms of a word. The presence or absence of the hyphen at the Error Position is enough to identi~r whether the word is correctly or incorrectly spelled. An example is the &errnan word bak-Jren, which is the form of the word used when it is hyphen:~te~; without the hyphen, the word is spelled backen. This information links the 25 hyphPn~te~ forrn with its unhyphel~ted form which would be the form normally uscd for such inforrnation retrieval tasks as in~xin~.
The Is Derivation Field 38 is used to identify whether a word is a derivation ~i.e., is a derived ;IForm of a root and therefore should use the derivation pattern to find the root form~ or a derivational root (in which case the derivation pattern is used to produce the 30 deri~ed forms of the root). For example, the word readable is a derived form of the deri~ational root read.
The Restricted/Word-Frequency Field 40 is used to store the word-frequency information about words in the word data table.
The POS Combination Index Field 44 stores an index into the part-of-speech - 3~ com~ination table 62, as illustrated in FIG. 4A. The part-of-speech combination table contains a list of parts-of-speech that a word can take. The parts-of-speech are stored with the most ~requent part-of-speech tag listed first in the part-of-speech combination table. The order of tihe other parts-of-speech in this table is unspecified, but implied to be in reverse W O 97l04405 PCT~US96/12018 frequency order. Fn~ h lists about 650 entries in this table, ~rench about 1900, Swedish about 200Q. Other languages fall within this range.
The Noun Inflection Pattern Field 46, the Verb Inflection Pattern Field 48, and the AdjectivelAdverb Inflection Pattern Field 50 give the respective pattern numbers used in 5 inflecting or uninflecting noun, verb, and adjective/adverb forms. The pattern number indexes a separate table of inflectional en-ling.~ and their parts-of-speech. Thus, there is an index to the noun inflection pattern of the word, an index to the verb inflection pattern of the word, and an index to the inflection pattern reprçs~nt~tive of the inflections of both the adjective and adverbial forms of the word.
The Derivation Pattern Field 52 contains information about how to derive or underive words from this particular word. Derivation patterns are much like inflection patterns. The derivation pattern is an index into a table of derivational endings and their parts-of-speech. The Is Derivation Field 38 described above tells whether the pattern should be used for deriving or underiving. If the bit contained within the Is Derivation Field 38 is not set, the wor~ is a derivational root.
The Compound Info Field 54 indexes another lookup table identifying rules regarding the compounding characteristics of the word. The lookup table contains fields, including a left-most compound component, a right-most compound component, that identify possible positions where the word can be used as a component in a compound word. This inforrnation is used for Germanic languages to decompose compounds into their constituents.
For examplie, the German compound Versicherungsgesellschaft (insurance company~ can be decomposed into Versicherung (its left-most compound component) and Gesellschaff (its right-most compound component).
The Error Position Field 56 specifies the position of a spelling-ch~n~in~
hyphen.
The LMCC Link Length Field 58 specifies the length of the compound link and is only used for words m~rkecl as being a Left-Most Compound Component. In the example above, the left-most compound component Versicherung has a Link Field of 1 since the single character s is used as its compound link.
The Field of Interest Field 60 describes the topic or domain of the given entry.For exa~n~le, field 60 can differentiate terms used exclusively in Medicine from those that are used exclusively in Law.
FI&UE~E 4A, 4B, and 4C illustrate other tables used by the multilingual text processor and stored in portions of either extern~l memory 14 or internal memory 22. In particular, FEG. 4A shows a Part-of-Speech Combination Table 62 cont~inin~ a list of indexes ~4~ a list of part-of-speech tags 66, and a list of OEM tags 68; FIG. 4B shows a Suffix Table 70 having a list of suffixes 72 and having a list of POS indexes 74 to the part-of-speech combin~tion table 62; and FIG. 4C shows a morphological file 71 having a list of CA 02227383 1998-01-l9 W O 97/04405 -17- PCT~US96/12018 paradigm numbers 73 each having a list of associated transformations identified by columns 75, 77 and 79.
These tables can be modified according to particular languages, such that the tables can provide linguistic informat;on for processing text in a particular language. Text processing system 10 can load tables associated with particular language databases when the database block of the application program interface 11 is initialized. This advantageously allows the databases to change without affecting the source code of the application program interface 11, the noun-phrase ana]yzer 13, or the morphological analyzer/generator 2. Thus, in effect the source code becomes independent of the language being processed. Further in accordance with this invention, multip~e languages can be processed by creating a database instance for each language being processed. The languages can be selected from either F,n~ h, Gerrnan, Spanish, Portuguese, French, Dutch, Italian, Swedish, Danish, Norwegian, or J~p:~nt?se. These particular languages are ,~l~selltative of languages having their own specific rules and tables for analy.~ing noun phrases, but are not included as a limitation of the invention.

THE PART OF SPEECH COM[BINATION TABLE
As shown in FIG. ~A, each entry in part-of-speech combination tablle 62 contains an index 64 having one or more associated part-of-speech tags 66 and having an associated, simpler OEM part-of-speech tag 68 used for display to users of the system. Each index 64 in table 62 identifies one or rnore part-of-speech tags 66. Thus, all words contained within the word data table are associated with one or more part-of-speech tag 66. If the part-of-speech tag entry 66 includes multiple part-of-speech tags, the most probable tag is the first tag in the entry 66. For example, as illustrated in FIG. 4A, if the Index 64 of a word is 1, the word has a single part-of-speech tag 66 of NN (used to identify generic singular nouns); and if the Index 64 of a word is 344, the word has five possible part-of-speech tags. Furthermore, a word indexed to 344 in the combination table has a most probable part-of-speech tag of ABN (used to identify pre-qualifiers such as half and all3, and also has part-of-speech tags of NN (used to identify generic singular nouns), NNS (used to identify generic plural nouns~, QL (used to identify qualifying adverbs), and RB (used to identify generic adverbs).

THE SUFFIX TABLE
FIGURE 4B illustrates a Suffix table 70 having a list of suffixes 72 and having a list of POS indexes 74 to the part-of-speech combination table 62. Thus, each entry in table 70 has a suffix 72 associated with a POS index 74. In operation, the suffix of a word contained in a streatn of text can be compared with suffix entries 72 in table 70. If a match is found for the suffix of the extracted word, then the word can be associated with a part-of-speech tag 66 in part-of-speech table 62 through POS index 74. For example, if a word in the stream of text co~Ltains a su~fix, le (as in ô'le), that word can be identified in table 7û and be CA 02227383 l998-Ol-l9 W 097/04405 PCTnUS96/12018 - lg-associated with a part-of-speech index "001 ". The part-of-speech index "001 " contains a part-of-speech tag NN (noun), as illustrated in FIG. 4A. Similarly, the word in the strearn of text having a suffix am (as in m'am~ can be associated with a part-of-speech tag of NN
through tables ~2 and 70.

THE MORPHOLOGICAL TABLE
FIGUR~ 4C illustrates an exemplary morphological file 71 where each horizontal line shown in the morphological file 71 is a separate morphological paradigm having one or more morphological transforms. Vertical column 73 identifies the numbering of the morphological par~irmc, and columns 75, 77, and 79 identify vertical columns cont~inin,~. different morphological transforms associated with any particular morphological paradigm. Each morphological transform is formed of a plurality of functional elements. In operation, the morphological file 71 of FIG. 4C describes how to produce a morphological transform given a baseform.
The morphological transforms identified by columns 75, 77 and 79 are all similarly structured. For example, each transform contains at least two functional elements that indicate one character string to be removed and one character string to be added to a candidate word. The similarity between the transforms allows processor 30 to uniformly apply the functional elements contained in any particular transform without having to consider exceptions to a discrete set of standard rules. The uniformity in the actions of processor 30, regardless of the transforrn being considered, allows for quick and easy processing.
As shown in FIG. 4C, every morphological transform identified in columns 75, 77 and 79 is structured as follows:
baseform part-of-speech ta~2; first character string to strip from the candidate word ~
second character string to add to the c~ntli~l~te word part-of-speech tag of morphological transform [optional field for ~ fix~ion].
Each morphological transform can thus be described as cont~ining a number of functional elements listed in sequence, as shown in FIG. 4C. In particular, the first functional element specifies the part-of-speech tag of the baseform of the candidate word, and the second functional element identifies the suffix to strip from the c~n~ te word to form an intermediate baseform. The third functional element identifies the suffix to add to the intermediate baseform to generate the actual baseform, and the fourth functional element specifies the part-of-speech of the morphological transforrn. The fifth functional element is an optional element indicating whether prefixation occurs.
FIG. 4C illustrates, in particular, a morphological file suited to inflection and uninflection. For example, inflection transforrn 001 (as identified by column 73) contains W O 97tO4405 PCT~US96/12018 three transforrnations shown in columns 75, 77 and 79, respectively. The column 75 transformation for inflection transform 001 contains the transform, VB ~d_VBN.
This transform contains rules specifying that: (1) the baseform part-of-speech is VB; (2) no suffix is to be stripped from the candidate word to form the interm~Ai~te baseform; (3) the suffix d is to be added to the intermediate baseform to generate the actual baseform, (4) the part-of-speech of the resulting inflected form is VBN; and (S) no prefixation occurs. The column 79 transforrnation for transform 001 contains the transforrn VB_e ~ ing_VBG.
This transforrn specifies: (1) the baseform part-of-speech is VB; (2) the suffix e is to be skipped from the candidate word to form the intermediate baseform; (3) the suffix ing is to be added to the intelmediate basefor~n to generate the actual baseform, (4) the part-of speech of the resulting inflected forrn is VBG; and (S) no pLcrlx~lion occurs.
A file similar to that shown in FIG. 4C can be constructed for derivation expansion and underivation (derivation reduction). A derivational file, however, will not contain a functional element in the transform identifying part-of-speech information used in specifying whether a candidate word is a derivation or a derivational baseform. Information regarding derivation baseforms is instead stored in the word data table 31 of FIG. 3 under the Is Derivation Field 38.
Morphological file 71 of FIG. 4C also illustrates the use of portm~nte~
par~flipm~ Portm~nt~ paradigrns provide a structure capable of mapping the morphological changes associated with words having complicated morphological p~tt~rn~ In particular,morphologicaltransfor~ms 133, 134, 135, 136and 137(asidentifiedincolumn73) contain portm~nt~ paradigm used fcr associating a plurality of paradigms with any particular candidate word.
Morphological transform 133 indicates that patterns "006" and "002", as identified in column 73, are used to inflect the candidate word associated with morphological transform 133. Accordingly, a candidate word associated with inflection transform 133 becomes further associated with inflection transforms 002 and 006. For in~t~nre, the po~ paradigm 133 identifies the two inflections of travel, that can be inflected as travelled and traveled, depending upon dialect. Portm~nte~n paradigm 133 can also be used to inflect install, which can also be spelled instal. The illustrated portm~nte~n paradigms illustrate one possible structure used for applying multiple paradigms to any particular candidate word.
Another possible structure for providing portm~nte~ll paradigms can be formed using word data table 31 and a representative entry 33, as shown in FIG. 3. For - 35 example, expression N2 in data table 31 points to a representative entry 33 having a noun inflection pattern 46, a verb inflection pattern 48, and an adjective/adverb inflection pattern 50. In addition, the patterns 46, 48, and 50 each point to a paradigm in a morphological file 71, as illustrated in FIG. 4C. Thus, a candidate word matched with the expression N2 can become associated with a plurality of par~r~T~m~

W O 97/04405 PCTrUS96/12018 ~ IG. 4C illustrates a further aspect of the invention wherein the applicants' system departs dramatically from the prior art. In particular, a morphological baseform in accordance with the invention can vary in length and does not need to remain invariant. By l1tili7ing baseforms of variable length, the invention removes many of the disadvantages S associated with earlier natural language processing techniques, including the need for a large exception dictionary.
The morphological file 71 includes transforms having a variable length baseform, such as paradigm numbers 001 and 004. For example, the column 75 and 77 transforms of paradigm 001 produce a baseform having no characters removed from the 10 candidate word while the column 79 kansform of paradigm 00 l produces a baseform having an e character removed. The column 75 transform of paradigm 004 produces a baseform having no characters removed while the column 77 and 79 transforms of paradigm 004 produce baseforms having ay character removed from the c~n~ te word. Thus, when processor 30 acts in accordance with the instructions of paradigms 001 or 004 to form all 15 possible baseforms of a candidate word, the processor will form baseforms that vary in length.
FIGURE S illustrates a ~l~t~b~e system stored in various portions of memory elements 14 and 22 showing a connection between tables 31, 62, and 70 for associating part-of-speech tags with various lexical expressions contained within a stream of text. ~n 20 Expression N2 contained within the stream of text can be identified in the word data table 31 as representative entry 33. Representative entry 33 encodes the information contained in a 32-byte prefix, of which bytes 16- 18 contain a code found in the part-of-speech combination table 62. This table in its turn relates this particular part-of-speech combination with index 343 in table 62, thereby associating the part-of-speech tags of ABN (pre-qualifier), NN
25 (noun), QL (qualifying adverb), and RB (adverb) with Expression N2 In accordance with a further aspect of the invention, a part-of-speech tag can be associated with an expression in the stream of text through the use of suffix table 70. For example, a first expression in stream of text might contain a suffix ole, and can be iclçntified in suffix table 70 as rel)les~ tive entry 63. A second expression in the stream of text might contain the suffix 61e, and can be identified in suffix table 70 as representative entry 65. The pointer in representative entry 63 points to index 1 in table 62, and the pointer in representative entry 65 points to index 1 in table 62. Thus, both the first and second expression in the stream of text become associated with the part-of-speech tag of NN.

THE NOUN PHRASE ANALYZER
FIGURE 6 shows a block diagrarn of a noun-phrase analyzer 13 for identifying noun phrases contained within a stream of natural language text. The analyzer 13 comprises a tokenizer 43, a memory element 45, and a processor 47 having: a part-of-speech identifier 49, a grammatical feature identifier 51, a noun-phase identifier 53, an agreement W O 97/~4405 PCT~US96/12018 checker 57, a disambiguator S9, and a noun-phrase truncator 61. Tnt~rn~l connection lines are shown both between the tokenizer 43 and the processor 47, and between the memory element 45 and the processor 47. FIG. 6 further illustrates an input line 41 to the tokenizer 43 from the application program interface 11 and an output line from the processor 47 to the S application program interface 11 .
Tokenizer 43 extracts tokens (i.e., white-space delimited strings with leading and trailing punctuation removed~ frolrn a stream of natural language text. The stream of natural language text is obtained f'rom text source 16 through the application program interface 11. Systems capable of removing and identif~ying white-space delimited strings are 10 known in the art and can be used herein as part of the noun-phrase analyzer 13. The extracted tokens are further processed by processor 47 to determine whether the exkacted tokens are members of a noun phrase.
Memory element 45, as illustrated in FIG. 5, can be a separate addressable memory element dedicated to the noun-phrase analyzer 13, or it can be a portion of either internal memory element 22 or external memory element 14. Memory element 5 provides a space for storing digital signals being processed or generated by the tokenizer 43 and the processor 47. For example, memory element 14 can store tokens generated by tokenizer 43, and can store various attributes identified with a particular token by processor 47. l[n another aspect of the invention, memory element 14 provides a place for storing a sequence of tokens 20 along with their associated characl:eristics, called a window of tokens. The window of tokens is utilized by the processor to identify characteristics of a particular candidate token by evaluating the tokens surrounding the candidate token in the window of extracted tokens.
Processor 47, as illustrated in FIG. 6, operates on the extracted tokens with various modules to form noun phrases. These modules can be hard-wired digital circuitry 25 performing functions or they can be software instructions implemented by a data processing unit performing the same fi~nctions. Particular modules used by processor 47 to implement noun-phrase analysis include modules that: identify the part-of-speech of the extracted tokens, identify the ~ Lical features of the extracted tokens, disambiguate the extracted tokens, identify agreement between extracted tokens, and identify the boundaries of noun 30 phrases.
FIGURE 8 depicts a processing sequence of noun-phrase analyzer 13 for forming noun phrases that begins at step 242. At step 243, the user-specified options are input to the noun-phrase analysis system. In particular, those options identified by ~he user through an input device, such as keyboard 18, are input to text processor 1 ~ and channeled 35 through the program interface 11 to the noun-phrase analyzer 13. The user selected options control certain processing steps within the noun-phrase analyzer as detailed below. At step 244, the user also specifies the text to be processed. The specified text is generally input from source text 16, although the text can additionally be int~ y generated within the digital colllpul~l 12. The specified text is channeled through the application program W O 97/04405 PCTAUS96/12~18 interface 11 to the noun-phrase analyzer 13 within the Buffer Block. Logical flow proceeds from box 244 to box 245.
At action box 245 tokenizer 43 extracts a token from the strearn of text specified by the user. In one embodiment, the tokenizer extracts a first token representative of the first lexical expression contained in the stream of natural language text and continues to extract tokens representative of each s~lccee~ling lexical expression contained ~n the identified stream of text. In this embodiment, the tokenizer continues extracting tokens until either a buffer, such as memory element 45, is full of the extracted tokens or until the tokenizer reaches the end of the text stream input by the user. Thus, in one aspect the tokenizer extracts tokens from the stream of text one token at a time while in a second aspect the tokenizer tokenizes an entire strearn of text without in~,l u~lion~
Decision box 246 branches logical control depending upon whether or not three sequential tokens have been extracted from the stream of text by tokenizer 43. At least three sequential tokens have to be extracted to identify noun phrases contained within the stream of text. The noun-phrase analyzer 13 is a context-l~l analysis system that identifies noun phrases based on a window of token cont~ining a c~nt~ te token and at least one token preceding the candidate token and one token following the candidate token in the strearn of text. If at least three tokens have not yet been extracted, control branches back to action box 245 for further token extraction, while if three tokens have been extracted logical flow proceeds to decision box 247.
At decision box 247 the system identifies whether the user-requested disambiguation of the part-of-speech of the tokens. If the user has not requested part-of-speech disambiguation control proceeds to action box 249. If the user has requested part-of-speech disambiguation, the logical control flow proceeds to decision box 248 wherein the system clct~rrnines whether or not disarnbiguation can be performed. The noun-phrase analyzer 13 disambiguates tokens within the stream of natural language text by performing further conte~t~ l analysis. In particular, the disambiguator analyzes a window of at most four sequential tokens to disambiguate part-of-speech of a candidate token. In one aspect the window of token contains the two tokens preceding an ambiguous candidate token, the ambiguous candidate token itself, and a token following the ambiguous ç~ntlil1~t~ token in the stream of text. Thus, in accordance with this aspect, if four sequential tokens have not been extracted logical flow branches back to action box 245 to extract further tokens from the stream of text, and if four sequential tokens have been extracted from the strearn of text logical flow proceeds to action box 249.
At action box 249, the part-of-speech identification module 49 of processor 47 determines the part-of-speech tags for tokens extracted from the stream of text. The part-o~-speech tag for each token can be determin~l by various approaches, including: table-driven, suffix-m~t~hing, and default tagging methods. Once a part-of-speech tag is determined for each token, the part-of-speech tag becomes associated with each respective token. After step 249, each token 21 in token list 17 preferably contains the most probable part-of-speech tag and contains a pointer to an address in a memory element cont~ining a list of other potential part-of-speech tags.
In accordance with the table driven aspect of the invention, the part-of-speech S tag of a token can be determined using the tables shown in Figures 3-5. For example, a representative lexical expression equivalent to the extracted token can be located in the word data table 31 of FIG. 2. As shown in FIG. 2 - FIG. 5, module 49 can then follow the pointer, contained in bytes 16-18 of t_e representative expression in word table 31, to an index 64 in the part-of-speech combination table 62. The index 64 allows module 49 to access a field 66 cont~ining one or more part-of-speech tags. Module 49 at processor 47 can then retrieve these part-of-speech tags or store the index to the part-of-speech tags with the extracted token.
Tl~is table-driven approach for identifying the part-of-speech tags of extractedwords advantageously provides a fast and efficient way of identifying and associating parts-l S of-speech with each extracted word. The word data table and the POS Combination Table further provide flexibility by providing the system the ability to change its part-of-speech tags in association with the various lan,guage .i~tzlh~ç~ For example, new tables can be easily downloaded into external memory 14 or memory 22 of the noun-phrase system without ch~nging any other sections of the mtlltilin~ual text processor 10.
In accordance with the suffix-m~t~ hing aspect of the invention, the part-of-speech tag of a token can be determined using the tables shown in Figures 4-S. For example, module 49 at processor 47 can identify a representative suffix consisting of the last end characters of the extracted token in su~fix table 70 of ~IG. 4B. Once a m~1~hing suffix is identified in suffix table 70, module 49 can follow the pointer in column 74 to an index 64 in part-of-speech combination table 62. The index 64 allows module 49 to access a field 66 cont~ining one or more part-of-speech tags. The index 64 allows module 49 to access a field 66 cont~inin~ one or more part-of speech tags. The part-of-speech identification module 49 can then retrieve these part-of-speech tags or store the index to the part-of-speech tags with the extracted token. Generally, the suffix-m~t~hin?~ method is applied if no ~ es~ iv~
entry in the word data table 31 was found for the extracted token.
A second alternative method for identifying the part-of-speech tags for the token involves default 1~ing Generally, default tagging is only applied when the token was not identified in the word data table 31 and was not identified in suffix table 70. Default tagging associates the part-of-speech tag of NN (noun) with the token. As a result, at the end of step 249 each token has a part-of-speech tag or part-of-speech index that in turn refers to either single or multiple part-of-speech tags. After step 249, logical control flows to action box 250.
At action box 250, the grammatical feature identification module 51 of the processor 9 ~letenninP~ the grammatical features for the tokens 21 contained in the token list CA 02227383 l998-Ol-l9 ~ W O 97/04405 PCT~US96tl2018 17. The grammatical features for each token can be obtained by identifying a representative entry for the token in the word data table 31 of FIG. 3. The identified representative entry contains inforrnation pertaining to the grammatical features of the word in fields 32, 34, 36, 38,40,42, 46, 48,50,52, 54, 56, 58 and 60. These fields in the representative entry either 5 contain digital data concerning the grammatical features of the token, or point to an address in a memory element co~ i . .i . ,3o; the grammatical features of the token. After box 250, control proceeds to decision box 251.
Decision box 251 queries whether the user requested disambiguation of the part-of-speech tags. If disambiguation was requested, control proceeds to action box 252. If disambiguation was not requested, control proceeds to action box 253. At action box 252, the part-of-speech tags of ambiguous tokens are disambiguated.

W O 97/04405 PCT~US96/12018 THE DISAMBIGUATOR
The disambiguator module 59 of the processor 47 identifies tokens having multiple part-of-speech tags as ambiguous and disambiguates the identified ambig~ous tokens. Accordingly, action box ~52 disambiguates those tokens identified as having multiple part-of-speech tags. For example, a first token extracted from the stream of text can be identified in the word data table 31 and thereby have associated with the first token an index 64 to the part-of-speech combination table 62. Furtherrnore, this index 64 can identify an entry having multiple part-of-speech tags in column 66 of table 62. Thus, the first token can be associated with multiple part-of-speech tags and be identi~led as ambiguous by processor 47.
Preferably, the first listed part-of-speech tag in table 62, called a primary part-of-speech tag, is the part-of-speech tag having the highest probability of occurrence based on frequency of use across dirre~ written genres and topics. The other part-of-speech tags that follow the primary part-of-speech tag in column 66 of table 62 are called the secondary part-of-speech tags. The secondary part-of-speech tags are so named because they have a lower probability of occurrence than the primary part-of-speech tag. The disambiguator can choose to rely on the primary part-of-speech tag as the part-of-speech tag to be associated with the ambiguous token. However, to ensure accurate identification of the part-of-speech for each token, this probabilistic method is not always reliable. Accordingly, in a pl~r~lcd aspect, the invention provides for a disambiguator module 59 that can disarnbiguate those tokens having multiple part-of-speech tags through context~ analysis of the arnbiguous token.
In particular, disarnLbiguator 59 identifies a window of sequential tokens cont~ining the arnbiguous token and then cle~ermines the correct part-of-speech tag as a function of the window of sequential tokens. In a first embodiment, the window of sequential tokens can include, but is not limil:ed to, the two tokens immediately prece-ling the ambiguous token and the token immediately following the ambiguous token. In a second embodiment, the window of sequential tokens includes the arnbiguous token, but excludes those classes of tokens not considered particularly relevant in disambiguating the ambiguous token. One class of tokens considered less relevant in disambiguating ambiguous tokens include those tokens having part-of-speech tags of either: adverb; qualifying adverb; or negative adverbs, such as never and not. This class of tokens is collect;vely referred to as tokens having "ignore tags". Under the second embodiment, for example, the disambiguator module 59 forms a window of sequential tokens cont~ining will run after skipping those words having ignore tags in the following phrases: will run; willfrequently run; will very ~ 35 frequently run; will not run; and will n~ver run. The second embodiment thus ensures, by skipping or ignoring a class of irrelevant tokens, an accurate and rapid contextual analysis of the ambiguous token without having to expand the number of tokens in the window of sequential tokens. Moreover, a window of four sequential tokens ranging from the two tokens immediately prececlin~ the ambiguous token and the token immediately following the CA 02227383 1998-01-l9 ambiguous token can be exp~ncle~l to include additional tokens by: (1) skipping those tokens contained within the original window of four sequential tokens that have ignore tags, and (23 replacing the skipped tokens with additional sequential tokens surrounding the ambiguous token.
S The functions or rules applied by module 59 identify the most accurate part-of-speech of the ambiguous token based both upon the window of sequential tokensconf~ininp; the ambiguous token and the characteristics associated with those tokens contained within the window of tokens. The characteristics associated with the tokens include, either separately or in combination, the part-of-speech tags of the tokens and the grammatical features of the tokens.
Once the disambiguator module 59 of the processor 47 has identified the most accurate part-of-speech tag, the processor places this part-of-speech tag in the position of the primary part-of-speech tag, i.e., first in the list of the plurality of part-of-speech tags associated with the ambiguous token. Thus, the ambiguous target token remains associated with a plurality of part-of-speech tags after the operations of processor 47, but the first part-of-speech tag in the list of multiple part-of-speech tags has been verified as the most conte~tll~lly accurate part-of-speech tag for the ambiguous token.
In one aspect, disambiguator 59 can determine that no disambiguation rules apply to the ambiguous token and can thus choose to not change the ordering of the plurality of part-of-speech tags associated with the ambiguous token. For example, a token having multiple part-of-speech tags has at least one part-of-speech tag identified as the primary part-of-speech tag. The primary part-of-speech tag can be identified because it is the first part-of-speech tag in the list of possible part-of-speech tags, as ill-lctr~t~d in FIG. 4A. If the disambiguator 59 determines that no disambiguation rules apply, the primary part-of-speech tag remains the first part-of-speech tag in the list.
In a further aspect, a disambiguation rule can be triggered and one of the secondary part-of-speech tags can be promoted to the pl;nl~y part-of-speech tag. In accordance with another aspect, a disambiguation rule is triggered and the primary part-of-speech tag of the ambiguous token is coerced into a new part-of-speech tag, not necessarily found amongst the secondary part-of-speech tags. An additional aspect of the invention provides for a method wherein a disambiguation rule is triggered but other conditions required to satisfy the rule fail, and the primary part-of-speech tag is not modified. Thus, after disambiguating, each token has a highly reliable part-of-speech tag identified as the primary part-of-speech tag.
FIGURE 9 illustrates an exemplary rule table used for disambiguating an extracted token in the F.ng~i~h language. As discussed with respect to the tables illustrated in FIG. 3 - FIG. 5, the disambiguation tables can differ from language to language.Advantageously, the tables can be added to the system 10 or removed from the system 10 to accommodate various languages without modifving the source code or hardware utilized in constructing the multilingual text processor 10 in accordance with the invention.
The illustrated table contains: (1) a colurnn of rules numbered 1-6 and identified with label 261; (2) a coluIILa reprçs.onting the ambiguous token [i] and identified S with label 264; (3) a column represçn~in~ the token ~i+1] imme~ tçly following the ambiguous token and identified with label 266; (4) a colurnn reprçs~nting the token [i- 1]
irnmediately prece-ling the ambiguous token and identified with the label 262, and (5) a column representing the token ~i-7~ immediately prece~lin~ the token [i-13 and identified with the label 260. Accordingly, the ta~ble illustrated in FIG. 9 represents a group of six 10 disambiguation rules that are applied by disambiguator 59, as part of the operations of the processor 47, to a window of sequential tokens cont~inin~ the ambiguous token [i]. In particular, each rule contains a set of requirements in colurnns 260, 262, 264, and 266, which if satisfied, cause the primary part-of-speech of the ambiguous token to be altered. In operation, processor 47 sequentially applies each rule to an ambiguous token in the stream of 15 text and alters the primary part-of speech tag in accordance with any applicable rule contained within the table.
For example, rule ] has a requirement and result labeled as item 268 in FIG. 9.
In accordance with rule 1, the processor 47 coerces the primary part-of-speech tag of the ambiguous token to NN (singular common noun) if the arnbiguous token [i~ is at the 20 beginning of a sentence and has a Capcode greater than 000 and does not have a part-of-speech tag of noun.
Rules 2-6, in FIG. 9, illustrate the promotion of a secondary part-of-speech tagto the primary part-of-speech tag as a function of a window of token surrounding the ambiguous token Li]. In particular, rule 2 promotes the secondary part-of-speech tag of 25 singular common noun to the primlary part-of-speech tag if: the token [i-2~ has a primary part-of-speech tag of article, as shown by entry 270; the token [i~ has a primary part-of-speech tag of either verb or second possessive pronoun or exclamation or verb past tense form, as shown by entry 272; and f he token [i] has a secondary part-of-speech tag of singular common noun, as shown by entry 272. Rule 3 promotes the secondary part-of-speech tag of 30 singular common noun to the primary part-of-speech tag îf: the token ti- 1] has a part-of-speech tag of verb infinitive or singular common noun, as shown by entry 274; and the token [i] has a primary part-of-speech tag of verb or second possessive pronoun or exclamation or verb past tense folm and has a secondary part-of-speech tag of singular common noun, as shown by entry 276. Rule 4 promotes the secondary part-of-speech tag of singular common 35 noun to the primary part-of-speech tag if: the token [i-1] has a part-of-speech tag of modal auxiliary or singular common noun, as shown by entry 278; the token [i~ has a primary part-of-speech tag of modal auxiliary and has a second part-of-speech tag of singular connmon noun, as shown by entry 280; and the token [i+l] has a part-of-speech tag of infinitive, as shown by entry 282.

CA 02227383 1998-01-l9 W O 97/04405 PCT~US96/12018 FIG. 9 thus illustrates one embodiment of the invention wherein the disambiguator 59 of the processor 47 modifies the ambiguous target token in accordance with a rule table. In particular, the illustrated rule table instructs processor 47 to modify the part-of-speech tags of the ambiguous token as a function of: the two tokens preceAing the 5 ambiguous target token in the stream of text, the token following the ambiguous target token in the stream of text, and the ambiguous target token itself. FIG. 9 further illustrates an embodiment wherein the ambiguous target token is modified as a function of the primary part-of-speech tag and the secondary part-of-speech tags of the ambiguous target token, and the part-of-speech tags of the other token surrounding the target token.
Disambiguation step 252 can also provide for a system that aids in identifying the elements of a noun phrase by checking whether or not the tokens in the stream of natural language text agree in gender, number, definiteness, and case. In particular, processor 47 can validate agreement between a candidate token and a token immediately adjacent (i.e., either immediately prece-lin~ or immediately following) the candidate token in the stream of text.
Agreement analysis prior to step 253, wherein the noun phrase is identified, operates in a single match mode that returns a success immediately after the first successful match. Thus, if agreement is being tested for token [i ~ and token [i-1] in the single match mode, processing stops as soon as a match is found. In accordance with this process, the processor selects the first part-of-speech tag from token [i], and tries to match it with each tag 20 for the token [i-l] until success is reached or all of the part-of-speech tags in token [i-l] are exh~11ct.oA If no match is found, then the processor 47 tries to match the next part-of-speech tag in the token [i] with each tag in token [i- 1 ] until success is reached or all of the part-of-speech tags in token [i-l] are ~h~11cteA This process continues until either a match is reached, or all of the part-of-speech tags in both token [i] and token [i-l] have been checked 25 with each other. A s~ ces~ful agreement found between two tokens indicates that the two tokens are to be treated as part of a noun phrase. If no agreement is found, then the two tokens are not considered to be a part of the same noun phrase.
First, the first POS tag from each token in checked for agreement.

Agreement Tags Agreement Tags Agreement Tags i-l ~.~fR~ Singular,Masculine ~ i,~; Singular,Masculine Plural,Masculine (Tagl & Tag2 &NumberMap) & ~Tag1 & Tag2 & GenderMap) fails fails W O 97/04405 - PCT~US96/12018 If this fails, the second POS tag from the token [i- 1 ] is checked for a match:
Agreement Tags Agreement Tags Agreement Tags i-l Plural,Masculine '',~,~iJ~ f~-'',';
r~ Singular,Masculine Plural,Masculine (Tagl & Tag2 & NumberMap) & (Tagl & Tag2 & GenderMap) 5passes fails At this point, all of the POS maps in the token [i-l] have been exhausted, and no successful match has been bound. The second POS tag in the token [i] must now be compared with all of the POS tags in the token [i-l].
The first POS tag from the token ri-l] and the second tag from the token [i] are checked for a match:

Agreement Tags Agreement Tags Agreement Tags , " ~ ,t..~ Singular,Masculine Singular,Feminine i~,iii.~ , Plural,Masculine 15 (Tagl & Tag2 & NumberMap) & (Tagl & Tag2 & GenderMap) fails passes If it fails, the second POS tag frorn the token [i-l] is checked for agreement-Agreemenl; Tags Agreement Tags ¦ AgreementTags i-l Plural,Masculi~ne ~
Singular,Feminine ~ Plural,Masculine (Tagl & Tag2 & NumberMap) ~: (Tagl & Tag2 GenderMap) passes passes At this point, a match has successfully been made, and all agreement processing stops. The ~ 25two tokens agree and Single Match mode processing is complete.
After Step 252, logical flow proceeds to Step 253. At step 253, the noun-phrase identifier module 53 of processor 47 identifies the boundaries of noun phrases contained within the stream of natural language text, and marks those tokens forming the noun phrase. In particular, processor 47 identifies the noun-phrase boundaries through W O 971044~5 PCTAUS96/12018 contextl~l analysis of each extracted token in the stream of text. In addition, module 53 marks those tokens forming the noun phrase by tagging tokens contained within the noun phrase. ~or exarnple, module 53 can associate with: the first token in the noun phrase a tag indicating "the beginnin~" of the noun phrase, the last token in the noun phrase a tag indicating "the end" of the noun phrase; and those tokens found between the first and last tokens in the noun phrase a tag indicating "the middle" of the noun phrase. Thus, module 53 of processor 47 identifies those tokens that it ~letermin~s are members of a noun phrase as either "the beginning", "the middle", or "the end" of the noun phrase.
According to one aspect of the invention, the noun-phrase identifier module 53 processor 47 forrns a window of sequential tokens to aid in identifying members of a noun phrase. Further in accordance with this aspect, the window of sequential tokens includes a token currently undergoing analysis, called a candidate token, and tokens prece-ling and following the candidate token in the stream of text. Preferably, the window of tokens includes the candidate token and one token immediately following the candidate token in the strearn of text and one token immediately preceding the candidate token in the stream of text.
Thus, the window contains at least three extracted tokens ranging from the token prece-ling the candidate token to the token following the candidate token inclusive. This window of sequential tokens provides a basis for contextually analyzing the candidate token to determinlo whether or not it is a member of a noun phrase.
The module 53 analyses characteristics of the window of sequential tokens to deterrnine whether the candidate token is a member of a noun phrase. The characteristics analyzed by processor 47 include, either separately or in conjunction, the part-of-speech tags and the grammatical features of each of the tokens contained within the window of tokens.
Module 5~ of processor 47 contextually analyzes the ç~ncli~1~te token by applying a set of rules or functions to the window of sequential tokens surrounding the candidate token, and the respective characteristics of the window of sequential tokens. By applying these rules, module 53 ~dentifies those c~nfii~1~t~ tokens which are members of noun phrases contained within the stream of text.
The noun-phrase identification rules are a set of hard-coded rules that define 3~ the conditions required to start, continue, and tennin~te a noun phrase. In general, noun phrases are forrned by conC~t~n~tinF together two or more contiguous tokens having parts-of-speech functionally related to nouns. Those parts-of-speech functionally related to nouns include the following parts-of-speech: singular cornmon noun (NN), adjective (JJ), ordinal num~er (ON), cardinal number (CD). In one embodiment, the noun-phrase rules apply these concepts arld form noun phrases from those sequential tokens having parts-of-speech functionally related to nouns.
Thus, for example, a set of four rules in pseudocode for identifying noun phrase is set forth in Table I below.

CA 02227383 1998-ol-lg - W o 97/04405 PCT/US96/12Q18 Table I

If the token is a member of Noun Phrase Tags 2 start to form a Noun Phrase.

3 If the tok:en is a stop list noun or adjective

4 If the Noun-phrase length is 0 don't start the Noun Phrase 6 else 7 ~reak the Noun Phrase.

8 If the token is a lowercase noun AND
9 the follo~ing token is an uppercase noun break the Noun Phrase.

11 If the token is a member of Noun-phrase Tags 12 continue the Noun Phrase.

In Table I, lines 1-2 represent a first rule and provide for identifying as a S "beginning of a noun phrase" those candidate tokens having a part-of-speech tag functionally related to noun word forms. That is, the first rule tags as the beginnin~ of a noun phrase those tokens having a part-o~-speech tag selected from the group of part-of-speech tags, including: singular common noun, adi~ective, ordinal number, cardinal number.
Lines 3-7, in Table I, represent a second rule. The second rule provides for 10 identify;ng as an "end of the noun phrase" those candidate tokens having a part-of-speech tag selected from the group consisting of stoplist nouns and adjectives. The defaultimplementation of the second rule contains the two stoplist nouns (i.e., one and ones) and one stoplist adjective (i.e., s2lch~. In particular applications, however, the user may introduce user-defined stoplist nouns and adjectives. For example, a user may chose to treat semantically 15 v~gue generic nouns such as use and ~pe as stoplist nouns.
In addition, lines 8-10 f~l~resellt a third rule. This third rules specifies that module 53 of processor 47 is to identify as an "end of the noun phrase" those selected tokens having a part-of-speech tag of noun and having a Capcode Field identification of "000" (i.e., lowercase), when the selected token is followed by an extracted token having a part-of-20 speech tag of nourl and having a Capcode Field identification of "001 " (initial uppelrcase) or "010" (i.e., all u~ c~se). Thus, in general, the third rule demonstrates identifying the end of a noun phrase through analysis of a group of tokens surrounding a candidate token, and the third rule demonstrates identifying the end of a noun phrase through analysis of the part-of-speech tags and grammatical features of tokens in the window of sequential tokens.
The fourth rule, represented by lines 11-12 in Table I, provides for identifyingas a "middle of the noun phrase" those selected tokens having a part-of-speech tag functionally related to noun word forms and following an extracted token identi~led as part of the noun phrase. For example, a token having a part-of-speech tag functionally related to noun word forms and following a token that has been identified as the beginning of the noun I 0 phrase is identified as a token contained within the middle of the noun phrase.
In operation, module 53 in conjunction with processor 47 applies each rule in Table I to each token extracted from the stream of natural language text. These rules allow module 53 to identify those tokens which are members of a noun phrase, and the relative position of each token in the noun phrase. The rules illustrated in Table ~ are not language-specific. However, other tables exist which contain language-specific rules for identifying noun phrases. Table II - VI, as set forth below, contain language-specific rules.

Table II - h',n~ h Lan~uage Noun-Phrase Rules If the token is uppercase AND
2 the token has a Part-of-speech Tag of Singular Adverbial Noun AND
3 the prece~ling token is a noun 4 break the Noun Phrase If the token is an adjective AND
the precerling token is a non-possessive noun 7 break the Noun Phrase 8 If the token is "of" or "&" AND
9 the prece.1ing token is an uplpel~ase noun AND
the following token is an uppercase noun 11 form a Noun Phrase starting with the prece-ling token and 12 continue the Noun Phrase as long as Noun Phrase Tags are 1 3 encountered.

W O 97/0440~ PCT~US96/12018 Tahle II contains a group of rules, in pseudocode, specific to the Fngli~h language. For example, lines 1-4 specify a first rule for identifying the end of a noun phrase, lines 5-7 recite a second rule ~or identifying the end of a noun phrase, and lines 8-13 specify a third rule for identifying the be~inning and for identifying the middle of a noun phrase.

Table III - Gerrnan Lan~ua~e Noun-Phrase Rules If the token is an adjective AND
2 the prece~1ing token is a noun AND
3 the following token is a member of Noun Phrase Tags 4 break the Noun Phrase Table III contains a group of rules, in pseudocode, specific to the Gerrnan Language. For example, lines 1-4 specify a rule for identifying the end of a noun phrase.

Table IV - Ttalian Language Noun-Phrase Rules If the token is "di" AND
2 the precediing token is a noun AND
3 the following token is a lowercase noun 4 forlm a Noun Phrase starting with the prece~ling token and corltinue the Noun Phrase as long as Noun Phrase Tags are 6 encoun~ered.

Table IV contains a group of rules, in pseudocode, specific to the Italian Language. For exarnple, lines 1-6 specify a rule for identifying the end of a noun phrase.

W Q 97/04405 PCTrUS96/12018 Table V - French and Spanish Noun Phrase Rules If the token is "de" AND
2 the prece-ling token is a noun AND
3 the following token is a lowercase noun 4 form a Noun Phrase starting with the prece-1ing token and continue Noun Phrase as long as Noun Phrase Tags are encountered.

Table V contains a group of rules, in pseudocode, specific to the French and Spanish Languages. For example, lines 1-5 recite a rule for identifying the beginning and the middle of a noun phrase.

Table VI - French and Spanish and Italian Noun-Phrase Rules If the token is an adjective AND
2 the prece~in~ token is a noun AND
3 the following token is a noun 4 break the Noun Phrase Table VI contains a group of rules, in pseudocode, specific to the French and Spanish and Italian languages. For exarnple, lines 1-4 recite a rule for identifying the end of 1~ a noun ph~ase.
After action box 253 of FIG. 8, control proceeds to decision box 254 of FIG.
8. At decision box 254 the processor 47 identifies whether the user requested application of the agreement rules to the noun phrase identified in action box 253. If the user did not request application of the agreement rules, control branches to decision box 256. If the user 20 did request applicativn of the agreement rules, logical control proceeds to action box 255 wherein the agreement rules are applied.
At action box 255 the agreement checking module 57 of the processor 47 ensures that the tokens within the identified noun phrase are in agreement. Although Fngli~h has no agreement ru~es, other languages such as German, French and Spanish require 25 agreement between the words contained within a noun phrase. For example, French and Spanish require gender and number agreement within the noun phrase, while Germanre~uires gender, number, and case agreement within the noun phrase. The gr~mms~ti~ ~1 W O ~7/04405 PCT~US96/12018 features concerning gender, number, and case agreement are supplied by the grammatical ~eature fields of the word data tab~e.
~ IGURE 10 illustrates a pseudocode listing that processor 47 executes to ensure agreement between the various members contained within an identified noun phrase.
S In particular, processor 47 iteratively checlcs whether a first identified part of a noun phrase agrees with a second identified palt of the noun phrase that immediately follows the first identified p~rt in ~he stream of texL. As described below, processor 47 ensures that each particular extracted token within the noun phrase agrees with all other extracted tokens contained in the noun phrase.
Pictorially, given a series of tokens with their associated agreement tags as shown below, where all tokens shown are valid candidates for being in the noun phrase, it would be possible to form a noun phrase that started with the token [i-2] and continued to the token Li+ 13 because they all agree with respect to the agreement tags of "Singular, Feminine".

Agreement Tags Agreement Tags Agreement Tags i-2 Plural,Masculine Singular, Masculine Singular, Feminine i-l Plural, Masculine Singular, ~eminine Plural, Feminine Singular, Feminine Singular, Masculine Plural, Masculine i~1 Singular, Feminine 1~
In one embodiment for checking agreement, two temporary array areas, templ and ~emp2, are proposed for storing the tokens while agreement is iteratively checked between the identified parts of the noun phrase.

20 ~ The token [i-2], identified as the "beginning of the noun phrase" has all of its agreement tags copied to a temporary area, templ .

templ Plwral, Singular, Singular, Masculine Masculine Feminine temp2 W 097/04405 PCTrUS96/12018 A11 agreement tags for the next token, token [i-l], whose values agree with templ area are placed in a second temporarv area, temp2.

templ Plural, Singular, Singular, Masculine Masculine Feminine temp2 Plural, Singular, Masculine Feminine As long as there are some identified agreement tags in templ and temp2, agreement has passed and the noun phrase can continue to be checked. If there is no match, agreement fails and the noun phrase is broken. When the noun phrase is broken, the last token that agrees with the previous tokens in the noun phrase is re-identified as the "end of the noun phrase".
In the current case being examined, there was agreement between templ and temp2, so that the contents of temp2 are copies of templ, and the next token is retrieved.

templ Plural, Singular, Masculine Feminine temp2 ~ All agreement tags for the next token [i] whose values agree with templ are 15 placed in the second temporary area, temp2. When this is done, the temporary areas contain:

temp I Plural, Singular, Masculine Feminine temp2 Singular, Plural, Feminine Masculine ~- Because token [i-2], token [i-l], and token [i] all have the above listed agreement tags in comrnon, the contents of the temp2 area are copied to templ, and the next 20 token is retrieved.

temp I Singular, Plural, Feminin~? Masculine temp2 . CA 02227383 1998-01-l9

5 PCT~US96/12018 ~ All agreement tags fox the next token [i+1] whose values agree with templ are placed in a second temporary area, temp2. When this is done, the second temporary areas contain:

templ Singular, Plural, Feminine Masculine temp2 Singular, Femmme ~ Because the token [i-2], token [i-1], token [i], and token ri+l] aIl have these agreement tags in common, the contents of the temp2 area are copied to templ, and the next token is retrieved.

templ Singular, Feminine ternp2 ~ At this point, noun phrase processing ends in our example. All the tokens from token [i-2] to token ri+l] had at least one agreement tag in com~non, and thus passed the agreement test.
In a further embodiment, the agreement checker 57 of the processor 47 creates 15 a "supertag" when checking agreement in accordance with action box 255 of FIG. 8. The supertags allow the agreement module 57 to quickly identify whether the extracted tokens fail to agree, or whether they may agree. In particular, a supertag is created for each extracted word contained within the identified noun phrase by logically OR'ing together all tihe agreement tags associated with each identified token in the noun phrase.
A supertag associated with one token in the noun phrase is then compared against the supertag associated with the following token in the noun phrase to see if any form of agreement is possible. A forrn of agreement is possible if the required number, gender, and case parameters agree or contain potential agreements between each of the supertags. If the re~uired number, gender, and case pararneters contained in the supertags do not agree, 25 then agreement is not possible. By making this comp~ on, it can be quickly determined whe~er or not agreement may exist between the tokens or whether agreement is impossible.
After action box 255, logical flow proceeds to decision box 256. At decision box 256 the processor 47 identifies whether the user requested application of the truncation rules to the noun phrase identified in action box 253. If the user did not request application 30 of the truncation rules, control branches to action box 258. If the user did request application W097104405 PCT~US96/120}8 ofthe truncation rules, logical control proceeds to action box 257 wherein the truncation rules are applied.
At action box 257, the truncator module 61 of the processor 47 truncates the identified noun phrases. In one aspect of the invention, as illustrated by the pseudocode S listing of F~GURE 11, truncator 61 truncates noun phrases exceerling two words in length which satisfy a specific set of rules. In accordance with another aspect of the invention, the truncator 61 removes tokens within the noun phrase that fail to agree with the other tokens within the noun phrase. Preferably, this operation is achieved by the truncator module 61 operating in conjunction ~,Yith the agreement checking module 57. For example, agreement module 57 identifies those tokens within the noun phrase that are in agreement and those tokens that are not in agreement, and truncator module 61 re-ex~min~ which tokens belong in the noun phrase based upon the agreement analysis of agreement checking module 57.
Thus truncator module 61 truncates from the noun phrase the set of tokens following, and including, a token that does not agree with the prece-iing members of the identified noun 1 5 phrase~
At action box 258, processor 47 outputs the tokens extracted from the input stream of natural language text into the output buffer 19 of the application program interface 11. Processor 47 also generates the token list 17 that correlates the input buffer of text 15 with the output buffer l 9, and places the token list 17 into the application program interface.
The generated token list 17 comprises an aTray of tokens that describe parameters of the input and output data. The parameters associated with each token include the part-of-speech tags, the grammatical fealules, and the noun-phrase member tags. With this data, processor 30 in digital computer 12 is able to output to display 20 the identified noun phrases contained within the input stream of natural language text~
FIGURE 12 illustrates an example of the operation of the noun-phrase analyzer 13 having arl input buffer 400, a token list 402, an output buffer 404, and identified noun phrases 40~ In particular, input buffer 400 contains a natural language text stream reading ~e caskf~ow is st~ong, the dividend yield is high, and. Token list 402 contains a list of tokens, wherein the tokens cas* and dividend are identified as the "beginnin~ of a noun phrase", and wherein the tokensflow and yield are identified as the "end of a noun phrase".
Output buffer 404 contains a list of the lexical ~x~le,~ions found in the input buffer 400, and box 406 contains the i~ientified noun phrases cash flow and dividend yield.
FIG~ 12 demonstrates the ability of the noun-phrase analyzer 10 to identify groups of words having a specific meaning when combined. Simply tokenizing the word in the stream of text and placing them in an index could result in many irrelevant retrievals.

W O 97/04405 PCT~US96/12018 MORPHOLOGICAL ANALYZER / GENE~RATOR
FIGURE 13 illustrates a pseudocode listing for implementing a moIphological analyzer/generator 2. In particular, the morphological analyzer can contain a processor 30 implementing the pseudocode listing of FIG. 13 as stored in memory 12. Additional tables, 5 as illustrated in FIG. 4A-4C, necessary for the implementation of morphological ~nalyzer/generator 2 can also be stored in memory element 12.
Lines I and 54 of the pseudocode listing in FIG. 13 forrn a first FOR-LOOP
that is operational until the noun form, the verb form, and the adverb/adjective form of the candidate word are each processed. In operation, processor 30 implements the conditions within the first FOR-LOOP of lines 1 and 54 by accessing the FIG. 3 representative entry 33 associated with the candidate word. The representative entry 33 includes a noun pattern field 46, a verb pattern field 48, and an adjective/adverb pattern field 50. Each of the fields (e.g., 46, 48, and 50) identifies a particular morphological transform in FIG. 4C.
Lines 2-4 of the pseudocode listing contain steps f'or checking whether morphological paradigms associated with each particular grarnmatical field being processed (i.e. noun, verb, adjective/adverb) exist. The steps can be implemerlted by proCeSS~Dr 30 ~cces.~in~ the FIG. 3 representative entry of the candidate word and identifying whether the fields 46, 48, 50 identify a valid morphological paradigm.
Lines 5-9 of the pseudocode of FIG. 13 include a logical IF-THEN-ELSE
construct for ~leterrnining the morphological paradigms associated with the candidate word.
In particular, these steps fonn a variable called "LIST" that identifies the locations of parS~ "LIST" can include one location in column 73 of FIG. 4C, or "LIST" can include a portm~ntezlll rule identifying a pllurality of locations in column 73.
Lines 10 and 53 of the pseudocode listing form a second FOR-LOOP nested within the first FOR-LOOP of lines l and 54. The second FOR-LOOP of lines 10 and 53 provide a logical construct for pra,cessing each of the paradigms contained in "LIST".
Lines 11 and 52 form a third nested FOR-LOOP that processes each candidate word once for each part-of-speech tag of the candidate word (identified as "POS tag" in FIG.
13~. The part-of-speech tags of the candidate word (i.e. "POS tag") are identified by the POS
3Q Com3~in~tion Index Field 44 of FIG. 3 that is associated with the c~n~ te word.
In one aspect of the invention, lines 12-18 include steps for identifying morphological transforms of the candidate word given a part-of-speech tag for the candidate word and given a morphological paradigm for the can~1id~te word. For example, the pseudocode instructions det--rrnine whether the baseform part-of-speech tag of the ~ 35 morphological transform ~identified as "BASE POS" in FIG. 13) matches the part-of-speech tag of the candidate word. rf a match is found, then the morphological transform is marked as a possible morphological transf'orm for the candidate word, and the candidate word can be identified as a baseform.

Lines 27 and 51 of FIG. 13, in accordance with another aspect of the invention, contain a further nested FOR-LOOP. This FOR-LOOP operates upon each of the morphological transforms listed in the particular paradigm from 'LIST' that is currently being processed.
Further in accordance with the invention, each morphological transform within the current paradigm being processed is inspected to determine whether the morphological transform is an ~y~rop~;ate morphological transform for the candidate word. In particular, as illustrated by pseudocode lines 28-31, processor 30 identifies an appropriate morphological L~ ro,m based upon whether a parameter of the candidate word matches a morphological pattern contained within a selected morphological transform For instance, line 28 of the pseudocode ~letermines whether the part-of-speech tag of the candidate word matches the part-of-speech tag of the morphological transform. If a match exists, the morphological transform is identified as an applicable transform for the candidate word.
In accordance with another embodiment of the invention, as shown in pseudocode lines 28-29 of FIG. 13, the processor 30 can identify an ayylopLiate morphological transform based upon various parameter of the candidate word m~tching various morphological patterns contained within a selected morphological transform. The parameters of the c~n~ te word can include: information contained within the representative entry 33, of FIG. 3; the length of the candidate word; and the identity of the character strings forming the c~n(li~l~te word, i.e. the suffixes, prefixes, and infixes in the czln~lirl~te word. While the morphological patterns of a selected morphological transform are generally selected from the functional elements contained in the morphological transform.
Thus, the morphological p~1tern.~ can be selected from: a functional element defining the part-of-speech tag of the baseform, a functional element defining the character string to strip from a candidate word; a functional element defining the character string to add to a ç~n~ te word; and a functional element defining the part-of-speech tag of the morphologically transformed candidate word.
For example, the processor 30 can compare the suffix of a c~n~ t~ word with the second functional element of the selected morphological transform, wherein the second functional element generally denotes the suffix to strip from the candidate word to form an intçrrne(1;slt~ baseforrn. In an :~ltt?rn~tive embodiment, the processor 30 can compare the prefix of the c~n~ tf~ word with the second functional element of the selected morphological transform. While in another embodiment the processor 30 compares the infix 3~ of $he candidate word with the second functional element of the selected morphological tr~ns~rm. Following the comparison step, processor 30 then identifies those morphological transforrns having morphological patterns matching the selected pararneter of the candidate word as an ~Lo~,l;ate tr~n~forrn for the candidate word.

~ W O 97/04405 PCT~US96/12018 Preferably, as illustrated in lines 28-31 ofthe FIG. 13 pseudocode listing the processor 30 only applies those transforms that both: (1 ) have a part-of-speech tag matching the part-of-speech tag of the candidate word; and (2) have a first character string to be removed from the candidate word that matches either a suffix, prefix, or infix in the candidate word.
According to a further embodiment of the invention, prefixation and infixation can be handled by separate structural elements in the system, as illustrated by pseudocode lines 32-35 of FIG. 13. Lines 32-35 illustrate a separate modular element for detel~Tninin~ an applicable transrorm based on prefixation. Lines 32-35 first identifies whether the current morphological transform has the prefix flag set, as described in the discussion of FIB. 4C. If the prefix flag is set, a separate morphological prefix table cont~ining morphological changes applicable to prefixes is referenced. The prefix table can be identified through the representative word entry 33 for the candidate word.
The prefix table will provide a list of baseforrn and inflection pref1x pairs. To handle ~l~r~x~Lion, the processor 30 will locate the longest m~tl hinp prefix from one column in the prefix table, remove it, and replace it with the prefix from the other column.
Preferably, these modifications v~ill only be done when a morphological transform is tagged as re~uiring a prefix change. An analogous system can be created to address infixation.
Prefixation and infixation morphology are particularly applicable in Germanic ~0 languages, such as German and I)utch. In these languages the morphology of the word can change based upon the alteration of a character string in the beg;nninf~, middle, or end of the word. For example, German verbs display significant alternations in the middle and end of words: the verb e~n~ringen {ein ~ bringen) forms its past participle as ein+ge+bracht, with the infixation (insertion~ of the string ge between the verbal prefix and stem; and the :25 transformation of the stem bringen into bracht.
~he morphological analyzer/generator 2 illustrated in FIG. 13 provides a system capable of morphological]y transforrning words found within natural language text.
For e~zlm~le, the multilingual text processor 10 of FIG. 1 can extract the ç~n~lid~te word ~ri7l~;s from a stream of text and forward the c~n~ te word to analyzer/generator 2 through intPrf~re 11 The text processor 10 can further identify a represPnt~tive entry 33 for the c~n~ te word. Once a rep~esçnl-~tive entry is located, the text processor 10 can provide information concerning the word drinks, such as the parts-of-speech and inflectional paradigms. In particular, the text processor 10 detPrrnine~ the parts-of-speech of drinks to be no~n plural and verb 3rd sing~lar present; and the text processor dete~nines the locations of 3~ a noun inflectional paradigm, a verb inflectional paradigm, an adjective/adverb paradigm, and a derivational paradigm.
After the text processor 10 obtains the data related to the candidate ~word dr~ks, the text processor can generate the a~lopliate morphological transforms in accordance with the pseudocode listing of FIG. 13. The morphological analyzer/generator 2 - W O 9710440~ PCT~US96/12018 first addresses the noun inflectional paradigm, and detrrmines that the noun paradigm has only one paradigm. Analyzer/generator 2 then proeesses the candidate word by applying the inflectional transforms contained within the identified noun paradigm to each part-of-speech of the eandidate word drinks. The inflectional transforrns within the noun paradigm are 5 applied by first deterrnining which inflectional transforms should be applied, and by then applying those inflectional transforms to generate inflectional baseforms.
For instance, the candidate word contains a part-of-speeeh of noun plural which must first be matched with particular inflectional transforms contained within the noun paradigm. The m~tching can be accomplished, in one embodiment, by comparing the parts-10 of-speeeh associated with a partieular transform to the part-of-speeeh of the eandidate words.
Thus, analyzer/generator 2 compares the current part-of-speech of the candidate word, i.e., no~n plural, to the part-of-speeeh tags assoeiated with the inflectional transforms stored in the noun infleetional paradigm. The analyzer deterrnines: (1) the baseform part-of-speech of the noun paradigm is no~n singular, that does not match the part-of-speech tag of the 15 e~n~ te word; (2) the first infleetional transform has as assoeiated part-of-speeeh tag of noun singular possessive, that does not mateh the part-of-speech tag of the e~n~lifl~te word;
and (3) the seeond inflectional transforrn has an assoeiated part-of-speech tag of noun plural, that does mateh the assoeiated part-of-speeeh tag of the eandidate word. These eomparison steps indieate that only the seeond infleetional transform matched the noun plural part-of-20 speech of the candidate word, and that therefore only the second inflectional transformcontained within the noun paradigm is applied.
Analyzerlgenerator 2 then continues to process the candidate word by applying the inflectional transforms contained within the identified verb p~r~'1igm and the i~entifie~l adjeetive/adverb p~ri~ m The verb paradigm eontains one paradigm having a 2~ baseform and two inflectional transforms, while the candidate word is associated with a potentially m~tching part-of-speech tag of verb 3rd singular present. The baseform part-of-speeeh tag of the verb infiectional paradigm is "verb infinitive", that does not mateh the part-of-speech tag of the ç~n~ ite word. The part-of-speeeh tag of the first infleetional kansform is verb present particip~e, that does not match the part-of-speech tag of the eandidate word.
30 But, the part-of-speeeh tag of the seeond infleetional transform is verb 3rd singular present, that does mateh the part-of-speeeh tag of the eandidate word. Thus, the infleetional transform contained within the second rule of the verb infleetional paradigm is applied to the candidate word.
After the applieation of the noun paradigm and the verb paradigm, the 35 analyzer 2 proeesses the transforrns contained within the adjective/adverb paradigm. In this partieular ease, the adjectiveladverb paradigm is blank, thereby eompleting the inflectional transformation of the candidate word drinks.
FIC~URE 14 depiets a processing sequenee for the uninfleetion module 5 for gel-L~ ;r-~; infleetional baseforms that begins at step 300. At step 302 the rzln~ te word for WO 97/~4405 PCTrUS96/12018 the inflectional analysis is obtained. Preferably, the candidate word is obtained from a strearn of natural language text by tokenizer 43 as described in connection with FIG. 6. After step 302,1Ogical flow proceeds to step 304.
At step 304 the processor 30 obtains data relevant to the candidate word. This S data is obtained by first finding a substantially equivalent expression to the candidate word in the word data table 31. The substantially equivalent expression in the word data table 31 is then accessed to obtain an associated representative entry 33. A representative ent}y 33 contains data such as the part-of-speech combination index, the noun inflection para-ligm~, the verb inflection par~-ligm~, and the adjective/adverb inflection paradigms. The data obtained from representative enky 33 can also identify portmanteau paradigms that act as branching points to multiple numbers of other pariq(li,~m~. At action box 310, the flow chart indicates the beginnin~ of the analysis of each paradigm.
At steps 312 and 314 the system determines whether the part-of-speech of the candidate word is in the same class as the current paradigm. For example, the processor rl~tetmine~ whether the part-of-speecll of the candidate word is the same as the part-of-speech of the paradigm identified by eith!er the noun field 46, the verb field 48, or the adjective/adverb field 50 in the representative entry 33. If the part-of-speech of the candidate word is not in the sarne class as the current paradigm, logical flow branches back to action block 312. If the part-of-speech tag of the candidate word agrees with the current paradigm, then logical flow proceeds to decision box 316.
Decision box 316 illustrates one L,lerell~d embodiment of the invention, wherein the ç~n~ te word is compared to the paradigm's baseform. If the c~n~ te word m~tch.os the paradigm baseform, logical flow proceeds to decision box 328. That is, if the c~n~ 1.qte word matches the subp;~radigm's baseform no uninflection is necessary. In many situations, however, the candidate word will not match the paradigm baseform. When the candidate word differs from the paradigm baseform, logical flow proceeds to action box 318.
Action box 318 begins another logical FOR-LOOP wherein each inflectional transfonn is processed. In accordance with FIG. 14, logical flow proceeds from box 318 to decision box 320.
At decision box 3~0 two aspects of the invention and a ~ d embodiment are illustrated. In particular, action box 320 indicates that the part-of-speech tag of the candidate word can be compared with the fourth functional element of the inflectional - transform (i.e. the functional element specifying the part-of-speech of the transform). If the part-of-speech tags m~t ~h~s, then logical flow proceeds to action box 322. However, if the ~ 35 part-of-speech tags differ, logical flow branches back to box 318. According to a further aspect of the invention, as illustrated in action box 320, the ending character strings of the candidate word and the second functional element of the inflectional transform (i.e. the fimctional element specifying the suffix to strip from the candidate word) are compared. If the character strings do not match, logical flow proceeds back to action box 318 wllile if the CA 02227383 1998-01-l9 W O 97/044QS PCT~US96/12018 character strings match, logical flow proceeds to action box 322. Preferably, as illustrated in FIG.14, the uninQectional module 5 compares the part-of-speech tags associated with the inflectional transforrn and the candidate word, and the uninflectional module S compares the character strings associated with the inflectional transforrn and the candidate word.
S According to this preferred embodiment, only if the part-of-speech tags match and the character strings match does logical flow proceed to action box 322.
At step 322, uninflection module 5 implements a strip and add algorithm to forrn the inflectional baseforrn of the candidate word. The strip and add algorithrn is obtained from the inflectional transforrn currently being processed. The transforrn currently being processed indicates a particular character string to be removed from the candidate word and a subsequent character string to be added to the character word to forrn the inflectional baseforrn. After step 322, logical flow proceeds to decision box 324.
Decision box 324 is an optional step involving prefixation. If prer~ ion operations are requested by the user, boxes 324 and 326 will be activated. At decision box 324 the processor 30 identifies whether the inflectional transforrn currently being considered has a prefixation rule associated with it. If the transform does contain the pref1xation rule logical flow proceeds to action box 326, otherwise logical flow proceeds to action box 328.
At action box 326 the prefix is removed from the baseforrn in accordance with the inflectional transforrn. Logical flow then proceeds to box 328.
Steps 328, 330,332, and 334 are optional steps demonstrating one implementation of the coupling between the inflection module 4, the uninflectional module 5, the derivation expansion module 6, and underivation (derivation reduction) module 7.
In particular, action box 328 identifies whether the user has requested underivation (derivation reduction). I~ underivation (derivation reduction) has been requested, logical flow proceeds to action box 330, otherwise flow proceeds to decision box 33~. At action box 330 the candidate word undergoes underivation (derivation reduction) in accordance with the flowchart identified in FIG. 16. Following underivation (derivation reduction), logical flow proceeds to decision box 332. At decision box 332 the processor tifies whether inflection has been requested. If inflection was requested, logical flow proceeds to action box 334, wherein the c~n~ te word undergoes inflection analysis in accordance with the steps illustrated in FIG. 15. If inflection was not requested, logical flow proceeds directly to action box 336.
At action box 336 the logical FOR-LOOP for the inflectional transform ends and at action box 338 the logical FOR-LOOP for the paradigms ends, thereby completing the uninflection routine.
FIGURE 15 depicts a processing sequence for the inflection module 4 of the morphological analyzer of FIG. 1. The inflection analysis begins at step 340 and logical control proceeds to action box 342. At action box 342 the inflection module 4 obtains an inflectional baseforrn of a candidate word. The inflectional baseform can be obtained, for CA 02227383 1998-01-l9 W O 97/04405 PCT~US96tl2018 example, from a candidate word which is processed by the uninflection module 5 in accordance with FIG. 14. After action box 342, logical flow proceeds to action box 344.
Box 344 begins a logical FOR-LOOP that is applied to each inflectional transform in the paradigm associated with the candidate word.
At action box 346 and 348 the inflection module attends to prefixing if prefixing processing was requested by the user of the text processing system 10. Decision box 346 determines whether a prefixing rule is contained within the inflectional transform, and if such a ~er~ g rule is present the rule is applied at action box 348. After boxes 346 and 348, logical flow proceeds to box 350.
At step 350 characters are removed from the baseform to form an intermediate baseform, and at step 352 characters are added to the intermediate baseform to form the inflected pattern. Thereafter, action box 354 assigns the part-of-speech tag associated with the applied inflectional transform to the newly generated inflected form. Action box 356 ends the FOR-LOOP begun at action box 344.
FIGURE 16 depicts a further processing sequence for the underivation (derivation reduction) module 7 ofthe morphological analyzer 2, that begins at step 360. At action box 362 underivation ~derivation reduction) module 6 obtains a baseform of the c~n(~ te word. The baseform can be obtained from the uninflection module ~. After action box 362, control proceeds to box 364.
Decision box 364 identifies whether the derivation paradigm is an empty set - or whether it contains morpholog;cal transforms. In particular, if derivational paradigms do not exist for this baseform, logical flow proceeds to action box 396 ending the underivation (derivation reduction) process. However, if the derivation paradigm is not blank, logical control continues to box 366.
Box 366 begins a logical FOR-I,OOP for processing each derivational paradigm. After box 366, control proceeds to decision box 368.
Decision box 368 examines whether the candidate word is a derivational route or not. Determin~tion of the derivation route characteristics of the word can be performed by analyzing the information contained within the representative entry 33 associated with the candidate word. For example, the Is Derivation Field 38 of FIG.3 identifies whether the c~n~ te word is a derivational route. If the candidate word is marked as a derivational route, logical flow proceeds to action box 394, otherwise logical flow proceeds to action box 376.
Action box 376 begins a logical FOR-LOOP for processing each derivational transform in the subparadigm. After action box 376, logical flow proceeds to decision box 378.
Decision box 378 detemlin~s whether the derivational transform includes a character string m~tchin~ the candidate word's ending string. If no match is found, logical flow will proceed to action box 376, otherwise logical flow will proceed onto box 380.

W O 97/044Q5 PCT~US96/12018 At action box 380, the derivational reduction module 7 implements the transform for chSlnging the candidate word into the derivational baseforrn of the word. This process is implemented by removing a first character string from the candidate word and adding a second character string to the candidate word in accordance with the derivational 5 transforrn. At box 382, the newly transformed word is marked as a derivational root. After box 382, flow proceeds to decision box 384.
Boxes 384 and 386 are optional boxes providing prefixing adjustments to the newly formed derivational root. For exarnple, decision box 384 deterrnines whether a prefixing rule exists within the derivational transform and if such a prefixing rule exists then insures that logical flow proceeds to action box 386. At action box 386, the prefix is removed to generate a more accurate derivational root. After the implementation of optional boxes 384 and 386, logical flow proceeds on to box 392.
At box 392, the FOR-LOOP which began with box 376 ends. E~ox 394 ends the logical FOR-LOOP associated with action box 366. Once each of the paradigms has been completely processed logical flow ~,vill proceed from box 394 to box 396. Box 396 indicates the end of the underivation (derivation reduction) module.
FIGURE 17 illustrates a processing sequence of derivation expansion module

6 for generating derivatives of the candidate word. Operation of the derivation expansion module begins at step 400, after which logical control proceeds to action box 402. At action 20 box 402 the derivation expansion module obtains the derivational root of the candidate word.
This root can be obtained from the underivation (derivation reduction) module 7 of FIG. 16.
After action box 402, control proceeds to action box 404. Box 404 provides a logical FOR-LOOP ~or each derivational transform in the paradigm associated with the derivational root obtained at action box 402. After action box 404, control proceeds to 25 decision box 406.
Boxes 406 and 408 illustrate optional plerlxillg control boxes. These control boxes are implemented if the user requests pl~rlxillg. Following action box 408 control proceeds to action box 410.
At action box 410, derivation expansion module 6 removes characters from 30 the derivational root in accordance with the derivational transforrn associated with the paradigm ~ ly being processed. After box 410, logical control passes to action box 412.
At action box 412, a string of characters is added to the interrn(?~ te root formed in action box 410 in accordance with the current derivational transforrn. After box 412 control proceeds to box 414. At action box 414 a part-of-speech tag is ~c~igned to the newly 35 generated derivational expansion in accordance with the derivational transforrn. Following box 414, control proceeds to action box 420. Action box 420 ends the FOR-LOOP
associated with action box 404, thereby ending the derivation expansion processing.

W O 9710440~ PCTAJS96/12018 THE TOKENIZER
Figure 1~ illustrates a detailed drawing of the advanced tokenizer I for extracting lexical matter from the stream of text and ~or filtering the stream of text.
Tokenizer I receives input either through the application program interface 11 or the input S line 41, shown in FrG. 6, in the form of a text stream consistin~ of altern~ting lexical and non-lexical matter; accordingly, ]exical tokens are separated by non-lexical matter. Lexical matter can be broadly defined as information that can be found in a lexicon or dictionary, and is relevant for Infornnation Retrieval lProcesses. Tokenizer 1 identifies the lexical matter as a token, and assigns the attributes of the token into a bit map. The attributes of the non-lexical 10 matter following the lexical token are mapped into another bit map and associated with the token. Tokenizer 1 also filters or identifies those tokens that are candidates for fulther linguistic proces~in~ This filtering effect by the tokenizer reduces the amount of data processed and increases the overall system throughput.
Tokenizer 1 includes a~ parser 430, an identifier 432 electronically coupled with the parser 430, and a filter 434 electronically coupled with the identifier 432. The parser 43~ parses the stream of natural language text and extracts lexical and non-lexical characters from the strearn of text. The identifier 432 identifies a set of tokens in the parsed stream of text output by the parser 430. The identifier 432 identifies tokens as a consecutive string of lexical characters bounded by non-lexical characters in the streatn of text. The filter 434 20 selects a candidate token from the tokens generated by the identifier 43~. The candidate tokens selected by the filter 434 are suited for additional linguistic processing.
Typically, the T<)keni7~r 1 is the first module to process input text in the multilingual text processor 10. The output from the tokenizer 1 is used by other linguistic processing modules, such as the noun phrase analyzer 13 and the morphological analyzer 2.
2~ Input to the tokenizer 1 is in the form of a text stream form the application program interface 11. ~he parser 430 of the tokeni~er 1 converts the input stream of text to lexical and non-lexical characters, after which the identifier 432 converts the lexical and non-lexical chala~ to tokens. The filter 434 tags those tokens requiring filrther linguistic processing.
The tokens are converted back to stream format upon output to the application program 30 interface 11. The filter can be implemented in either electronic h~-lw~ or software instructions executed by a multi-purpose computer. Flow charts and descriptions of the software sufficient to enable one skilled in the art to generate a filter 434 are described below.
Tlle tokenizer can Ibe implemented using conventional progr~mming and numerical analysis techniques on a general purpose digital data processor. The tokenizer can 3~ ~e implemente~l on a data processor by writing computer instructions by hand that implement the tokenizer as detailed herein, or by forming a tokenizer as a finite state m~ehin~.
Preferably, a finite state m~hine is used to implement the tokenizer. The finite state m~ehine operates by recognizing one or more characters in the stream of text and entering a state in the m~hine based upon this recognition, and by performing operations on tokens within the W O 97/04405 PCT~US96112018 stream of text based upon the current state of the machine. The code for the finite state machine must keep track of the current state of the machine, and have a way of ch~ngin~
from state to state based on the input stream of text. The tokeni~er must also include a memory space for storing the data concerning the processed stream of natural language text.
In particular, for each token processed by the tokenizer 1, the filter 434 creates a tag to a memory location that stores a data structure including the parameters for the processed token. For instance, the data structure can include the following parameters:

pInStream Input: A pointer to the null-tt?rmin~tçd input stream from which the tokenizer creates a token. The input stream might contain 8-bit or Unicode characters.

pMoreText Input: A pointer to a flag that indicates if more text follows after the end of buffer is reached. This determines whether the tokenizer will process a partial token, or request more text before processing a partial token.
pCharsProcd Output: The number of characters that the tokenizer processed on this call.
This is the total number of characters that define the current token; this includes the length of the token and the non-lexical matter that follows it .
The calling routine should increment the input buffer by this value to dçtt~rrnine where to continue proces~ing.
p~MCBuf Output: A pointer to the output text buffer of the tokenizer.

pLexTokLen Output: The total number of characters that define the current token; this includes only the length of the token and not the non-lexical matter that precedes or follows it.

pAttrib Output: A 4-byte BitMap that contains the lexical and non-lexical attributes of the returned token. Preferably, the pAttribu parameters are stored in a 32-bit BitMap.

This impl~rn~nt~tion of tokenizer 1 has several benefits. It achieves high throughput, it generates information about each token during a first pass across the input stream of text; it elimin~tçc and reduces multiple scans per token; it does not require the 3~ accessing of a ~l~t~h~e; it is sensitive to changes in language; and it generates sufficient information to perforrn sophisticated linguistic processing on the stream of text. Moreover, tolcenizer ~ allows the non-lexical matter following each token to be processed in one call.
Additionally, tokenizer 1 achieves these goals while simultaneously storing the properties of the non-lexical string in less space than is required to store the actual string.

. CA 02227383 1998-01-19 W O 97~04405 PCTAJS96/12018 The filter 434 also includes a character analyzer 440 and a contextual analyzer 442 to aid in selecting a candidate tok.en from the set of tokens generated by the identif~ying element 432. The filter selects a candidate token based upon an analysis of characters in the strearn of text. The filter can either compare a particular character in the stream of text with entries in a character table, or the filter can analyze the particular character in the stream of text in view of the characters surrounding the particular characters in the stream of text.
For example, in one aspect the contextual analyzer 442 can select tokens for additional linguistic processing b y analyzing those characters surrounding probable terminator characters, strippable punctuation characters, lexical punctuation characters, hyphen characters, apostrophe characlers, parentheses characters, dot/period characters, slash characters, ellipse characters, and a series of hyphen characters.
The contextn~l analyzer in another aspect selects tokens for additional procescing based on where the selected character is located relative to the suspect token. The character may be located in the "beginning", "middle", or "end" of the token. The term "Beginning" refers to a character that immediately precedes a lexical character, and the term "Middle" refers to a character occurring between two lexical characters, and the term "End"
refers to a character irnmediately following a lexical character. In particular, in the case of strippable punctuation, the punctuatiom may be stripped from a token if it is found at the either the beginning or end of the token. If it occurs in the middle of a token, it does not cause the token to be split, and the punctuation is instead included within the token.
Furthermore, the location of the character relative to its position in the suspect token is applicable for analysis of probable tf~rrninS~tor characters, lexical punctuation characters, hyphen characters, apostrophe characters, parenth~sçs characters, dotlperiod characters, slash charaGters, ellipse ch~r~cters, and a series of hyphen characters The contextual analyzer can also select tokens for additional processing based upon the existence of similarly related characters. In the case of paret~th~se~, the existence of nn~t~hin~ parentheses ~i.e. left and right hand parentheses) within a particular token, as opposed to m~tchin~ par~nth~ses ~spanning two or more tokens, effects the linguistic proces~in~ performed on the particular token.
Further in accordance with the invention, the character analyzer 440 scans the stream of text for selected characters and identifies tokens having the selected characters as c~ntlicl~t~ tokens for additional linguis~ic processing. The character analyzer 440 utilizes a comparator and an associator for achieving this analysis. The comparator compares a selected character in the stream of text with entries in a character table. If the selec~ed character and entry match, then additional linguistic processing is al3~LoL"iate. After a sllcces~fill match, the associator associates a tag ~,vith a token located proximal to the selected character, the tag identifying the al3~ropliate additional linguistic processing.
One example of a table of characters used by the comparator includes characters selected from the group con~i~ting of probable tennin~tQr characters, strippable punctuation characters, lexical punctuation characters, hyphen characters, apostrophe characters, parentheses characters, dot/period characters, slash characters, ellipse characters, or a series of hyphen characters.
In a further example of the filter 434, both the character analyzer 440 and the 5 contextual processor 442 are used in selected tokens for additional linguistic processing. For instance, the filter 434 may use both the character analyzer 440 and the contextual processor 442 when filtering text that includes: (1.) apostrophes, (2.) parenthesis, (3.) dots/periods, (4.) ~ hes, (5.) ellipsis, and (6.) a series of hyphens.

( 1.) Apostrophes The character analyzer scans the stream of text for apostrophes because they can indicate pre-clitics, post-clitics, and contractions. The contextual analyzer ~leterrnines whether the apostrophe causes a token to be apl)rol~liate for additional linguistic because if the apostrophe occurs at the beginning or end of a token, it is stripped off, and the a~ ,;ate 15 flag is set. While if the apostrophe occurs between two lexical characters, it is included within the token, and the intern~l apostrophe flag is set.

~2.) Parenthesi~
Parentheses are also analyzed by both the character analyzer and the 20 contextual analyzer because parentheses behaves differently depending upon their relative location within the token and the relative location of matching parenthesis. The key rule is that if matching left and right parentheses are found within a lexical token, neither of the parentheses are stripped. For example, if the parenthesis character is located in the following positions relative to the token, the following actions occur:
Beginning: [~womanO
The left parenthesis is stripped.
The Pre Noun Phrase Break flag is set.
The resultant token is: [(Iwoman10 [(wo)manO
Both parentheses are ignored.
The Tntt?rn~l Parenlheses flag is set.
The resnlt~nt token is: [(wo)man10 3~
[(woman)O
Both parenthPs~ are ~l;~ed.
The Pre Noun Phrase Break flag is set.
The Post Noun Phrase Break flag is set.

CA 02227383 1998-01-l9 WO 97/04405 - 51 - PCT/US96/1:2018 The resultant token is: [(Iwomanl)O

Middle: ~wo(man30 Both parentheses are ignored.
S The Tnt~rn~l Parentheses flag is set.
The resultant token is: [wo(man)10 [wo(m)anO
Both pa~nthesçs are ignored.
The Internal Parentheses flag is set.
The resultant token is [wo(m)an10 [wo(manO
The left parentht~sic is ignored.
No flags are set. I he token is not split.
The resultant token is: [wo(man10 End: Lwoman(O
The left parenthe~ix is stripped.
The Post Noun Phrase Break flag is set.
The resultant token is: [womanl(O

~womanOO
Both parcntheses are stripped.
The Post Noun Phrase Break flag is set.
The rcsult~nt token is: [woman100 Possi~le Flags Set:
Tntern~l Par~nth~ses 30 Pre Noun Phrase Break Post No~ Phrase Break - The right parenth~si~ behaves exactly like the mirror image of the left parenth~ Again, the key rule is that if matching left and right parentheses are found within ~ 35 a lexical token, neither of the parentheses are stripped. In addition, an Tnt~rn~l Par2ntheses flag is set.

Be~inning: [)womanO
The right par(?nthe~i~ is stripped.

CA 02227383 1998-01-l9 W 097/04405 PCT~US96/12018 The Pre Noun Phrase Break flag is set.
The resultant token is: [)Iwoman10 [)(womanO
Both parenthPses are stripped.
The Pre Noun Phrase Break flag is set.
The resultant token is: [)(Iwoman10 Middle: [wo)manO
The right parçnth~ is ignored.
No flags are set The resultant token is: [wo)manl(O

[wo3m(anO
lS Both parenthPses are ignored.
No flags are set.
The resultant token is: [wo)m(an10 [wo)(manO
~O Both parentheses are ignored.
No flags are set.
The resultant token is: [wo)(man10 End: [woman)O
The right parenth~ci~ is stripped.
The Post Noun Phrase Break flag is set.
The resultant token is: [womanl)O

[woman)(O
Both parenthPses are stripped.
The Post Noun Phrase Break flag is set.
The resultant token is: [womanl)(O

Possible Flags Set:
35 Pre Noun Phrase Break Post Noun Phrase Break CA 02227383 1998-01-l9 WO 97/n4405 PCT~US96/1~018 (3.) Periods The Period can either indicate the end of a sentence, abbreviations, or numeric tokens, depending on the context. Accordingly, the period is analyzed by both the character and contextual analyzer of the fi lter 434.
s (4.) Slash The slash can be ii~llcl~leted in one of two ways. Under most circumstances, it is used to separate two closely related words, such as male/female. In such cases, it is 11n~lerstood that the user is referring to a male or a female. However, the slash can also be 10 used within a word to create a non-splittable word such as I/O. I/O c~nnot be split apart like male/female. ~e tokenizer preferably recognizes a slash character in the strearn of text and performs a co~ xl~l analysis tc~ det~rmine how the slash character is being used, thereby identifying the apl)ru~l;ate additional linguistic procesqing 15 (~.) Ellipsis It is also beneficial to perforrn both conte~rt~7~1 and character analysis on the Points of Ellipsis (POE). The POE is defined by either a series of three or four dots. Two c~ots and over four dots can be cklssified as non-Points of Ellipsis that are to be either stripped or ignored. While a valid POE at the end of a sentence may indicate sentence termin~tion ~0 The behavior of the POE depends upon its relative position to the lexical token, as is demonstrated below.

Be~inning: [....abcO
The POE is stripped.
The Pre Noun Phrase Break flag is set.
The resultant tokem is: [....labc10 Middle: [abc.. def0 The POE is treated like an IIWSPC class character: The token is split.
~he Post Noun Phrase Break flag is set ~or the "abc" token.
The resultar~t token is: [abcl....ldef~O

End: [abc.. 0 The POE is stripped.
The Prob~le Termin~tion flag is set.
The Post Noun Phrase Break flag is set.
The Stripped End of Word Period flag is not set.
The resultant token is: [abc1....0 WO 97/04405 PCT~US96/12018 The three dot POE is treated in the same manner as the four dot POE.
However, variations such as two dots and five or more dots in series are treated as follows:

Beginning: [..abcO
Exactly the same as a valid leading POE.

Middle: [abc.. defO
The dots are ignored: The token is not split.
No flags are set.
The resultant token is: [abc.. de~O

End: [abc.. O
The dots are stripped.
The Post Noun Phrase Break flag is set.
The Stripped End of Word Period flag is not set.
The reslllt~nt token is: ~abcl..O

~6.) ~Iyphen Series Because any other than two hyphens in series can be either stripped or 20 ignored, both contextual analysis and character analysis are ~plo~liate in the case of a series of hyphens.
Further aspects of the invention provide for an associative processor 436 for associating with a seleGted candidate token a tag identifying additional linguistic proce~in~, or for associating with a selected group of c~nc1i~te tokens a plurality of tags identifying 25 additional linguistic proces~inE The additional linguistic processing identified by a tag can include: pre noun phrase breaks, post noun phrase breaks, probable token termination, pre-c}itic processing, post-clitic proces~ing, apostrophe proccs~inF, hyphen procec~ing, token verification, parentheses proce~ing, unconvertible character proces~in~, and capitalization procescing This list of advanced linguistic processing is intt?n(led as an example of 30 additional processing and not as a limitation on the invention.
The filter 434 can also include a modifying processor 438. The modifying processor changes a selected token based on the tags identifying further linguistic proces~in~ for the selected token. The modifving processor includes sub-processors capable 35 of either sp~itting tokens, stripping characters from tokens, ignoring particular characters, or merging tokens. The modi~ying processor 438 is capable of acting based upon flags potentially set during the process of selecting the token, as described above. The modifying processor 438 is also capable of acting based upon flags set in the parameters associated with a selected token. In particular, the modifying processor operates as a function of the . CA 02227383 1998-01-19 attributes associated with each se] ected candidate token. The attributes associated with each token are identified by the pAttrib flag discussed above.
One sub-group of attributes identii~y the lexical attributes of the token.
In particular, this sub-groups includes the internal character attribute, the special processing attribute, the end of sentence attribute, and the noun phrase attribute. Another sub-group of attributes identifies the non-lexical attributes of the token. The non-lexical attributes include:
contains white space, single new line, and multiple new line.
The InternaLl Characters attributes signify the presence of a special character within a lexical token. The internal character attributes include: leading apost~ophe, intern~l apostrophe, trailing apostrophe, leading hyphen, internal hyphen, trailing hyphen, int~ l slash, and intP~ l parcnth~-sçc.
The special processing attributes signals that the token must undergo special processing either inside or outside the tokenizer 1. These attributes include:
numbers, possible pre-clitic, possible post-clitic, and unicode error.
1~ The end of sentence and noun phrase attributes are used by both the Sentence Boundary Det~rrniners and t~e Noun Phrase Analyzer. These attributes include:
probable sentt?nce tPrrninSItion~ pre nolm phrase break, post noun phrase break, attached end of word period, stripped end of word period, capitalization codes, and definite non s~nten(~e t~rmin~tion.
The above identified attributes are detailed below. The detailed desGriptions of the attributes identify both the operations of the modifying processor 438 and the associating processor 436. In particular, the descriptions identify how the associating processor 436 identifi~s when a pluralnty of tokens becomes associated with a plurality of tags identifying additional Iinguistic proGescing Furtherrnore, the descriptions below 2~ identify how the modifying processor modifies tokens in the stream of text as a function of the tag identified with a selected c~nrli~l~t~ token. Modifying functions described below include splitting tokens, stripping characters from tokens, ignoring characters within tokens, and merging tokens.

~-eading,4postrop}ze (7II,~APO~) The IILEADAPO bit is set only if:
1. An apostrophe immediately precedes a lexical character AND
2. The apostrophe does not occur between two lexical characters.
If these conditions are met:
1. The IILEA~APO flLag is set.

W O 9~/04405 PCTAJS96/12018 2. The leading apostrophe is stripped. An exception occurs if an IIDIGIT class character immediately follows the apostrophe.

Examples:
s String Actions Flags Set Token ['twasO Apostrophe stripped. IILEADAPO twas ['sGravenschag Apostrophe stripped. IILEADAPO sGravenschage eO
[;'defO Semi-colon stripped. IILEADAPO def Apostrophe stripped.
[abc+'defO Non-lexical characters ignored. None. abc+'def Token not split.
["defO Bothapostrophes stripped. III EADAPO def [-'defO Hyphen stripped. IILEADAPO def Apostrophe stripped.
['4~ersO Special because IIDIGIT IINUMBER '49ers immediately follows apostrophe.
Apostrophe not stripped.
Token not split.
['940 SpeciaI because IIDIGIT IINUMBER '94 im me~i~t~ly follows apostrophe.
Apostrophe not stripped.
Token not split.

Internal Apostrophe ~I~NTAPO) The IINTAPQ bit is set if:
10 1~ An apostrophe occurs between two lexical characters.

If this condition is met:
1. The IINTAPO flag is set.

Examples:

String Actions Flags Set Token [llenfantO Token not split. IINTAPO l'enfant [d'aujour'huiO Token notsplit. IINTAPO d'aujour'hui Cjack-o'-lantern Non-lexical characters ignored. None. jack-o'-lantern O Token not split.
Internal Apostrophe flag not set.
[Amy'sO Token not split. IINTAPO Amy's [abc"defO Non-lexicalcharacters ignored. None. abc"def Token not split.
~ntern~l Apostrophe flag not set.
[abc'-'defO Non-lexical c:haraeters ignored. None. abc'-'def Token not split.
Internal Apostroplhe flag not set.
abc.'.defO Non-lexical c]:laractersignored. None. abc.'.def Token not split.
Internal Apostrophe flag not set.

S Trai~ingApostropke ~IITR~PO) The IITRLAPO flag is set only if:
1~ An apostrophe immediately follows a lexical character AND
2. The character following the apostrophe is an IIWSPC class çh~r?ct~r or the 10 apostrophe is the last character in ~e entire text strearn. The end of the text stream is represen~ed by either an IINULL class character as defined by the OEM, or by the Lnd of File character.

1~ .... Ift~ese conditions are met:
1. The lITRLAPO flag is set.
2. The trailing apostrophe is stripped.

Examples:

Strin~ Actions Flags Set Token ~Jones'O Aposkophe stripped. IITRLAPO Jones ~Jones';O Apostrophe stripped. IITRLAPO Jones Semi-colon stripped.
[Jones"O Bothaposkophes stripped. IITRLAPO Jones [abc"defO Both apostrophes ignored. None. abc"def Token not split.

S Leading Hyphen (~H~

The IILEADHYP bit is set only if:
1. A hyphen imm~ tely precedes a lexical character AND
2. The hyphen does not occur between two lexical characters.
If these conditions are met:
l. The IILEA~HYP flag is set.
2. The leading hyphen is stripped.

15 In~ern~ ~Iyphe~ HYP) The IINTHYP bit is set if one of the following two conditions occur:
l. The hyphen is between two lexical characters.
2. The hyphen imrnediately follows a valid form of an abbreviation, and is 20 followed by a lexical character. The special case of "U.S.A.-based" is handled by this condition. Valid forms of abbreviations include "U.S.A." and "J.", but not "Jr."
If these conditions are met:
1. The IINTHYP flag is set.
25 2. The token is not split. However, the presence/absence of an Em-Dash must be verified.

~razli~g Hyphen fI~ L~YPJ

30 The IITRLHlYP bit is set only if:
1. The hyphen follows a lexical character AND

W O 97/044Q5 PCT~US96/12018 -59-2. The character following the hyphen is an IIWSPC class character or the trailing hyphen is the last character in the entire text stream. The end of the text stream is est;llled by either an IINULL class character as defined by the OEM, or by the End of File character.
s If these conditions are met:
1. The IITRlHYP flag is set.
2. The trailing hyphen is stripped.

0 Infernal Slas~ (IINTSL~S~) The IINTSLASH flag is set only if a slash occurs between 2 lexical characters.

~nternal Paren~eses ~IINTPAR~
The IINTPAREN flag is set only if a LPAREN and a RPAREN occur in that order within a lexical token. In summary, two forrns of a word can be indicated by using pairedparentheses: i.e. (wo)man can be used to represent both man and woman. In one case, the text within the parenth~s( c is disregarded, and in the second form, the text is included In 20 order to simplify the proce~ing for the Output Manager, only tokens that contain parentheses in this form are marked.

Digit Flag fI~NUMBEI~) 25 T~e IINUMBER flag is set any time an IIDIGIT class character occurs within a lexical token.
Nurnbers may contain periods, commas, and hyphens as in the case of catalog part numbers.
An external module wil~ handle all tokens with the IINUMBER flag set: they may be in~1P~e-1, or may be treated as non-indexable terms.

30 Special ?~ rhm~nt rules are used in t~le following two cases:
1. If a period is immediately followed by an IIDIGT, the period is left att~h~cl to the token.
~ i.e. .7~
2. If an apostrophe is immediately followed by an IIDIGIT, the apostrophe is left 35 attached to the token.
i.e. '49ers In both cases, the period~apostrophe must be preceded by the beginning of the buf~er or an II~SPC character.

W O 97/04405 PCT~US96/12018 Possible Pre-Clitic (lIPR~CL~C3 The IIPRECLTC bit is set if:
1. The language is French, C~t~l~n, or Italian AND
2. An apostrophe is found after a lexical character AND
3. The number of characters preceding the apostrophe doesn't exceed the maximum pre-clitic length as defined in the language structure, AND
4. The lexical character immediately preceding the apostrophe is found in a table of pre-clitic-termin~tin~ characters as defined in the language structure.

Possible l'ost-Cli~ic (IIPOSC~LTC) The IIPOSCLTC bit is set if:
I. The language is French, Catalan, or Portuguese AND
l. A hyphen (or apostrophe for Catalan) is found AND
2. The number of characters preeeding the hyphen (apostrophe:Catalan) exceeds the minim~lm stem length as defined in the language structure, AND
20 3. The character immediately following the hyphen (apostrophe:Catalan) is lexical AND
4. It's found in a table of post-clitic initial characters as defined in the language structure.

25 II. The larlguage is Spanish or Italian AND
1. The length of the token exceeds the minimum post-clitic length as defined in the language stmeture AND
2. A right to left scan (L <= R) of the token m~tche~ a post-clitic in the table of post-clitics defined in the language structure. Note that exact implementation is te~ Jol~y.
Unico~e ~;rror f~UN~CF~R~

Unconvertible Unicode characters are treated exactly like IIALPHA lexical characters. They do not cause a token to break: upon encountering such a character, the IIUNICERR flag must 35 be set.

CA 02227383 1998-01-l9 W O 97/04405 PCT~US96/12018 Probable ~.exical Termination !'IIP~TERM) If an IIPTERM or a point of ellipsis is encountered, this flag is set. It indicates to an external module to examine the token both preceding and following the current token. In particular, it 5 indicates that the CapCode of the following token should be examined to see if the sentence has really ~f~rmin:~ted.

Pre Nou~ Phrase Brealc (I~P~NPBRK) 10 The Pre Noun Phrase Break flag is set when the current token contains characters that guarantee that it cannot be comb~inedl with the previous token to form a noun phrase.

Post No2-n Phrase Break ~IIPO~NPBR~) 15 The Post Noun Phrase Break flag is set when the current token contains characters that guarantee that it cannot be combined with the following token to form a noun phrase.

Attached ~nd of Word Period (IIAEOWPER) 20 This flag is set when the token is a valid abbreviation that ended in a period followed by IIWSPC. It cannot be determin~d if the abbreviation ends the sentence or not without ~x~.,.il.ing the current token to see if it's a valid abbrev;ation, and the following token for its CapCode. In any case, the period is ~tt~che~l to the token.

2~ Strippe~E~nd of Word Pe~iod (~IS~OWPER) This flag is set when a per~od is found at the end of a token. The period is stripped, and the flag is set.

3r~ Ca~Codes Two bits will be used to defime the capCode as it exists.

Proba~le No~ riC~ ermination (~IPNLTE~M) If an I~PT~RM or a point of ellipsis is encountered in the middle of non-lexical matter, this flag is set.

~on~ains Wh~te Space CA 02227383 1998-01-l9 W O 97/04405 PCT~US96/12018 -62-Set when non-lexical ma~ter contains characters of the IIWSPCS class.

Single Line Feed (SN) The IISNLN bit is set only if a single NewLine 'character' occurs within the non-lexical matter following a token.

FIGURES 7A-7C are flow charts illustrating the operation of tokenizer 1.
10 FIG. 7A generally illustrates the main trunk ofthe tokeni7~tion operation, FIG. 7B illustrates the token identification steps, FIG. 7C illustrates the token lengthening steps, and FIG. 7D
illustrates the trailing attributes steps of the tokenization method according to the invention.
FIG. 7A shows steps 80 -134 in the operation ofthe tokenizer 1. The operation of tokenizer 1 begins at step 80. After step 80, the operation proceeds to step 82.
At step 82, the tokenizer reserves space in memory 12 for the token. The reserved memory space will be used to hold a data structure that includes the parameters for the token being processed. These parameters, as discussed above, can include a pointer to the null-termin~t~l input stream, a pointer to a flag indicating if more text follows, the number of characters processed by the tokenizer, a pointer to the output of the tokenizer, the total 20 number of characters that define the current token, and a bitmap including the lexical and non-lexical attributes of the current token. After step 82, logical flow proceeds to step 84.
At step 84, the parser module 430 of tokenizer 1, gets an input character from the streann of natural language text. After which, at step 86, the tokenizer identifies whether the end of the text buffer is reached. If the end of buffer is reached, then logical flow 25 proceeds to step 88. If the end of buffer is not reached, then logical flow branches to step 110.
When an end of buffer is icl~ntified in step 86, the tokenizer identifies whether a token is currently under construction, at step 88. If there is no token cull~,nlly under construc~on, then control proceeds to step 90 and the tokenizer executes a return to the 30 procedure calling the tokenizer. If a token is ~;ull~nlly under construction at step 88, then the logical flow of the tokenizer proceeds to decision box 92.
At decision box g2, the tokenizer queries whether the end of the document has been reached. The tokenizer can identify the end of the document by sç~qnnin~ for particular codes in the stream of text than identify the end of the document. If the tokenizer is not at the 35 en~ of the document, then control branches to action box 94, otherwise control proceeds to action box 98.
At action box 94, the tokenizer removes the space reserved for a token back in action box 8~. After action box 94, the tokenizer proceeds to step 96 where the tokenizer executes a return instruction.

W O 97~04405 PCT/US96/12018 At action box 98, the tokenizer caps the token string and then executes a for-loop starting with box 100 and ending with box 106. The for-loop modifies attributes of the token or the token itself as a function of each significant pattern identified within ~he token.
~n particular, boxes 100 and 106 identify that the for-loop will execute once for every S significant pattern. Decision box 102 queries whether a pattern is located in the token. If a pattern is found in the token, then control proceeds to action box 104. If a pattern is not found in the token, then control proceeds directly to action box 106. At action box 104, the tokt?ni7t?r modifies the token and/or the token's attributes in accordance with pattems associated with the token. After box 106, the tokenizer executes a return instruction.
Steps 100-1~)6 are executed by filtering element 434 oftokenizer 1. The filter 434 can further include sub-processors called the character analyzer 440, the contextual processor 442 and the modifying processor 438. The character analyzer 440 and the contextual processor 442 are closely related with steps 100 and 1~2. The modifying processor 43~ is associated with step 104. In particular, the character analyzer and the contextual processor identify significant patterns formed by characters in the input strearn o;f text. While, the modifying processor provides the tokenizer with the capability to modif,v the token and/or the token attributes as a function of significant patterns associated with the token ~;u~lclllly being processed.
A.t step 1 10 the to]ceni2er translates the input character code identified in step 84 to an internal character code suitable for the tokenizer's use. After step 1 10, logical flow proceeds to step 112.
Steps 112-134 illustrate various steps for identifying tokens within the stream of text. In general, steps 1 12-134 include steps for (letPrminin~ those characters forming the be~innin~ of a token, ~he end of a token, and the middle of a token. In particular, at step 112, 2~ if a tolcen is not ~ Lly under construction control branches to step 136, in Figure 7B. At step ~12, if a token is ~ elllly under construction, then control proceeds to decision box 114.
At decision box 1 ] 4, control branches to step 180 in Figure 7C if the current character is not a whitesr~ce. However, if the current character is a whitespace, then control proceeds to decision box 116.
At decision box 1 l 6, the tokenizer queries whether the current character beingprocessed is next in a pattern. The tokenizer performs these operations by relying on the character analyzer 441~ and the contextual processor 442. If the character is not in a pattern, then logical flow ~oranches to action box 124. If the character is identified as part of a pattern, then flow proceeds to action box 1 18.
3~ At action box 118, the tokenizer obtains an additional character. At decision box 120, the tokenizer queries whether the pattern is now completed. If the pattern is completed, then the tokenizer modifies the a~plopliate token attributes at action box 122. If the pattern is not completed, then flow proceeds to action box 124.

W O 97/044~ PCTAJS96/12018 Steps 124-132 are equivalent to steps 98-106. For instance, at steps 124-132 the tokenizer identifies patterns in the stream of text and modifies tokens and token attributes in view of the identified patterns. After step 132, the token is identified as complete at step 134 and control returns to step 84.
FIG. 7B shows steps 136-178 in the operation of tokenizer 1. In particular, at decision box 136 the tokenizer ~lueries whether the token is complete. If the token is complete, logical flow proceeds to decision box 138. If the token is not yet complete, then logical flow proceeds to decision box 154.
Steps 138 - 152 are perforrned within the confines of various token sub-processors. In particular, the modifying processor 438, the character analyzer 440, and the context~ l processor 442 each play a part in performing steps 138- 152. For instance, at decision box 138, the tokenizer and its character analyzer sub-processor query whether the current character starts a token. If the current character starts a token, then flow proceeds to action box 140. If the current character does not start a token ,then flow proceeds to decision 1 S box 142.
At action box 140, the tokenizer backs up to the last whitespace and then branches to step 212 of ~IG. 7D.
At decision box 142, the tokenizer queries whether the attributes of the currentcharacter modify tokens to the left. If the character analyzer identifies that the current character modi~les tokens to the left, then logical flow proceeds to step 144. At step 144, the modifying processor mod~fies the token attributes, and then the tokenizer branches back to step 84 of FI~;. 7A. If the character analyzer, in step 142, identifies that the current character is not modiiying tokens to the left, then logical flow branches to step 146.
Steps 146-152 are identical to steps 116-122 as shown in FIG. 7A. Following step 152, the tokenizer branches back to step 84 of FIG. 7A.
At step 154, in FIG. 7B, the tokenizer deterrnines whether the current character is a whitespace. If the character is a whitespace, then control proceeds to step 156.
At step 156, the token string is cleared and process returns to step 84 of FIG. 7A. If the character is not a whitespace, then control branches to step 158.
A.t step 158, another sub-processor ofthe tokenizer acts. In particular, at step158 the identifier 432 ~ppends the current character to the token being forrned. The identifier thus acts throughout the tnkeni7Pr process described in FIGs. 7A-7D to identify tokens fo~ned of lexica~ characters bounded by non-lexical characters. In addition, at step 158, the tokenizer marks the a~plo~-;ate token attributes as a function of the character appended to the token. After step 158, control proceeds to step 160.
From step 160 through step 166, the tokenizer executes a for-loop starting with box 161~ and ending with box 166. The for-loop modifies attributes of the token or the token itself as a function of each significant pattern identified within the token. In particular, boxes 160 and 166 identify that the for-loop will execute once for every significant pattern.

CA 02227383 1998-01-l9 W O 97/04405 PCT~US96/1~,018 Decision box 162 ~ueries whether a pattern is found in the previous character. If a pattern is fowld in the previous character, then control proceeds to action box 164. If a pattern is not found in the token, then control proceeds directly to action box 166. At action box 164, the tokenizer modifies the token's attributes in accordance with patterns associated with the token.
Steps 160-166 are executed by sub-processors within the tokenizer called the character analyzer 440, the contextual processor 442 and the modifying processor 438. In particular, the character analyzer 440 and the context~ processor 442 are closely related with steps 160 and 162, while the modifying processor 438 is associated with step 164. After step 166, control proceeds to step 168.
Steps 168-174 are identical to steps 146-152 and proceed in the same manner.
Af~er step 174, control proceeds to decision box 176. At decision box 176, the tokenizer queries whether the current character can start a token. If the current character is not ~1o~,liate for starting a token t]hen control returns to step 84 of FIG. 7A. If the current 1 $ character can start a token, then at step 178 the current character is identified as the beginning of a token. After step 178, control returns to step 84 of FIG. 7A.
l~IG. 7C shows ste,ps 180-210 in the operation of tokenizer 1 At s~ep 180, the tokenizer appends the current character to the token string being formed and updates the attributes associated w}th the token string in view of the newly appended character. After step 180, control proceeds to decision box 182.
At decision box 182, the tokenizer addresses whether the current token being forrned is too long. If the token is too long, control proceeds to step 184 where the length of the token string is capped, a3nd fr~m there to steps 186 and 188 where the tokenizer advances to the beginnin~ of the next token and executes a return instruction. If the token does not 2~ exceed a predetl~nninPcl length, then control branches from decision box 182 to decision box lgO.
Steps 190-196 are identical to steps 168-174 of FIG. 7B. For instance, steps 190-196 identify p;.~ c formed by characters in the stream of text and update token at~ibutes effected by the identified p~1tt?rnc After step 196, logical control proceeds to step 1~8.
Step lg8 begins a for-loop that is t~rrnin~f.?~l by either step 206 or by step 210.
The for-loop iteratively reviews the significant patterns in the token currently being formed until it is clçtPrminf~ t~at either: the token is complete under step 206, or there are no additional significant patterns in the token under step 210. After step 198, logical flow 3~ proceeds to decision box 200.
At decision box 200, the tokenizer identifies whether a pattern was found in the character. If no palttern is found, then control jumps to step 84 of FIG. 1. If a pattern is found, then control proceeds to decision box 202.

CA 02227383 1998-01-l9 W 097/04405 PCT~US96/12018 -66-At decision box 202, the tokenizer queries whether the pattern is a breaking pattern. If a breaking pattern is found then control branches to step 204 If no breaking pattern is found, then control first flows to action box 208 where the token attributes are modified in view of the patte~n found, after which conkol flows to box 210 which continues S the for-loop that started at step 198.
At action box 204, the token attributes are modified and the token is broken before the pattern identif1ed in step 200. After step 204, the tokenizer flags the identified token as complete in step 206 and then branches to step 212 of FIG. D.
FIG. 7~ shows steps 212-228 in the operation oftokenizer 1. Steps 212-218 10 execute a for-loop that executes until all attributes in the token that can modify the token have been processed. In particular, the for-loop begins at step 212 and then proceeds to step 214.
At steps 214 and 216 the tokenizer modifies the token in accordance with the attribute currently being processed. At step 218 the tokenizer completes its processing on the current attribute and branches back to step 212 if additional attributes remain, otherwise control 1 ~ flows to step 220.
At step 220 another for-loop that ends with step 226 begins executing. This for-loop is identical to the for-loop of steps 100-106, of FIG. 7A. After completing execution of the for-loop of steps 22~-226, the tokenizer executes a return instruction at step 228.
While the invention has been shown and described having reference to specific 20 preferred embo-liment~, those skilled in the art will understand that variations in form and detail may be made without departing from the spirit and scope of the invention. Having described the invention, what is claimed as new and secured by letters patent is:

Claims

1. A computerized tokenizer for identifying a token formed of a string of lexical characters found in a stream of digitized natural language text, the computerized tokenizer comprising:
parsing means for extracting lexical and non-lexical characters from the stream of digitized text, identifying means coupled with said parsing means for identifying a set of tokens, each token being formed of a string of parsed lexical characters bounded by non-lexical characters, and filtering means coupled with said identifying means for selecting a candidate token from said set of tokens, said candidate token being suitable for additional linguistic processing.

2. A tokenizer according to claim 1, wherein said filtering means further comprises an associative processing element for associating with said candidate token a tag identifying additional linguistic processing for said candidate token.

3. A tokenizer according to claim 2, wherein said associative processing element further includes a group processing element for associating with a plurality of tokens, as a function of said candidate token, a plurality of tags identifying additional linguistic processing for said plurality of tokens.

4. A tokenizer according to claim 2, further comprising a modifying processor for modifying said candidate token as a function of said tag associated with said candidate token.

5. A tokenizer according to claim 1, wherein said filtering selects said candidate token from said set of tokens during a single scan of the parsed stream of text.

6. A tokenizer according to claim 1, wherein said filtering means further comprises a character analyzer for selecting said candidate token from said set of tokens, said character analyzer including comparing means for comparing a selected character in the parsed stream of text with entries in a character table, and associating means for associating a first tag with a first token located proximal to said selected character, when said selected character has an equivalent entry in the character table.

7. A tokenizer according to claim 6, wherein said character table includes entries representative of a plurality of languages such that said tokenizer operates in the plurality of languages.

8. A tokenizer according to claim 1, wherein said filtering means further comprises a contextual processor for selecting said candidate token from said set of tokens as a function of a contextual analysis of the lexical and non-lexical characters surrounding a selected character in the parsed stream of text.

9. A tokenizer according to claim 8, wherein said contextual processor includes a set of rules applicable in a plurality of languages such that said tokenizer operates in the plurality of languages.

10. A tokenizer according to claim 1, further comprising a memory element for storing and retrieving the digitized stream of natural language text and for storing and retrieving a data structure that includes parameters for each token.

11. A tokenizer according to claim 10, wherein said parameters include an input stream flag identifying the location of a digitized stream of natural language text in said memory element.

12. A tokenizer according to claim 10, wherein said parameters include a flag identifying the number of lexical characters and non-lexical characters forming a token.

13. A tokenizer according to claim 10, wherein said parameters include an output flag identifying the location of an output signal generated by said tokenizer.

14. A tokenizer according to claim 10, wherein said parameters include a flag identifying the number of lexical characters forming a token.

15. A tokenizer according to claim 10, wherein said parameters include the lexical and non-lexical attributes of a token.

16. A tokenizer according to claim 15, wherein said lexical attributes are selected from the group consisting of internal character attributes, special processing attributes, end of sentences attributes, and noun phrase attributes.

17. A computerized data processing method for identifying a token formed of a string of lexical characters found in a stream of digitized natural language text, said method comprising the steps of extracting lexical and non-lexical characters from the stream of text, identifying a set of tokens, each token being formed of a string of extracted lexical characters bounded by extracted non-lexical characters, and selecting a candidate token from said set of tokens, said candidate token being suitable for additional linguitic processing.

18. A computerized data processing method according to claim 17, wherein said candidate token is selected from said set of tokens during a single scan of the parsed stream of text.

19. A computerized data processing method according to claim 17, wherein said selecting step further comprises the steps of character a selected character in the parsed stream of text with entries in a character table, and associating a first tag with a first token located proximal to said selected character, when said selected character has an equivalent entry in the character table.

20. A computerized data processing method according to claim 19, further comprising the steps of comparing a selected non-lexical character with entries in the character table, and associating said first tag with a token preceding said selected non-lexical character, when said selected non-lexical character has an equivalent entry in the character table.

21. A comperized data processing method according to claim 19, further comprising the step of forming a character table having entries representative of a plurality of languages.

22. A computerized data processing method according to claim 17, further comprising the steps of selecting said candidate token from said set of tokens as a function of a contextual analysis of the lexical and non-lexical characters surrounding a selected character in the parsed stream of text.

23. A computerized data processing method according to claim 17, further comprising the step of associating with said candidate token a tag identifying additional linguistic processing for said candidate token.

24. A computerized data processing method according to claim 23, further comprising the step of modifying said candidate token as a function of said tag associated with said candidate token.

25. A computerized data processing method according to claim 23, further comprising the steps of storing in a first location of a memory element attributes of said candidate token, said attributes identifying the additional linguistic processing suitable for said candidate token, and causing the tag to point to the first location.

26. A computerized data processing method according to claim 25, further comprising the step of storing in the first location attributes selected from the group consisting of lexical attributes and non-lexical attributes.

27. A computerized data processing method according to claim 26, further comprising the step of selecting the lexical attributes from the group consisting of internal character attributes, special processing attributes, end of sentence attributes, and noun phrase attributes.

28. A computerized data processing method according to claim 26, further comprising the step of selecting the non-lexical attributes from the group consisting of:
contains white space, single new line, and multiple new line.

29. A programmed data processing method for generating a morphologically related word from a candidate word using a first addressable table containing a list of lexical expressions and a second addressable table containing a list of paradigms, each paradigm having at least one transform that includes at least a first morphological pattern and a second morphological pattern, said method comprising the steps of:
locating in the first addressable table a first lexical expression substantiallyequivalent to the candidate word, identifying a first paradigm in the second addressable table as a function of the located first lexical expression, matching a transform in the identified first paradigm with the candidate word, forming an intermediate baseform by stripping a first character string from the candidate word, the first character string being defined by the first morphological pattern included with the matched transform, and generating a morphological baseform of the candidate word by adding a second character string to the formed intermediate baseform, the second character string being defined by the second morphological pattern included with the matched transform.

30. A method in accordance with claim 29, wherein said matching step further comprises:

identifying a parameter of the candidate word, selecting a morphological pattern for each transform, and matching a transform with the candidate word when the identified parameter of the candidate word matches the selected morphological pattern.

31. A method in accordance with claim 30, wherein the identified parameters of the candidate word are selected from the group consisting; of: part-speech-tags, grammatical features, the length of the candidate word, suffixes, prefixes, and infixes.

32. A method in accordance with claim 30, wherein the second addressable table includes part-of-speech tags associated with each transform and said matching step further comprises:

identifying the parameter of the candidate word as a part-of-speech tag of the candidate word, selecting a part-of-speech tag as the morphological pattern for each transform and matching a transform having an associated first part-of-speech tag with the candidate word when the first part-of-speech tag matches the identified part-of-speech tag of the candidate word.

33. A method in accordance with claim 29, wherein the intermediate baseform formed varies as a function of the matched transform.

34. A method in accordance with claim 29, further comprising the steps of:

locating a portmanteau paradigm in the second addressable table as a function of the first lexical expression, the portmanteau paradigm including the locations of a plurality of paradigms, and identifying at least a first paradigm selected from the plurality of paradigms included in the portmanteau paradigm.

35. A method in accordance with claim 34, wherein said portmanteau paradigm includes the location of a noun paradigm, a verb paradigm, and an adjective/adverb paradigm.

36. A method in accordance with claim 29, wherein said step of forming anintermediate baseform further comprises stripping a suffix character string from the end of the candidate word.

37. A method in accordance with claim 29, wherein said step of forming anintermediate baseform further comprises stripping a prefix character string from the front of the candidate word.

38. A method in accordance with claim 29, wherein said step of forming anintermediate baseform further comprises stripping an infix character string from the middle of the candidate word.

39. A programmed data processing method for generating an inflectional baseform from a candidate word using a first addressable table containing a list of lexical expressions and a second addressable table containing a list of paradigms, each paradigm having at least one inflectional transform that includes at least a first inflectional pattern and a second inflectional pattern, said method comprising the steps of:

locating in the first addressable table a first lexical expression substantiallyequivalent to the candidate word, identifying a first paradigm in the second addressable table as a function of the located first lexical expression, matching an inflectional transform in the identified first paradigm with the candidate word, forming an intermediate baseform by stripping a first character string from the candidate word, the first character string being defined by the first inflectional pattern included with the matched transform, and generating an inflectional baseform of the candidate word by adding a second character string to the formed intermediate baseform, the second character string being defined by the second inflectional pattern included with the matched inflectional transform.

40. A method in accordance with claim 39, further comprising the step of creating the inflected forms of the generated inflected baseform by applying the list of transforms in the identified first paradigm to the generated inflectional baseform.

41. A method in accordance with claim 38 wherein applying the list of transforms further comprises:

forming a second intermediate baseform by stripping a first inflectional character string from the generated inflectional baseform in accordance with a selected transform, and generating an inflected form of the generated inflectional baseform by adding a second inflectional character string to the second intermediate baseform.

42. A programmed data processing method for generating a derivationally related word from a candidate word using a first addressable table containing a list of derivational paradigms, each derivational paradigm having at least one derivational transform and an associated derivational pattern, said method comprising the steps of:

identifying a first derivational paradigm in the first addressable table as a function of the candidate word, matching a derivational pattern in the identified first derivational paradigm with the candidate word, forming an intermediate derivational baseform by stripping a first character string from the candidate word in accordance with a first derivational transform associated with the matched derivational pattern, and generating the derivational baseform of the candidate word by adding a second character string to the formed intermediate derivational baseform in accordance with the first derivational transform.

43. A method in accordance with claim 42, wherein said derivational pattern matching step further comprises identifying a derivational pattern that matches a suffix pattern in the candidate word.

44. A method in accordance with claim 42, wherein said derivational pattern matching step further comprises identifying a derivational pattern that matches a prefix pattern in the candidate word.

45. A method in accordance with claim 42, wherein said derivational pattern matching step further comprises identifying a derivational pattern that matches an infix pattern in the candidate word.

46. A method in accordance with claim 42 wherein said intermediate derivational baseform varies as a function of the derivational patterns contained within an identified derivational paradigm.

47. A method in accordance with claim 42 wherein said identifying step further comprises:

locating a first lexical expression substantially equivalent to the candidate word in a second table containing a list of lexical expressions, and identifying the first derivational paradigm in the first addressable table as a function of the located first lexical expression.

48. A method in accordance with claim 42, wherein said identifying step further comprises:
locating a first lexical expression substantially equivalent to the candidate word in a second table containing a list of lexical expressions, determining the inflectional baseform of the first lexical expression, and identifying the first derivational paradigm in the first addressable table as a function of the determined inflectional baseform.

49. A method in accordance with claim 42, further comprising the step of creating the derivational expansions of the generated derivational baseform by applying the list of derivational transforms in the identified first derivational paradigm to the generated derivational baseform.

50. A method in accordance with claim 46 wherein applying the list of derivational transforms further comprises:
forming a second intermediate derivational baseform by stripping a first derivational character string from the generated derivational baseform in accordance with a selected derivational transform, and generating a derivational expansion of the generated baseform by adding a second derivational character string to the second intermediate derivational baseform.

51. An apparatus for generating morphologically related forms of a candidate word, said apparatus comprising:

A) a digital memory element, including a first addressable table having a list of lexical expressions, a second addressable table having a list of paradigms, each paradigm having at least one morphological transform that includes at least a first morphological pattern and a second morphological pattern, and wherein each lexical expression listed in said first addressable table is associated with at least one paradigm listed in said second table, B) a digital data processing element coupled with said digital memory element, said digital data processor including first processing means for identifying a first paradigm for the candidate word by locating a lexical expression representative of the candidate word in said first addressable table, second processing means for matching a first morphological transform in the first paradigm with the candidate word, third processing means for forming an intermediate baseform by stripping a first character string from the candidate word, the first character string being defined by the first morphological pattern included with the matched transform, and fourth processing means for generating a morphological baseform of the candidate word by adding a second character string to the formed intermediate baseform, the second character string being defined by the second morphological pattern included with the matched transform.

52. An apparatus according to claim 51, wherein said second processing means further comprises:
identifying means for identifying a parameter of the candidate word, selecting means for selecting a morphological pattern for each transform, and comparing means for comparing the parameter of the candidate word with the selected morphological pattern such that a transform is matched with the candidate word.

53. An apparatus according to claim 52 wherein the second addressable table includes part-of-speech tags associated with each transform and the selected morphological pattern is the part-of-speech tag associated with each transform, the second processing means further comprising:
means for identifying a part-of-speech tag of the candidate word, and means for comparing the part-of-speech tag of the candidate word with the part-of-speech tag associated with each transform, such that the candidate word is matched with a transform having an equivalent part-of-speech tag.

54. An apparatus according to claim 51, wherein the third processing means forms an intermediate baseform that varies as a function of the first morphological transform matching the candidate word.

55. An apparatus according to claim 51, further comprising:
portmanteau paradigm means stored in the second addressable table for identifying the location of a plurality of paradigms, and fifth processing means for identifying a paradigm selected from the plurality of paradigms included in the portmanteau paradigm.

56. A data processing method for identifying noun phrases in a stream of words, the method comprising the steps of:
extracting a sequence of tokens from the stream, storing the sequence of tokens in a first memory element, determining the most probable part-of-speech tag and grammatical features for each token, and identifying parts of a noun phrase by inspecting the part-of-speech tags and thegrammatical features of a window of extracted tokens, the window of extracted tokens, including a selected candidate token and a first token preceding the selected candidate token and a second token following the selected candidate token.

57. A method in accordance with claim 56 wherein the identifying step further comprises:
identifying as a beginning of the noun phrase a candidate token having a part-of-speech tag functionally related to noun word forms, identifying as a middle of the noun phrase a candidate token when the candidate token has a part-of-speech tag functionally related to noun word forms and when the first token is an identified part of the noun phrase, and identifying as an end of the noun phrase a candidate token when the candidate token has a part-of-speech tag of noun and has a grammatical feature of lowercase and when the second token has a part-of-speech tag of noun and has a grammatical feature of uppercase.

58. A method in accordance with claim 57, wherein the identifying step further comprises identifying as an end of the noun phrase a candidate token having a part-of-speech tag selected from the group consisting of stop list nouns and adjectives.

59. A method in accordance with claim 57, wherein those tokens having a part-of-speech tag functionally related to noun word forms are selected from the group of part-of-speech tags consisting of: nouns, adjectives, ordinal numbers, cardinal numbers.

60. A data processing method for identifying noun phrases in a stream of words, the method comprising the steps of:
extracting a sequence of tokens from the stream, storing the sequence of tokens in a first memory element, determining the most probable part-of-speech tag and grammatical features for each token, identifying parts of a noun phrase by inspecting the part-of-speech tags of successive tokens, and iteratively checking agreement between a first identified part of the noun phrase and a second identified part of the noun phrase immediately following the first identified part in the stream of text.

61. A method in accordance with claim 60 wherein the iterative checking step further comprises:
monitoring gender agreement between the first identified part of the noun phrase and the second identified part of the noun phrase, and monitoring number agreement between the first identified part of the noun phrase and the second identified part of the noun phrase.

62. A method in accordance with claim 61 wherein the successive checking step further comprises monitoring case agreement between the first identified part of the noun phrase and the second identified part of the noun phrase.

63. A data processing method for identifying noun phrases in a stream of words, the method comprising the steps of:
extracting a sequence of tokens from the stream, storing the sequence of tokens in a first memory element, determining at least one part-of-speech tag for each token, disambiguating the at least one part-of-speech tag of an ambiguous token by inspecting the part-of-speech tags of a window of sequential tokens containing the ambiguous token, and identifying parts of a noun phrase by inspecting the part-of-speech tags of successive tokens.

64. A method in accordance with claim 63, wherein the determining step further comprises locating at least one lexical expression representative of an extracted token in a first addressable table, the first addressable table containing a list of lexical expressions with each lexical expression being associated with at least one part-of-speech tag.

65. A method in accordance with claim 64, wherein the determining step further comprises:

forming a target suffix from the last three characters of the extracted token, and locating a stored suffix matching the target suffix in a second addressable table, the second addressable table containing a list of stored suffixes with each stored suffix being associated with at least one part-of-speech tag.

66. A method in accordance with claim 63, further comprising the step of forming a window of sequential tokens, including the ambiguous token and a token immediately following the ambiguous token in the stream of words and at least two tokens immediately preceding the ambiguous tokens in the stream of words.

67. A method in accordance with claim 63, wherein the disambiguating stepfurther comprises:
identifying a primary part-of-speech tag of the ambiguous token, identifying a secondary part-of-speech tag of the ambiguous token, and promoting the secondary part-of-speech tag to the primary part-of-speech tag as a function of the part-of-speech tags of the window of sequential tokens.

68. A method in accordance with claim 67, wherein the promoting step further comprises promoting the secondary part-of-speech tag to the primary part-of-speech tag as a function of the identified primary part-of-speech tag of the ambiguous token and as a function of the identified secondary part-of-speech tag of the ambiguous token.

69. A method in accordance with claim 63, wherein the disambiguating stepfurther comprises generating a primary part-of-speech tag by operating upon the part-of-speech tags of the window of sequential tokens with a predetermined rule, and replacing the at least one part-of-speech tag of the ambiguous tokens with the generated primary part-of-speech tag, such that the primary part-of-speech tag is contextually accurate.

70. A method in accordance with claim 63, further comprising the step of truncating the identified noun phrase.

71. An apparatus for identifying noun phrases contained in a stream of words, the apparatus comprising:
tokenizing means for extracting a sequence of digital signals representative of a sequence of tokens contained in the stream, first addressable memory means containing a list of lexical expressions with each lexical expression being associated with a part-of-speech tag and grammatical features, data processing means coupled with the tokenizing means and with the first addressable memory means, the data processing means, including:
means for determining a part-of-speech tag and grammatical features for each token by identifying in the first addressable memory means at least one lexical expression representative of each token, and means for identifying parts of a noun phrase by inspecting the part-of-speech tags of a first window of tokens, and means for generating an output signal representative of the tokens forming the identified noun phrase.

72. An apparatus according to claim 71, wherein the means for identifying parts of a noun phrase further comprises:
means for identifying as a beginning of the noun phrase a token having a part-of-speech tag functionally related to noun word forms, means for identifying as a middle of the noun phrase a token having a part-of-speech tag functionally related to noun word forms, and having immediately followed a token identified as part of the noun phrase, and means for identifying as an end of the noun phrase a token having a part-of-speech tag of noun and having a grammatical feature of lowercase, and having immediately preceded an extracted tokens having a part-of-speech tag of noun and having a grammatical feature of uppercase.

73. An apparatus according to claim 71, wherein the data processing meansfurther comprises means for iteratively checking agreement between a first identified part of the noun phrase and a second identified part of the noun phrase immediately following the first identified part in the stream of text.

74. An apparatus according to claim 73, wherein the means for iterativelychecking agreement further comprises:
means for monitoring gender agreement between the first identified part of the noun phrase and the second identified part of the noun phrase, and means for monitoring number agreement between the first identified part of the noun phrase and the second identified part of the noun phrase.

75. An apparatus according to claim 74, wherein the means for iterativelychecking agreement further comprises means for monitoring case agreement between the first identified part of the noun phrase and the second identified part of the noun phrase.

76. An apparatus according to claim 71 further comprising a second addressable memory means coupled with the data processing means, the second addressable memory means containing a list of stored suffixes with each stored suffix being associated with at least one part-of-speech tag, and wherein the data processing means further comprises a means for determining a part-of-speech tag for each token by identifying in the second addressable memory means at least one stored suffix representative of the last three characters of each token.

77. An apparatus according to claim 71, wherein the data processing meansfurther comprises a means for disambiguating the part-of-speech tag of an ambiguous token by inspecting the part-of-speech tags of a second window of sequential token containing the ambiguous token.

78. An apparatus according to claim 77, further comprising a means for forming the second window of words such that the second window of words includes the ambiguous token and a token immediately following the ambiguous token in the stream of words and at least two tokens immediately preceding the ambiguous token in the stream of words.

79. An apparatus according to claim 77, wherein the disambiguating means further comprises:
means for identifying a primary part-of-speech tag of the ambiguous token, means for identifying at least one secondary part-of-speech tag of the ambiguous token, and means for promoting the at least one secondary part-of-speech tag to the primary part-of-speech tag as a function of the part-of-speech tags of the second window of sequential token.

80. An apparatus according to claim 79, wherein the promoting means further comprises a means for promoting the at least one second part-of-speech tag to the primary part-of-speech tag as a function of the identified primary part-of-speech tag of the ambiguous token and as a function of the identified at least one secondary part-of-speech tag of the ambiguous token.

81. An apparatus according to claim 77, wherein the disambiguating means further comprises:
means for generating a primary part-of-speech tag by operating upon the part-of-speech tags of the second window of sequential tokens with a predetermined rule, and means for replacing the at least one part-of-speech tag of the ambiguous token with the generated primary part-of-speech tag, such that the primary part-of-speech tag is contextually accurate.

82. An apparatus according to claim 71, wherein the data processing meansfurther comprises a means for truncating the identified noun phrase.