|Publication number||US5873061 A|
|Application number||US 08/642,018|
|Publication date||Feb 16, 1999|
|Filing date||May 2, 1996|
|Priority date||May 3, 1995|
|Also published as||CN1130688C, CN1153567A, DE69607913D1, DE69607913T2, EP0769184A1, EP0769184B1, WO1996035207A1|
|Publication number||08642018, 642018, US 5873061 A, US 5873061A, US-A-5873061, US5873061 A, US5873061A|
|Inventors||Reinhold Hab-Umbach, Peter Beyerlein, Eric Thelen|
|Original Assignee||U.S. Philips Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (15), Non-Patent Citations (4), Referenced by (40), Classifications (14), Legal Events (5)|
|External Links: USPTO, USPTO Assignment, Espacenet|
The invention relates to a method for representing a new word for speech recognition as based on a stored inventory of models of sub-word units. The method includes:
presenting a plurality of utterances that all intendedly conform to the new word, representing the utterances by a sequence of feature vectors and building a word model from the utterances.
In the environment of the invention, a word is understood to mean any linguistic entity of more or less restricted length, and thus may include brief sentences, personal and other names, and other items that warrant machine recognition upon their being presented in the form of speech. In particular, the invention addresses the problem of finding an acoustic representation, hereinafter also called transcription, of an unknown word as a sequence of sub-word units. This is effected through providing only a few sample utterances of the unknown word(s), and furthermore, an inventory of speaker-independent sub-word unit models.
A problem arises if a user wants to add one or more additional vocabulary words to a speaker-independent recognition system, by training the system with only a few utterances of the new word. Speaker-independent recognition is used when the number of envisaged speakers to use a particular type of system is relatively large and/or the system is relatively inexpensive. A typical example would be a speech actuated telephone device that normally may recognize the ten digits and a few standard terms, and which the user may train to recognize in addition such names or other labels that pertain to frequently called telephone extensions.
Another example would be useful with a speaker-independent speech recognition system that could have only a limited standard set of recognizable words, such as only twenty. This system then should have been trained on many different speakers. The system may now have to be extended with extra words, for which only a very limited number of training speakers, such as no more than three, is available, but for which extra words the same recognition robustness is required as for the original set.
Still another example would be with a grapheme-to-phoneme conversion, wherein a new word from keyboard entry is transcribed into an acoustic model. To improve reliability, the keyboard entry is then supplemented by acoustic entry of the same word. The parallel presentations again improve robustness, and under particular circumstances would also solve reliability problems due to orthographic errors, or due to the existence of two correct pronunciations of a single written word which then would have respective different meanings.
In particular, it is a requirement that the minimally necessary number of training utterances should remain low, such as no more than three, for nevertheless attaining a reliable performance at later recognizing. The problem also is generally restricted to systems that allow adding only a limited set of words, say, up to ten words. If the number of words added becomes too high, the transcription could render confusable results. On the other hand, the set of standard words may be either small or large.
In consequence, amongst other things, it is an object of the present invention to provide a method according to the preamble, wherein deviations between the various training utterances are rendered harmless, in that they would not lead to training the system on a spuriously misshaped utterance by the user person. In consequence, according to one of its aspects, the invention is characterized by
first, training a whole-word model that is independent of the models of the sub-word units, using said utterances, said whole-word model having a length corresponding to the average length of the utterances,
second, interpreting a first sequence of Markov states and associated probability densities of acoustic events of the whole-word model as a reference template represented by a second sequence of averaged feature vectors, and
third, recognizing said second sequence by matching to models of sub-word units in said inventory and storing a recognition result as a model of said utterances.
In contradistinction, a publication by L. R. Bahl et al, A Method for the Construction of Acoustic Markov Models for Words, IEEE Trans. on Speech and Audio Processing Vol I, No. 4, Oct. 1993, pp. 443-452 describes another method for producing a transcription on the basis of a plurality of utterances. According to the reference, each utterance results in its own transcription. Subsequently, all respective transcriptions are mapped onto all utterances, to find the optimum transcription, which also implies the best utterance. The present inventors have found that such procedure requires an excessive amount of processing when all acoustic distortions, time warpings, scale variations, and other non-conformance among the utterances is taken into account. In particular, when considering the envisaged use of the invention in a device of limited capability, such as for example, but not limited to, a handheld telephone, this would tie up the device for a long interval, during which it is unavailable for other functions. In fact, the known procedure needs a processing time that is proportional to the square of the number of utterances, whereas the present invention needs only a processing time that is linear in the number of utterances.
Other relevant references are EP-B1-285 222, corresponding to U.S. Pat. No. 5,058,166, on modelling, and EP-A3-533 260, corresponding to U.S. patent application Ser. No. 07/944,554 (PHD 91.138) on the execution of a tree search operation in speech recognition, all being assigned to the assignee of the present application, and all being incorporated by reference.
Furthermore, an advantageous embodiment of the present invention combines with the teachings of Bahl et al, in that the transcription based on the average utterance as attained in the present invention, would in a subsequent operation, compete with the various one-utterance transcriptions produced according to the method of Bahl et al. Under certain circumstances this brings about further improvement to the result acquired according to the foregoing.
The invention also relates to an device for implementing the method. Further advantageous aspects are recited in dependent claims.
These and other aspects and advantages of the invention will be described in detail hereinafter with reference to preferred embodiments, and in particular with reference to the appended Figures, that show:
FIG. 1 a system for effecting recognition;
FIG. 2 a system for effecting transcription;
FIG. 3 a flowchart for effecting transcription;
FIG. 4 a device for effecting transcription.
First, a brief formulation of the inventive method is given. The start of the procedure is the providing of a stored inventory of models of sub-word units. Now, in the method, a separate whole-word model with single-density emission probabilities is derived from the n given utterances, wherein n may be an appropriate number, such as three or five. This model can be interpreted as an `average utterance` Y obtained from the various utterances, where the observation vectors of this average utterance are the mean vectors of the state-specific emission probability densities. The transcription of the unknown word is given by the subword unit sequence which had most likely produced this average utterance Y:
Tavg =argmaxsεS P(Y.linevert split.s)
This subword unit sequence can in its turn be found by a continuous-speech recognition procedure. In particular, the selected transcription Tavg can be any one out of the set S of sub-word unit sequences.
In a somewhat broader aspect, the invention first uses the various utterances to train a whole-word model which is independent of the models of the sub-word units and has a length that is equal to the average length of the utterances. Next, the whole-word model is used to therefrom derive a sequence of Markov states and associated probability densities of acoustic events. By itself, such representation as a hidden Markov process has been in wide use in speech recognition technology. This representing also leads to the distinguishing of sub-word units in the utterance, so that the result of the interpretation can be used as a template consisting of a sequence of averaged feature vectors. In their turn, elements of this latter sequence can be matched to models of the sub-word units in the inventory, so that the recognition string is in fact a model of the averaged utterances, and can subsequently be used to recognize further utterances.
FIG. 1 shows a system for effecting recognition of spoken words. Herein, block 20 represents a spoken word to be recognized, wherein time runs from left to right in the Figure, and the contents of the block are the time-varying feature vectors (FV)s of the speech as sampled. First, the word is broken down into sub-word units shown as smaller stocks. Various such units may be phones, diphones, syllable parts, full syllables, or even single such feature vectors. It is not necessary that a sub-word unit has an acoustic relevance. This breaking down is effected on the basis of various ones of the above feature vectors, and may be done in a conventional manner. The separation is indicated by vertical lines, that need not be equidistant, as shown in the Figure, inasmuch as a mix of longer and shorter sub-word units is feasible. Next, starting from one end of the word in question, the subword units are matched to the items stored in the inventory 22 that has been set up from analyzing a large body of speech. By itself, such analyzing falls outside of the present invention, that takes the inventory as given. Each successive sub-word unit may produce a match with one or more of the inventory items.
Now, each match also brings about a score, which score can depend on the degree of non-correspondence between the two matched items, so that a lower score indicates a better match and vice versa. Furthermore, the score may depend on a language model that indicates the probability for a particular concatenation of words. Again, a low probability gives a high score and vice versa. According to one strategy, in this way a tree is built up from the various possible concatenations of recognized subwords that has branch-wise scoring. A pruning mechanism is provided for terminating and cutting down such branches of the tree that through their high score indicate a very low probability for the associated recognition. When at a certain point in time there are no parallel possibilities anymore for a different particular assignments, the word or word part in question has been recognized, unless the associated score indicates an unrealistical low probability. The latter leads to discarding of a part of the speech as unrecognized and resetting of the scoring. By way of symbolizing, various paths have been indicated, each dot symbolizing the assignment of a subword to an item from the associated row of storage system 22. It should be clear that the representation in the Figure is completely symbolic. Various other scoring and searching strategies are feasible.
FIG. 2 shows a system for effecting transcription, as based on three utterances (30, 32, 34) of (presumably) the same word by a user person. The user person may be the same one or not. First, the lengths of the utterances are determined, for example by counting the number of uniformly-spaced samples for each utterance, and an average thereof is calculated. Next, a whole-word model (36) that is independent of the sub-word model inventory, with the average length of the utterances is trained using the various utterances. This training may be effected by means of a standard speech recognition training method. Next, this training result is interpreted by means of a sequence of Markov states and associated probability densities of acoustic events of the whole word model. The result (38) is a reference template expressed as a series of averaged feature vectors. This acoustic model then operates as a template, i.e. as some kind of averaged utterance. This latter sequence is then recognized by matching to the various sub-word unit models in inventory 22, and the result (40) is a model for later utterances.
Finally, the transcription for the new word is derived by finding the sequence of sub-word units that best matches the sequence of averaged feature vectors, similarly to the procedure applied in FIG. 1. In contradistinction to FIG. 1 however, often no language model is applied now, because no a priori knowledge on the character of the utterance may be present: it may be a single word, or rather a composite term, such as a given name plus a surname of the person to be called in the telephone system considered supra. In another environment, the usage of a kind of such language model could however be used for possible sequences of subword units.
Also, an acoustic model is used for the recognizing, inasmuch as the acoustic model generally does not necessarily depend on the meaning of the underlying syllables. If the recognition score of the averaged utterances is excessively high, and therefore would represent an excessively low recognition probability, the user person is signalled, such as by an audio message or an indicator light, that the training has been unsuccessful, and that a new training must be undertaken. If the training has been successful, the recognized string of subword units is stored in the inventory, symbolized by block 40, whereupon the user is asked, also by an audio message or an indicator light, to specify the telephone extension number to be subsequently associated with the string 40 upon its later recognition. Such feature then subsequently allows for speech-controlled dialling, in which the transcription is used for recognizing the word to which the system has now been trained.
FIG. 3 shows a flowchart for effecting transcription. In block 60 the system is initialized, inter alia by providing the sub-word model inventory and resetting operational parameters. In block 62 the system detects whether an utterance was received. If negative, a waiting loop brings the system back to block 62 again. If positive, in block 64 the utterance is sampled, and stored after representing the samples as feature vectors. In block 66, the system detects whether the number of intendedly conforming utterances had attained an appropriate level, such as three. Another stratagem is to detect an `execute` command. If negative, the system again reverts to block 62 for expecting a further utterance. In block 68, by means of the various utterances received, a whole-word model of the utterances is trained, without recourse to the inventory, at a length that is the average of the various utterances. Subsequently, in block 70 the whole-word model, represented as a sequence of Markov states and associated probability densities of acoustic events is interpreted as a reference template given as sequence of averaged feature vectors. Subsequently, in block 72, the latter sequence is recognized by matching to the models of the sub-word units in the inventory. This allows to subsequently identify later received utterances as one of the utterances that have now been represented by a string of sub-word models. Next, in block 76 the system detects whether any new utterance should still be received for in its turn to be processed. If not yet ready, the system then reverts to block 62. If ready, or alternatively, by some `finished` command, the system exits by way of terminating block 78.
FIG. 4 shows a device for effecting transcription according to the invention. Block 80 is some kind of microphone or other device for speech input. Block 82 effects periodic sampling of the utterance received, and may comprise a speech detector for effecting sampling only when speech is effectively received. Block 84, from the stream of samples develops feature vectors, of speech characteristics. Consequently the device comprises various conventional elements, such as band filters, A/D converters, so that ultimately the feature vectors are a useful vehicle for subsequent processing. In block 86, the feature vector strings of the utterances are stored in the respective sub-blocks A, B, C. Through combined accessing of this storage, training block 88 trains a whole-word model with a length corresponding to the average length of the utterances. This also requires aligning of the various utterance representations. By itself, modelling to a single utterance has been disclosed in the cited art, supra. Next, block 90 interprets the whole-word model received from block 88 which model in particular comprises a sequence of Markov states and associated probability density distributions of acoustic events in the whole-word model. The result from the interpretation is a reference template comprising a sequence of averaged feature vectors. Finally, block 92 accesses the stored inventory of sub-word unit models in storage block 96 and so recognizes the template received from block 90 as matching to a particular sequence of the stored sub-word unit models. Interface (I/F) block 94 interfaces to various other devices according to necessity. For example, it may contain additional storage for storing the series of sub-word models from block 96 that pertain to the utterance for subsequent recognition, so that subsequent reception of a single utterance after translation to feature vectors may be matched to particular sub-word unit models directly.
Several additional but optional user interface devices have been shown in the Figure. Block 98 is a keyboard for telephone usage. Often, this will be a twelve-key device. Keying-in of a telephone extension number, after storing of the sequence of sub-word unit models pertaining to a subsequent utterance, will link this utterance to the keyed-in number and in consequence, will subsequently allow activating of this number under control of speech input of the associated label utterance. Block 100 is an alphabetic keyboard, such as alphanumerical QWERTY. Inputting of a text in block 102 results in grapheme-to-phoneme conversion, which by itself is a conventional feature. The phonemes used have been modelled in correspondence with the modelling of sub-word units in inventory storage 96. The output of converter 102 is presented to recognizer block 92. In this way, the grapheme-to-phoneme conversion may be strengthened by the speech-to-phoneme representation according to the invention, for example, in case of a foreign name that has a different spelling. Also, strengthening in the other direction is possible, in that a particular word could be difficult to recognize in speech.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4819271 *||Dec 16, 1987||Apr 4, 1989||International Business Machines Corporation||Constructing Markov model word baseforms from multiple utterances by concatenating model sequences for word segments|
|US5129001 *||Apr 25, 1990||Jul 7, 1992||International Business Machines Corporation||Method and apparatus for modeling words with multi-arc markov models|
|US5165007 *||Jun 12, 1989||Nov 17, 1992||International Business Machines Corporation||Feneme-based Markov models for words|
|US5170432 *||Sep 21, 1990||Dec 8, 1992||Alcatel N.V.||Method of speaker adaptive speech recognition|
|US5315689 *||Dec 21, 1992||May 24, 1994||Kabushiki Kaisha Toshiba||Speech recognition system having word-based and phoneme-based recognition means|
|US5390278 *||Oct 8, 1991||Feb 14, 1995||Bell Canada||Phoneme based speech recognition|
|US5429513 *||Feb 10, 1994||Jul 4, 1995||Diaz-Plaza; Ruth R.||Interactive teaching apparatus and method for teaching graphemes, grapheme names, phonemes, and phonetics|
|US5454062 *||Dec 31, 1992||Sep 26, 1995||Audio Navigation Systems, Inc.||Method for recognizing spoken words|
|US5502774 *||Sep 6, 1994||Mar 26, 1996||International Business Machines Corporation||Automatic recognition of a consistent message using multiple complimentary sources of information|
|US5502791 *||Sep 1, 1993||Mar 26, 1996||International Business Machines Corporation||Speech recognition by concatenating fenonic allophone hidden Markov models in parallel among subwords|
|US5528728 *||Jul 12, 1993||Jun 18, 1996||Kabushiki Kaisha Meidensha||Speaker independent speech recognition system and method using neural network and DTW matching technique|
|US5613034 *||Sep 26, 1994||Mar 18, 1997||U.S. Philips Corporation||Method and apparatus for recognizing spoken words in a speech signal|
|US5621859 *||Jan 19, 1994||Apr 15, 1997||Bbn Corporation||Single tree method for grammar directed, very large vocabulary speech recognizer|
|US5638487 *||Dec 30, 1994||Jun 10, 1997||Purespeech, Inc.||Automatic speech recognition|
|EP0344017A2 *||May 26, 1989||Nov 29, 1989||Kabushiki Kaisha Toshiba||Speech recognition system|
|1||"A Method for the Construction of Acoustic Markov Models for Words", IEEE Transactions on Speech and Audio Processing, vol. 1, No. 4, Oct. 1993 by L.R. Bahl et al.|
|2||"Automatic Recognition of Keywords in Unconstrained Speech Using Hidden Markov Models" IEEE Transactions on Acoustics, Speech, and Signals, Processing, vol. 38, No. 11 Nov. 1990, by Jay G. Wilpon et al.|
|3||*||A Method for the Construction of Acoustic Markov Models for Words , IEEE Transactions on Speech and Audio Processing, vol. 1, No. 4, Oct. 1993 by L.R. Bahl et al.|
|4||*||Automatic Recognition of Keywords in Unconstrained Speech Using Hidden Markov Models IEEE Transactions on Acoustics, Speech, and Signals, Processing, vol. 38, No. 11 Nov. 1990, by Jay G. Wilpon et al.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US6190173 *||Jun 2, 1998||Feb 20, 2001||Scientific Learning Corp.||Method and apparatus for training of auditory/visual discrimination using target and distractor phonemes/graphics|
|US6210166 *||Jun 16, 1998||Apr 3, 2001||Scientific Learning Corp.||Method for adaptively training humans to discriminate between frequency sweeps common in spoken language|
|US6224384 *||Jun 27, 2000||May 1, 2001||Scientific Learning Corp.||Method and apparatus for training of auditory/visual discrimination using target and distractor phonemes/graphemes|
|US6328569 *||Jun 26, 1998||Dec 11, 2001||Scientific Learning Corp.||Method for training of auditory/visual discrimination using target and foil phonemes/graphemes within an animated story|
|US6331115 *||Jun 30, 1998||Dec 18, 2001||Scientific Learning Corp.||Method for adaptive training of short term memory and auditory/visual discrimination within a computer game|
|US6334776 *||Jun 27, 2000||Jan 1, 2002||Scientific Learning Corporation||Method and apparatus for training of auditory/visual discrimination using target and distractor phonemes/graphemes|
|US6334777 *||Jun 24, 2000||Jan 1, 2002||Scientific Learning Corporation||Method for adaptively training humans to discriminate between frequency sweeps common in spoken language|
|US6358056 *||Jun 21, 2000||Mar 19, 2002||Scientific Learning Corporation||Method for adaptively training humans to discriminate between frequency sweeps common in spoken language|
|US6434547||Oct 28, 1999||Aug 13, 2002||Qenm.Com||Data capture and verification system|
|US6599129||Sep 24, 2001||Jul 29, 2003||Scientific Learning Corporation||Method for adaptive training of short term memory and auditory/visual discrimination within a computer game|
|US6801891||Nov 13, 2001||Oct 5, 2004||Canon Kabushiki Kaisha||Speech processing system|
|US6801893 *||Jun 22, 2000||Oct 5, 2004||International Business Machines Corporation||Method and apparatus for expanding the vocabulary of a speech system|
|US6873993||May 24, 2001||Mar 29, 2005||Canon Kabushiki Kaisha||Indexing method and apparatus|
|US6882970||Oct 25, 2000||Apr 19, 2005||Canon Kabushiki Kaisha||Language recognition using sequence frequency|
|US6961703 *||Sep 13, 2000||Nov 1, 2005||Itt Manufacturing Enterprises, Inc.||Method for speech processing involving whole-utterance modeling|
|US6973427 *||Dec 26, 2000||Dec 6, 2005||Microsoft Corporation||Method for adding phonetic descriptions to a speech recognition lexicon|
|US6990448||Aug 23, 2001||Jan 24, 2006||Canon Kabushiki Kaisha||Database annotation and retrieval including phoneme data|
|US7054812||Apr 25, 2001||May 30, 2006||Canon Kabushiki Kaisha||Database annotation and retrieval|
|US7212968||Oct 25, 2000||May 1, 2007||Canon Kabushiki Kaisha||Pattern matching method and apparatus|
|US7240003||Sep 28, 2001||Jul 3, 2007||Canon Kabushiki Kaisha||Database annotation and retrieval|
|US7257533||Sep 22, 2005||Aug 14, 2007||Canon Kabushiki Kaisha||Database searching and retrieval using phoneme and word lattice|
|US7260532||Nov 6, 2002||Aug 21, 2007||Canon Kabushiki Kaisha||Hidden Markov model generation apparatus and method with selection of number of states|
|US7289957 *||Oct 24, 2000||Oct 30, 2007||Siemens Aktiengesellschaft||Verifying a speaker using random combinations of speaker's previously-supplied syllable units|
|US7295980||Aug 31, 2006||Nov 13, 2007||Canon Kabushiki Kaisha||Pattern matching method and apparatus|
|US7310600||Oct 25, 2000||Dec 18, 2007||Canon Kabushiki Kaisha||Language recognition using a similarity measure|
|US7337116 *||Nov 5, 2001||Feb 26, 2008||Canon Kabushiki Kaisha||Speech processing system|
|US7676365||Apr 20, 2005||Mar 9, 2010||Microsoft Corporation||Method and apparatus for constructing and using syllable-like unit language models|
|US8756058 *||Feb 22, 2007||Jun 17, 2014||Nec Corporation||Speech recognition system, speech recognition result output method, and speech recognition result output program|
|US9436675 *||Feb 11, 2013||Sep 6, 2016||Continental Automotive Gmbh||Method and device for phonetizing data sets containing text|
|US9570069 *||Sep 9, 2014||Feb 14, 2017||Disney Enterprises, Inc.||Sectioned memory networks for online word-spotting in continuous speech|
|US20020022960 *||Apr 25, 2001||Feb 21, 2002||Charlesworth Jason Peter Andrew||Database annotation and retrieval|
|US20020052740 *||Aug 23, 2001||May 2, 2002||Charlesworth Jason Peter Andrew||Database annotation and retrieval|
|US20020082831 *||Dec 26, 2000||Jun 27, 2002||Mei-Yuh Hwang||Method for adding phonetic descriptions to a speech recognition lexicon|
|US20030163313 *||Nov 6, 2002||Aug 28, 2003||Canon Kabushiki Kaisha||Model generation apparatus and methods|
|US20050187769 *||Apr 20, 2005||Aug 25, 2005||Microsoft Corporation||Method and apparatus for constructing and using syllable-like unit language models|
|US20070150275 *||Aug 31, 2006||Jun 28, 2007||Canon Kabushiki Kaisha||Pattern matching method and apparatus|
|US20100217598 *||Feb 22, 2007||Aug 26, 2010||Nec Corporation||Speech recognition system, speech recognition result output method, and speech recognition result output program|
|US20150302001 *||Feb 11, 2013||Oct 22, 2015||Continental Automotive Gmbh||Method and device for phonetizing data sets containing text|
|US20160071515 *||Sep 9, 2014||Mar 10, 2016||Disney Enterprises, Inc.||Sectioned memory networks for online word-spotting in continuous speech|
|US20170032779 *||Apr 25, 2016||Feb 2, 2017||Samsung Electronics Co., Ltd.||Method and device for updating language model and performing speech recognition based on language model|
|U.S. Classification||704/254, 704/E15.028, 704/252, 704/251|
|International Classification||G10L15/06, G10L15/14, G10L15/02, G10L15/10, G10L15/08|
|Cooperative Classification||G10L15/02, G10L15/144, G10L15/142, G10L15/06|
|Jul 16, 1996||AS||Assignment|
Owner name: U.S. PHILIPS CORPORATION, NEW YORK
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HAB-UMBACH, REINHOLD;BEYERLEIN, PETER;THELEN, ERIC;REEL/FRAME:008330/0279;SIGNING DATES FROM 19960605 TO 19960625
|Jul 24, 2002||FPAY||Fee payment|
Year of fee payment: 4
|Sep 6, 2006||REMI||Maintenance fee reminder mailed|
|Feb 16, 2007||LAPS||Lapse for failure to pay maintenance fees|
|Apr 17, 2007||FP||Expired due to failure to pay maintenance fee|
Effective date: 20070216