CA2486128C - System and method for using meta-data dependent language modeling for automatic speech recognition - Google Patents

System and method for using meta-data dependent language modeling for automatic speech recognition Download PDF

Info

Publication number
CA2486128C
CA2486128C CA2486128A CA2486128A CA2486128C CA 2486128 C CA2486128 C CA 2486128C CA 2486128 A CA2486128 A CA 2486128A CA 2486128 A CA2486128 A CA 2486128A CA 2486128 C CA2486128 C CA 2486128C
Authority
CA
Canada
Prior art keywords
data
meta
tree
speech
language model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
CA2486128A
Other languages
French (fr)
Other versions
CA2486128A1 (en
Inventor
Michiel A. U. Bacchiani
Brian E. Roark
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nuance Communications Inc
Original Assignee
AT&T Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by AT&T Corp filed Critical AT&T Corp
Publication of CA2486128A1 publication Critical patent/CA2486128A1/en
Application granted granted Critical
Publication of CA2486128C publication Critical patent/CA2486128C/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/08Speech classification or search
    • G10L15/18Speech classification or search using natural language modelling
    • G10L15/183Speech classification or search using natural language modelling using context dependencies, e.g. language models
    • G10L15/19Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
    • G10L15/197Probabilistic grammars, e.g. word n-grams
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/23Clustering techniques
    • G06F18/232Non-hierarchical techniques
    • G06F18/2321Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/08Speech classification or search
    • G10L15/18Speech classification or search using natural language modelling
    • G10L15/183Speech classification or search using natural language modelling using context dependencies, e.g. language models
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/22Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G10L2015/226Procedures used during a speech recognition process, e.g. man-machine dialogue using non-speech characteristics

Abstract

Disclosed are systems and methods for providing a spoken dialog system using meta-data to build language models to improve speech processing. Meta-data is generally defined as data outside received speech; for example, meta-data may be a customer profile having a name, address and purchase history of a caller to a spoken dialog system. The method comprises building tree clusters from meta-data and estimating a language model using the built tree clusters. The language model may be used by various modules in the spoken dialog system, such as the automatic speech recognition module and/or the dialog management module.
Building the tree clusters from the meta-data may involve generating projections from the meta-data and further may comprise computing counts as a result of unigram tree clustering and then building both unigram trees and higher-order trees from the meta-data as well as computing node distances within the built trees that are used for estimating the language model.

Description

SYSTEM AND METHOD FOR USING META-DATA DEPENDENT LANGUAGE
MODELING FOR AUTOMATIC SPEECH RECOGNITION
BACKGROUND OF THE INVENTION

1. Field of the Invention
[0002] The present invention relates to speech recognition and more specifically to using meta-data dependent language modeling for speech recognition.

2. Introduction
[0003] Speech recognition has gained momentum as a viable technology over time. As computer processors improve and as the language models and recognition algorithms improve, the recognition accuracy for systems that perform speech recognition also improve.
One way used by those of skill in the art to improve automatic speech recognition (ASR) is to use language models that have many recognition examples and samples within a certain "domain." For example, if a speech recognition engine is to recognize speech from a person who will be talking about making a flight reservation, then there is a known world of words and phrases that are expected to be spoken by the user. The speech recognition engine and the experts designing the system can make use of commonly used words for that domain to improve recognition.
[0004] A language model used by a speech recognition engine needs to be "trained" to recognize speech. This typically involves experts recording average or expected discussions between a person and the system, labeling that data, and creating the language model. For example, the expert may record a person saying "I want to fly from LA to New York". The person would then label the terms used "LA", "New York" and so forth so that the system, I

when faced with similar sounding words in live use, will associate the appropriate sound with the appropriate word. With enough training data that covers many scenarios, the language model is ready for live use to listen to real users.
[0005] There are still many challenges with providing a speech recognition application that approaches acceptable recognition accuracy such that its use will become widespread. While many technologies exist to help improve accuracy, there is a need in the art to consider extending the information used by a recognition engine beyond what is presently available to further improve the technology.

SUMMARY OF THE INVENTION
[0006] Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The features and advantages of the invention may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. These and other features of the present invention will become more fully apparent from the following description and appended claims, or may be learned by the practice of the invention as set forth herein.
[0007] The present invention provides for systems, methods and computer-readable media for using meta-data to condition language modeling in spoken dialog systems or any other application for speech recognition. Disclosed are systems and methods for providing a spoken dialog system using meta-data to build language models to improve speech processing. Meta-data is generally defined as data outside received speech;
for example, meta-data may be a customer profile having a name, address and purchase history of a caller to a spoken dialog system.
[0008] The method embodiment of the invention comprises building tree clusters from meta-data and estimating a language model using the built tree clusters. The language model may be used by various modules in the spoken dialog system, such as the automatic speech recognition module and/or the dialog management module. Building the tree clusters from the meta-data may involve generating projections from the meta-data and further may comprise computing counts from the meta-data and building both unigram trees and higher-order trees from the meta-data as well as computing node distances within the built trees that are used for estimating the language model.
[0009] Building the unigram tree and higher-order trees and generating projections from the meta-data provide a process where meta-data containing many parameters may be used for estimating language models for automatic speech recognition.

BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to describe the manner in which the above-recited and other advantages and features of the invention can be obtained, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered to be limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
[0011] FIG. I illustrates the modules used in a spoken dialog system;
[0012] FIG. 2 illustrates a method embodiment of the invention;
[0013] FIG. 3 illustrates recognition performance on the SSNIFR test set of data; and
[0014] FIG. 4 illustrates recognition performance on the ScanMail test set.

DETAILED DESCRIPTION OF THE INVENTION
[0015] Various embodiments of the invention are discussed in detail below.
While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without parting from the spirit and scope of the invention.
[0016] Throughout this description, there terms ASR and speech recognition engine will be used. An ASR module or speech recognition engine typically is a component in a larger spoken dialog system that includes, as is shown in FIG. 1, an ASR module 102, a spoken language understanding (SLU) module 104, a dialog manager 106, a language generation module 108 and a text-to-speech module 110. These modules will recognize the user's speech utterances, identify (or understand) the meaning of the text, determine an appropriate response in the spoken dialog, generate the text of the appropriate response and from that text, generate the audible "speech" from the spoken dialog system that the user then hears. In this manner, the user can carry on a natural language dialog with the system.
Those of skill in the art will understand the programming languages and means for generating and training the ASR module 102 or any of the other modules in the spoken dialog system.
Further, these modules may operate independent of a full dialog system. For example, a computing device such as a smartphone (or any computing device having a phone capability) may have an ASR
module wherein the user may say "call mom" and the smartphone acts on the instruction without a "spoken dialog." Similarly, each of these components may use a language model generated by the method disclosed herein and have independent application depending on the computing device with which they are associated.
[0017] Those of skill in the art know of the various approaches and technologies that may be employed to deploy such a dialog system. This invention has application to many if not all of these modules in the dialog system. The invention will mainly be described with reference to the ASR module but the basic principles are also applicable to improved text-to-speech, dialog management, or any of the processes in any module associated with speech recognition. Furthermore, the principles of the invention may be applied to any use of ASR
beyond any specific spoken dialog system.
[0018] As introduced above, speech recognition is improved by focusing the language models on words and phrases within a known domain. In addition to knowing or expecting that certain words may be spoken, those designing speech recognition engines may also use other information to help modify and adjust speech recognition algorithms to improve accuracy. For example, if the engine knows that the speaker is a man or woman, or if the engine knows that the user is a teenager, for example. This kind of information can assist the engine in making appropriate assumptions about speech patterns, dialect, and so forth that aid in improving the recognition accuracy.
[0019] For many ASR applications, information in addition to the speech that is to be recognized is available. For example, a company receiving speech from a customer might have a database of information and customer profile information associated with that customer. This additional information may be referred to as meta-data.
Although this meta-data contains information useful for the recognition task (for example geographical location, previous purchases and product preferences) it cannot be used directly in the language model as the information is too fragmented providing little training data for each unique customer profile.
[0020] ASR can occur where meta-data is available, such as when access to a customer database record is available during a call. Conditioning the ASR models directly on this information to improve the transcription accuracy is hampered because, generally, the meta-data takes on many values and a training corpus will have little data for each meta-data condition. For example, taking the flight reservation example above, using a person's name and address that would be found in their record database may not assist much in recognizing natural language speech during a dialog.
[0021] The present invention, however, presents a system and method of constructing language models conditioned on such meta-data. The method preferably uses tree-based clustering of the training data to automatically derive meta-data projections, useful as language model conditioning contexts. The method has been tested on a multiple domain voicemail transcription task. The performance of an adapted system aware of the domain shift has been compared to a system that only has meta-data to infer that fact. The meta-data that may be used include the callerlD strings associated with the voicemail messages. The meta-data adapted system matched the performance of the system adapted using the domain knowledge explicitly.
[0022] Adaptation of ASR models to the operating environment and context has been shown repeatedly to be of enormous benefit. This is particularly true of acoustic models, where many robustness issues that arise when dealing with variations in channel and speaker characteristics have been successfully addressed by use of unsupervised self-adaptation and normalization techniques. Adaptation has also been shown to be effective for language modeling. In light of the popularity of the Maximum Entropy (ME) algorithm, some language model adaptation efforts have looked at using long distance history features, such as triggers or lexico-syntactic features, and external features such as topic.
Most of the adaptation efforts have focused on adapting on the test data itself (unsupervised self-adaptation) or on a sample representative of the context of interest.
[0023] Some ASR systems use contextual information, associated with but external to the speech signal such as gender or topic to shape the model distributions. This external information can be either explicitly given or inferred implicitly from the speech signal. For example, a gender dependent system might use the test data likelihood as modeled by gender dependent models to infer the speaker gender. Information external to the speech signal will be referred to herein, regardless of whether it was given or inferred, as meta-data.
[0024] Many applications have a large amount of meta-data available, e.g. from databases associated with the speech to be recognized. For example, a company receiving a call from a customer might have access to a database record of that customer, revealing their geographical location, product preferences, purchasing history, address, web-surfing history, and so forth. Until the introduction of the present invention, the models used in ASR
systems have not used this type of information.
[0025] If the meta-data is sparse in the sense that it can take on few values, meta-data dependent models can be trained directly. An example of a "sparse" meta-data record may be one that includes only gender information, i.e., either male or female.
Another approach is to adapt a meta-data independent model using the data corresponding to a specific meta-data value. For example, one approach of using sparse gender meta-data is to build a separate model on the gender-dependent data sets (or data sets based on any type of data within the meta-data). Another approach is to adapt a gender (or other type of data) independent model built on all data towards the gender of interest by using the gender dependent data. For example, increase the weight of the female data compared to the male data to obtain a model that is skewed towards female speech but still benefits from the larger data pool that does include male data. This weighing or skewing of the type of data within the meta-data can therefore provide improvement in speech recognition.
[0026] In many scenarios the meta-data space will be large, i.e. it can take on many values (for example, name, gender, address, language, dialect, age, etc). As a result, there will be little data available for a specific meta-data value, making direct conditioning infeasible.
However, if the highly fragmented meta-data can be projected to a small number of conditioning contexts, it may be used to condition the ASR models. Take the gender example: it might not be possible to create speaker-identity dependent models but if a projection can be found that provides clusters representing gender, conditioning on the speaker identity will be beneficial.
[0027] The present invention is especially useful to construct meta-data conditional language models suitable for highly fragmented meta-data. The invention may be embodied as a spoken dialog system, a spoken dialog module such as an ASR module, a spoken dialog module training process, a method for generating a conditional language model and/or a computer-readable medium. In one aspect of the invention, the method uses divisive likelihood-based clustering techniques to find projections of the meta-data, which is a good choice in the face of such fragmentation. A meta-data conditional model is then obtained by merging the meta-data conditional counts, weighted based on the given meta-data value.
Experimental results for a voicemail transcription task show promise for this approach.
[0028] The language model estimation algorithm comprises preferably two parts.
FIG. 2 illustrates generally the steps for generating a language model using a tree clustering approach. The first part is a tree clustering step which may be performed at training time.
The clustering result produced by the first step is then preferably used at test time to estimate the language model used in recognition or in another spoken dialog module.
Both steps are described in detail below.
[0029] The first part of the process relates to identifying projections based on the meta-data.
An example identification method is tree clustering which is more fully explained next. For an n-gram model of order k, the tree-clustering step involves estimating models for each order <_ k beginning with the unigram model. The method used for the unigram tree is different from this higher order trees, and will be presented first. First, a process of computing counts (202) is performed followed by a process of building a unigram tree (204).
The output of the unigram clustering is used for building the higher-order trees (206), which are built in two steps to control the greediness of the algorithm, allowing different meta-data dependencies for different histories.
[0030] The tree clustering algorithm groups history and meta-data dependent n-gram count distributions. As used herein, "h" is a history of words. For example, for the word sequence "flight to LA", the words "flight to" are an observed history for the word `LA" as it was seen preceding that word. Counting not just the frequency of words "w" but the frequency of "w"
in different histories "h" fragments the counts. For the example, the system may see LA in various histories "flight to LA", "moving to LA", "is from LA" etc. The meta-data label x indicates different meta-data conditions. For example, x could be first name, gender and age.
Making counts not only of history but also of meta-data dependent further fragments the counts. In another example, assume that one now has different counts for "flight to LA" as spoken by Mary, female, age 33 and for "flight to LA" as spoken by Rob, male, age 22.
Clearly the number of unique context dependent counts C(wjh, x) becomes very large and as a result frequencies become poor estimators due to data sparsity.
[0031] For a given vocabulary V let w E V denote words, and h E V' denote a history, or conditioning sequence of k words. As mentioned above, x represents the set of possible meta-data values such as name, gender, language spoken, age, etc. Let C(w I h, x) denote the raw count distribution across words w E V following history h E V *, in meta-data condition x E V. Then, C(w I h) _ ~XEx C(w I h, x).
[0032] The clustering algorithm preferably uses a likelihood objective.
Likelihoods are computed based on smoothed probability distribution estimates to account for unobserved events. The smoothing technique preferably uses Good-Turing discounting and Katz backoff estimation to provide probability estimates from counts, in the standard way.
See, I. Good, "The population frequencies of species and the estimation of population parameters,"
Biometrica V, vol. 40, no. 3,4, pp. 237-264, 1953; and S. Katz, "Estimation of probabilities from sparse data fro the language model component of a speech recognizer,"
IEEE
Transactions on Acoustic, Speech, and Signal Processing, vol. 35, no. 3, pp.
400-401, 1987.

It is understood that other approaches for clustering and smoothing may also be used. Let d(f) denote the discounting fraction for frequency f, such that 0 < d(f) < 1, where d(f) is calculated once using, for example, Katz backoff estimation on the sample of meta-data independent counts. For frequencies 6 and higher, d(f) may equal 1.
[0033] The meta-data dependent count distributions are clustered by building trees in increasing Markov order, starting with unigrams. The results of the unigram tree clustering (the calculated counts) are used both for subsequent unigram probability estimation as well as for building the higher order trees.
[0034] Since for the unigram count distributions h E V (i.e. there is no word history), the count distributions used in building this tree are C(w I x) for all x c X. The tree is built by greedily splitting the leaf that provides the largest likelihood gain. Leaf splits are evaluated using, for example, the Chou algorithm which will be described here briefly.
The evaluation of the merit of a split is an iterative process. See, P. Chou, "Optimal partitioning for classification and regression trees," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, no. 4, pp. 340-354, 1991. Let A c X

denote the set of meta-data values that are assigned to the leaf, the split gain is evaluated as follows:

1. Estimate a parent probability distribution C(wIA)=JC(wIx) xEA (1) C(A) = I C(W I A) XEV (2) d(C(w I A))C(w I A) if C(w I A) > 0 P(w A) = C(A) otherwise (3) choosing 6 to ensure proper normalization and estimate the parent data log likelihood LP = Y C(w I A) log(P(w I A)) WEV (4) set m = 1.

2. Randomly partition set A into two disjoint subsets LO and RO, i.e. A = L0 U
R0 and L0nR0-Q).

3. For Q in {Lm-i , R._, } compute, (~ C(w l x) C(w I Q) = XEQ (5) = YC(w I Q) C(Q) _ wÃV (6) d(C(w I Q))C(w I Q) if C(w I Q) > 0 P(w I Q) = C(Q) aP(w I A) otherwise (7) where a is chosen to ensure a normalized distribution.

4. Set L,, = R,,, = 0. For each member x E A, evaluate L, (x) _ I C(w I x) log(P(w I Lm-, )) WEV (8) Lr(x) YC(w I x) log(P(w I R,,,-,)) and wEV (9) and assign x to Lm if Li(x) > Lr(x), to Rm otherwise.
5. Compute total likelihood L(A) _ L, (x) + E Lr (x) 7 xEL,,, xER. (10) If in > 1 and L(A) = B goto 7.

6. Set in = in + 1. Set B = L(A). Go to 3.
7. SetG=B - Lp. SetL=Lm, R=Rm.
[0035] After termination of this iterative process, a partition of A into two subsets L and R is defined providing a likelihood gain G associated with that partition.
[0036] The unigram tree is built by iteratively splitting the leaf with the largest likelihood gain and evaluating the likelihood gain of the new leaves created by those splits. Once the likelihood gain of the best leaf split falls below a given threshold Tõnigram, the unigram tree building step terminates. The unigram tree will then define N leaf sets, grouping the observed meta-data contexts X. The sets defined in this N-way partition will be denoted as S={s1, s2, ..., SN I.
[0037] Like the unigram tree, the higher order (h E V', k > 0) trees define a partition of the history and meta-data dependent n-gram count distributions. The count distributions used in building these trees are C (w I h, x) for all x E X and h E V". The higher order trees, like the unigram tree, are built by greedily splitting the leaf that provides the largest likelihood gain. Unlike the unigram tree, the higher order trees are built in two stages.
[0038] In the first stage, the count distributions are partitioned allowing only history dependent splits. In other words, only subsets of V1 are considered, no subsets of X are allowed. As a result, all meta-data dependent occurrences of a history h are forced to fall in the same leaf. This clustering stage again uses the Chou algorithm for the evaluation of the likelihood gains from splitting leaves. Once this algorithm terminates based on a given likelihood gain threshold ThIstory, the leaves of the tree will contain groups of histories.
These leaves or nodes are referred to as the history nodes of the tree.
[0039] In the second stage, the leaves are split further but now only via domain splits. In other words, only partitions of X are allowed at this stage. The gain from splitting leaves is evaluated similarly to the Chou algorithm, however instead of repartitioning based on likelihood (step 4 of the Chou algorithm), only partitions based on class memberships are considered. The classes that are considered are those defined by the unigram tree leaves, i.e.
S. The gains for partitioning on each class in S are computed and the partition that results in the largest likelihood increase is used if that leaf is split. Again, a given likelihood threshold Tngram determines when the tree growing algorithm terminates.
[0040] After termination of the tree growing algorithm, each history h E V"-' is assigned to one or more leaf nodes. The number of leaf nodes the history appears in determines the number of distinct meta-data projections for that history.
[0041] The final step in the tree clustering stage of the algorithm is a distance computation (208). Let J denote the set of nodes containing the root node of the unigram tree and the history nodes of the higher order trees. For each node k e J, let Uk denote the set of leaves that are descendants of k. Then for each k E J:
[0042] 1. For each node m E Uk compute P(w I m) as in step 1 of the Chou algorithm, i.e.
estimate the leaf conditional distribution based on the subset of histories and meta-data values that were assigned to that leaf.
[0043] 2. For each node m E Uk compute a distance D(m, p) to all p E Uk. The distance is defined as the Kullback-Liebler distance D(m,p)= P(wIm)log PP(wIm) (11) WELD ` (4' 1 P)
[0044] The language model estimation step is performed preferably at test time using the trees designed in the training phase. It assumes a meta-data condition v is given for the test data that is to be recognized.
[0045] Finally, using the data obtained above, the language model is estimated (210). The language model is constructed by estimating history dependent distributions for all histories h seen in the training data. Together with the meta-data value v E X, each history h identifies a leaf node m E Uk, for some k E J. The set of nodes p E Uk define the various meta-data projections for history h. Given the meta-data value v, in is identified as the projection applicable to the current test data. Estimating a language model on the subset of the training data represented by in will likely produce a less accurate model due to sparse data resulting from the reduction of the training set size. To prevent this, all data from all meta-data projections p E Uk are used, but weighted based on the distance from in to p, i.e. D(m, p).
[0046] The history dependent distribution is estimated in a two step process.
[0047] 1. Estimate the discounting factors for the observed words w based on the unweighted, summed counts C(w) _ YC(w I q) (12) gEUA

z(w) = d(C(w)) (13)
[0048] 2. Estimate the history dependent distribution based on weighted, discounted counts P(w I h) z z(w)C(w j q) aEUA g if C(W) > 0 =
l gEUA l weV ZgC(W I q) yP(,A, h') otherwise (14) where h' is the history h without its initial word. The value of y is chosen to ensure proper normalization.
[0049] The count scales 'rq are derived from the distances D(m, p) as 1 if 1 <(D
iq = D(m, q) D(m, q) cD otherwise (15) where 1 is a parameter of the algorithm. One can interpret this parameter as setting the maximum distance for a sibling node to be considered "in-domain", and hence receive the same weight as the node in itself. If (D = 5, then another node must be within KL-distance 0.2 to be considered in-domain.
[0050] The invention was evaluated in a controlled experiment. For this experiment, a scenario was set up using two corpora. The ScanMail corpus is a general voicemail corpus.
For more information on the ScanMail application, see M. Bacchiani, "Automatic transcription of voicemail at AT&T," in Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2001.

The SSNIFR corpus (the messages from the target domain) consists of voicemail message received at a network center customer care voicemail box. Although both corpora consist of voicemail messages, the language differs significantly. Adaptation of a language model built on ScanMail messages to the SSNIFR domain provides as much as a 7% accuracy improvement.
[0051] The experiment is controlled in the sense that an accuracy improvement can be obtained if it is known that the SSNIFR corpus differs in distributional characteristics to the ScanMail corpus. The question is how much of this accuracy improvement can be obtained if the data partition is not known and only a highly fragmented meta-data variable related to that partition is given. The meta-data used in this experiment are callerlD
strings provided by the telephone switch for every incoming message. For a subset of both corpora, this callerlD
information is available. Using these subsets training and test sets were constructed for both the ScanMail and SSNIFR domains using random partitions. Statistics for these sets are given in table 1. Out of the 169 CallerID strings associated with the ScanMail test messages, 136 were seen in the training set. Out of the 120 CallerlD strings associated with the SSNIFR test messages, 78 were seen in the training set. There were no CallerID strings overlapping between the ScanMail and SSNIFR corpora.

Corpus Set Message Unique Word Count Speech Name Type Count CallerID Duration Strings minutes ScanMail Train 6489 2072 803838 4302 ScanMail Test 169 149 21736 114 SSNIFR Train 300 183 33386 195 SSNIFR Test 120 95 13164 79 Table 1 Corpus statistics
[0052] For the experiments, three conventional Katz backoff trigram models were trained on different data sets. The model trained on the SSNIFR training set will be referred to as SSNIFR, the one trained on the ScanMail training set will be referred to as SM. The model referred to as MRG was obtained by estimation on the combined SSNIFR and SM
training sets. The fourth model, referred to as MAP was obtained using the weighted count merging approach. The weight parameter was set to 5 which was empirically determined to be the optimal value. The results obtained using these models give the performance bounds using the domain information assuming this is known.
[0053] Two meta-data dependent models were trained using the proposed algorithm on the combined SSNIFR and ScanMail training sets. As meta-data the callerlD strings were used, hence the size of the set X was 2255 corresponding to the number of unique callerlD strings seen in the combined corpora. Both meta-data dependent models used the same threshold values Thistory = 1000 and Tngram = 0 and used the same scaling parameter c =
5. The likelihood threshold Tunigram, however, was set to 1500 for the model referred to as SpkrS and 1000 for the model referred to as SpkrL. These models represent the scenario where the domain shift is not known but a highly fragmented meta-data variable is available possibly revealing that fact.
[0054] At test time, the meta-data dependent language models were created on a per message basis, using the callerlD strings associated with the test messages. If a callerlD string was not seen in the training data, the system would default to the MRG model. In the absence of any knowledge about the heterogeneity of the corpus, this would be the most appropriate model.
[0055] The run-time vs. accuracy curves showing the performance of all the models on the SSNIFR test set are shown as graph 300 in figure 3. The performance of the SM, MRG, SpkrS and SpkrL models on the ScanMail test set are shown as graph 400 in Figure 4. It can be seen that on the SSNIFR test data, the MAP model gives an additional gain over the MRG model. Both models outperform the SSNIFR model. It also shows that the SpkrS model matches the MAP model performance. The SpkrL model does not perform as well as the SpkrS model but still provides a performance gain over the MRG
model.
[0056] On the ScanMail test data, both the SM and MRG model give equal performance. The SpkrS and SpkrL models perform as well or show a small improvement over the baseline models.
[0057] The experimental results show that the algorithm succeeds in finding meta-data projections useful for conditioning the language model. When provided only with very fragmented meta-data and no explicit knowledge about a domain shift for a small subset of the data, the algorithm successfully created a model that matched the performance of a model adapted using the domain shift knowledge.
[0058] The weight estimation based on the inverse Kullback-Liebler distance computation appears to provide a reasonable estimate of the proximity of different meta-data projections.
This is shown by the small performance difference between SpkrS and SpkrL. In the SpkrS
model, 182 out of the 183 SSNIFR CallerID strings were in a single leaf of the unigram tree, along with some ScanMail messages. This means the model effectively found the SSNIFR
subset from the rest of the data. In the SpkrL model, due to the lower Tunigram parameter, that leaf was split further into 3 subsets. The resulting model performed almost as well since the distance between these subsets was found to be small and hence the SSNIFR data was weighted approximately equally even though it was partitioned into multiple subsets.
[0059] The fact that the meta-data dependency did also provide a small accuracy improvement on the ScanMail data, where little or no gain from meta-data conditioning was expected reinforces the view that the distance is appropriate.
[0060] That the SpkrS model matches the MAP performance shows that the algorithm is using the leaf distances to appropriately weight the contributions of various meta-data projections. Note that only 78 out of the 120 test messages used a meta-dependent model as the other messages defaulted to the MRG model due to a novel CallerlD string.
[0061] Various aspects of the algorithm can possibly benefit from further investigation. First, the set definitions inferred from the unigram tree might not be the optimal choice for use in the higher order trees. Second, other distance and weight relationships can be considered that might give better performance. Besides algorithmic improvements, one can envision many empirical trials testing the conditioning benefit of various meta-data sources.
[0062] Embodiments within the scope of the present invention may also include computer-readable media for carrying or having computer-executable instructions or data structures stored thereon. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM
or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code means in the form of computer-executable instructions or data structures. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or combination thereof) to a computer, the computer properly views the connection as a computer-readable medium. Thus, any such connection is properly termed a computer-readable medium. Combinations of the above should also be included within the scope of the computer-readable media.
[0063] Computer-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Computer-executable instructions also include program modules that are executed by computers in stand-alone or network environments. Generally, program modules include routines, programs, objects, components, and data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.
[0064] Those of skill in the art will appreciate that other embodiments of the invention may be practiced in network computing environments with many types of computer system configurations, including personal computers, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. Embodiments may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination thereof) through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
[0065] Although the above description may contain specific details, they should not be construed as limiting the claims in any way. Other configurations of the described embodiments of the invention are part of the scope of this invention. For example, meta-data may be clustered using any other algorithm or other objectives. Further, a meta-data dependent language model may be constructed using other adaptation approaches then those outlined above. Also, while FIG. 1 illustrates a spoken dialog system, there are many applications that may use an ASR module, such as appliances or computing devices that have an ASR module to listen for audible input to take an action but do not engage in a dialog with the speaker. Accordingly, the appended claims and their legal equivalents should only define the invention, rather than any specific examples given.

Claims (27)

1. A method of generating a language model using non-speech meta-data, the method comprising:
applying a tree growing algorithm to highly fragmented non-speech meta-data related to a caller to a spoken dialog system but does not describe physical characteristics of the caller;
identifying a first quantity of leaf nodes in which a history appears as a result of applying the tree growing algorithm;
identifying a second quantity of projections based on the highly fragmented non-speech meta-data wherein the first quantity equals the second quantity; and estimating a conditional meta-data dependent language model using the identified projections.
2. The method of claim 1, wherein identifying the second quantity of projections based on the highly fragmented non-speech meta-data further comprises using divisive likelihood-based techniques to identify the non-speech meta-data projections.
3. The method of claim 2, wherein identifying the second quantity of projections based on the highly fragmented non-speech meta-data further comprises using tree-based clustering to identify the highly fragmented non-speech meta-data projections.
4. The method of claim 1, wherein the conditional meta-data dependent language model is estimated by merging non-speech meta-data conditional counts that are weighted based on a given non-speech meta-data value.
5. The method of claim 1, wherein the estimated language model is used for automatic speech recognition.
6. The method of claim 1, wherein identifying the second quantity of projections based on the highly fragmented non-speech meta-data is performed at training time.
7. The method of claim 6, wherein estimating the conditional meta-data dependent language model using the identified projections is performed at test time.
8. The method of claim 1, wherein identifying the second quantity of projections based on the highly fragmented non-speech meta-data further comprises estimating a unigram model to generate a unigram clustering output.
9. The method of claim 8, wherein the unigram clustering output is used for building a higher order tree.
10. The method of claim 8, wherein the step of estimating a unigram model to generate a unigram clustering output further comprises:
generating a unigram tree by splitting a leaf with the largest likelihood gain; and evaluating the likelihood gain of new leaf created by the splits.
11. The method of claim 10, wherein splitting the leaf with the largest likelihood gain is iterative.
12. The method of claim 11, wherein the likelihood gain of a best leaf split falls below a threshold value, terminating the iterative splitting process, wherein the unigram tree defines N leaf sets.
13. The method of claim 9, wherein building the higher order tree further comprises:
generating history nodes for the higher order tree; and splitting leaves on the higher order tree via domain splits.
14. The method of claim 13, wherein generating history nodes for the higher order tree further comprises:
partitioning count distributions allowing only history dependent splits;
and terminating the splitting based on a threshold value.
15. The method of claim 9, further comprising:
computing a distance between a root node of a unigram tree and at least one history node of a higher order tree.
16. The method of claim 15, wherein estimating the conditional meta-data dependent language model using the identified projections further comprises weighing the identified projections using the computed distance.
17. The method of claim 1, wherein estimating the conditional meta-data dependent language model using the identified projections further comprises estimating the conditional meta-data dependent language model by estimating history dependent distributions for histories in training data.
18. A method of building a language model used for automatic speech recognition (ASR), the method comprising:
applying a tree growing algorithm to highly fragmented non-speech meta-data related to a caller to a spoken dialog system but does not describe physical characteristics of the caller;
identifying a first quantity of leaf nodes in which a history appears as a result of applying the tree growing algorithm;

building tree clusters from non-speech meta-data by generating a second quantity of projections based on the highly fragmented non-speech meta-data wherein the first quantity equals the second quantity; and estimating a meta-data dependent language model using the built tree clusters and speech data, the language model used for ASR.
19. The method of claim 18, wherein building tree clusters from non-speech meta-data further comprises:
computing counts from the non-speech meta-data;
building a unigram tree from the computed counts;
building at least one higher-order tree using the built unigram tree; and computing node distances, wherein the computed node distances are used to estimate the meta-data dependent language model.
20. The method of claim 18, wherein ASR is used in a spoken dialog system.
21. The method of claim 18, wherein different language models are estimated based on different types of data within the non-speech meta-data.
22. The method of claim 18, wherein the meta-data dependent language model is estimated by weighing non-speech meta-data towards a type of data within the non-speech meta-data.
23. The method of claim 22, wherein the non-speech meta-data includes gender information and is weighed towards one gender.
24. A tangible computer-readable medium storing instructions for controlling a computing device to operate an automatic speech recognition (ASR) module using a meta-data dependent language model, the meta-data dependent language model generated by the steps:

applying a tree growing algorithm to highly fragmented non-speech meta-data related to a caller to a spoken dialog system but does not describe physical characteristics of the caller;
identifying a first quantity of leaf nodes in which a history appears as a result of applying the tree growing algorithm;
building tree clusters from non-speech meta-data by generating a second quantity of projections based on the highly fragmented non-speech meta-data, wherein the first quantity equals the second quantity; and estimating the meta-data dependent language model using the built tree clusters and speech data, the language model used by the ASR module.
25. The tangible computer-readable medium of claim 24, wherein building tree clusters from non-speech meta-data further comprises:
computing counts from the non-speech meta-data;
building a unigram tree from the computed counts;
building at least one higher-order tree using the built unigram tree; and computing node distances, wherein the computed node distances are used to estimate the meta-data dependent language model.
26. A spoken dialog system having a plurality of modules for processing speech, at least one module of the plurality of modules uses a language model estimated by the steps:
applying a tree growing algorithm to highly fragmented non-speech meta-data related to a caller to a spoken dialog system but does not describe physical characteristics of the caller;
identifying a first quantity of leaf nodes in which a history appears as a result of applying the tree growing algorithm;
building tree clusters from non-speech meta-data by generating a second quantity of projections based on the highly fragmented non-speech meta-data, wherein the first quantity equals the second quantity; and estimating a meta-data dependent language model using the built tree clusters.
27. The spoken dialog system of claim 26, wherein building tree clusters from non-speech meta-data further comprises:
computing counts from the non-speech meta-data;
building a unigram tree from the computed counts;
building at least one higher-order tree using the built unigram tree; and computing node distances, wherein the computed node distances are used to estimate the meta-data dependent language model.
CA2486128A 2003-10-30 2004-10-27 System and method for using meta-data dependent language modeling for automatic speech recognition Expired - Fee Related CA2486128C (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US51578103P 2003-10-30 2003-10-30
US60/515,781 2003-10-30

Publications (2)

Publication Number Publication Date
CA2486128A1 CA2486128A1 (en) 2005-04-30
CA2486128C true CA2486128C (en) 2011-08-23

Family

ID=34421830

Family Applications (1)

Application Number Title Priority Date Filing Date
CA2486128A Expired - Fee Related CA2486128C (en) 2003-10-30 2004-10-27 System and method for using meta-data dependent language modeling for automatic speech recognition

Country Status (4)

Country Link
US (2) US7752046B2 (en)
EP (1) EP1528538B1 (en)
CA (1) CA2486128C (en)
DE (1) DE602004028322D1 (en)

Families Citing this family (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7184929B2 (en) * 2004-01-28 2007-02-27 Microsoft Corporation Exponential priors for maximum entropy models
US9240188B2 (en) 2004-09-16 2016-01-19 Lena Foundation System and method for expressive language, developmental disorder, and emotion assessment
US9355651B2 (en) 2004-09-16 2016-05-31 Lena Foundation System and method for expressive language, developmental disorder, and emotion assessment
US8938390B2 (en) * 2007-01-23 2015-01-20 Lena Foundation System and method for expressive language and developmental disorder assessment
US10223934B2 (en) 2004-09-16 2019-03-05 Lena Foundation Systems and methods for expressive language, developmental disorder, and emotion assessment, and contextual feedback
US8311822B2 (en) * 2004-11-02 2012-11-13 Nuance Communications, Inc. Method and system of enabling intelligent and lightweight speech to text transcription through distributed environment
US20080091423A1 (en) * 2006-10-13 2008-04-17 Shourya Roy Generation of domain models from noisy transcriptions
GB0623932D0 (en) * 2006-11-29 2007-01-10 Ibm Data modelling of class independent recognition models
EP2126901B1 (en) * 2007-01-23 2015-07-01 Infoture, Inc. System for analysis of speech
US8332207B2 (en) * 2007-03-26 2012-12-11 Google Inc. Large language models in machine translation
EP4318463A3 (en) 2009-12-23 2024-02-28 Google LLC Multi-modal input on an electronic device
US11416214B2 (en) 2009-12-23 2022-08-16 Google Llc Multi-modal input on an electronic device
US8352245B1 (en) 2010-12-30 2013-01-08 Google Inc. Adjusting language models
US8296142B2 (en) 2011-01-21 2012-10-23 Google Inc. Speech recognition using dock context
US9786281B1 (en) * 2012-08-02 2017-10-10 Amazon Technologies, Inc. Household agent learning
US9626960B2 (en) 2013-04-25 2017-04-18 Nuance Communications, Inc. Systems and methods for providing metadata-dependent language models
US9842592B2 (en) 2014-02-12 2017-12-12 Google Inc. Language models using non-linguistic context
US9412365B2 (en) 2014-03-24 2016-08-09 Google Inc. Enhanced maximum entropy models
US10515151B2 (en) * 2014-08-18 2019-12-24 Nuance Communications, Inc. Concept identification and capture
US10134394B2 (en) 2015-03-20 2018-11-20 Google Llc Speech recognition using log-linear model
US9641680B1 (en) 2015-04-21 2017-05-02 Eric Wold Cross-linking call metadata
US9672814B2 (en) * 2015-05-08 2017-06-06 International Business Machines Corporation Semi-supervised learning of word embeddings
US9978367B2 (en) 2016-03-16 2018-05-22 Google Llc Determining dialog states for language models
US10832664B2 (en) 2016-08-19 2020-11-10 Google Llc Automated speech recognition using language models that selectively use domain-specific model components
US10311860B2 (en) 2017-02-14 2019-06-04 Google Llc Language model biasing system
US10529357B2 (en) 2017-12-07 2020-01-07 Lena Foundation Systems and methods for automatic determination of infant cry and discrimination of cry from fussiness

Family Cites Families (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7349976B1 (en) * 1994-11-30 2008-03-25 Realnetworks, Inc. Audio-on-demand communication system
US6778970B2 (en) * 1998-05-28 2004-08-17 Lawrence Au Topological methods to organize semantic network data flows for conversational applications
US6415248B1 (en) 1998-12-09 2002-07-02 At&T Corp. Method for building linguistic models from a corpus
US7058947B1 (en) * 2000-05-02 2006-06-06 Microsoft Corporation Resource manager architecture utilizing a policy manager
US7716358B2 (en) * 2000-09-12 2010-05-11 Wag Acquisition, Llc Streaming media buffering system
US6904408B1 (en) * 2000-10-19 2005-06-07 Mccarthy John Bionet method, system and personalized web content manager responsive to browser viewers' psychological preferences, behavioral responses and physiological stress indicators
US20020073136A1 (en) * 2000-12-07 2002-06-13 Tomoaki Itoh Data reproduction method, data receiving terminal and data receiving method
GB0031187D0 (en) * 2000-12-21 2001-01-31 Deeke Georg W An internal combustion engine
US20020087315A1 (en) 2000-12-29 2002-07-04 Lee Victor Wai Leung Computer-implemented multi-scanning language method and system
US7089237B2 (en) * 2001-01-26 2006-08-08 Google, Inc. Interface and system for providing persistent contextual relevance for commerce activities in a networked environment
US6754626B2 (en) 2001-03-01 2004-06-22 International Business Machines Corporation Creating a hierarchical tree of language models for a dialog system based on prompt and dialog context
JP4318407B2 (en) * 2001-05-17 2009-08-26 パイオニア株式会社 Receiving apparatus and flow control method thereof, transmitting apparatus and flow control method thereof
US7143035B2 (en) * 2002-03-27 2006-11-28 International Business Machines Corporation Methods and apparatus for generating dialog state conditioned language models
US20030191639A1 (en) 2002-04-05 2003-10-09 Sam Mazza Dynamic and adaptive selection of vocabulary and acoustic models based on a call context for speech recognition
US7398209B2 (en) * 2002-06-03 2008-07-08 Voicebox Technologies, Inc. Systems and methods for responding to natural language speech utterance
US7092615B2 (en) * 2002-06-05 2006-08-15 Matsushita Electric Industrial Co., Ltd. Content reproducing apparatus for reproducing content that is stream data divided into a plurality of reply segments, and content transmitting/receiving system
US20040098265A1 (en) * 2002-11-05 2004-05-20 Sean Kelly Dialog management system
US20040199484A1 (en) * 2003-04-04 2004-10-07 Laurence Smith Decision tree analysis
US7739715B2 (en) * 2003-06-24 2010-06-15 Microsoft Corporation Variable play speed control for media streams
US7340398B2 (en) * 2003-08-21 2008-03-04 Hewlett-Packard Development Company, L.P. Selective sampling for sound signal classification
US7861246B2 (en) * 2004-06-17 2010-12-28 Platform Computing Corporation Job-centric scheduling in a grid environment
JP4333655B2 (en) * 2005-08-24 2009-09-16 ソニー株式会社 Content receiving apparatus and content receiving method

Also Published As

Publication number Publication date
US8069043B2 (en) 2011-11-29
EP1528538A1 (en) 2005-05-04
US20050096907A1 (en) 2005-05-05
US20100241430A1 (en) 2010-09-23
DE602004028322D1 (en) 2010-09-09
US7752046B2 (en) 2010-07-06
EP1528538B1 (en) 2010-07-28
CA2486128A1 (en) 2005-04-30

Similar Documents

Publication Publication Date Title
CA2486128C (en) System and method for using meta-data dependent language modeling for automatic speech recognition
US10453117B1 (en) Determining domains for natural language understanding
Yu et al. Calibration of confidence measures in speech recognition
EP0708960B1 (en) Topic discriminator
JP4195428B2 (en) Speech recognition using multiple speech features
EP0771461B1 (en) Method and apparatus for speech recognition using optimised partial probability mixture tying
US9653093B1 (en) Generative modeling of speech using neural networks
Wang et al. Acoustic segment modeling with spectral clustering methods
US10490182B1 (en) Initializing and learning rate adjustment for rectifier linear unit based artificial neural networks
JP2017228160A (en) Dialog act estimation method, dialog act estimation device, and program
US8682669B2 (en) System and method for building optimal state-dependent statistical utterance classifiers in spoken dialog systems
US20090112600A1 (en) System and method for increasing accuracy of searches based on communities of interest
US11081104B1 (en) Contextual natural language processing
US10872601B1 (en) Natural language processing
US11276403B2 (en) Natural language speech processing application selection
US11195522B1 (en) False invocation rejection for speech processing systems
JPWO2007105409A1 (en) Standard pattern adaptation device, standard pattern adaptation method, and standard pattern adaptation program
JPH11338491A (en) Speaker including maximum method based on native voice and environment adaptation
US20220269850A1 (en) Method and device for obraining a response to an oral question asked of a human-machine interface
JP6910002B2 (en) Dialogue estimation method, dialogue activity estimation device and program
CN112397053B (en) Voice recognition method and device, electronic equipment and readable storage medium
WO2007118324A1 (en) Method and apparatus for building grammars with lexical semantic clustering in a speech recognizer
JPH07261785A (en) Voice recognition method and voice recognition device
Bacchiani et al. Meta-data conditional language modeling
Wang Strategies for statistical spoken language understanding with small amount of data-an empirical study.

Legal Events

Date Code Title Description
EEER Examination request
MKLA Lapsed

Effective date: 20201027

MKLA Lapsed

Effective date: 20201027

MKLA Lapsed

Effective date: 20201027

MKLA Lapsed

Effective date: 20201027