US 7406416 B2 Abstract A method and apparatus are provided for storing parameters of a deleted interpolation language model as parameters of a backoff language model. In particular, the parameters of the deleted interpolation language model are stored in the standard ARPA format. Under one embodiment, the deleted interpolation language model parameters are formed using fractional counts.
Claims(20) 1. A method of storing parameters of a deleted interpolation language model, the method comprising:
obtaining a set of parameters for the deleted interpolation language model, wherein the parameters of the deleted interpolation language model allow an N-gram probability to be determined as a linear interpolation of a relative frequency estimate for the N-gram and a probability for a lower order n-gram; and
storing at least one parameter for the deleted interpolation language model as a parameter for a backoff language model, wherein the backoff language model replaces an N-gram probability with a lower order n-gram and a backoff weight for any N-gram that cannot be located in the backoff language model.
2. The method of
3. The method of
4. The method of
5. The method of
6. The method of
7. The method of
8. The method of
9. The method of
10. The method of
11. A computer-readable storage medium having encoded thereon computer-executable instructions for performing steps comprising:
identifying a parameter for a deleted interpolation language model that forms probabilities through interpolations of values; and
placing the parameter in a data structure as a backoff parameter for a backoff language model that substitutes a weighted lower order n-gram probability for an N-gram probability when the N-gram cannot be located in the backoff language model.
12. The computer-readable storage medium of
13. The computer-readable storage medium of
14. The computer-readable storage medium of
15. The computer-readable storage medium of
16. The computer-readable storage medium of
17. The computer-readable storage medium of
18. A method for constructing a language model, the method comprising:
using deleted interpolation to train parameters for a language model;
storing at least some of the trained parameters in a data structure conforming to the ARPA format for backoff language models.
19. The method of
20. The method of
Description The present invention relates to language models. In particular, the present invention relates to storage formats for storing language models. Language models provide probabilities for sequences of words. Such models are trained from a set of training data by counting the frequencies of sequences of words in the training data. One problem with training language models in this way is that sequences of words that are not observed in the training data will have zero probability in the language model, even though they may occur in the language. To overcome this, back-off modeling techniques have been developed. Under a back-off technique, if a sequence of n words is not found in the training data, the probability for the sequence of words is estimated using a probability for a sequence of n−1 words and a back-off weight. For example, if a trigram (w N-gram language models that use back-off techniques are typically stored in a standard format referred to as the ARPA standard format. Because of the popularity of back-off language models, the ARPA format has become a recognized standard for transmitting language models. However, not all language models have back-off weights. In particular, deleted interpolation N-gram models do not have back-off weights because they use a different technique for handling the data sparseness problem associated with language models. As a result, deleted interpolation language models have not been stored in the standard ARPA format. Because of this, it has not been easy to integrate deleted interpolation language models into language systems that expect to receive the language model in the ARPA format. A method and apparatus are provided for storing parameters of a deleted interpolation language model as parameters of a backoff language model. In particular, the parameters of the deleted interpolation language model are stored in the standard ARPA format. Under one embodiment, the deleted interpolation language model parameters are formed using fractional counts. The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, telephony systems, distributed computing environments that include any of the above systems or devices, and the like. The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention is designed to be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules are located in both local and remote computer storage media including memory storage devices. With reference to Computer The system memory The computer The drives and their associated computer storage media discussed above and illustrated in A user may enter commands and information into the computer The computer When used in a LAN networking environment, the computer The present invention provides a technique for storing a language model generated through deleted interpolation in the standard ARPA format. In deleted interpolation, an N-gram probability is determined as a linear interpolation of a relative frequency estimate for the N-gram probability and a probability for a lower order n-gram. The probability of the lower order n-gram is similarly defined as an interpolation between the relative frequency probability estimate for the lower order n-gram and next lower order n-gram. This continues until a unigram probability is determined. Thus, the interpolation is determined recursively according to:
Using the recursion of Equations 1 and 2, the probability for the N-gram becomes an interpolation of relative frequencies for different orders of n-grams that are below the N-gram of interest. For example, for a trigram, the recursive interpolation produces: Under some embodiments, the counts used to determine the relative frequencies are not limited to integer valued counts and may include fractional values that are computed as the expected values of the counts. This is one advantage of deleted interpolation over other back−off methods, such as the Katz back-off method, which cannot be used on fractional (real valued) counts. For example, beginning at node The probability at the next higher node This recursive summation continues upward until it reaches node As can be seen from In step At step
Where [λ As noted above, the weights are functions of the contexts of the n-gram probabilities they are used to determine. To counteract data sparseness (which would lead to unreliable estimates) and at the same time reduce the computational complexity of the EM training, these weights are grouped into buckets based on the frequency counts of the context. Under one embodiment, ranges of frequency counts are grouped into the same weights. Thus, one λ Note that since the weights are maximized against check data Under some embodiments, the training text The interpolation represented by Equations 1 and 2 is substantially different from the techniques used with the more widely accepted backoff language models, which are typically represented in the standard ARPA format. Instead of using a linear interpolation to determine the probability for an N-gram, the more widely accepted backoff language models use a substitute probability for any N-gram that cannot be located in the model. This substitute probability is based on a lower order model and a backoff weight associated with the context of the probability that can not be located. Thus, instead of performing an interpolation, the more standard backoff language models simply replaces an N-gram probability with a lower order n-gram probability. Once a probability is found for an n-gram at step As can be seen in Below list Below each heading for the different orders of n-grams, there is a list of entries, one for each n-gram of that order. Each entry includes the probability of the n-gram, the n-gram, and for n-grams of orders other than the top order, a backoff weight. For example, under unigram heading For entries under top order n-gram heading Comparing In step If the relative frequency of the L-gram is not greater than zero, the probability for the L-gram is not stored in the standard ARPA format. After the probability for the L-gram has been stored at step Once all of the L-grams for the top order of L-grams have been processed at step At step Thus, an L-gram is stored if its relative frequency is greater than zero, i.e. it was seen in the training data, and if it appears as a context for a higher order L-gram. By limiting the L-grams that are stored to those that meet these criteria, this embodiment of the present invention creates a compact language model in the backoff format. An L-gram can appear as a context while having a relative frequency of zero in the training text if the relative frequencies are determined by setting the relative frequencies to zero if their initial relative frequency is below a threshold. For example, if an L-gram has a relative frequency of 0.02 and the threshold is set to 0.02, the relative frequency for the L-gram would be set to zero. This is done to reduce the size of the interpolation model. The reason for storing an L-gram if it appears as a context in a higher order L-gram even though it has a relative frequency of zero is that since the L-gram appears as a context for a higher order L-gram, a backoff weight for this context will be needed in the language model. After step When there are no more L-grams for the current order at step Thus, the method of Although the present invention has been described with reference to particular embodiments, workers skilled in the art will recognize that changes may be made in form and detail without departing from the spirit and scope of the invention. Patent Citations
Non-Patent Citations
Referenced by
Classifications
Legal Events
Rotate |