US 7003158 B1 Abstract The invention performs handwriting recognition using mixtures of Bayesian networks. A mixture of Bayesian networks (MBN) consists of plural hypothesis-specific Bayesian networks (HSBNs) having possibly hidden and observed variables. A common external hidden variable is associated with the MBN, but is not included in any of the HSBNs. Each HSBN models the world under the hypothesis that the common external hidden variable is in a corresponding one of its states. The MBNs encode the probabilities of observing the sets of visual observations corresponding to a handwritten character. Each of the HSBNs encodes the probabilities of observing the sets of visual observations corresponding to a handwritten character and given a hidden common variable being in a particular state.
Claims(61) 1. A handwritten character recognition network for inferring parts of handwriting from visual observations with a common hidden variable having plural discrete states, comprising:
at least one mixture of Bayesian networks (MBN) that encodes probabilities of observing the visual observations corresponding to a handwritten character, the probabilities employed for inferring parts of handwriting;
the MBN comprising:
a plurality of hypothesis-specific Bayesian networks (HSBNs), each of the HSBNs encoding probabilities of observing the visual observations corresponding to a handwritten character and given the common hidden variable being in a respective one of its states;
a first aggregator that combines outputs of the HSBNs to produce an MBN output of the MBN;
each one of the HSBNs comprises:
plural nodes, each of the plural nodes corresponding to an associated visual observation element, and
at least some of the plural nodes having dependencies with others of the plural nodes within the one HSBN,
a second aggregator connected to outputs of the nodes of the one HSBN to provide the output of the one HSBN, the nodes having inputs that receive the state of a respective one of the visual observation elements of a current one of the visual observations.
2. The handwriting recognition network
3. The handwriting recognition network of
4. The handwriting recognition network of
5. The handwriting recognition network of
6. The handwriting recognition network of
7. The handwriting recognition network of
8. The handwriting recognition network of
9. The handwriting recognition network of
10. The handwriting recognition network of
11. The handwriting recognition network of
12. The handwriting recognition network of
13. The handwriting recognition network of
14. The handwriting recognition network of
15. The handwriting recognition network of
each of the HSBNs further comprises an inference input defining observed data corresponding to the visual observations and an inference output corresponding to the likelihood of an visual observation corresponding to a handwritten character and given the common hidden variable being in one of its states corresponding to the HSBN; and
the mixture of Bayesian networks further comprises a weight multiplier which weights the inference output of each HSBN by a corresponding HSBN score and combines the weighted HSBN inference outputs into a single inference output of the mixture of Bayesian networks.
16. The handwriting recognition network of
17. The handwriting recognition network of
18. The handwriting recognition network of
19. The handwriting recognition network of
20. The handwriting recognition network of
21. The handwriting recognition network of
22. A handwriting recognition system, comprising:
means for encoding probabilities of observing sets of visual observations for a predetermined character;
the means for encoding comprising plural means for modeling a hypothesis that a common hidden variable corresponding to a handwritten character associated with the means for encoding is in a respective one of a plurality of discrete states;
each of the means for modeling comprising plural means for storing probability parameters that define relationships between visual features of the handwritten character;
at least some of the means for storing having dependencies with others of the means for storing in each respective means for modeling;
each means for modeling further comprising means for aggregating outputs from the plural means for storing and providing outputs of each respective means for modeling; and
means for aggregating the outputs of the respective means for modeling to produce a corresponding output of each respective means for encoding indicative of the probability that an input visual observation corresponds to the handwritten character defined by the common hidden variable.
23. A system for training a mixture of Bayesian networks (MBN), comprising:
the mixture of Bayesian network comprising a plurality of hypothesis-specific Bayesian networks (HSBNs), each of the HSBNs modeling a hypothesis that a common hidden variable corresponding to a handwritten character associated with the MBN is in a respective one of a plurality of discrete states;
each of the HSBNs comprising a plurality of nodes that correspond to visual features of the handwritten character, each one of the plurality of nodes in a respective HSBN storing probability parameters that indicate how a visual feature of the one node depends on the handwritten character;
a parameter search component that identifies a set of changes in the probability parameters that improve the goodness of each of the HSBNs in predicting visual observations of the handwritten character;
a parameter modification component that modifies the probability parameters based on the identified set of changes;
a scoring system that computes a structure score for each HSBN in the MBN that reflects the goodness of the each respective HSBN in predicting visual observations according to a structure of each HSBN;
a network adjuster that searches for changes in dependencies between nodes of each HSBN that improve the structure score and modifies the dependencies so as to improve the structure score.
24. The system of
25. A method of training a mixture of Bayesian networks (MBN) to facilitate recognition of handwritten characters, the MBN encoding probabilities of observing the sets of visual observations corresponding to a handwritten character and comprising a plurality of hypothesis-specific Bayesian networks (HSBNs), each HSBN including a plurality of nodes having probability parameters with dependencies between at least some of the nodes to model a hypothesis that a common hidden variable corresponding to a handwritten character is in a respective one of a plurality of discrete states, the method comprising:
for each one of the HSBNs:
conducting a parameter search for a set of changes in the probability parameters which improves the goodness of the one HSBN in predicting the visual observations, and
modifying the probability parameters of the one HSBN accordingly; and
for each one of the HSBNs:
computing a structure score of the one HSBN reflecting the goodness of the one HSBN in predicting the visual observations,
conducting a structure search for a change in the causal links which improves the structure search score, and
modifying the causal links of the one HSBN based on the structure search.
26. The method of
computing expected complete model sufficient statistics (ECMSS) based on the visual observations;
computing sufficient statistics for the one HSBN based on the ECMSS; and
computing the structure score based on the sufficient statistics.
27. The method of
computing the probability of each combination of states of discrete hidden and observed variables of the nodes of the one HSBN;
forming a vector for each observed case in the set of visual observations, each entry in the vector corresponding to a particular one of the combinations of the states of the discrete variables; and
summing the vectors over plural cases of the visual observations.
28. The method of
(a) the probability of the one combination of the states of the discrete variables,
(b) sub-entry vectors representing the states of the continuous variables.
29. The method of
30. The method of
31. The method of
(a) mean,
(b) scatter,
(c) sample size.
32. The method of
33. The method of
repeating the conducting a parameter search, the computing the structure score and the conducting a structure search until a structure search convergence criteria is met.
34. The method of
35. The method of
36. The method of
37. The method of
38. The method of
39. The method of
40. The method of
41. The method of
42. The method of
43. The method of
44. The method of
repeating the conducting a parameter search, the computing the structure score and the conducting a structure search until a structure search convergence criteria is met.
45. The method of
attempting different modifications of the dependencies at each node of the one HSBN;
for each one of the different modifications, computing the structure score of the one HSBN; and
saving those modifications providing improvements to the structure score.
46. The method of
47. The method of
48. The method of
49. The method of
50. The method of
51. The method of
(a) defining a causal link from each node corresponding to a hidden variable to each node corresponding to a continuous observed variable; and
(b) initializing the probability parameters in each of the nodes.
52. The method of
53. The method of
54. The method of
55. The method of
56. A computer-readable medium storing computer-executable instructions for performing the method of
57. A system for inferring a handwritten character from visual observations, comprising:
mixtures of Bayesian networks (MBNs), each MBN encoding the probabilities of the visual observations associated with a handwritten character, the probabilities employed for inferring parts of handwriting, each of the MBNs associated with a common hidden variable;
each of the MBNs comprising a plurality of hypothesis-specific Bayesian networks (HSBNs) that model a hypothesis that the common hidden variable corresponding to a handwritten character is in a respective one of a plurality of discrete states, the common hidden variable is a common external hidden variable not included in any of the HSBNs.
58. The system of
59. The system of
60. The system of
61. The system of
Description This application is related to U.S. application Ser. No. 08/985,114 filed Dec. 4, 1997, U.S. Pat. No. 6,534,669 by Bo Thiesson et al., and entitled MIXTURES OF BAYESIAN NETWORKS, and to U.S. application Ser. No. 09/220,197 filed Dec. 23, 1998, U.S. Pat. No. 6,336,108 by Bo Thiesson et al., and entitled SPEECH RECOGNITION WITH MIXTURES OF BAYESIAN NETWORKS. The present invention relates generally to data processing systems and, more particularly, to the generation and use of Bayesian networks for handwriting recognition. Various approaches have been developed to recognize patterns associated with handwritten characters. Most handwriting recognition systems use recognizers based on Neural Nets, Hidden Markov Models (HMM) or a K-Nearest-Neighbor (KNN) approach. These systems perform reasonably well at the task of classifying characters based on their total appearance. For example, a level of similarity can be determined by generating a distance measure between patterns. The advent of artificial intelligence within computer science has brought an abundance of decision-support systems. Decision-support systems are computer systems in which decisions, typically rendered by humans, are recommended and sometimes made. In creating decision-support systems, computer scientists seek to provide decisions with the greatest possible accuracy. Thus, computer scientists strive to create decision-support systems that are equivalent to or more accurate than a human expert. Applications of decision-support systems include medical diagnosis, troubleshooting computer networks, or other systems wherein a decision is based upon identifiable criteria. One of the most promising new areas for research in decision-support systems is Bayesian networks. A Bayesian network is a representation of the probabilistic relationships among distinctions about the world. Each distinction, sometimes called a variable, can take on one of a mutually exclusive and exhaustive set of possible states. A Bayesian network is expressed as an acyclic-directed graph where the variables correspond to nodes and the relationships between the nodes correspond to arcs. The arcs in a Bayesian network convey dependence between nodes. When there is an arc between two nodes, the probability distribution of the first node depends upon the value of the second node when the direction of the arc points from the second node to the first node. For example, node In other words, sets of variables X and Y are said to be conditionally independent, given a set of variables Z, if the probability distribution for X given Z does not depend on Y. If Z is empty, however, X and Y are said to be independent as opposed to conditionally independent. If X and Y are not conditionally independent, given Z, then X and Y are said to be conditionally dependent given Z. The variables used for each node may be of different types. Specifically, variables may be of two types: discrete or continuous. A discrete variable is a variable that has a finite or countable number of states, whereas a continuous variable is a variable that has an uncountably infinite number of states. All discrete variables considered in this specification have a finite number of states. An example of a discrete variable is a Boolean variable. Such a variable can assume only one of two states: true or false. An example of a continuous variable is a variable that may assume any real value between −1 and 1. Discrete variables have an associated probability distribution. Continuous variables, however, have an associated probability density function (density). Where an event is a set of possible outcomes, the density p(x) for a variable x and events a and b is defined as:
The automobile troubleshooting Bayesian network also provides a number of examples of conditional independence and conditional dependence. The nodes operation of the lights There are two conventional approaches for constructing Bayesian networks. Using the first approach (the knowledge-based approach), a person known as a knowledge engineer interviews an expert in a given field to obtain the knowledge of the expert about the field of expertise of the expert. The knowledge engineer and expert first determine the distinctions of the world that are important for decision making in the field of the expert. These distinctions correspond to the variables of the domain of the Bayesian network. The domain of a Bayesian network is the set of all variables in the Bayesian network. The knowledge engineer and the expert next determine the dependencies among the variables (the arcs) and the probability distributions that quantify the strengths of the dependencies. In the second approach (called the data-based approach), the knowledge engineer and the expert first determine the variables of the domain. Next, data is accumulated for those variables, and an algorithm is applied that creates a Bayesian network from this data. The accumulated data comes from real world instances of the domain. That is, real world instances of decision making in a given field. Conventionally, this second approach exists for domains containing only discrete variables. After the Bayesian network has been created, the Bayesian network becomes the engine for a decision-support system. The Bayesian network is converted into a computer-readable form, such as a file and input into a computer system. Then, the computer system uses the Bayesian network to determine the probabilities of variable states given observations, determine the benefits of performing tests, and ultimately recommend or render a decision. Consider an example where a decision-support system uses the Bayesian network of By utilizing both expert knowledge and empirical data, the network generator provides an improved Bayesian network that is more accurate than conventional Bayesian networks. In addition, the exemplary embodiment facilitates the use of continuous variables in Bayesian networks and handles missing data in the empirical data that is used to construct Bayesian networks. Expert knowledge consists of two components: an equivalent sample size or sizes (sample size), and the prior probabilities of all possible Bayesian-network structures (priors on structures). The effective sample size is the effective number of times that the expert has rendered a specific decision. For example, a doctor with 20 years of experience diagnosing a specific illness may have an effective sample size in the hundreds. The priors on structures refers to the confidence of the expert that there is a relationship between variables (e.g., the expert is 70 percent sure that two variables are related). The priors on structures can be decomposed for each variable-parent pair known as the prior probability of the variable-parent pair. Empirical data is typically stored in a database. An example of acquiring empirical data can be given relative to the Bayesian network of An example of a case can more clearly be described relative to the Bayesian network of The following presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the invention. This summary is not an extensive overview of the invention. It is intended to neither identify key or critical elements of the invention nor delineate the scope of the invention. Its sole purpose is to present some concepts of the invention in a simplified form as a prelude to the more detailed description that is presented later. The present invention provides a system and method to perform handwriting recognition using mixtures of Bayesian networks. A mixture of Bayesian networks (MBN) includes plural hypothesis-specific Bayesian networks (HSBNs) having possibly hidden and observed variables. A common external hidden variable is associated with the MBN, but is not included in any of the HSBNs. The number of HSBNs in the MBN corresponds to the number of states of the common external hidden variable, and each HSBN models the world under the hypothesis that the common external hidden variable is in a corresponding one of those states. In accordance with an aspect of the present invention, the MBNs encode the probabilities of observing the sets of visual observations corresponding to a handwritten character (e.g., input ink). The HSBNs encode the probabilities of observing the sets of visual observations corresponding to the handwritten character and given a hidden common variable being in a particular state. Each HSBN has one or more nodes corresponding to the attributes of features of the visual observations. These nodes store probability parameters corresponding to the probabilities with causal links representing dependencies between nodes in each HSBN. In accordance with an aspect of the present invention, the MBNs can be trained by conducting a parameter search for a set of changes in the probability parameters which improves the goodness of each HSBN in predicting the observed data (e.g., handwritten characters), and then modifying the probability parameters of each HSBN accordingly. Then, for each HSBN, a structure score is computed reflecting the goodness of the HSBN in predicting the observed data. A structure search is conducted for a change in the causal links that improves the structure search score, and the causal links of each HSBN are adjusted accordingly. The MBN structure is initialized as a collection of identical HSBNs whose discrete hidden variables are connected to all observed variables and whose continuous hidden variables are connected only to each of the continuous observed variables, the directionality being from hidden variable to observed variable. To facilitate training of the MBN, the parameter searching and modifications can be interleaved with the structure searching and modifications. In constructing the MBN, the parameters of the current HSBNs are improved using, for example, an Expectation-Maximization (EM) process applied for training data. The EM process is iterated to improve the network performance in predicting the training data, until some criteria has been met. Early in the process, for example, the criteria may be a fixed number of iterations which may itself be a function of the number of times the overall learning process has iterated. Later in the process, the criteria can be convergence of the parameters to a near optimum network performance level. Then, expected complete-model sufficient statistics are generated from the training data. After computation of the expected complete-model sufficient statistics for the MBN, the structures of the HSBNs are searched for changes which improve the HSBN's score or performance in predicting the training data given the current parameters. The MBN score can be determined by the HSBN scores, the score for the common hidden external variable, and a correction factor. If the structure of any HSBN changes as a result of this fast search, the prior steps can be repeated. The foregoing is iteratively repeated until the network structure stabilizes, such as based on the MBN score. At this point the current forms of the HSBNs are saved as the MBN for the class variable. An MBN is thus generated for each possible combination of number of states of the hidden discrete variables, including the common external hidden variable, so that a number of MBNs is produced in accordance with the number of combinations of numbers of states of the hidden discrete variables. Overview The present invention generally relates to systems and methods to facilitate recognition of handwritten characters. In particular, Mixtures of Bayesian Networks (MBNs) encode the probabilities of the visual observations associated with a handwritten character. Each of the MBNs is associated with a common hidden variable, which can be an external common hidden variable having a predetermined number of discrete states. Each of the MBNs consists of a plurality of hypothesis-specific Bayesian networks (HSBNs) that model a hypothesis that the common hidden variable corresponding to a handwritten character is in a respective one of its states. Introduction to Mixtures of Bayesian Networks The MBN generator An analyzer As one example, C can have two possible values. In this case, the conditional distribution of sets of variables O and H given C=0 might be represented by a Bayesian network, such as shown in A naοve method for learning a single Bayesian network with hidden variables is to (1) fix the structure of the Bayesian network (2), use the expectation maximization (EM) algorithm to find good (e.g., ML or MAP) parameter values for the Bayesian network, and (3) use the parameters obtained from step 2 to compute a score for the model, such as by using the Cheeseman-Stutz, BIC or other approximation of the posterior probability for the model. There are two difficulties with this approach. First, the EM algorithm is an iterative algorithm that can be too computationally expensive to run on many models. Second, the approximate scores for models with hidden variables (C and H in the case of a mixture of Bayesian networks) usually do not factor into scores for individual nodes. If it did factor, one could use previously calculated scores to make search more efficient. These and other deficiencies can be solved in by interleaving the EM algorithm's search for parameters with a search for the structure of the Bayesian networks in accordance with an aspect of the present invention. By interleaving the search for Bayesian networks and the search for parameters, scores are created that factor according to the model and thus allows for efficient search of model structure. In addition, a method according to an aspect of the invention can independently search for each of the Bayesian networks in the mixture of Bayesian networks. Let H From the foregoing definition, the vector T(x Example Involving Complete Data The following is a working example in which a complete database with 2 cases is given as:
The variables O1 and O2 are continuous. The remaining variables are discrete. According to an aspect of the invention, all possible configurations of the discrete variables are indexed in some fixed way, an example of which is given in the table below:
From the foregoing tables, the complete model statistics vector for case 1 is:
The complete model statistics vector for case 2 is:
The expected complete model sufficient statistics is a vector ECMSS where
The expectation of T(x Example Involving Incomplete Data The following is a working example in which an incomplete database is given. The incomplete data base is given in the following table, in which the variables O1 and O2 are continuous and O3, H1, and C are binary discrete and the symbol ? denotes unknown data:
The vectors T(x In accordance with an aspect of the present invention, the expected complete model sufficient statistics can be employed as complete model sufficient statistics to perform searching among alternative Bayesian networks, such as using the methods described below. One way to do this is to form the expected complete model sufficient statistics for each value of C. Hence, for each value of C, the expected complete model sufficient statistics for O and H is formed, which is denoted ECMSS_c. The expected complete model sufficient statistics for O and H can then be used for searching for Bayesian networks. Since the expected complete model sufficient statistics for each value of C are distinct (and we have assumed parameter independence) the statistics for each value of C can be used to search for the respective Bayesian network independently of other Bayesian networks. By creating the complete model sufficient statistics (in essence) new scores are created that factor according to the Bayesian networks, as discussed herein. For instance, let the indexing of the discrete configurations be as described in the table below.
Using the index Δ, ECMSS_j is derived from ECMSS by selecting the appropriate triples from ECMSS. For this index we would have
In this case, the triple_j is the triple <N_j,R_j,S_j> from ECMSS. Specifically, for example, triple From the foregoing, a general process for learning a mixture of Bayesian networks (MBN) in accordance with an aspect of the invention is as follows: -
- 1. Choose the number of possible states for the variables C and Hd.
- 2. Initialize a hypothesis-specific Bayesian-network structure for each hypothetical value of C to be a graph in which each variable in H points to each variable in Oexcept for the restriction that no continuous variable may point to a discrete variableand in which there are no additional arcs. Choose initial values for the parameters in each of these hypothesis-specific Bayesian networks. The parameter values can be set at random, with agglomerative methods, marginal+noise, or other methods. Choose values for the parameters θc, e.g., choose them to be uniform.
The marginal+noise initialization method can be implemented as follows: Take the initial MBNs, which is a Bayesian network with discrete hidden variables. Remove all hidden nodes and adjacent arcs and adjust the local distributions, creating model s -
- 3. Use the EM algorithm to do one E step and one M step to improve the parameter estimates for the current model.
- 4. If some convergence criterion is not satisfied then go to step 3.
- 5. Using the current MBN, create the expected complete-model sufficient statistics ECMSS and ECMSS_c for each hypothesis-specific Bayes net corresponding to C=c. For every C=c, translate ECMSS_c to expected sufficient statistics N
_{ijk}, sample mean, scatter matrix, and sample size (for use in the structure search step that follows). Those practiced in the art will recognize that this step can be performed with standard techniques. - 6. Using the expected complete model sufficient statistics for each value of C, search for structures that improve the score. The result is a new network structures with new parameter values.
- 7. If some Convergence criterion is not satisfied then go to step 3.
- 8. Save the model that is selected. Choose another number of possible states for the variable C and Hd. Go to step 2. Repeat this step and compare the models that are selected. Use the corrected version of the Cheeseman-Stutz score.
The choice in step 4 of whether to go to step 3 or step 5 can be decided in a variety of ways, e.g., by checking the convergence of the likelihood, by limiting the number of iterations allowed, and so forth. There are various modifications of this process that are useful, including having the process adaptively prune out hypothesis-specific Bayesian networks for which θc (the support for the HSBN corresponding to C=c) falls below some threshold (e.g., 1/N). The following is a description of different modes of the foregoing process. -
- ((EM)
^{#}ES*M)*^{: }
- ((EM)
In this mode of the process, the EM step is iteratively repeated (steps 3 and 4) a limited number (#) of times while the remainder of the process including the search for optimum structure is carried out to convergence. -
- ((EM)*ES*M)*
^{: }
- ((EM)*ES*M)*
In this mode of the process, the EM steps are iterated until convergence before performing the remainder of the algorithm and the structure search is also carried to convergence. -
- ((EM)
^{#(iteration)}ES*M)*
- ((EM)
In this version of the process, the iteration of the EM step (steps 3 and 4) is carried out over a limited number (#) of iterations which is a function of the number of iterations of the structure search step (step 6). -
- ((EM)
^{#}ES^{#}M)*:
- ((EM)
In this version of the process, the number of iterations of the EM step is a fixed number, the number of iterations of the structure search step is a fixed (possibly different) number. -
- ((EM)*ES
^{#}M)*
- ((EM)*ES
In this version of the process, the EM steps are always iterated to convergence, while the structure search is iterated a limited number of times. -
- ((EM)
^{#(iteration)}ES^{#}M)*
- ((EM)
In this version of the process, the number of iterations of the EM step is a function of the number of iterations of the structure search step performed thus far, while the number of iterations of the structure search is a fixed number. The foregoing examples use discrete variables in the Bayesian network where all of the conditional probabilities in the Bayesian network are represented as full tables. It is to be understood and appreciated that decision graphs could be employed instead of the tables in accordance with an aspect of the present invention. Implementations of Mixtures of Bayesian Networks In accordance with an aspect of the present invention, the model depicted in As indicated in the drawing of _{i}, Y_{j}) pairs, with each coordinate corresponding to a feature. It is to be understood and appreciated that the stroke position features are simply for purposes of ease of illustration and that other character features, such as parts of the character strokes located between the start-stop positions, curvature of character strokes, can be used in accordance with an aspect of the present invention. For example, the particular features used in modeling a given character generally will depend on the character being modeled.
_{1 }to HSBN_{n}, where n is positive integer. Thus, the MBN of Each HSBN includes a plurality of nodes in which each respective node corresponds to a feature or visual observation element for one of the four strokes (e.g., X1 . . . Y8). Dependencies and interrelationships between nodes are represented by arcs interconnecting certain nodes, which defines the structure of each HSBN and, collectively, of the MBN. For example, in HSBN1, the arc between X1 and X4 represents the relationship between the start position of the first stroke to the end position of the second stroke. The other arcs can represent similar types of relationships. One restriction on the dependencies between nodes in a given HSBN can be the character completion order. For example, each HSBN shown in By way of illustration, the MBN has a score determined by the set of probability parameters at each node of each HSBN. That is, each node stores a set of probability parameters that define the distribution of probabilities across different states of the visual observation for a given character. If all cases of characters in the empirical data database are drawn from a multivariate normal distribution, the scoring takes advantage of the fact that a set of variables has a multivariate normal distribution if and only if each particular variable is an independent (univariate) normal distribution, when conditioned on the variables that precede the particular variable in some ordering (e.g., the restriction on completion order). The set of parameters at each node of each HSBN are functionally related to a normal distribution having a mean, a variance, and coefficients. Each of the coefficients, sometimes referred to as partial regression coefficients, indicates the strength of the connection between the mean of a first variable and the value of a second variable, as represented by the respective arcs. Thus, a coefficient for a given pair of variables is equal to zero if and only if there is no arc connecting nodes for the given pair of variables. The scores for each MBN have been previously computed, as described herein, at the time each respective MBN is generated before inferencing is performed. Each MBN output is weighted by the corresponding MBN score by a multiplier An ink processor For the language of Japanese, as an example, handwriting recognition scientists have identified groups of stroke counts into which all characters may be divided. The number of characters in each k-stroke group ranges from approximately 200 to 3000 characters depending on the stroke count (k). The MBN array Initially, the MBNs In accordance with one aspect of the present invention, the number m of components or HSBNs in each MBN is determined in accordance with techniques described above in this specification, and each MBN In accordance with an aspect of the present invention, each HSBN can have a restricted form. This restricted from is related to the order in which the strokes of the character are written. For example, as mentioned above, each HSBN is not allowed to have any arcs from variables associated with the visual patterns for later stroke to variables associated with the visual pattern for an earlier stroke. A system for character completion uses a set of MBN arrays. A character completion system identifies characters with the same or more strokes than the user has written. For instance, if a user inputs two strokes of a character, this character completion system obtains probabilities for characters with two strokes, three strokes, etc. . . . and provides the user with a list of characters that have the highest probabilities. The procedure for computing the probabilities for a user's strokes can be identical to the procedure described above when using an MBN array for characters with the same number of strokes. Those skilled in the art will appreciate that when using an MBN array for characters with more strokes than the user has written, one needs to use probabilistic inference to compute these probabilities. Referring back to A multiplier Referring between While the example of As indicated in A Structure Search: Searching for Optimum Structure and Scoring the Hypothesis-Specific Network Structure After receiving the information, the scoring mechanism For example, the scoring mechanism The network adjuster Although the above example has been described as iterating many times between the scoring mechanism In order to provide additional context for various aspects of the present invention, The illustrated aspects of the invention may also be practiced in distributed computing environments where certain tasks (e.g., processing input characters or generating models) are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules for carrying one or more aspects of the present invention can be located in local and/or remote memory storage devices. While various aspects of the present invention have been described above in the general context of computer-executable instructions that may run on one or more computers or other microprocessor-based equipment, it is to be recognized that the invention also may be implemented in combination with other program modules and/or as a combination of hardware and software. Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. With reference to The system bus The computer It is to be appreciated that the computer (or other device) A number of program modules may be stored in the drives and RAM A user may enter commands and information into the computer The computer When used in a LAN networking environment, the computer In view of the exemplary systems shown and described above, methodologies, which may be implemented in accordance with the present invention, will be better appreciated with reference to the flow charts of Next, at At At At Next, at At The process begins at Next, at Next, at In carrying out a training process of The calculate discrete score process, the calculate continuous score process and the calculate mixed score process are based upon a common concept, Bayes' theorem. The score that each scoring process produces is proportional to the posterior probability of the test network. That is, probability distributions and densities can be of two types: prior and posterior. The prior probability distribution or density is the probability distribution or density before data is observed. The posterior probability distribution or density is the probability distribution or density after data is observed. Bayes' theorem states that the posterior probability of a test network is proportional to the prior probability of a test network multiplied by the probability of the empirical data database given the test network and the expert knowledge. Calculate Discrete Score Process The calculate discrete score process scores a test network containing all discrete variables. The calculate discrete score process takes advantage of the fact that the probability of the empirical data database given a test network and expert knowledge is the product over all cases of the probability of a particular case given a test network, expert knowledge, and previous cases (i.e., cases observed prior to the particular case). The computation of the probability of a case given a test network, expert knowledge, and previous cases is based on the assumption that the empirical data database represents a multinomial sample from the test network. That is, the empirical data database contains a sequence of observations that form a multinomial distribution as described in DeGroot, -
- i is the variable index, i=1 . . . n, where n is the number of variables in the test network;
- j is the parent-instance index; j=1 . . . q
_{i,}, where q_{i }is the number of instances of the parents; - k is the variable state index; k=1 . . . r
_{i}, where r_{i} is the number of states of the variable i.
The parameter θ Next, at After the subscore for a variable is calculated and stored, the calculate discrete score process determines at Calculate Continuous Score Process The calculate continuous score process calculates scores for test networks containing all continuous variables and is based on Bayes' theorem. The calculate continuous score process assumes that all cases in the empirical data database are drawn from a multivariate normal distribution. The calculate continuous score process takes advantage of the fact that a set of variables have a multivariate normal distribution if and only if each particular variable is an independent (univariate) normal distribution, when conditioned on the variables that precede the particular variable in some ordering:
The term p(x In addition, the calculate continuous score process is based on three assumptions. First, the calculate continuous score process assumes that the prior distributions for the mean and precision matrix of the multivariate normal distribution with all dependencies between variables being possible (i.e., B Next, at At The term p(D The determinant of an n by n matrix (A) is the sum over all permutations p=(i After the calculate continuous score process of the exemplary embodiment calculates a subscore for one variable, a determination is made at Calculate Mixed (Discrete and Continuous) Score Process The calculate mixed score process is programmed and/or configured to calculate a score for a mixed network having both discrete and continuous variables in accordance with an aspect of the present invention. The calculate mixed score process is based on Bayes' theorem. In calculating a score for a mixed network, process enforces a restriction that the initial network be constructed under the assumption that all dependencies among variables are possible. This restriction, for example, is enforced by the knowledge engineer. The calculate mixed score process also follows a restriction that the prior network and all mixed test networks correspond to a collection of conditional Gaussian distributions. This restriction is enforced by the knowledge engineer and the network adjuster, respectively. In the following discussion of The process begins at After calculating the continuous subscores at At Network Adjuster The network adjuster process begins at At After the best change for the selected nodes has been identified, the network adjuster determines at At What has been described above includes examples of the present invention. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the present invention, but one of ordinary skill in the art will recognize that many further combinations and permutations of the present invention are possible. Such changes may include parallelization of some of the computations described herein or the use of other probability distributions, for example. Accordingly, the present invention is intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims. Patent Citations
Non-Patent Citations
Referenced by
Classifications
Legal Events
Rotate |