US 6289330 B1 Abstract The present invention provides a system for learning from and responding to regularly arriving information at once by quickly combining prior information with concurrent trial information to produce useful learned information. A parallel embodiment of the system performs can perform updating operations for memory elements of matrix through the coordinated use of parallel feature processors and a joint access memory, which contains weighted values and provision for connecting feature processors pairwise. The parallel version also performs feature function monitoring, interpretation and refinement operations promptly and in concert with concurrent operation. A non-parallel embodiment of the system uses a single processor to perform the above operations however, more slowly than the parallel embodiment, yet faster than available alternatives.
Claims(60) 1. A parallel processing system for computing output values from input values received during a time trial, comprising:
a plurality of processing units, each of said processing units operative to receive, during a time trial and in parallel, individual input values from an input vector, said processing units comprising analogs components for computing arithmetic operations of said system;
a plurality of interconnected conductors, operative to connect each of said processing units to every other processing unit of said system and operative to transfer weighted values among said processors;
a plurality of switching junctions located along said interconnected conductors, said switching junctions being operable for uniquely pairing each of said processors to every other processor of said system; and
each of said processing units operative to provide, during said time trial, an expected output value based on said weighted values and each of said processing units operative to update, during said time trial, said weighted values based on said input values.
2. The system of claim
1 wherein analog components are operative to calculate values for updating said connection weights to transmitting to other processors of said system.3. The system of claim
1 wherein said switching junctions comprise analog components for computing connection weight at said switching junctions.4. The system of claim
1 wherein said switching junction have memory elements.5. The system of claim
4 wherein said memory elements are resistive-capacitance elements.6. The system of claim
1 wherein each of said conductors is a single wire conductor.7. The system of claim
1 wherein said analog components receive input analog signals and produce an analog output having a magnitude corresponding to the magnitude of the inputs as operated on by the function associated with the analog component.8. The apparatus of claim
3 further comprising a plurality of memory elements, each of said memory elements being individually coupled to a separate switching junction and each of said memory elements containing a weighted memory value.9. A computer processing system for updating a matrix of elements for use in determining the relationship among a plurality of variables, comprising:
a plurality of processors, each of said processors operative to receive an input variable of a plurality of input variables in parallel received during a time trial;
an ordered array of memory elements associated with each processor, each of said memory elements defining elements of a matrix, each of said memory elements containing values that are operative to define a relationship among received input variables and each of said memory elements operative to be updated by each of said processors;
each of said processors operative to be uniquely paired with every other processor of said computer system;
said uniquely paired processors sharing data via a memory element associated with each unique pair of processors; and
each of said processors updating said memory elements to reflect the relationship among input variables received during a time trial.
10. The system of claim
9 wherein said memory elements contain values that represent a matrix inverse.11. The system of claim
10 wherein said matrix inverse is an inverse of a matrix of second order derivatives.12. The system of claim
11 wherein said matrix of elements are updated according to the equation:^{T}x[DFP]+b[DFP]y[DFP]^{T}y[DFP], where
^{T } and
^{T}, wherein said ω[DFP,IN], equals ω[DFP,OUT] from the previous measurement trial and wherein ω[DFP,IN] equals an initialization matrix before the first time trial.
13. The system of claim
12 wherein the inner product C[DFP]x[DFP]^{T}x[DFP] is computed in a first step; and the inner product b[DFP]y[DFP]^{T}y[DFP] is computed in a second step.14. The system of claim
9 wherein said matrix of elements are updated according to the equation:^{T}x), where
and
^{T}=xe[IN]wherein said ω[IN] equals ω[OUT] from the previous measurement trial and wherein ω[IN] equals an initialization matrix before the first time trial.
15. The system of claim
9 wherein said matrix is a connection weight matrix used for predicting outputs based on input variables received.16. The system of claim
9 further comprising a plurality of switching junctions for uniquely pairing each of said processors to every other processor of said system.17. The apparatus of claim
16 wherein each of said switching junctions selectively connects each of said processors to only one other of said processors during a time interval, thereby forming multiple paired sets of said processors, for communicating said weighted values during a time interval.18. The apparatus of claim
17 wherein each of said memory elements is individually coupled to a separate switching junction.19. The apparatus of claim
17 wherein said switching junctions successively connect different sets of multiple paired sets of said processors during multiple time intervals.20. The system of claim
19 wherein said switching junctions successively connect all possible combinations of said different sets of multiple paired sets of said processors in a minimum number of steps.21. A computer processing system for computing a matrix of elements, comprising:
a plurality of processing units, each of said processing units operative to receive, during a time trial and in parallel, individual input values from an input vector e[IN];
an ordered array of memory elements associated with each processor, each of said memory elements defining elements of a symmetric matrix ω;
each of said processors operative to be uniquely paired with every other processor of said computer system, thereby forming a processor pair ƒ and g;
said uniquely paired processors sharing data via a memory element uniquely associated with each unique pair of processors; and
each of said processors computing said memory elements during a time trial by multiplying said input vector e[IN] by said symmetric matrix ω.
22. The system of claim
21 further comprising a plurality of switching junctions for uniquely pairing each of said processors to every other processor of said system.23. The apparatus of claim
22 wherein each of said switching junctions selectively connects each of said processors to only one other of said processors during a time interval, thereby forming multiple paired sets of said processors, for communicating said weighted values during a time interval.24. The apparatus of claim
22 wherein each of said memory elements is individually coupled to a separate switching junction.25. The apparatus of claim
22 wherein said switching junctions successively connect different sets of multiple paired sets of said processors during multiple time intervals.26. The system of claim
25 wherein said switching junctions successively connect all possible combinations of said different sets of multiple paired sets of said processors in a minimum number of steps.27. A computer processing system for modifying the elements ω{ƒ,ƒ} of a connection weight matrix used to produce an output based on inputs received during a time trial, comprising:
a plurality of processing units, each of said processing units operative to receive, during a time trial and in parallel, individual input variables from an input vector ω<ƒ>;
an ordered array of memory elements associated with each processor, each of said memory elements defining elements of a matrix ω<ƒ,ƒ>, each of said memory elements containing values that are operative to define a relationship among received input variables and each of said memory elements operative to be modified by each of said processors;
each of said processors operative to be uniquely paired with every other processor of said computer system;
said uniquely paired processors sharing data via a memory element uniquely associated with each unique pair of processors; and
each of said processors updating said memory elements to reflect the relationship among input variables received during a time trial.
28. The system of claim
27 wherein said matrix ω{ƒ,ƒ} is updated in parallel by said processors according to the following process ω{ƒ,ƒ}=ω<ƒ,ƒ>−ω<ω>^{T}ω<ω>/ω(ƒ,ƒ), where ω <ƒ> represents an input vector, ƒ<ƒ,ƒ> is a symmetric matrix of memory elements, and ω(ƒ,ƒ) is a constant.29. The system of claim
28 wherein said matrix of element is the inverse of a matrix.30. The system of claim
27 further comprising a plurality of switching junctions for uniquely pairing each of said processors to every other processor of said system.31. The apparatus of claim
30 wherein each of said switching junctions selectively connects each of said processors to only one other of said processors during a time interval, thereby forming multiple paired sets of said processors, for communicating weighted values during, a time interval.32. The apparatus of claim
30 wherein each of said memory elements is individually coupled to a separate switching junction.33. The apparatus of claim
30 wherein said switching junctions successively connect different sets of multiple paired sets of said processors during multiple time intervals.34. The system of claim
33 wherein said switching junctions successively connect all possible combinations of said different sets of multiple paired sets of said processors in a minimum number of steps.35. A computer processing system for updating a matrix of elements for use in determining the relationship among a plurality of variables, comprising:
a plurality of processors, each of said processing units operative to receive, during a time trial and in parallel, individual input values from an input vector m[IN];
a plurality of interconnected conductors, operative to connect each of said processing units to every other processing unit of said system and operative to transfer weighted values among said processors;
a variance-covariance matrix of elements ν associated with a processor, each of said memory elements being variance-covariance matrix values ν defining elements of a matrix, each of said memory elements containing values that are operative to define a relationship among received input variables and each of said memory elements operative to be updated, by each of said processors, according to the equation:
^{T}m[IN]+ν[IN]/(1+l), where l is a learning weight; and
wherein said matrix of elements is inverted for use in providing output values from said input values.
36. The system of claim
35 wherein each of said processors is operative to be uniquely paired with every other processor of said computer system; andsaid uniquely paired processors sharing data via a memory element uniquely associated with each unique pair of processors.
37. The system of claim
36 further comprising a plurality of switching junctions for uniquely pairing each of said processors to every other processor of said system.38. The apparatus of claim
37 further comprising each of said switching junctions selectively connects each of said processors to only one other of said processors during a time interval, thereby forming multiple paired sets of said processors, for communicating said weighted values during a time interval.39. The apparatus of claim
37 wherein each of said memory elements is individually coupled to a separate switching junction.40. The apparatus of claim
37 wherein said switching junctions successively connect different sets of multiple paired sets of said processors during multiple time intervals.41. The system of claim
40 wherein said switching junctions successively connect all possible combinations of said different sets of multiple paired sets of said processors in a minimum number of steps.42. A method for updating a matrix of elements for use in determining the relationship among a plurality of variables in a computer processing system, comprising:
receiving individual input values from an input vector m[IN]; and
updating a variance-covariance matrix of elements ν associated with a processor according to the equation:
^{T}m[IN]+ν[IN]/(1+l), where l is a learning weight,
said elements being variance-covariance values ν defining elements of a variance-covariance matrix ν, each of said elements containing values that are operative to define the relationship among received input variables.
43. The method of claim
42 further comprising inverting said matrix of elements for use in providing output values from said input values.44. A processing system for computing output values from input values received during a series of time trials, comprising:
a processing unit operative to receive an input vector comprising a plurality of input data values for each time trial;
a memory unit, connected to the processing unit, containing elements defining a connection weight matrix that is the inverse of a covariance matrix based on regression analysis applied to input data values received during prior time trials;
said processing unit operative to provide, during each time trial, one or more output values calculated as a function of the input data values for the time trial and the elements of the connection weight matrix; and
said processing unit operative to update the elements of the connection weight matrix, during each time trial, based on covariance relationships among the input values received during the time trial.
45. The processing system of claim
44, wherein the output values for each time trial correspond to a weighted sum of the input data values for the time trial, in which the weights are defined by the elements of the connection weight matrix.46. The processing system of claim
44, wherein the processing unit is operative to access each element of the connection weight matrix in a sequential order.47. The processing system of claim
44, wherein the processing unit is operative to access the elements of the connection weight matrix in parallel.48. The processing system of claim
44, wherein the processing unit is operative to impute output values corresponding to missing input data values, during each time trial, based on the input data values for the time trial and the elements of the connection weight matrix.49. The processing system of claim
44, wherein the processing unit is operative to impute output values corresponding to deviant input data values, during each time trial, based on the input data values for the time trial and the elements of the connection weight matrix.50. The processing system of claim
44, wherein the input values correspond to measurements, and the output values correspond to expected values for the measurements.51. The processing system of claim
44, wherein the processing unit updates each element of the connection weight matrix, for each time trial, using a learning weight reflecting a weight given to the input values for the time trial relative to weights given to corresponding input values for previous time trials.52. The processing system of claim
51, further comprising a learning weight controller operative for changing the learning weight for each element of the connection weight matrix from time trial to time trial.53. The processing system of claim
52, wherein the learning weight controller is operative for disabling learning in response to detecting abnormal deviation of input values.54. The processing system of claim
52, wherein the learning weight controller is operative for receiving user inputs defining one or more of the learning weights.55. The processing system of claim
44, further comprising a feature function controller for receiving measurements and computing the input vector based on the measurements.56. The processing system of claim
55, wherein the feature function controller is operative for receiving user inputs defining values for the elements of the connection weight matrix.57. The processing system of claim
56, wherein the connection weight matrix is initialized through the user defined values for the elements of the connection weight matrix.58. The processing system of claim
55, wherein the feature function controller is operative for receiving output feature values from the processing unit and computing the output values based on the output feature values.59. The processing system of claim
55, wherein the feature function controller is operative for computing statistics to identify sudden changes in input values.60. The processing system of claim
55, wherein the feature function controller is operative for computing statistics to identify sudden changes in output values.Description The present application is a continuation-in-part of U.S. patent application Ser. No. 08/333,204 filed on Nov. 2, 1994. Generally, the present invention relates to the field of parallel processing neurocomputing systems and more particularly to real-time parallel processing in which learning and performance occur durling a sequence of measurement trials. Conventional statistics software and conventional neural network software identify input-output relationships during a training phase, and each apply the learned input-output relationships during a performance phase. For example, during the training phase a neural network adjusts connection weights until known target output values are produced from known input values. During the performance phase, the neural network uses connection weights identified during the training phase to impute unknown output values from known input values. A conventional neural network consists of simple interconnected processing elements. The basic operation of each processing element is the transformation of its input signals to a useful output signal. Each interconnection transmits signals from one element to another element, with a relative effect on the output signal that depends oil the weight for the particular interconnection. A conventional neural network may be trained by providing known input values and output values to the network, which causes the interconnection weights to be changed. A variety of conventional neural network learning methods and models have been developed for massively parallel processing. Among these methods and models, back propagation is the most widely used learning method and the multi-layer perceptron is the most widely used model. Multi-layer perceptrons have two or more processing element layers, most commonly an input layer, a single hidden layer and an output layer. The hidden layer contains processing elements that enable conventional neural networks to identify nonlinear input-output relationships. Conventional neural network learning and performing operations can be performed quickly during each respective stage, because neural network processing elements can perform in parallel. Conventional neural network accuracy depends on data predictability and network structure that are pre-specified by the user, including the number of layers and the number of processing elements in each layer. Conventional neural network learning occurs when a set of training records is imposed on the network, with each such record containing fixed input and output values. The network uses each record to update the network's learning by first computing network outputs as a function of the record inputs along with connection weights and other parameters that have been learned up to that point. The weights are then adjusted depending on the closeness of the computed output values to the training record output values. For example, suppose that a trained output value is 1.0 and the network computed value is 0.4. The network error will be 0.6 (1.0−0.4=0.6), which will be used to determine the weight adjustments necessary for minimizing the error. Training occurs by adjusting weights in the same way until all such training records have been used, after which the process is repeated until all error values have been sufficiently reduced. Conventional neural network training and performance phases differ in two basic ways. While weight values change during training to decrease errors between training and computed outputs, weight values are fixed during the performance phase. Additionally, output values are known during the training phase, but output values can only be predicted during the performance phase. The predicted output values are a function of performance phase input values and connection weight values that were learned during the training phase. While input-output relationship identification through conventional statistical analysis and neural network analysis may be satisfactory for some applications, both Such approaches have limited utility in other applications. Effective manual data analysis requires extensive training and experience, along with time-consuming effort. Conventional neural network analysis requires less training and effort, although the results produced by conventional neural networks are less reliable and harder to interpret than manual results. A deficiency of both conventional statistics methods and conventional neural network methods results from the distinct training and performance phases implemented by each method. Requiring two distinct phases causes considerable learning time to be spent before performance can begin. Training delays occur in manual statistics methods because even trained expert analysis takes considerable time, and training delays occur in neural network methods because many training passes through numerous training records are needed. Thus, conventional statistical analysis is limited to settings where (a) delays are acceptable between the time learning occurs and the time learned models are used, and (b) input-output relationships are stable between the time training analysis begins and performance operations begin. Thus, there is a need in the art for an information processing system that may operate quickly to either learn or perform or both within a time trial. Generally described, the present invention provides a data analysis system that receives measured input values for variables during a time trial and learns relationships among the variables gradually by improving learned relationships from trial to trial. Additionally, in some embodiments if any input values are missing, the present invention provides, during the time trial, an (imputed) output value for the missing value that are based on the prior learned relationships among the analyzed variables. More particularly, an embodiment of the present invention may provide the imputed values by implementing a mathematical regression analysis of feature values that are functions of the input values. The regression analysis may be performed by utilizing a matrix of connection weights to predict each feature value as a weighted sum of other feature values. Connection weight elements are updated during each trial to reflect new connection weight information from trial input measurements. Also, a component learning weight is also utilized during each trial that determines the amount of impact that the input measurement vector has on learning relative to prior vectors received. Embodiments of the present invention may process the input values in parallel or process the values sequentially. The different input values may be provided in the form of vectors. Each of the values of the input feature vector is operated on individually with respect to prior learned parameters. In the parallel embodiment, a plurality of processors process the input values, with each processor dedicated to receive a specific input value from the vector. That is, if the system is set up to receive sixteen input feature values (i.e., corresponding to a vector of length sixteen), sixteen processing units are used to process each of the input feature values. In the sequential embodiment, one processor is provided to successively process each of the input feature values. In a parallel embodiment of the present invention, each of the processing units is operative to receive, during a time trial, individual input values from an input vector. A plurality of conductors connect each of the processing units to every other processing unit of the system. The conductors transfer weighted values among each of the processor units according to processes of the present invention. Each of the processing units provide, during the time trial, an imputed output value based upon the weighted values. Also, during the same time trial, each of the processing units is operative to update connection weights for computing the weighted values based on the input values received. Due to the limited number of outputs that a particular processor may drive, when interconnecting many processing units in parallel for the processing of data, the number of processing units that may be interconnected or driven by a single processing unit can be substantially limited. However, the present invention provides a plurality of switching junctions located along the conductors for interconnecting the processors to alleviate the problem associated with a single processor communicating with many others. The switching junctions are operable for uniquely pairing each of the processors to every other processor of the system. Additionally, in embodiments of the present invention, output values and learned values may be evaluated and controlled by controller units within the information processing system. A learning weight controller may be provided that automatically adjusts the learning weight from trial to trial in a manner that generally regulates the relative effect that each input vector has on prior learning. Additionally, a user may interface with the system to provide desired learning weights different than the learning weights that may be automatically provided by the system. Also, the present invention may provide a feature function controller that is operative to convert measurement values initially received by the system to input feature vectors for imputing and learning use by the system. The feature function controller is also operative to either provide default initial connection weights or receive connection weight elements externally so that a user of the system may supply initial weights as desired. While the above embodiments of the present invention are advantageous in updating connection weights in neural network type applications, alternative operations of the system include: (1) updating a coefficient matrix that is used by the Davidon-Fletcher-Powell (DFP) numerical optimization algorithm; (2) multiplying a symmetric matrix by a vector; and (3) adjusting the connection weight matrix for deleted features during feature function control. With respect to the numerical optimization application, the DFP method is one of several iterative methods for finding the maximum (or minimum) independent variable values for a function of several variables. Numerical optimization methods are generally useful but are also generally slow. For example, numerical optimization methods are used to find optimum values associated with five-day weather forecasts, but generally take many hours to converge, even on supercomputers. Among the numerical optimization methods, the DFP is especially useful in a variety of applications, because the DFP method learns derivative information during the iterative search process that may not be readily available. Just as the parallel system process is used to implement a fast concurrent information processing system, a modified version of is used for a fast new numerical optimization system. The DFP method continuously updates the inverse of a matrix as part of normal operation. Instead of updating the inverse of a covariance matrix as in the CIP system, the DFP algorithm updates the inverse of an estimated matrix of second-order derivatives, which is called the information matrix. Although the formula for updating the DFP inverse is distinct from the formula for updating the CIP inverse, an extension to the parallel CIP system algorithm can be used for DFP updating. The present invention also includes a less computationally involved embodiment that may be implemented where multiplication of a symmetric matrix by a vector is performed repeatedly and quickly. The parallel system embodiment may be simplified to compute Such products, by preserving only operations that are needed to compute Such products and removing all others. As, with the parallel CIP system embodiment and all other tailored versions, using parallel processing instead of sequential processing will produce results that are faster. Thus, it is an object of the present to provide an information processing system that provides accurate learning based on input values received. It is a further object of the present invention to convert input measurement values to input feature values during a single time trial. It is a further object of the present invention to provide learning and performance (measurement and feature value imputing) during a single time trial. It is a further object of the present invention to impute missing values from non-missing values. It is a further object of the present invention to identify unusual input feature deviations. It is a further object of the present invention to provide a system for learning and performing quickly during a single time trial. It is a further object of the present invention to identify sudden changes in input feature values. It is a further object of the present invention to provide a system for quickly processing input feature values in parallel. It is a further object of the present invention to provide a system for quickly processing input feature values sequentially. It is a further object of the present invention to provide a system that enables multiple parallel processing units to communicate among each of the processing units of the system quickly. It is a further object of the present invention to provide a system that enables multiple parallel processing units to be accessed in pairs. It is a further object of the present invention to provide communication between paired processors in a minimal number of steps. It is a further object of the present invention to provide processes that accomplish the above objectives. These and other objects, features, and advantages of the present invention will become apparent from reading the following description in conjunction with the accompanying drawings. FIG. 1 illustrates the preferred embodiment of the present invention. FIG. 2 is a block diagram that illustrates a parallel processor embodiment of the preferred embodiment of the present invention. FIG. 3 is a block diagram that illustrates a sequential computer embodiment of the preferred embodiment of the present invention. FIG. 4 shows an array of pixel values that may be operated on by the preferred embodiment of the present invention. FIG. 5 shows a circuit layout for the joint access memory and processors used in the parallel embodiment of the present invention. FIG. 6 FIG. 6 FIG. 7 shows timing diagrams for joint access memory control during intermediate matrix/vector operations of the parallel embodiment. FIG. 8 shows timing diagrams for joint access memory control timing for updating operations associated with a switching junction of the joint access memory of the preferred embodiment of the present invention. FIG. 9 shows processing time interval coordination for parallel embodiment of the preferred embodiment of the present invention. FIG. 10 shows a block diagram of the overall system implemented in the parallel embodiment of the preferred embodiment of the present invention. FIG. 11 shows a block diagram of the overall system implemented in the sequential embodiment of the preferred embodiment of the present invention. FIG. 12 shows communication connections for a controller used in the parallel embodiment of the preferred embodiment of the present invention. FIG. 13 shows communication connections for another controller used in the parallel embodiment of the preferred embodiment of the present invention. FIGS. 14 through 22 are flow diagrams showing preferred steps for the processes implemented by the preferred embodiment of the present invention. Referring to the figures, in which like numerals refer to like parts throughout the several views, a concurrent learning and performance information processing (CIP) neurocomputing system made according to the preferred embodiment of the present invention is shown. Related U.S. patent application Ser. No. 08/333,204 is incorporated herein by reference. Referring to FIG. 1, a CIP system Generally, when the CIP system The CIP system Upon receiving an input measurement record at the beginning of a trial, the CIP system The CIP system is useful in many applications, such as continuous and adaptive: (a) instrument monitoring in a chemically or radioactively hostile environment; (b) on-board satellite measurement monitoring; (c) missile tracking during unexpected excursions; (d) in-patient treatment monitoring; and (e) monitoring as well as forecasting competitor pricing tactics. In some applications high speed is less critical than in others. As a result, the CIP system has provision for either embodiment on conventional (i.e., sequential) computers or embodiment on faster parallel hardware. Although speed is not a major concern in some applications, CIP high speed is an advantage for broad utility. Sequential CIP embodiment is faster than conventional statistics counterparts for two reasons: first, the CIP system uses concurrent updating instead of off-line training; second, the CIP system updates the inverse of a certain covariance matrix directly, instead of the conventional statistics practice of computing the covariance matrix first and then inverting the covariance matrix. Concurrent matrix inverse updating allows for fast CIP implementation. When implemented using a sequential process, CIP response time increases as the square of the number of data features utilized increases. However, when implemented using a parallel process, CIP response time increases only as the number of features utilized increases. In the parallel system, a processor is provided for each feature. As a result, parallel CIP response time is faster than sequential CIP response time by a factor of the number of features utilized. Referring to FIG. 2, a parallel embodiments of the basic subsystems of the CIP system The basic components of the transducer The input processor The input measurement vector j[IN] received by the transducer input processor Prior to concurrent operation, the transducer input processor Feature viability elements are computed as products of corresponding measurement plausibility elements. Every CIP system input measurement value is treated as an average of non-missing quantum measurement values from a larger set, some of which may be missing. The corresponding plausibility value of each input measurement is further treated as the proportion of component quanta that are non-missing within the larger set. From probability theory, if an additive or product composite feature function is made up of several such input measurements and if the distributions of missing quanta are independent between measurements, the expected proportion of terms in the composite for which all quantum measurements are non-missing is the product of the component measurement plausibility values. Since feature viability values within the CIP system have this expected proportion interpretation, the feature viability values are computed as products of component measurement plausibility values. After input measurement values j[IN] and plausibility values p have been converted to input feature vectors m[IN] and viability values ν by the transducer input processor Kernel processors The kernel input learning weight l is a non-negative number that—like input plausibility values and viability values—is a quantum/probabilistic measure. The learning weight I for each trial is treated by the CIP system as a ratio of quantum counts, the numerator of which is the number of quantum measurement vectors for the concurrent trial, and the denominator of which is the total of all quantum measurements that have been used in prior learning. Thus, if the concurrent input feature vector m[IN] has a high learning weight l value, the input feature vector will have a larger impact on learned parameter updating than if the input feature vector has a lower learning weight l value, because the input feature vector in[IN] will contain a higher proportion of the resulting plausible quantum measurement total. Normally, the learning weight l is supplied as an input variable during each trial, but the learning weight can also be generated optionally by the CIP system manager Kernel imputing, memory updating and monitoring operations are based on a statistical regression framework for predicting missing features as additive functions of non-missing feature values. Within the regression framework, the weights for imputing each missing feature value from all others are well-known. Formulation for the weights used for imputing are functions of sample covariance matrix inverses. In the conventional approach to regression, the F by F covariance matrix is computed first ν based on a training sample, followed by inverting the covariance matrix and then computing regression weights as functions of the inverse. The conventional approach involves storing and operating with a training set that includes all measurements received up to the current input trial. Storing all prior measurements is typical for conventional systems, because all prior measurements are needed in order to first calculate present covariances from which the inverse matrix may be obtained. Unlike conventional statistics operations, CIP kernel The process of updating the inverse elements of ν is the CIP counterpart to conventional learning. CIP fast updating capability from trial to trial provides a statistically sound and fast improvement to conventional learning from off-line training data. As a result, the CIP System With continuous reference to FIG. 2, the joint access memory Once the kernel Once imputed measurement valuesj[OUT] have been produced as outputs, the outputs, can be useful in several ways, including: (a) replacing direct measurement values, such as during periods when instruments break down; (b) predicting measurement values before the measurements occur, such as during econometric forecasting operations; and (c) predicting measurement values that may never occur, such as during potentially faulty product classification operations. The manager The CIP system may also perform feature value monitoring operations, which are performed by the kernel In addition to specifying feature imputing, feature value monitoring and learned parameter updating operations concurrently, the CIP manager As with CIP imputing and learned parameter updating operations, CIP feature function monitoring and control operations are based on a statistical regression framework. For example, all of the necessary partial correlation coefficients and multiple correlation coefficients for identifying redundant or unnecessary features can be computed from the elements of ν inverse that reside in the joint access memory The probability/quantum basis for learning weight interpretation allows learning weight schedules to be computed that will produce: (a) equal impact learning, through which each input feature vector will have the same overall impact on parameter learning; (b) conservative learning, through which less recent input feature vectors will have higher overall impact on parameter learning than more recent input feature vectors; and (c) liberal learning, through which more recent input feature vectors will have lower overall impact. When the learning weight controller Referring to FIG. 3, a block diagram illustrating the CIP system Just as in the parallel system embodiment, the sequential system receives an input vector j[IN] and a plausibility value p. As in the parallel system, the input vectors j[IN] are also converted to input feature values m[IN]. Plausibility values p are converted to viability values ν as discussed above. The kernel process The executive block During each concurrent trial, the executive In addition to concurrent operations, the sequential embodiment may utilize occasional refinement operations, as discussed above in connection with the parallel system. In the sequential version the executive An Example of CIP Imputing The following example illustrates some CIP operations. Referring to FIG. 4, three binary pixel arrays that could represent three distinct CIP input measurement vectors are shown. Each array has nine measurement variables, labeled as x(1,1) through x(3,3). The black squares may be represented by input binary values of 1, while the white squares may be represented by binary input values of 0. The three arrays can thus be represented as CIP binary measurement vectorsj[B] having values of (1, 0, 0, 0, 1, 0, 0, 0, 1,),(0, 0, 1, 1, 1, 0, 0, 0, 0) and (1, 0, 1, 0, 1, 0, 0, 0, 0). As noted above, the CIP system uses each plausibility value in p to establish missing or non-missing roles of its corresponding measurement value inj[N]. Thus, if all nine p values corresponding to the FIG. 4 measurements are 1 then all nine j[IN] values will be used for learning. However, if two p elements are 0, indicating that the two corresponding j[IN] values are missing, the two corresponding J[OUT] values will be imputed from the other seven j[IN] values that have corresponding values of 1. With continuing reference to FIG. 4, assume that at trial number If the CIP system has been set up for equal impact learning operation, the pattern that occurred most between the possible 70a and 70c patterns in previous trials number 1 through 100 would be imputed. Suppose, for example, that in previous trials 1 through 100 all nine values of p were 1 for each such trial, and j[IN] values corresponded to types 70a, 70b and 70c for 40, 19 and 41 such trials, respectively. In this example, the unknown upper right and lower right pixels during trial Transducer Input Operation Input measurements can be termed arithmetic, binary and categorical. Arithmetic measurements such as altitude, temperature and time of day have assigned values that can be used in an ordered way, through the arithmetic operations of addition, subtraction, multiplication and division. When a measurement has only two possible states, the measurement may be termed binary and may be generally represented by a value of either 1 or 0. Non-arithmetic measurements having more than two possible states may be termed categorical. CIP measurement vectorsj may thus be grouped into arithmetic, binary and categorical sub-vectors (j[A], j[B],j[C]), where each sub-vector can be of any length. Depending on how options are specified, the CIP system either (a) converts arithmetic and binary measurement values to feature values in the transducer input processor After the input processor Recent features can be used to both monitor and impute (or forecast) concurrent feature values as a function of previously observed feature values stored in the recent feature memory Plausibility and Viability Details. In addition to operating with binary plausibility values as discussed in connection with FIG. 4, the CIP system can be implemented to operate with plausibility and viability values between 0 and 1. Values between 0 and 1 can occur naturally in several settings, such as preprocessing operations outside the CIP system. Suppose, for example that instead of 9 measurements only 3 measurements corresponding to average values for array rows Plausibility values between 0 and 1 also may be used in settings where CIP system users wish to make subjective ratings of measurement reliability instead of calculating plausibility values objectively. The CIP system treats a plausibility value between 0 and 1 as a weight for measurement learning, relative to previous values of that measurement as well as concurrent values of other measurements. The processes and formulations for plausibility based weighting schemes are discussed below. Kernel Learning Operation As noted above, in order for the CIP system to provide useful data based on a set of measurements, the CIP system identifies relevant parameters and implements an accurate process for learning the relationships among the multiple input measurements. The CIP kernel learns during each trial by updating learned parameter estimates, which include: a vector μ of feature means, a matrix ω of connection weights, a vector ν[D] of variance estimates (diagonal elements of the previously described variance-covariance matrix ν) and a vector λ of learning history parameters. Updating formulas for the learned parameters are discussed below, but simplified versions will be discussed first for some parameters to illustrate basic properties. The mean updating formula takes on the following simplified form if all prior and concurrent viability values are 1:
The term μ[OUT] represents the mean of all prior measurements up to and including the current measurement value. Equation (1) changes μ values toward m[IN] values more for higher values of l than for lower values of l, in accordance with the above learning weight discussion. Equation (1) is preferably modified, however, because Equation (1) may not accurately reflect different plausibility histories for different elements of μ[IN]. Instead, Equation (1) is modified to combine μ[IN] with m[IN] according to elements of a learning history parameter λ, which keeps track of previous learning history at the feature element level. Equation (1) can be justified and derived within the quantum conceptual framework for the CIP system, as follows: suppose that the learning weight l is the ratio of concurrent quantum counts for m[IN] to the concurrent quantum counts q[PRIOR] associated with μ[IN], as the CIP system does; suppose further that μ[IN] is the mean of q[IN] prior quantum counts and that m[IN] is the mean of q[PRIOR] concurrent quantum counts; algebra can show that Equation (1) will be the overall mean that is based on all q[IN] prior counts along with all q[PRIOR] concurrent counts. Equation (1) only applies when all viability values are 1. The CIP system preferably uses a more precise function than Equation (1), in order to properly weight individual elements of μ[OUT] differentially, according to the different viability histories of μ[OUT] elements. The mean updating formula is, for any element μ(ƒ) of μ,
(ƒ=1, . . . , F—such “ƒ” labeling is used to denote array elements in the remainder of this document). Equation (2) resembles Equation (1), except a single learning weight l as in Equation (1), which would be used for all feature vectors elements during a trial, is replaced by a distinct learning weight l[C](ƒ) for each component feature vector element. Thus, each feature vector element may be individually rather than each feature vector element have the same weight as in Equation (1). These component learning weights, in turn, depend on concurrent and prior learning viability values of the form.
The learning history parameter λ is also updated, after being used to update feature means, to keep a running record of prior learning for each feature, as follows: λ[OUT](ƒ)=λ[IN](ƒ)(1+l)/(1+l[C](ƒ)). (4) The remaining learned parameters ν[D] and ω are elements of the covariance matrix ν and ν inverse respectively. Also, ν depends on deviations of features values from means values instead of features values alone commonly known as errors. As a result, ν[D] and (ω) may be updated, not as functions of features values alone, but instead as functions of error vectors having the form,
An appropriate formula for updating the elements of ν might be,
(the T superscript in Equation (6), as used herein, denotes vector transposition). Equation (6) is similar in general form to Equation (3), because Equation (3) is based on the same CIP quantum count framework. Just as Equation (3) produces an overall average μ[OUT] of assumed quantum count values, Equation (6) produces an overall average ν[OUT] of squared deviance and cross-product values from the mean vector μ[OUT]. An appropriate formula for updating the elements of (ω) based on Equation (6) is
where
Equation (7) is based on a standard formula for updating the inverse of a matrix having form (6), when the inverse of the second term in (6) is known. Equation (7) is also based on the same quantum count rationale as Equations (3) and (6). Equations (5) through (8) are only approximate versions of the preferred CIP error vector and updating formulas. However, different and preferred alternatives are used for four reasons: first, the CIP system counterpart to the error vector formula Equation (5) is based on μ[IN] instead of μ[OUT], since the CIP kernel can update learned parameters more quickly utilizing μ[IN], thus furthering fast operation. Second, the preferred CIP embodiment equation to Equation (5) reduces each element of the error vector e toward 0 if the corresponding concurrent viability ν(ƒ) is less than 1. This reduction gives each element of the e vector an appropriately smaller role in updating elements of ν and ω if the e vector's corresponding element of m[IN] has a low viability value. Third, the CIP kernel does not require all elements of ν but uses the elements of ν[D] instead where [D] represents diagonal elements. Finally, Equation (6) and Equation (7) are only accurate if previous μvalues that have been used to compute previous μ[OUT] and ω[OUT] values are the same as μ[OUT] for the current trial. Since all such μ values change during Equation (7) the CIP system uses an appropriate modification to Equation (7) in the preferred CIP embodiment. The preferred alternatives to Equation (5) through (8) are,
and
where
and
The ω[IN], μ[IN] and ν[D,IN] values represent values that were the output values from the previous trial that were stored in the learned parameter memory. Kernel Imputing Operations CIP feature imputing formulas are based on linear regression theory and formulations, that fit CIP storage, speed, input-output flexibility and parallel embodiment. Efficient kernel operation is enabled because regression weights for imputing any feature from all other features can be easily computed from elements of ν inverse, which are available in ω. The CIP kernel imputes each missing m[IN] element as a function of all non-missing m[IN] elements, where missing and non-missing m[IN] elements are indicated by corresponding viability ν element values of 0 and 1, respectively. If a CIP application utilizes F features and the viability vector for a trial contains all 1 values except the first element, which is a 0 value, the CIP kernel imputes only the first feature value as a function of all others. The regression formula for imputing that first element is, m[OUT](1)=μ(1)−{[m[IN](2)−μ(2)]ω(2,1) + . . . +[m[IN](F)−μ(F)]ω(F,1)}/ω(1,1)] (15) Formulas for imputing other in[IN] elements are similar to Equation (15), provided that only the m[IN] element being imputed is missing. The CIP kernel uses improved alternatives to Equation (15), so that the CIP system can operate when any combination of m[IN] elements may be missing. When any element is missing, the kernel imputes each missing m[IN] element by using only other in[IN] elements that are non-missing. The kernel also replaces each m[OUT] element by the corresponding m[IN] element whenever m[IN] is non-missing. The regression formulas used by the CIP system are also designed for parallel as well as efficient operation that makes maximum use of other kernel computations. For example, the kernel saves time and storage by using the elements of e[IN] and x from Equation (9) and Equation (14) for imputing, because e[IN] and x are also used for learning. The kernel imputing formula is,
Monitoring Operations The kernel produces several statistics for feature value monitoring and graphical display. these include the learned feature mean vector μ[OUT], feature variances ν[D] and d from Equation (15), which is a well-known statistical monitoring measure called Mahalanobis distance. The kernel also produces another set of regressed feature values, which arc the imputed values that each feature would have if the feature was missing. These regressed values have the form, {circumflex over (m)}(ƒ)=μ[IN](ƒ)−x(ƒ)/ω(ƒ,ƒ)+e[IN](ƒ). (17) Given the above monitoring statistics from the kernel, the CIP system can use the statistics in several ways, as specified by user options. One use is to plot deviance measures as a function of trial number, including the Mahalanobis distance measure d, standardized squared deviance values from learned means,
and standardized squared deviance values between regressed values,
The Mahalanobis distance measure d and the three deviance measures of equations (17), (18), and (19) are useful indices of unusual input behavior. The Mahalanobis distance measure d is a useful global measure for the entire feature vector, because d is an increasing function of the squared difference between each observed feature vector and the feature learned mean vector. The standardized deviance measures d[1](ƒ) are component feature counterparts to the global measure d, which can help pinpoint unusual feature values. The standardized residual measures d[2](ƒ) indicate how input features deviate from their regressed values, based not only oil their previously learned means but also on other non-missing concurrent feature values. The CIP system can also use special features in conjunction with their monitoring statistics to produce useful information about unusual feature trends. For example, for any feature of interest, a new feature can be computed that is the difference between the concurrent feature value and the feature value from the immediately preceding trial. The resulting deviance measure from Equation (17) provides a useful measure of unusual feature value change. The CIP system can also use a similar approach based on second-order differences instead of first-order differences to identify unusual deviations from ordinary feature changes. The CIP system can thus provide a variety of graphical deviance plots for manual user analysis outside the CIP system. The CIP system can also use deviance information internally to control learning weights and schedule feature modification operations. For example, the system can establish a preselected cutoff value for the global distance measure d. The system can then treat a d value exceeding the preselected cutoff value as evidence of a data input device problem, and the system can set future learning weight values to 0 accordingly until the problem is fixed. Likewise, the component deviance measures d[1](ƒ) and d[2](ƒ) can be used to set measurement plausibility or feature viability values to 0 after the component deviance measures have exceeded pre-specified cutoff values. Setting the learning weight to zero prevents input problems from adversely affecting the accuracy of future CIP operations. The firm statistical basis of the CIP system enables the CIP system to be useful for such decision applications, because the distance measures follow chi-square distributions in a variety of measurement settings. As a result, distance cutoff values can be deduced from known chi square cumulative probability values. Learning Weight Control Operation The CIP system uses the learning weight l as part of the system learning. The learning weight l is the ratio of a quantum count associated with the concurrent feature vector to the quantum count associated with prior parameter learning. From that basis, the CIP system produces equal impact learning weight sequences, that is, sequences based on an equal number of quantum counts for each trial. If a learning weight sequence is labeled by l(1), l(2) and so on, equal impact schedules have the form,
l(3)=1/(R+2) (22) and so on. The constant R is the ratio of the common quantum count for all such trials to an initial quantum count. The role of this ratio and the initial quantum count is discussed below. In addition to providing equal impact learning weight sequences, CIP users or the CIP system can generate sequences that are either liberal or conservative. Liberal sequences give more impact to more recent trial feature values, while conservative sequences give more impact to less recent trial feature values. For example, a learning sequence with all learning weights set to 1 is liberal, while one that sets all but the first learning weight to 0 is conservative. Liberal sequences are appropriate when the input CIP data are being generated according to continuously changing parameter values, and conservative sequences are appropriate when more recent information is not as reliable as less recent information. Learned Parameter Initialization The CIP system treats initial values for learned regression parameters μ, ν[D] and ω as if they were generated by observed feature values. During the first trial the CIP kernel combines initial values with information from the first feature vector to produce updated parameter values, according to Equations (2), (10) and (11); and during the second trial the CIP kernel combine initial values and first trial values with information from the second vector to produce new updated parameter values. This process is repeated for subsequent trials. After any trial having a positive learning weight l, the impact of initial parameter values on overall learning will be less than the initial parameter impact before the trial. As a result, effects due to particular initial regression parameter values will be small after a small number of learning trials, unless very conservative learning weight sequences are provided to the kernel. In some applications where accurate imputing by the CIP system may be required from the first trial on, initial values for learned regression parameters can be important. For accurate early imputing in Such applications, the CIP system may accept user-supplied initial regression parameter values from a keypad The CIP system provides default initial values for learned regression parameters as follows: the default value for each element of the mean vector μ is 0; the default value for the connection weight matrix ω is the identity matrix; and the default value for each element of the variance vectorν[D] is 1. Using the identity matrix as the initial default value for ω produces initial imputed feature values that do not depend initially on other feature values. The initial identity matrix also enables the CIP system to impute feature values from the first trial onward. By contrast, conventional statistical approaches require that at least F learning trials (where F is the number of features) before any imputing can occur. In addition to initializing learned regression parameters, the CIP system initializes elements of the learning history parameter vector λ. The learning history parameter vector dictates how much an input feature vector element will affect learning, relative to previous learning. The default initial value for each element of the learning history vector λ is 1, which gives each input feature vector element the same impact on learning during the first learning trial. Feature Function Monitoring Operation The CIP system alternatively may implement three kinds of feature function monitoring statistics for graphical display: a vector of squared feature multiple correlations χ[M], a vector of tolerance band ratio values r and an array of partial correlations χ[P]. Each element χ[M](ƒ) of χ[M] is the squared multiple correlation for imputing the corresponding feature vector element m(ƒ) from the other elements in m. When optionally implemented in the CIP system, squared multiple correlations can be interpreted according to well-known statistical properties. Such statistical properties imply that each feature can be predicted by other features if the feature's squared correlation is near the maximum possible value of 1 instead of the minimum possible value of 0. Each squared multiple correlation χ[M](ƒ) also may be optionally used to compute and supply the corresponding tolerance band ratio element r(ƒ). Each element of r can be expressed as a ratio of two standard deviations. The numerator standard deviation is the square root of ν[D](ƒ), while the denominator standard deviation is the standard deviation of {circumflex over (m)}(ƒ). Since error tolerance band widths are routinely made proportional to standard deviations, it follows that each r(ƒ) value is the tolerance band width for imputing m(ƒ) if all other m[IN] elements are not missing, relative to the tolerance band width for imputing m(ƒ) if all other m[IN] values are missing. The partial correlation array χ[P] contains a partial correlation χ[P](ƒ,g) for each possible pair of features ƒ and g(ƒ=1, . . . , F−1; g=1, . . . F) Each partial correlation is an indexχof how highly two features are correlated, once they have been adjusted for correlations with all other features. As a result, users can examine the partial correlations to decide if any given feature is unnecessary for imputing any other given feature. Users can also examine rows of the partial correlation matrix to identify if a pair of features can be combined to produce an average, instead of being used separately. For example, suppose that two features are needed to impute a third feature and each partial correlation for the first feature is the same as the corresponding partial correlation for the second feature. Both such feature values can then be replaced by their average value for imputing the third feature value, without loss of imputing accuracy. An advantage provided by the CIP system is concurrent operation capability in conjunction with occasional feature function assessment by the manager
the system may use the following formula for tolerance band values, r(ƒ)=(1−χ[M](ƒ)) the system may use the following formula for partial correlation values,
In addition to supplying the above feature function assessment statistics to users, the CIP system can also supply the connection weight matrix ω to users for user modification and interpretation. For example, users can compute and assess principal component coefficients, orthogonal polynomial coefficients and the like from ω to identify essential features to fit user needs. Once a user has identified essential features, the user can either reformulate the input transducer functions or supply features outside the CIP system, accordingly. Feature Function Control Operation In addition to supplying feature assessment statistics and ω elements for manual external use, the CIP system can use the statistics internally and automatically, through its feature function controller (FIGS. In addition to modifying transducer operations during changes in feature function specification, the CIP system can also modify elements of the connection weight matrix ω. The elements of ω can be adjusted for the removal of an unnecessary feature, for example, feature ƒ, to produce a new, adjusted connection weight matrix with one less row and one less column, say ω{ƒ,ƒ}, as follows. If the submatrix of ω excluding row ƒ and column ƒ is labeled by ω<ƒ,ƒ> and the deleted row ƒ is labeled by ω<ƒ>, then an appropriate adjustment formula based on a standard matrix algebra function is,
Parallel Kernel Operation As noted above regarding the CIP System discussed in connection with FIG. 2, the parallel CIP kernel Because processors have a limited number of outputs that may be driven and a processor is utilized to process feature values, implementing a larger number of features could readily exceed the number of outputs that a processor can drive. The parallel CIP system solves the output problem by providing a joint access memory Referring to FIG. 5, a circuit layout of the conductors and interconnections among parallel kernel processors is shown, for F=16. The illustrated circuitry enables each feature processor to be connected to elements of the joint access memory In addition to the sixteen feature processors The circuitry illustrated in FIG. 5 may be implemented in a silicon chip layout, containing a lower bus layer, an upper bus layer, a set of semiconductor layers between and connecting the lower and upper bus layers and a control bus layer above all of the other layers. Lines With continuing reference to FIG. 5, each of the circles M(2,1), M(3,1) and M(3,2) through M(16,15) represents a JAM memory and switching node containing a switching junction, switching logic and a memory register, all of which may lie within semiconductor layers between the lower bus layer and the upper bus layer. The circles, M(16, 1) through M(16,15) within the feature processors Each of the sixteen lower buses Each of the lower buses Each of the upper buses The lower and upper buses and the interconnections include one line of conductor for each storage bit that is implemented by the CIP kernel Referring additionally to FIGS. 6 FIG. 6 FIG. 6 Switches S Five basic switching operations ((a)-(e)) are performed that implement the circuitry illustrated in the joint access memory Parallel kernel With continuing reference to FIG. 5, in computing χwhen F=16, x(1) through x(16) are computed by feature processors Each feature processor F computes its feature processor x(F) value by first initializing the x value at 0 and then accessing joint access memory nodes along the feature processor's lower and upper bus, one at a time. During each access, each feature processor F performs the following sequence of operations: first, fetching the stored ω element along that node; second, fetching the e[IN] element that is available at that node; third, multiplying the two elements together to obtain a cross-product; and fourth, adding the cross-product for the processor to the running sum for x(F) implemented in the processor. For example, the feature processor of focus may be processor FIG. 6 illustrates the control unit timing for the x updating step described above. The top signal illustrates a CIP system clock pulse as a function of time and the next 7 plots below the graph show the switch control values along lines C With respect to updating the elements of ω, when updating ω(16,15) according to Equation (11), ω[IN](16,15), x(15) and x(16) are all first available in a single processor. Tile single processor then computes ω[OUT](16,15) according to equation (11), after which the processor sends the updated value to the storage cell for ω(16,15). Referring to FIG. 8, control timing for the ω(16,15) updating sequence of operations is shown. The system clock pulses are shown as a function of time and the four plots below the clock pulse show control values along lines C Referring to FIG. 9, a coordination scheme for x and ω updating processor operations is shown. Each entry in FIG. 9 shows the time interval during which a processor is performing by itself or with one other feature processor at every interval in the overall x or ω updating process. The triangular table labeled as Feature processor pairing at any time interval is determined by locating in FIG. 9 the time interval along the processor's bus or within its processor. For example, the bus for processor The entries The numbers in FIG. 9 form a systematic pattern that can be used to identify processor operation steps during x computations. The following formula identifies the processor g that is accessed by processor ƒ during iteration i, as part of computing the matrix-vector product x during kernel step
For example, if F=16 then at time interval The number patterns in FIG. 9 also indicate a systematic pattern of control lines that can be used to implement processor operations during x computations. For example, when using sixteen-features, each given time interval number in the sequence falls along a line from the lower left boundary of the The same coordination that is formulated in Equation (27) for computing x is used by the CIP system for updating the elements of ω, with one exception. While x updating implements the computing the operations indicted in FIG. 7 at each Equation (27) interval i, ω updating implements computing operations indicated in FIG. 8 at each Equation (27) interval i. After x is computed, the values x(1) through x(F) and e[IN ](1) through e[IN]F will be residing in feature processors 1 through F, respectively. The Mahalanobis distance d is then computed according to Equation (14) as follows: first, the products x(1)×e[IN](1) through x(1)×e[IN](F) will be computed by processors The design of the joint access memory and connected processors is advantageous for compact embodiment in highly integrated circuitry. A compact embodiment is advantageous for optimal speed during each trial. Thus, the kernel The CIP parallel kernel also satisfies other design concerns: (a) a signal degradation concern (commonly known as fall-out)—minimizing the maximum number of inputs that a single feature processor of JAM elements supplies at any given time; and (b) a space utilization concern-minimizing the number of required conductors for communicating between feature processors and JAM elements. The CIP parallel kernel satisfies these various design concerns through the JAM bus and switching structure, along with parallel kernel feature processing coordination discussed above. It should be appreciated by those skilled in the art that the kernel Sequential Kernel Operation The sequential kernel utilizes the basic kernel operations that are discussed above in connection with the parallel kernel. Thus, sequential kernel operations produce the same outputs as parallel kernel operations whenever both kernel receive the same inputs. However, sequential operations will generally be slower, because they are obtained using only one processor instead of using F processors. Some of the sequential kernel operations are implemented in a different manner for efficiency, rather than identically simulating parallel kernel operations. The x computing step and the ω[OUT] computing step of the sequential kernel operation are implemented for optional storage and speed. Both steps are based on storing the elements of ω as a consecutive string containing ω(1,1), followed by ω(2,1), followed by ω(2,2), followed by ω(3,1), followed by (3,2), followed by (3,3) and so on to ω(F,F). The x computing step and ω[OUT] computing steps access the consecutively stored elements of ω from the first element to the last element. The overall effect is to make both such steps far faster than if they were to be performed conventionally, using a nested loop. The sequence of operations for computing x and ω updating steps are discussed in connection with FIG. Parallel System Operation Referring to FIG. 10, at the system level, separate subsystems can simultaneously perform: input transducer operations Referring to FIG. Management operations, which can be performed occasionally over a period of several trials, include learning weight control operations Within the learning weight controller After kernel Along with concurrent imputing, monitoring and learning operations, the CIP system occasionally monitors feature functions. The monitoring operations include receiving learned connection weight values and learned variance values from the kernel Changing feature specification is implemented by controlling the modification switching operation Feature function monitoring statistics satisfying Equation (23) through Equation (25), along with learned means and learned connection weights can be interpreted in a straightforward way, because the CIP system utilizes quantum-counts, “easy Bayes”, as discussed herein. Suppose that each learning weight from trial 1 to trial t, labeled by l(1) through l(t), is a ratio of quantum counts for each trial to the quantum counts for previous trials as follows: if the quantum counts for the initial parameter values along with features from trial 1 through t are labeled by q(0) through q(t), then l(1)=q(1)/q(0),
Suppose further that the initial mean vector is an average of q(0) quantum initial vectors, and input feature vectors m[IN] for trials 1 through t are averages of q(1) quantum vectors for trial 1 through q(t) quantum vectors for trial t, respectively. It then follows from statistical theory that all concurrently learned regression parameter values and all concurrently available refinement parameters can be interpreted as average statistics based on equally weighted quantum counts from an overall sample size of q(0)+q(1)+ . . . +q(t). For example, the learned feature mean vector at the end of trial 10 has the interpretation of an average among q(0)+q(1)+ . . . +q(10) quantum values, and the learned feature mean vector after any other number of trials has the same interpretation. As a second example, equal impact sequences satisfying Equation (20) through Equation 22) can be generated by setting q(1) =q(2)=q(3)=R, where R, is a positive constant; in that case, from algebra based on Equation (28), l(1)=1/R, 1(2)=1/(R+1), l(3)=1/(R+2) and so on as in Equation (20) through Equation (23), where R+q(0)/R As a result of using quantum-counts, “Easy Bayes,” used by the CIP system, feature regression parameters and feature function monitoring parameters can be concurrently evaluated from trial to trial and may be implemented more easily than alternative parameters that are available from either conventional statistics procedures or conventional neurocomputing procedures. As discussed above, the parallel CIP system can operate more quickly through the use of buffered communication. Also, discussed above, implementing as many CIP operations as possible on one chip can avoid considerable inter-chip communication time loss. As a result, implementing several CIP subsystems on different layers of a single chip and communicating between the layers through parallel buffering can maximize overall CIP operation speed. FIG. 11 shows memory locations Referring to FIG. 12, buffering to the feature function controller is shown. FIG. 12 shows: memory locations The geometrically aligned buffering in FIG. Sequential System Operation Referring to FIG. 13, the subsystems of the sequential kernel are shown. FIG. 13 shows more specifically than FIG. 3, the vectors and parameters that are transferred among the subsystems. The various inputs and outputs of the learning weight controller The sequential CIP system can perform all operations associated with the parallel system discussed above, although not as fast, because only one CIP operation is performed at time using a single available processor. Also, at the subsystem level simultaneous operations are not implemented as in the parallel kernel embodiment, because only one processor is available for kernel operations. Beyond speed concerns, however, the CIP system is no less powerful when implemented sequentially than it is when implemented using parallel processors. Also, the sequential CIP embodiment has at least two advantages over parallel embodiment: the sequential embodiment is generally less expensive because it may be embodied in a conventional computer rather than specially designed parallel circuits; and sequential embodiment can accommodate many more features per trial on conventional computers than the parallel embodiment can accommodate on specialized circuits. As a result, the sequential CIP embodiment is useful, in applications where trial occurrence rates are low relative to the number of features per trial. Alternative Kernel Implementations Alternative operations of the kernel include: (1) updating a coefficient matrix that is used by the Davidon-Fletcher-Powell (DFP) numerical optimization algorithm; (2) multiplying a symmetric matrix by a vector; (3) adjusting the connection weight matrix for deleted features during feature function control; and (4) training the kernel to become an input transducer. All four related applications are discussed below based on kernel modifications. Beginning with the numerical optimization application, the DFP method is one of several iterative methods for finding the maximum (or minimum) independent variable values for a function of several variables. Numerical optimization methods are generally useful but are also generally slow. For example, numerical optimization methods are used to find optimum values associated with five-day weather forecasts, but generally take many hours to converge, even on supercomputers. Among the numerical optimization methods, the DFP is especially useful in a variety of applications, because the DFP method learns derivative information during the iterative search process that may not be readily available. Just as the parallel kernel process is used to implement a fast concurrent information processing system, a modified version of can be used for a fast new numerical optimization system. In particular, if sequential DFP updating based on F independent variables takes s seconds for convergence to an optimal solution, then parallel DFP updating will require only about s/F seconds to converge. For example, suppose that five-day weather forecasting required optimizing a function of 2,000 variables, which in turn took 20 hours to converge using the conventional (sequential) DFP method. If the same optimization problem could be solved with a parallel counterpart to the DFP method resembling the parallel Kernel, convergence would take about 18 seconds. The DFP method continuously updates the inverse of a matrix as part of normal operation. Instead of updating the inverse of a covariance matrix as in the CIP system, the DFP algorithm updates the inverse of an estimated matrix of second-order derivatives, which is called the information matrix. Although the formula for updating the DFP inverse is distinct from the formula for updating the CIP inverse, an extension to the parallel CIP kernel algorithm can be used for DFP updating. The DFP information matrix inverse updating formula is,
where
and
The DFP Kernel updating formulas (29) through (33) may be tailored to suit DFP updating. In particular, the DFP counterpart to the kernel process utilizes the same number of steps, as the parallel CIP kernel A less computationally involved tailored kernel embodiment may be implemented where multiplication of a symmetric matrix by a vector is performed repeatedly and quickly. The kernel embodiment may be simplified to compute such products, of which Equation (13) is one example, by preserving only operations that are needed to compute such products and removing all others. As with the parallel CIP kernel embodiment and all other tailored versions, using parallel processing instead of sequential processing will produce results that are faster by about a factor of F. Regarding tailored kernel counterparts within the CIP system, the feature removal adjustment formula (26) is a simplified version of the kernel updating formula (11), in that: (a) the Equation (26) second term constant coefficient does not utilize a distance function, and (b) only an outer product among 2 vectors is needed to compute the Equation (26) second term, without first requiring a matrix-vector product as in Equation (13). As a result, the parallel CIP kernel can be simplified to solve Equation (26). Regarding tailored parallel CIP kernel operations for feature function modification and input transducer processing, “student input transducers” can first be “taught” to use only useful features, after which the operations can be used to produce features. For example, suppose that a CIP system is needed to forecast one dependent variable value feature 1 as a function of several independent variable values for feature 2 through feature 100. During a series of conventional learning trials a modification of the kernel process can learn to identify the 99 optimal connection weights for imputing feature 1 from feature 2 through feature 100. After the learning has occurred, the trained module can be used in place of an input transducer having 99 inputs corresponding to features 2 through 100 and only one output corresponding to feature 1. When used as an input transducer, the module would differ from the kernel in that the module's learning and updating operations would be bypassed. Thus, the only modifications of the kernel needed to implement such a module are an input binary indicator for learning versus feature imputing operation, along with a small amount of internal logic to bypass learning during feature imputing operation. Referring to FIG. At step Referring to FIG. 15, a discussion of the preferred steps of the processes of the preferred embodiment of the present invention continues. FIG. 15 illustrates the preferred process by which the intermediate matrix/vector product is calculated as discussed above. At step The process proceeds to step Referring to FIG. 16, the steps of the preferred embodiment of the present invention that compute the output values of the kernel subsystem The imputed feature vector m[OUT] is then computed at step Referring to FIG. 17, the steps of the processes for updating the connection weight matrix element ω(ƒ, g) in the preferred embodiment of the present invention are shown. At step At step If the final interval for the coordination time scheme has not been reached, then at step If at step Referring to FIG. At step Referring to FIG. 19, the preferred steps of the sequential CIP kernel processes implemented in the present invention for computing the intermediate matrix/vector x are shown. The process discussed in connection with FIG. 19 provides a method of calculating the intermediate matrix/vector x without performing a conventional double loop (i.e., one loop for all possible row values of a matrix and one loop for all possible column values of a matrix). The matrix ω elements are stored in a single string in consecutive order corresponding from ω(1,1) to ω(2,1) to ω(2,2) to ω(3,1) to ω(3,2) to ω(3,3) and so on to ω(F, F). At step At step At step Referring to FIG. 20, the steps of the preferred embodiment of the present invention which compute the output values of the sequential kernel subsystem The imputed feature evaluation m[OUT] is then computed at step Referring to FIG. 21, the steps of the process for updating ω for the sequential kernel processes are shown. At step If at step Referring to FIG. 22, processes of the preferred embodiment of the present invention for system monitoring are shown. At step The foregoing relates to the preferred embodiment of the present invention, and many changes may be made therein without departing from the scope of the invention as defined by the following claims. Patent Citations
Non-Patent Citations
Referenced by
Classifications
Legal Events
Rotate |