US20040225587A1

US20040225587A1 - Fusion classification for risk categorization in underwriting a financial risk instrument

Info

Publication number: US20040225587A1
Application number: US10/832,003
Authority: US
Inventors: Richard Messmer; Piero Bonissone; Kareem Aggour; Rajesh Subbu; Weizhong Yan; Naresh Iyer
Original assignee: General Electric Co
Current assignee: General Electric Co
Priority date: 2003-04-30
Filing date: 2004-04-23
Publication date: 2004-11-11
Also published as: US20040220837A1; WO2004099946A2; WO2004099946A3; US20090048876A1; US7383239B2; US8214314B2

Abstract

A system, process and computer program product for underwriting a financial risk instrument application represented by at least one risk attribute is provided. Decision engines examine the at least one risk attribute associated with the financial risk instrument application and assign the application to one of a predetermined set of risk classes. A fusion engine compares the risk classes assigned by each of the decision engines and fuses the assigned risk classes into an aggregated result representative of the risk of the financial risk instrument application. The fusion engine includes a first multi-classifier fusion module that uses an associative function to fuse the assigned risk classes into a first aggregated result and a second multi-classifier fusion that uses a non-associative function to fuse the assigned risk classes into a second aggregated result. A comparison engine selects one of the first aggregated result generated from the first multi-classifier fusion module and the second aggregated result generated from the second multi-classifier fusion module and compares it with a production result generated from the production decision engine. The comparison engine generates an underwriting decision for the financial risk instrument application according to the comparison.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent application Ser. No. 10/425,721, entitled “System And Process For A Fusion Classification For Insurance Underwriting Suitable For Use By An Automated System”, filed Apr. 30, 2003.[0001]

BACKGROUND OF THE INVENTION

The present invention relates to a system, process and computer program product for risk categorization of financial risk instrument applications, and more particularly to a system, process and computer program product for providing expert assistance to the underwriting of such financial risk instrument applications based on a fusion classification.

Classification is the process of assigning an input pattern to one of a predefined set of classes. Classification problems exist in many real-world applications, such as medical diagnosis, machine fault diagnosis, handwriting character recognition, fingerprint recognition, and credit scoring, to name a few. Broadly speaking, classification problems can be categorized into two types: dichotomous classification, and polychotomous classification. Dichotomous classification deals with two-class classification problems, while polychotomous classification deals with classification problems that have more than two classes.

Classification consists of developing a functional relationship between the input features and the target classes. Accurately estimating such a relationship is key to the success of a classifier. Instrument underwriting of financial risk instruments such as financial credit or loan applications is another area where these classification problems exist. The underwriting process of a financial risk instrument application may consists of assigning a given instrument application, described by its financials, credit rating, corporate structure, market and other key input, to one of several risk categories (also referred to as risk or rate classes). A trained individual or individuals traditionally perform financial risk instrument underwriting. A given application for a financial risk instrument (also referred to as an “instrument application”) may be compared against a variety of underwriting rules/standards set by a financial company. Using the underwriting rules/standards enables the instrument application to be classified into one of several risk categories available for a type of coverage requested by an applicant. The risk categories can affect the payment structure (in terms of amount and timing) paid by the applicant, e.g., the higher the risk category, the higher the overall payment. A decision to accept or reject the application for the instrument may also be part of this risk classification, as risks above a certain tolerance level set by the financial company may simply be rejected.

One problem associated with this approach in underwriting an instrument application is that there are a large number of features (financials, credit rating, corporate structure, market) and rules/standards that the underwriters have to take into account, as well as several risk categories (or rate classes). With the large number of features, rules/standards and risk categories, it is very difficult and time consuming to go over all of the information necessary to make a decision and furthermore, the results are often inadequate in consistency and reliability. The inadequacy of this process becomes more apparent as the complexity of instrument applications increases.

Another problem with this underwriting process is that the underwriting standards typically do not cover all possible cases and variations of an application for an instrument. The underwriting standards may even be self-contradictory or ambiguous, leading to an uncertain application of the standards. As a result, the subjective judgment of the underwriter will likely play a role in the process. Variation in factors such as underwriter training and experience, and a multitude of other effects can cause different underwriters to issue different, inconsistent decisions. Sometimes these decisions can be in disagreement with the established underwriting standards of the issuing company, while sometimes they can fall into a “gray area” not explicitly covered by the underwriting standards.

Further, there may be an occasion in which an underwriter's decision could still be considered correct, even if it disagrees with the written underwriting standards. This situation can be caused when the underwriter uses his or her own experience to determine whether the underwriting standards should be adjusted. Different underwriters may make different determinations about when these adjustments are allowed, as they might apply stricter or more liberal interpretations of the underwriting standards. Thus, the judgment of experienced underwriters may be in conflict with the desire to consistently apply the underwriting standards.

SUMMARY OF THE INVENTION

In one embodiment of the invention, there is a system for underwriting a financial risk instrument application represented by at least one risk attribute. In this embodiment, there is a plurality of decision engines that each examines the at least one risk attribute associated with the financial risk instrument application and assigns the application to one of a predetermined set of risk classes. A fusion engine compares the risk classes assigned by each of the plurality of decision engines and fuses the assigned risk classes into an aggregated result representative of the risk of the financial risk instrument application. The fusion engine comprises a first multi-classifier fusion module that uses an associative function to fuse the assigned risk classes into a first aggregated result and a second multi-classifier fusion that uses a non-associative function to fuse the assigned risk classes into a second aggregated result. The system also comprises a production decision engine that assigns the financial risk instrument application to one of a predetermined set of risk classes according to the at least one risk attribute associated with the application and generates a production result representative of the risk of the application. A comparison engine selects one of the first aggregated result generated from the first multi-classifier fusion module and the second aggregated result generated from the second multi-classifier fusion module and compares with the production result generated from the production decision engine. The comparison engine generates an underwriting decision for the financial risk instrument application according to the comparison.

In a second embodiment, there is a system for underwriting a financial risk instrument application represented by at least one risk attribute. In this embodiment, there is a plurality of decision engines that each examines the at least one risk attribute associated with the financial risk instrument application and assigns a preference from a set of predetermined risk classes for the application. The preference of risk classes provides a conviction in applicability of each risk class assigned to the financial risk instrument application. A fusion engine compares the preferences of risk classes generated by each of the plurality of decision engines and fuses the preferences of risk classes into an aggregated result representative of the risk of the financial risk instrument application. The fusion engine comprises a first multi-classifier fusion module that uses an associative function to fuse the preferences of risk classes into a first aggregated result and a second multi-classifier fusion that uses a non-associative function to fuse the preferences of risk classes into a second aggregated result. A production decision engine assigns the financial risk instrument application to a preference from a set of predetermined risk classes according to the at least one risk attribute associated with the application and generates a production result representative of the risk of the application. A comparison engine selects one of the first aggregated result generated from the first multi-classifier fusion module and the second aggregated result generated from the second multi-classifier fusion module and compares with the production result generated from the production decision engine. The comparison engine generates an underwriting decision for the financial risk instrument application according to the comparison.

In another embodiment, there is a computer-implemented process and computer readable medium for underwriting a financial risk instrument application represented by at least one risk attribute. In this embodiment, the at least one risk attribute associated with the financial risk instrument application is examined with a plurality of decision engines. Each decision engine is then used to assign the application to one of a set of predetermined risk classes. The assigned risk classes are fused into an aggregated result representative of the risk of the financial risk instrument application. The fusing comprises applying an associative function to fuse the assigned risk classes into a first aggregated result and applying a non-associative function to fuse the assigned risk classes into a second aggregated result. Then one of the first aggregated result and the second aggregated result is selected. The selected aggregated result is then compared with a production result generated from a production decision engine. An underwriting decision for the financial risk instrument application is generated according to the comparison of the selected aggregated result with the production result.

In still another embodiment, there is a computer-implemented process and computer readable medium for underwriting a financial risk instrument application represented by at least one risk attribute. In this embodiment, the at least one risk attribute associated with the financial risk instrument application is examined with a plurality of decision engines. Each decision engine is used to assign a preference from a set of predetermined risk classes for the application. The preference of risk classes provides a conviction in applicability of each risk class assigned to the financial risk instrument application. The preferences of risk classes are fused into an aggregated result representative of the risk of the financial risk instrument application. The fusing comprises applying an associative function to fuse the preferences of risk classes into a first aggregated result and applying a non-associative function to fuse the preferences of risk classes into a second aggregated result. Then one of the first aggregated result and the second aggregated result is selected. The selected aggregated result is compared with a production result generated from a production decision engine. An underwriting decision for the financial risk instrument application is generated according to the comparison of the selected aggregated result with the production result.

In yet another embodiment of the invention, there is a system for underwriting a financial risk instrument application represented by at least one risk attribute. In this embodiment, there is a plurality of decision engines that each examines the at least one risk attribute associated with the financial risk instrument application and assigns the application to one of a predetermined set of risk classes. A fusion engine compares the risk classes assigned by each of the plurality of decision engines and fuses the assigned risk classes into an aggregated result representative of the risk of the financial risk instrument application. The fusion engine comprises a first multi-classifier fusion module that uses an associative function to fuse the assigned risk classes into a first aggregated result and a second multi-classifier fusion that uses a non-associative function to fuse the assigned risk classes into a second aggregated result. A comparison engine selects between the first aggregated result generated from the first multi-classifier fusion module and the second aggregated result generated from the second multi-classifier fusion module. The selected result is representative of an underwriting decision for the financial risk instrument application.

In a sixth embodiment, there is a computer-implemented process and computer readable medium for underwriting a financial risk instrument application represented by at least one risk attribute. In this embodiment, the at least one risk attribute associated with the financial risk instrument application is examined with a plurality of decision engines. Each decision engine is then used to assign the application to one of a set of predetermined risk classes. The assigned risk classes are fused into an aggregated result representative of the risk of the financial risk instrument application. The fusing comprises applying an associative function to fuse the assigned risk classes into a first aggregated result and applying a non-associative function to fuse the assigned risk classes into a second aggregated result. Then one of the first aggregated result and the second aggregated result is selected. An underwriting decision for the financial risk instrument application is generated according to the selected aggregated result.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the architecture of a quality assurance system based on the fusion of multiple classifiers according to an embodiment of the invention. [0014]
FIG. 2 illustrates a table of an outer product using the function T(x,y) according to an embodiment of the invention. [0015]
FIG. 3 illustrates the disjointed risk classes within the universe of risk classes according to an embodiment of the invention. [0016]
FIG. 4 illustrates the results of the intersections of the risk classes and the universe according to an embodiment of the invention. [0017]
FIGS. 5-9 illustrate the results of T-norm operators according to an embodiment of the invention. [0018]
FIGS. 10-14 illustrate the normalized results of T-norm operators according to an embodiment of the invention. [0019]
FIG. 15 illustrates a summary of the fusion of two classifiers according to an embodiment of the invention. [0020]
FIG. 16 illustrates a penalty matrix for a particular fusion module according to an embodiment of the invention. [0021]
FIG. 17 illustrates a summary of the fusion of two classifiers with disagreement according to an embodiment of the invention. [0022]
FIG. 18 illustrates a summary of the fusion of two classifiers with agreement and discounting according to an embodiment of the invention. [0023]
FIGS. 19-23 illustrate the results of T-norm operators according to an embodiment of the invention. [0024]
FIGS. 24-28 illustrate the normalized results of T-norm operators according to an embodiment of the invention. [0025]
FIG. 29 illustrates a Dempster-Schaefer penalty matrix according to an embodiment of the invention. [0026]
FIG. 30 illustrates a comparison matrix according to an embodiment of the invention. [0027]
FIG. 31 illustrates fusion as a function of a confidence threshold for Type G cases according to an embodiment of the invention. [0028]
FIG. 32 illustrates fusion as a function of a confidence threshold for Type H cases according to an embodiment of the invention. [0029]
FIG. 33 illustrates a Venn diagram for fusion for Type G cases according to an embodiment of the invention. [0030]
FIG. 34 illustrates a Venn diagram for fusion for Type H cases according to an embodiment of the invention. [0031]
FIG. 35 schematically illustrates the classes of fusion aggregation for the multi-classifier fusion modules shown in FIG. 1. [0032]
FIG. 36 is an example illustrating the operation of a multi-classifier fusion module in FIG. 1 that uses a non-associative function. [0033]
FIG. 37 is a flowchart illustrating the operation of the system shown in FIG. 1.[0034]

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

A system, process and computer program product for underwriting of financial risk instrument applications that is suitable for use by a computer rather than by human intervention is described. The system, process and computer program product make use of existing risk assignments made by human underwriters to categorize new applications in terms of the risk involved. The term “financial risk instrument” application generally refers to financial credit (or loan) applications. It will be recognized, that the principles disclosed herein may extend beyond the realm of financial credit underwriting and that it may be applied to any risk classification process where there is a determination of the proper payments to cover a given risk. Therefore, the ultimate domain of this invention may be considered risk classification. As such, it can be applied as well to risk categorization of service agreements (e.g., service contracts for aircraft engines or power system installations or water purification installations). Therefore, as used herein, the term “financial risk instrument” refers to financial credit (or loan) applications and/or applications for service agreements. “Instrument application” is used as a short phrase to invoke the same concept. [0035]
One technical effect of the invention is to provide an automated process for consistent and accurate underwriting decisions for instrument applications. Another technical effect of the invention is to provide accurate decision support to underwriters when complexity or inadequate information preclude the making of a final decision automatically. Various aspects and components of this system, process and computer product are described below. [0036]
An aspect of the invention provides a system, process and computer product for fusing a collection of classifiers used for an automated financial risk instrument underwriting system and/or its quality assurance. While the design method is demonstrated for quality assurance of automated financial risk instrument underwriting, it is broadly applicable to diverse decision-making applications in business, commercial, and manufacturing processes. A process of fusing the outputs of a collection of classifiers is provided. The fusion can compensate for the potential correlation among the classifiers. The reliability of each classifier can be represented by a static or dynamic discounting factor, which will reflect the expected accuracy of the classifier. A static discounting factor represents a prior expectation about the classifier's reliability, e.g., it might be based on the average past accuracy of the model. A dynamic discounting represents a conditional assessment of the classifier's reliability, e.g., whenever a classifier bases its output on an insufficient number of points, the result is not reliable. Hence, this factor could be determined from the post-processing stage in each model. The fusion of the data will typically result in some amount of consensus and some amount of conflict among the classifiers. The consensus will be measured and used to estimate a degree of confidence in the fused decisions. A particular fusion aggregation approach may be too strict to gain a high consensus with limited cases available. In order to overcome this potential drawback, a second fusion aggregation with fewer classifiers or a different mode of aggregation is used to provide greater coverage. The details of the second fusion depend on the context of the particular instrument application being underwritten. A comparison engine, with context knowledge, can provide for a selection of the fusion approach or provide a decision by a combination procedure. [0037]
According to an embodiment of the invention, a fusion module (also referred to as a fusion engine) combines the outputs of several classifiers (also referred to as decision engines) to determine the correct rate class (also referred to as a risk class) for an instrument application. Using a fusion module with several decision engines may enable a classification to be assigned with a higher degree of confidence than is possible using any single model. According to an embodiment of the invention, a fusion module function may be part of a quality assurance (“QA”) process to test and monitor a production decision engine (“PDE”) that makes the risk class assignment in real-time. For example, at periodic intervals, e.g., every week, the fusion module and its components may review the decisions made by the PDE during the previous week. The output of this review will be an assessment of the PDE performance over that week, as well as the identification of cases with different level of decision quality. [0038]
The fusion module may permit the identification of the best cases of application classification, e.g., those with high-confidence, high-consensus decisions. These best cases in turn may be likely candidates to be added to the set of test cases used to tune/re-train the PDE. Further, the fusion module may permit the identification of the worst cases of application classification, e.g., those with low-confidence, low-consensus decisions. These worst cases may be likely candidates to be selected for a review by an auditing staff and/or by senior underwriters. [0039]
The fusion module may also permit the identification of unusual cases of application classification, e.g., those with unknown confidence in their decisions, for which the models in the fusion module could not make any strong commitment or avoided the decision by routing the instrument application to a human underwriter. These cases may be candidates for a blind review by senior underwriters. In addition, a fusion module may also permit an assessment of the performance of the PDE, by monitoring the PDE accuracy and variability over time, such as monitoring the statistics of low, borderline and high quality cases as well as the occurrence of unusual cases. These statistics can be used as indicators for risk management. [0040]
According to an embodiment of the invention, the fusion module may leverage the fact that except for the unusual situation where all the components (e.g., models) contain the same information (e.g., an extreme case of positive correlation), each component should provide additional information. This information may either corroborate or refute the output of the other modules, thereby supporting either a measure of consensus, or a measure of conflict. These measures may define a confidence in the result of the fusion. In general, the fusion of the components' decisions may provide a more accurate assessment than the decision of each individual component. [0041]
According to an embodiment of the invention, the fusion module may leverage the fact that except for the unusual situation where all the components (e.g., models) contain the same information (e.g., an extreme case of positive correlation), each component should provide additional information. If this information refutes the output of the other modules, thereby supporting a measure of conflict, the fusion module provides decision support information to the human underwriter on the nature of the conflicts that need to be resolved. [0042]
The fusion module is described in relation to various types of decision engines, including a case-based decision engine, a dominance-based decision engine, a multi-variate adaptive regression splines engine, and a neural network decision engine respectively. However, the fusion module may use any type of decision engine. According to an embodiment of the invention, the fusion module will support a quality assurance process for a production decision engine. However, it is understood that the fusion module could be used for a quality assurance process for any other decision making process, including a human underwriter. [0043]
According to an embodiment of the invention, a general method for the fusion process, which can be used with decision engines that may exhibit any kind of (positive, neutral, or negative) correlation with each other, may be based on the concept of triangular norms (“T-norm”), a multi-valued logic generalization of the Boolean intersection operator. The fusion of multiple decisions, produced by multiple sources, regarding objects (e.g., classes) defined in a common framework (e.g., the universe of discourse) consists of determining the underlying of degree of consensus for each object (e.g., class) under consideration, i.e., the intersections of their decisions. With the intersections of multiple decisions, possible correlation among the sources needs to be taken into account to avoid under-estimates or over-estimates. This is done by the proper selection of a T-norm operator. [0044]
According to an embodiment of the invention, each model is assumed to be solving the same classification problem. Therefore, the output of each classifier is a weight assignment that represents the degree to which a given class is selected. The set of all possible classes, referred to as U, represents the common universe of all answers that can be considered by the classifiers. The assignment of weights to this universe represents the classifier's ignorance (i.e., lack of commitment to a specific decision). This is a discounting mechanism that can be used to represent the classifier's reliability. [0045]
According to an embodiment of the invention, the outputs of the classifiers may be combined by selecting the generalized intersection operator (e.g., the T-norm) that better represents the possible correlation between the classifiers. With this operator, the assignments of the classifiers are intersected and a derived measure of consensus is computed. This fusion may be performed in an associative manner, e.g., the output of the fusion of the first two classifiers is combined with the output of the third classifier, and so on, until all available classifiers have been considered and a non-associative manner. [0046]
Thus, according to an embodiment of the invention, a fusion module only considers weight assignments made either to disjoint subsets that contain a singleton (e.g., a risk class) or to the entire universe of classes U (e.g., the entire set of risk classes), as will be described in greater detail below. Once compensation has been made for correlation and fusion has been performed, the degree of confidence C is computed among the classifiers and used to qualify the decision obtained from the fusion. Further, the confidence measure and the agreement or disagreement of the fusion module's decision is used with the production engine's decision to assess the quality of the production engine. As a by-product, the application cases may be labeled in terms of the decision confidence. Thus, cases with low, high, or unknown confidence may be used in different ways to maintain and update the production engine. [0047]
Triangular norms (T-norms) and Triangular conorms (T-conorms) are the most general families of binary functions that satisfy the requirements of the conjunction and disjunction operators, respectively: T-norms T(x,y) and T-conorms S(x,y) are two-place functions that map the unit square into the unit interval, i.e., T(x,y): [0,1]×[0,1]→[0,1] and S(x,y): [0,1]×[0,1]→[0,1]. They are monotonic commutative and associative functions. Their corresponding boundary conditions, i.e., the evaluation of the T-norms and T-conorms at the extremes of the [0,1] interval, satisfy the truth tables of the logical AND and OR operators. They are related by the DeMorgan duality, which states that if N(x) is a negation operator, then the T-conorm S(x,y,) can be defined as S(x,y)=N(T(N(x), N(y))). [0048]
As described in Bonissone and Decker (1986) the contents of which are incorporated by reference in their entirety, six parameterized families of T-norms and their dual T-conorms may be used. Of the six parameterized families, one family was selected due to its complete coverage of the T-norm space and its numerical stability. This family has a parameter p. By selecting different values of p, T-norms with different properties can be instantiated, and thus may be used in the fusion of possibly correlated classifiers. [0049]
Various articles discuss the fusion and the different features associated therewith, include proofs as to the development of algorithms associated with the present invention. Chibelushi et al. (Chibelushi, C. C., Deravi, F., and Mason, J. S. D., “Adaptive Classifier Integration for Robust Pattern Recognition,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 29, no. 6, 1999, the contents of which are incorporated herein by reference) describe a linear combination method for combining the outputs of multiple classifiers used in speaker identification applications. [0050]
Fairhurst and Rahman (Fairhurst, M. C., and Rahman, A. F. R., “Enhancing consensus in multi expert decision fusion,” [0051] IEE Proc.-Vis. Image Signal Process, vol. 147, no. 1, 2000, the contents of which are incorporated herein by reference) describe ENCORE, a multi-classifier fusion system for enhancing the performance of individual classifiers for pattern recognition tasks, specifically, the task of hand written digit recognition. Kuncheva and Jain (Kuncheva, L. I., and Jain, L. C., “Designing Classifier Fusion Systems by Genetic Algorithms,” IEEE Transactions on Evolutionary Computation, vol. 4, no. 4, 2000, the contents of which are incorporated herein by reference) describe a genetic algorithm approach to the design of fusion of multiple classifiers.
Xu et al. (Xu, L., Krzyzak, A., and Suen, C. Y., “Methods of Combining Multiple Classifiers and Their Applications to Handwriting Recognition,” [0052] IEEE Transactions on Systems, Man, and Cybernetics, vol. 22, no. 3, 1992, the contents of which are incorporated herein by reference) describe several standard approaches for classifier decision fusion, including the Dempster-Shafer approach, and demonstrate fusion for handwritten character recognition.
Arthur Dempster (A. P. Dempster, “Upper and lower probabilities induced by a multivalued mapping,” [0053] Annals of Mathematical Statistics, 38:325-339, 1967, the contents of which are incorporated herein by reference) describes a calculus based on lower and upper probability bounds. Dempster's rule of combination describes the pooling of sources under the assumption of evidential independence. Glenn Shafer (G. Shafer, “A Mathematical Theory of Evidence”, Princeton University Press, Princeton, N.J., 1976, the contents of which are incorporated herein by reference) describes the same calculus discovered by Dempster, but starting from a set of super-additive belief functions that are essentially lower bounds. Shafer derives the same rule of combination as Dempster. Enrique Ruspini (E. Ruspini, “Epistemic logic, probability, and the calculus of evidence. Proc. Tenth Intern. Joint Conf. on Artificial Intelligence, Milan, Italy, 1987, the contents of which are incorporated herein by reference) goes on to describe a possible-world semantics for Dempster-Shafer theory.
B. Schweizer and A. Sklar (B. Schweizer and A. Sklar, “Associative Functions and Abstract Semi-Groups”, [0054] Publicationes Mathematicae Debrecen, 10:69-81, 1963, the contents of which are incorporated herein by reference) describe a parametric family of triangular T-norm functions that generalize the concept of intersection in multiple-valued logics. Piero Bonissone and Keith. Decker (P. P. Bonissone and K. Decker, “Selecting Uncertainty Calculi and Granularity: An Experiment in Trading-off Precision and Complexity” in Kanal and Lemmer (editors) Uncertainty in Artificial Intelligence, pages 217-247, North-Holland, 1986, the contents of which are incorporated herein by reference) describe an experiment based on Schweizer and Sklar's parameterized T-norms. They show how five triangular norms can be used to represent an infinite number of T-norms for some practical values of information granularity. Piero Bonissone (P. P. Bonissone, “Summarizing and Propagating Uncertain Information with Triangular Norms”, International Journal of Approximate Reasoning, 1(1):71-101, January 1987, the contents of which are incorporated herein by reference) also describes the use of Triangular norms in dealing with uncertainty in expert system, Specifically he shows the use Triangular norms to aggregate the uncertainty in the left-hand side of production rules and to propagate it through the firing and chaining of production rules.
FIG. 1 illustrates the architecture of a quality assurance system based on the fusion of multiple classifiers (decision engines) according to an embodiment of the invention for underwriting an instrument application that is represented by at least one risk attribute. A risk attribute as used in this invention is a contributing factor to the overall risk of the instrument application that must be evaluated to provide input to the assessment the overall risk. An illustrative, but non-exhaustive list of risk attributes includes items such as financial statements, credit ratings, corporate structure, market and other key input. For the service agreement scenario, one type of risk attribute could be the percentage of time that equipment is run at peak power. One of ordinary skill in the art will recognize that there are other possible risk attributes for the service agreement scenario. [0055]
[0056] System 100, as illustrated in FIG. 1, includes a number of quality assurance decision engines 110 that comprise a case-based reasoning decision engine 112, a MARS decision engine 114, a neural network decision engine 116, and a dominance-based decision engine 118. The case-based reasoning decision engine 112, MARS decision engine 114, neural network decision engine 116, and dominance-based decision engine 118 use classifiers based on a case-based reasoning model, MARS model, neural network model and dominance-based model, respectively. U.S. patent application Ser. Nos. 10/170,471 and 10/171,190 provide a more detailed description of the case-based reasoning model classifier; U.S. patent application Ser. No. 10/425,722 provides a more detailed description of the MARS model classifier; U.S. patent application Ser. No. 10/425,610 provides a more detailed description of the neural network model classifier; and U.S. patent application Ser. No. 10/425,610 provides a more detailed description of the dominance-based model classifier It is understood, however, that other types of quality assurance decision engines 110 could be used in addition to and/or as substitutes for those listed in the embodiment of the invention illustrated in FIG. 1.
In this embodiment for FIG. 1, each of the [0057] decision engines 120 examines the at least one risk attribute associated with the financial risk instrument application and assigns the application to one of a predetermined set of risk classes. Risk classes are categorized by the letters A, B, C, D, E and represent the financial risk of the application, in which A represents the lowest risk and E the highest risk. These letter categories can be thought of as representing very low risk, moderately low risk, moderate risk, moderately high risk, high risk, respectively, for example; very high risk would be considered for decline. One of skill in the art will recognize that there are other ways of labeling and designating risk classes.
[0058] Post processing modules 122, 124, 126, and 128 receive the outputs from the various quality assurance decision engines 120 and perform processing on the outputs. The results of the post-processing are input into multi-classifier fusion engine or module 129 that compares the risk classes assigned by each of the decision engines 110 and fuses the assigned risk classes into an aggregated result representative of the risk of the financial risk instrument application. The multi-classifier fusion engine 129 comprises multi-classifier fusion modules 130 and 131. Multi-classifier fusion engine 130 uses an associative function to fuse the assigned risk classes into a first aggregated result and multi-classifier fusion engine 131 uses a non-associative function to fuse the assigned risk classes into a second aggregated result. The first and second aggregated results are representative of a fusion risk class decision 135 and 136, respectively.
The [0059] multi-classifier fusion engine 129 also comprises a fusion confidence estimation engine that estimates a degree of confidence in the aggregated result of the fusion engine. In particular, the fusion confidence estimation engine estimates a degree of confidence in the first aggregated result generated from the multi-classifier fusion module 130 and a degree of confidence in the second aggregated result generated from the multi-classifier fusion module 131. The confidence measures estimated by the fusion estimation engine are shown in FIG. 1 as confidence measures 140 and 141.
A comparison module or [0060] engine 150 receives the fusion risk class decision 135 and 136 and confidence measures 140 and 141 from the multi-classifier fusion engine 129. The comparison engine 150 uses the confidence measures 140 and 141 to select between the risk class decisions 135 and 136 generated from multi-classifier fusion modules 130 and 131. The comparison engine 150 may also receives input from a production decision engine 145 that includes a production risk class decision 147 and a production confidence measure 149. In this embodiment, the production decision engine 145 is a fuzzy logic rule-based automatic decision engine, however, it is within the scope of the invention to use a human as the decision engine. The comparison engine 150 then compares the production result generated from the production decision engine 145 with the risk class decision and confidence measure selected from either the multi-classifier fusion modules 130 and 131. After a comparison has been made, the comparison engine 150 outputs a compared risk class decision 151 and a compared confidence measure 153. The compared risk class decision 151 and compared confidence measure 153 are representative of an underwriting decision for the financial risk instrument.
An [0061] evaluation module 155 may evaluate the case confidence and consensus regarding the compared risk class 151 and the compared confidence measure 153. Those cases evaluated as “worst cases” are stored in case database 160 for standard underwriting, and may be candidates for auditing. Those cases evaluated as “unusual cases” are stored in case database 165, and may be candidates for standard underwriting. Those cases evaluated as “best cases” are stored in case database 170, and may be candidates for using with the test sets. An outlier detector and filter 180 may ensure that any new additions to the best-case database 170 will be consistent, with the existing cases, preventing logical outliers from being used.
As the details of the present invention are explained, one of ordinary skill in the art will recognize that there are other possible implementations and therefore the invention is not meant to be limited to the particular configuration shown in FIG. 1. For example, it is possible to implement the invention without employing the [0062] production decision engine 145. In this case, the risk class and confidence measure that the comparison engine 150 selects in its choice from the ones generated from the multi-classifier fusion modules 130 and 131 will be representative of the underwriting decision for the instrument application.
Below is a more detailed description of the quality assurance run-[0063] time decision engines 110, production decision engine 145, multi-classifier fusion modules 130 and 131 and the comparison module 150 and the functionality performed by each respective module. Other aspects of System 100 of FIG. 1 are described in U.S. patent application Ser. No. 10/425,721 with regard to insurance underwriting, but may be extended to underwriting of an instrument application.
Each [0064] decision engine 110 generates an output vector I=[I(1), I(2), . . . I(N+1)] where I(i)∈[0,M], where M is a large real value and N is the number of risk classes. In the embodiment of the invention illustrated in FIG. 1, each vector I is identified by a superscript associated with the quality assurance decision module 120 that generates the vector. Therefore, IC is generated by case-based reasoning decision engine 112, IM is generated by MARS decision engine 114, IN neural network decision engine 116, and ID is generated by dominance-based decision engine 118. Further, each entry I(i), for i=1, . . . , N, can be considered as the (un-normalized) degree to which the case could be classified in risk class i. The last element, I(N+1) indicates the degree to which the case cannot be decided and the entire universe of risk classes is selected.
For illustration purposes, assume that five risk classes are used, i.e., N=5, namely:[0065]
Risk Class={A, B, C,D, E, No Decision (Send to UW)}
By way of this example, assume that the output of the first decision engine (CBE) is: IC=[0.3, 5.4, 0.3, 0, 0, 0]. This indicates that the second risk class (e.g., B) is strongly supported by the decision engine. Normalizing IC to see the support as a percentage of the overall weights, Î[0066] ^C=[0.05, 0.9, 0.05, 0, 0,0], shows that 90% of the weights is assigned to the second risk class.
Further, to represent partial ignorance, i.e., cases in which the decision engine does not have enough information to make a more specific risk classification, discounting may be used. According to an embodiment of the invention, discounting may involve the assignment of some weight to the last element, corresponding to the universe U=(No Decision: Send to UW). For example, the previous assignment of I[0067] ^Ccould be changed such that I^C=[0.3, 1.4, 0.3, 0, 0, 4], and its normalized assignment would be Î^C=[0.05, 0.23, 0.05, 0, 0, 0.67]. This example shows how 67% of the weights have now been assigned to the universe of discourse U (the entire set of risk classes). This feature allows a representation of the lack of commitment by individual modules. According to an embodiment of the invention, if it is necessary to discount a source because it is not believed to be credible, competent, or reliable enough in generating the correct decision, a portion of the weight is transferred to the universe of discourse (e.g., “any of the above categories”). The determination of the discount may be derived from meta-knowledge, as opposed to object-knowledge. Object knowledge is the level at which each decision engine is functioning, e.g., mapping input vectors into decision bins. Meta-knowledge is reasoning about the decision engine's performance over time. Discounting could be static or dynamic. Static discounting may be used a priori to reflect historical (accuracy) performance of each decision engine. Dynamic discounting may be determined by evaluating a set of rules, whose Left Hand Side (“LHS”) defines a situation, characterized by a conjunct of conditions, and whose Right Hand Side (“RHS”) defines the amount by which to discount whichever output is generated by the classifier. According to an embodiment of the invention, postprocessing may be used to detect lack of confidence in a source. When this happens, all the weights may be allocated to the universe of discourse, i.e., refrain from making any decision.
According to an embodiment of the invention, each decision engine will independently perform a post-processing step. For purposes of illustration, the post processing used for the neural network decision engine will be described. According to an embodiment of the invention, to further improve the classification performance of a neural network module, some post-processing techniques may be applied to the outputs of the individual networks, prior to the fusion process. For example, if the distribution of the outputs did not meet certain pre-defined criteria, no decision needs to be made by the decision engine. Rather, the case will be completely discounted by allocating all of the weights to the entire universe of discourse U. The rationale for this particular example is that if a correct decision cannot be made, it would be better not to make any decision rather than making a wrong decision. Considering the outputs as discrete membership grades for all risk classes, the four features that characterize the membership grades may be defined as follows, where N is the number of risk classes and I the membership function, i.e., the output of the decision engine. [0068]
1. Cardinality [0069] $C = \sum_{1}^{N} I (i)$
2. Entropy [0070] $E = \frac{1}{E_{\max}} \sum_{1}^{N} I (i) \times \log (I (i)), where E_{\max} = - \log (1 / N)$
3. Difference between the highest and the second highest values of outputs. [0071]

D = I_max1− I_max2
4. Separation between the rank orders of the highest and the second highest values of outputs [0072]

S = RankOrder(I_max1) − RankOrder(I_max2)
With the features defined for characterizing the network outputs, the following two-step criteria may be used to identify the cases with weak decisions: [0073]
Step 1: C<τ[0074] ₁OR C>τ₂OR E>τ₃
Step 2: D<τ[0075] ₄AND S≦1
where τ[0076] ₁, τ₂, τ₃, and τ₄are the thresholds. The value of the thresholds is typically dataset dependent. However, in some embodiments, the value of the thresholds may be independent of the dataset. In the present example related to a neural network decision engine (which in turn is described in greater detail below), the value of the thresholds may be first empirically estimated and then fine-tuned by a global optimizer, such as an evolutionary algorithm. As part of this example, the final numbers are shown below in Table 1. Other optimization methods may also be used to obtain the thresholds.

TABLE 1

Type G Type H

Thresholds Application Application

τ₁ 0.50 0.30

τ₂ 2.00 1.75

τ₃ 0.92 0.84

τ₄ 0.10 0.21
Thus, post-processing may be used to identify those cases for which the module's output is likely to be unreliable. According to an embodiment of the invention, rather than rejecting such cases, the model assignment of normalized weights to risk classes may be discounted by assigning some or all of those weights to the universe of discourse U. [0077]
As described previously, the [0078] fusion modules 130 and 131 may perform the step of determining a combined decision via either the associative or non-associative fusion of the decision engine models' outputs. Associative fusion means that given three or more decision engines, any two of the engines may be fused, and then fusing the results with the third classifier, and so on, regardless of the order. On the other hand, non-associative fusion means the fusion cannot be done in a piece-wise manner because the order of the operation is critical. For example, consider the simple averaging operator acting on three values 1, 3 and 5. To get the average, all the values are summed and divided by the number of values, giving 9/3=3 as the average. The non-associativity can be seen by averaging the values 1 and 3 to obtain 4/2=2 and then forming the average of the result with 5 to obtain (2+5)/2=3.5. This can be compared to averaging 3 and 5 to obtain 4 and then averaging the result with 1 to obtain 2.5. Thus the averaging operator is not associative.
Below is a description of the associative fusing operations performed by the [0079] multi-classifier fusion module 130.
By way of example of determining a combined decision, define m decision engine S[0080] ₁, . . . S_m, such that the output of decision engine S_jis the vector I^jshowing the normalized decision of such decision engine to the N risk classes. Recall the last (N+1)^thelement represents the decision engine's lack of commitment, i.e., I^j=[I^j(1), I^j(2), . . . , I^j(N+1)], where: $I^{j} (i) \in [0, 1] and \sum_{i = 1}^{N + 1} I^{j} (i) = 1$
The un-normalized fusion of the outputs of two decision engines S[0081] 1 and S2 is further defined as:
F(I ¹ ,I ²)=Outerproduct(I ¹ ,I ² ,T)=A
where the outer-product is a well-defined mathematical operation, which in this case takes as arguments the two N-dimensional vectors I[0082] ¹and I²and generates as output the N×N dimensional array A. Each element A(i,j) is the result of applying the operator T to the corresponding vector elements, namely I¹(i) and I²(j), e.g.,
A(i,j)=T[I ¹(i), I ²(j)]
As illustrated in FIG. 2, [0083] matrix 200 illustrates classes 202 and values 204 for vector I¹and classes 206 and values 208 for vector I². Intersection 210 illustrates one intersection between the vector I¹and vector I². Other intersections and representations may also be used.
The operator T(x,y) may be referred to as a Triangular Norm. Triangular Norms (also referred to as “T-norms”) are general families of binary functions that satisfy the requirements of the intersection operators. T-norms are functions that map the unit square into the unit interval, i.e., T: [0,1]×[0,1]→[0,1]. T-norms are monotonic, commutative and associative. Their corresponding boundary conditions, i.e., the evaluation of the T-norms at the extremes of the [0,1] interval, satisfy the truth tables of the logical AND operator. [0084]

As there appear to be an infinite number of T-norms, the five most representative T-norms for some practical values of information granularity may be selected. According to an embodiment of the invention, the five T-norms selected are:



T-Norm	Correlation Type

T₁(x, y) '2 max(0, x + y − 1)	Extreme case of negative
	correlation
T_1,5(x, y) '2 max(0, x^0.5+ y^0.5− 1)²	Partial case of negative correlation
T₂(x, y) '2 x * y	No correlation
T_2.5(x, y) '2 (x⁻¹+ y⁻¹− 1)⁻¹	Partial case of positive correlation
T₃(x, y) = min(x, y)	Extreme case of positive
	correlation

The selection of the best T-norm to be used as an intersection operation in the fusion of the decision engines may depend on the potential correlation among the engines to be fused. For example, T[0086] 3 (the minimum operator) may be used when one decision engine subsumes the other one (e.g., extreme case of positive correlation). T2 may be selected when the decision engines are uncorrelated (e.g., similar to the evidential independence in Dempster-Shafer). T1 may be used if the decision engines are mutually exclusive (e.g., extreme case of negative correlation). The operators T_1.5and T_2.5may be selected when the decision engines show intermediate stages of negative or positive correlation, respectively. Of course, it will be understood by one of ordinary skill in the art that other T-norms may also be used. However, for the purposes of the present invention, these five T-norms provide a good representation of the infinite number of functions that satisfy the T-norm properties.
Because the T-norms are associative, so is the fusion operator, i.e.,[0087]
F(I ¹ ,F(I ² , I ³))=F(F(I ¹ , I ²),I ³)
Each element A(i,j) represents the fused assignment of the two decision engines to the intersection of risk classes r[0088] _iand r_j. FIG. 3 illustrates that each risk class is disjointed and that U 300, is the universe of all (risk) classes. In this example, risk classes r₁ 302, r ₂ 304 to r _n 306 are shown. Given that the risk classes are disjoint, there are five possible situations:
(a) When i=j and i<(N+1) then r[0089] _i∩r_j=r_j∩r_i=r_i
(b) When i=j and i=(N+1) then r[0090] _i∩r_j=U (the universe of risk classes)
(c) When i≠j and i<(N+1) and j<(N+1) then r[0091] _i∩r_j=φ (the empty set)
(d) When i≠j and i=(N+1) then U∩r[0092] _j=r_j
(e) When i≠j and j=(N+1) then r[0093] _i∩U=r_i
FIG. 4 depicts a [0094] chart 400 that illustrates the result of the intersections of the risk classes and the universe U, according to an embodiment of the invention. The chart demonstrates the intersection according to those situations set forth above, such that when situation (a) occurs, the results are tabulated in the main diagonal identified as 410 in FIG. 4. Further, when situation (b) occurs, the results are tabulated in the appropriate areas identified as 420 in FIG. 4. When situation (c) occurs, the results are tabulated in the appropriate areas identified as 430, while when situations (d) or (e) occur, the results are tabulated in the appropriate areas identified as 440 in FIG. 4. By way of example, when one application is rated r1 in the first instance and r2 in the second instance, the intersection may be tabulated at 450, where the column for r1 and the row for r2 intersect. In this example, the intersection of r1 and r2 is the empty set φ. The decisions for each risk class can be gathered by adding up all the weights assigned to them. According to the four possible situations described above, weights may be assigned to a specific risk class only in situation a) and d), as illustrated in FIG. 4. Thus, there will be:
Weight (r _i)=A(i,i)+A(i,N+1)+A(N+1,i)
Weight (U)=A(N+1,N+1)
To illustrate the fusion operator based on T-norms, an example will now be described. Assume that [0095]
I[0096] ¹=[0.8, 0.15, 0.05, 0, 0, 0] and I²=[0.9, 0.05, 0.05, 0, 0, 0]
This indicates that both decision engines are showing a strong preference for the first risk class (e.g., “A”) as they have assigned them 0.8 and 0.9, respectively. Fusing these decision engines using each of the five T-norm operators defined above will generate the corresponding matrices A that are shown in the tables in FIGS. 5-9, such that FIG. 5 illustrates an extreme positive correlation, FIG. 6 illustrates a partial positive correlation, FIG. 7 illustrates no correlation, FIG. 8 illustrates a partial negative correlation and FIG. 9 illustrates an extreme negative correlation. If the results are normalized so that the sum of the entries is equal to one, the matrices Â are generated, as shown in the tables in FIGS. 10-14 in a manner corresponding to the un-normalized results. During the process, the un-normalized matrices A (FIGS. 5-9) may be used to preserve the associative property. At the end, the normalized matrices Â are used (FIGS. 10-14). Using the expressions for weights of a risk class, the final weights for the N risk classes and the universe U from FIGS. 10-14 can be computed. An illustration of the computation of the final weights is illustrated in the chart of FIG. 15. [0097] Chart 1500 illustrates the five classes 1510, the five T-norms 1520, and the fused intersection results 1530.
According to an embodiment of the invention, the confidence in the fusion may be calculated by defining a measure of the scattering around the main diagonal. The more the weights are assigned to elements outside the main diagonal, the less is the measure of the consensus among the classifiers. This concept may be represented by defining a penalty matrix P=[P(i,j)], of the form: [0098] $P (i, j) = {\begin{matrix} {\max (0, (1 - W * \langle i - j \rangle))}^{d} & for 1 \leq i \leq N and 1 \leq j \leq N \\ 1 & for i = (N + 1) or j = (N + 1) \end{matrix}$
This function rewards the presence of weights on the main diagonal, indicating agreement between the two decision engines, and penalizes the presence of elements off the main diagonal, indicating conflict. The conflict increases in magnitude as the distance from the main diagonal increases. For example, for W=0.2 and d=5 we have the penalty matrix set forth in FIG. 16. [0099] Matrix 1600 intersects the column classes 1610 with the row classes 1620 to determine the appropriate penalty.
Other functions penalizing elements off the main diagonal, such as any suitable non-linear function of the distance from the main diagonal, i.e., the absolute value |i-j|, could also be used. The penalty function is used because the conflict may be gradual, as the (risk) classes have an ordering. Therefore, the penalty function captures the fact that the discrepancy between risk classes r[0100] ₁and r₂is smaller than then the discrepancy between r₁and r₃The shape of the penalty matrix P in FIG. 16 captures this concept, as P1600 shows that the confidence decreases non-linearly with the distance from the main diagonal. A measure of the normalized confidence Ĉ is the sum of element-wise products between Â and P 1600, e.g.: $\hat{C} = Normalized Confidence (\hat{A}, P) = \sum_{i = 1}^{N + 1} \sum_{j = 1}^{N + 1} \hat{A} (i, j) * P (i, j)$
where Â is the normalized fusion matrix. The results of the fusion of decision engines S[0101] 1 and S2, using each of the five T-norms with the associated normalized confidence measure 1540, are shown in FIG. 15.
In a situation in which there is a discrepancy between the two decision engines, this fact may be captured by the confidence measure. For instance, consider a situation different from the assignment illustrated in FIGS. 5-14, in which the decision engines agreed to select the first risk class. Now for example, assume that the two decision engines are showing strong preferences for different risk classes, the first decision engine is selecting the second risk class, while the second decision engine is favoring the first class: [0102]
I[0103] ¹=[0.15, 0.85, 0.05, 0, 0, 0] and I²=[0.9, 0.05, 0.05, 0, 0, 0]
The results of their fusion are summarized in the table of FIG. 17, where the [0104] chart 1700 illustrates the risk classes 1710, the T-norms 1720 and the fused intersection results 1730. None of the risk classes have a high weight and the normalized confidence 1740 has dropped.
According to an embodiment of the invention, it may be desirable to be able to discount the one of the decision engine, to reflect our lack of confidence in its reliability. For example, the second decision engine (S[0105] 2) in the first example (in which the decision engines seemed to agree on selecting the first risk class) may be discounted:
I[0106] ¹=[0.8, 0.15, 0.05, 0, 0, 0] and I²=[0.9, 0.05, 0.05, 0, 0, 0]
This discounting is represented by allocating some of the decision engine's weight, in this instance 0.3, to the universe of discourse U, (U=No decision: Sent_to_UW): [0107]
I[0108] ¹=[0.8, 0.15, 0.05, 0, 0, 0] and I²=[0.6, 0.05, 0.05, 0, 0, 0.3]
The results of the fusion of I[0109] ¹and I²are summarized in FIG. 18 below. Summarization chart 1800 illustrates the classes 1810, T-norms 1820, the fused intersection results 1830 and the confidence measure 1840. The risk classes have a slightly lower weight (for T3, T2.5, T2), but the normalized confidence is higher than with respect to FIG. 17, as there is less conflict. Fusion matrices A are shown in the tables of FIGS. 19-23, while the tables of FIGS. 24-28 illustrate matrices Â. According to an embodiment of the invention, a fusion rule based on Dempster-Shafer corresponds to the selection of:
a) T-norm operator T(x,y)=x*y; and [0110]
b) Penalty function using W=1 (or alternatively d=∞) [0111]
Constraint b) implies the [0112] penalty matrix P 2900 illustrated in FIG. 29. Therefore, the two additional constraints a) and b) required by Dempster-Shafer theory (also referred to as “DS”) imply that the decision engines to be fused must be uncorrelated (e.g., evidentially independent) and that there is no ordering over the classes, and any kind of disagreement (e.g., weights assigned to elements off the main diagonal) can only contribute to a measure of conflict and not, at least to a partial degree, to a measure of confidence. In DS, the measure of conflict K is the sum of weights assigned to the empty set. This corresponds to the elements with a 0 in the penalty matrix P 2900 illustrated in FIG. 29.
According to an embodiment of the invention, the normalized confidence C described above may be used as a measure of confidence, i.e.: [0113] $\hat{C} = Normalized Confidence (\hat{A}, P) = \sum_{i = 1}^{N + 1} \sum_{j = 1}^{N + 1} \hat{A} (i, j) * P (i, j)$
The confidence factor Ĉ may be interpreted as the weighted cardinality of the normalized assignments around the main diagonal, after all the classifiers have been fused. In the case of DS, the measure of confidence Ĉ is the complement (to one) of the measure of conflict K, i.e.: Â=1−K, where K is the sum of weights assigned to the empty set. [0114]
An additional feature of the present invention is the identification of cases that are candidates for a test set, auditing, or standard reference decision process via the comparison module. As illustrated in FIG. 1, the [0115] comparison engine 150 has four inputs. These inputs include the decision of the production engine 145, which according to an embodiment of the invention, is one of five possible risk classes or a no-decision (e.g., “send the case to a human underwriter”), i.e.:
D(PDE)=r1 and r1∈{A, B, C, D, E, Sent_to_UW}
As mentioned above, additional input to the [0116] comparison engine 150 comprises the decision of the multi-classifier fusion modules 130 and 131, which according to an embodiment of the invention, is also one of five possible risk classes or a no-decision (e.g., “send the case to a human underwriter”), i.e.:
D(FUS)=r2 and r2∈{A, B, CD, E, Sent_to_UW}
An additional input may comprise the degree of confidence in the production engine decision. A detailed description of the computation of the confidence measure is described in the U.S. patent application Ser. Nos. 10/173,000 and 10/171,575, entitled “A Process/System for Rule-Based Insurance Underwriting Suitable for Use by an Automated System”. This measure may be equated to the degree of intersection of the soft constraints used by a production decision engine (“PDE”). This measure may indicate if a case had all its constraints fully satisfied (and thus C(PDE)=1) or whether at least one constraint was only partially satisfied (and therefore C(PDE)<1). [0117]
An additional input may comprise the degree of confidence in the fusion process. The normalized confidence measure Ĉ is C(FUS). According to an embodiment of the invention, the first test performed is to compare the two decisions, i.e., D(PDE) and D(FUS). FIG. 30 illustrates all the possible comparisons between the decision of the production engine and the [0118] multi-classifier fusion module 130. Comparison matrix 3000 illustrates the D(PDE) classes 3010 and the D(FUS) classes 3020. From the table it can be seen that label I shows that D(PDE)=D(FUS) and they both indicate the same, specific risk class. Further, label II shows that the fusion module made no automated decision and suggested to send the application to a human underwriter, i.e. D(FUS)=No Decision. Label III shows that D(PDE)≠D(FUS) and that both D(PDE) and D(FUS) indicate a specific, distinct risk class. In addition, label IV shows that D(PDE)≠D(FUS), and in particular, that the PDE made no automated decision and suggested to send the application to a human underwriter, while the Fusion module selected a specific risk class. Label V shows that D(PDE)=(FUS) and that both D(PDE) and D(FUS) agree not to make any decision.
A second test may be done by using this information in conjunction with the measures of confidence C(PDE) and C(FUS) associated with the two decisions. With this information, the performance of the decision engine may be assessed over time by monitoring the time statistics of these labels, and the frequencies of cases with a low degree of confidence. According to an embodiment of the invention, a stable or increasing number of label I's would be an indicator of good, stable operations. An increase in the number of label II's would be an indicator that the fusion module (with its models) needs to be retrained. These cases might be shown to a team of senior underwriters for a standard reference decision. An increase in the frequency of label III's or of cases with low confidence could be a leading indicator of increased classification risk and might warrant further scrutiny (e.g., auditing, retraining of the fusion models, re-tuning of the production engine). An increase in label IV's may demonstrate that either the production engine needs re-tuning and/or the fusion modules needs retraining. An increase in label V's may demonstrate an increase in unusual, more complex cases, possibly requiring the scrutiny of senior underwriters. Thus, the candidates for the auditing process will be the ones exhibiting a low degree of confidence (C(FUS)<T[0119] 1), regardless of their agreement with the PDE and the ones for which the Fusion and the Production engine disagree, i.e., the ones labeled C.

The candidates for the standard reference decision process are the cases for which the

multi-classifier fusion module

130 shows no decisions (labeled II or V). The candidates to augment the test set may be selected among the cases for which the fusion module and the production engine agree (label I). These cases may be filtered to remove the cases in which the production engine was of borderline quality (C(PDE)<T2) and the cases in which the confidence measure of the fusion was below complete certainty (C(FUS)<T1). Thresholds T1 and T2, may be data dependent and must be obtained empirically. By way of example, T1=0.15 and T2=1. Table 2 below summarizes the conditions and the quality assurance actions required, according to an embodiment of the invention. Dashes (“-‘) in the entries of the table may indicate that the result of the confidence measures are not material to the action taken and/or to the label applied.

TABLE 2


Decision	Confidence
Label from	Measures

FIG 30	C(PDE)	C(FUS)	ACTION

I	≧T2	≧T1	Candidate to be added to data set for
			tuning of PLE
II	—	—	Candidate for Stand Ref Dec. Process.
			After enough cases are collected, re-
			tune the classifiers
III	—	—	Candidate for Auditing
IV	—	—	Candidate for Stand Ref Dec. Process.
			After enough cases are collected, re-
			tune the classifiers
V	—	—	Candidate for Stand Ref Dec. Process.
			After enough cases are collected, re-
			tune the classifiers
—	—	<T3	Candidate for Auditing

According to an embodiment of the invention, the [0121] multi-classifier fusion module 130 may be implemented using software code on a processor. By way of an example of the results of an implementation of the present invention, a fusion module was tested against a case base containing a total of 2,879 cases. After removing 173 UW cases, the remaining 2,706 cases were segmented into 831 Type H applications, with three risk classes, and 1,875 Type G applications, with five risk-classes. These cases were then used to test the fusion process. Because the cases for which the production engine had made no decision were removed, use of a comparison matrix similar to the one of Table 1400 will only have labels I, II, III. The fusion was performed using the T-norm T2(x,y)=x*y.
FIG. 31 illustrates the effect of changing the threshold T[0122] 1 on the measure of confidence Ĉ, were 0≦Â≦1. Table 3100 displays decisions 3110, confidence thresholds 3120 and the case distributions 3130 based on the confidence threshold 3120. Each column shows the number of cases whose measure of confidence Ĉ is≧T1. As the threshold is raised, the number of “No Fusion Decision” increases. A “No Fusion Decision” occurs when the results of the fusion are deemed too weak to be used. When the threshold T1 is 0, no case is rejected on the basis of the measure of conflict. This leaves 36 cases for which no decision could be made. As the threshold is increased, decisions with a high degree of conflict are rejected, and the number of “No Fusion Decisions” increases.
“Agreements” occur when the fused decision agrees with the [0123] production decision engine 145 and with the Standard Reference Decision (SRD). “False Positives” occur when the fused decision disagrees with the production decision engine 145, which in turn is correct since the production decision engine agrees with the Standard Reference Decision (“SRD”). “False Negatives” occur when the fused decision agrees with the production decision engine 145, but both the fusion decision and the production decision engine are wrong, as they disagree with the SRD. “Corrections” occur when the fused decision agrees with the SRD and disagrees with the production decision engine. Finally, “Complete Disagreement” occurs when the fused decision disagrees with the production decision engine, and both the fused decision and the production decision engine 145 disagree with the SRD. Further, similar results were obtained for Type H applications, and these results are illustrated in FIG. 32, with table 3200 displaying decisions 3210, confidence thresholds 3220 and the case distributions 3230 based on the confidence thresholds 3220.
FIG. 33 illustrates a Venn diagram [0124] 3300 illustrating the situation for the threshold T1=0.15 (i.e., for C≧0.15) for the Type G applications, while FIG. 34 illustrates a Venn diagram 3400 illustrating the situation for the threshold T1=0.15 (i.e., for C≧0.15). In the case of the Type G applications (for T1=0.15) the following labels result:
A: 1,588+27=1,615 (86.13%) in which [0125] 3310 D(FUS)=D(PDE); (e.g., agreements 3310 and false negative 3320)
B: =36 (1.92%) in which the fusion did not make any decision (from Ĉ=0); [0126]
C[0127] 1: 212−36=176 (9.39%) in which the fusion was too conflictive (Ĉ<0.15); and
C[0128] 2: 22+25+1=48 (2.56%) in which D(FUS)≠D(PDE) (e.g., false positive 3330, corrections 3340 and complete disagreements 3350).
In the case of the Type H applications (for T[0129] 1=0.15), the following labels result:
A: 729+15=744 cases (89.5%) in which D(FUS)=D(PDE); (e.g., [0130] agreements 3410 and false negatives 3420);
B: =37 cases (4.5%) in which the fusion did not make any decision (from Ĉ=0); [0131]
C[0132] 1: 68−37=31 cases (3.7%) in which the fusion was too conflictive (Ĉ<0.15); and
C[0133] 2: 16+3=19 cases (2.3%) in which D(FUS)≠D(PDE) (e.g., false positives 3430, corrections 3440 and complete disagreements 3450).
According to the present example, since there is no SRD in production, there can only be reliance on the degree of conflict and the agreement between the fused decision and the [0134] production decision engine 145. If the disagreement between production decision engine and FUS (e.g., subset C2) is used, it can be observed that the number of cases in which the fusion will disagree with the production decision engine, and make a classification, is 48/1875 (2.56%) for Type G applications and 19/831 (2.3%) for Type H applications . This may be considered a manageable percentage of cases to audit. Further, this sample of cases may be augmented by additional cases sampled from subsets C1.
A further analysis of set C[0135] 2 in the case of Type G applications shows that out of 48 cases, the fusion module called 22 of them correctly and 26 of them incorrectly. From the 26 incorrectly called cases, 14 cases were borderline cases according to the production decision engine. This illustrates that the problematic cases may be correctly identified and are good candidates for an audit.
A further analysis of set C[0136] 2 in the case of Type H applications shows that out of 19 cases, the multi-classifier fusion module 130 incorrectly called 16. Of these 16 cases, 6 cases were borderline cases, i.e., the production decision engine 145 only had partial degree of satisfaction of the intersection of all the constraints e.g., C(PDE)<0.9. Furthermore, 11 cases had a conflict measure Ĉ<0.4. If the union of these two subsets (e.g., the borderline cases and the conflict measure cases) is taken, the results are 13 cases that are either borderline (from the PDE) or have low confidence in the fusion, and the remaining 3 cases were ones that the CBE could not classify (i.e., it could not find enough similar cases). This again demonstrates that the problematic cases may be generally correctly identified and are worth auditing.
The set B (4.5%) illustrates a lack of commitment and is a candidate for a review to assign an SRD. The set A may be a starting point to identify the cases that could go to the test set. However, set A may need further filtering by removing all cases that were borderline according to the PDE (i.e., C(PDE)<T[0137] 2), as well as removing those cases whose fusion confidence was too low (i.e., C(FUS)<1). Again T2 will be determined empirically, from the data.
Various aspects of the [0138] multi-classifier fusion module 131 will now be discussed in greater detail below. Instead of using an associative function such as a T-Norm operator as with the multi-classifier fusion module 130, the multi-classifier fusion module 131 employs convex combination and averaging operators such as a geometric average, an arithmetic average, a majority vote, etc. FIG. 35 schematically illustrates the classes of fusion aggregation generated from multi-classifier fusion modules 130 and 131. In FIG. 35, the intersection operators used in obtaining the intersection of sources is represented by reference numeral 3510, with some explicit examples of such fusion operators given by reference numeral 3570. These are the operators used by the multi-classifier fusion module 130. As the number of sources or as the complexity of the problem increase, it is more difficult to end up with non-empty intersections. For instance, it would be enough to have one source disagreeing with the remaining (n−1 sources) to experience a total conflict. Therefore it is necessary to also consider compensating tradeoff operators, which are situated between the intersection and union operators and represented by reference numeral 3550. Explicit examples of such fusion operators are shown in FIG. 35 by reference numeral 3590.
This invention has recognized that sometimes there may be a situation where there is a lack of consensus for a large number of cases for n decision engines and therefore no decision is possible and there is good evidence that one or more classifiers may not apply for a sub-set of applications. This invention has overcome this potential drawback by using the [0139] multi-classifier fusion module 131 in addition to the multi-classifier fusion module 130. The multi-classifier fusion module 131 is operative with m decision engines, where m may be a sub-set of decision engines used by the multi-classifier fusion module 130. Furthermore, multi-classifier fusion module 131 is extended to allow the aggregation of decision engines beyond the intersection operation that uses the T-Norm approach such as the multi-classifier fusion module 130. In particular, multi-classifier fusion module 131 may aggregate the decision engines according to convex combinations, such as a geometric average, an arithmetic average, a majority vote, etc. which are shown in FIG. 35 by reference numerals 3550 and 3590. The relationships among the possible aggregation approaches are illustrated in FIG. 35.
FIG. 36 is an example illustrating the operation of the [0140] multi-classifier fusion module 131. In this example, FIG. 36 shows that there are three decision engines 110 referenced as S1, S2, and S3 that have ordered a set of four alternatives risk classes, A, B, C, D from the set {A,B,C,D} and that decision engine 145 had an initial ordering of {A,D,B,D}. The initial ordering from the decision engine 145 needs to be reconciled with the three decision engines S1, S2, and S3. This ordering implies that the decision engine 145 initially prefers A over D over B over C. These preferences may be stated strictly in an ordinal manner as just described or may also be expressed as a cardinal number that adds additional information about the preferences. The preferences from the decision engines S1, S2, and S3 are as follows:
S1→{D,A,B,C}
S2→{B,A,C,D}
S3→{C,A,B,D}
The [0141] multi-classifier fusion module 131 takes these individual ordered sets for each of the decision engines S1, S2 and S3 and produces a fused ordered set according to the type of convex combination and averaging operator selected.
One convex combination and averaging operator that the [0142] multi-classifier fusion module 131 may use is a majority vote operator that employs an average majority-voting rule to assign for each pair-wise ordering whether or not a majority voted for that ordering. Referring to FIG. 36, both S1 and S3 rank alternative A higher than B so that the fusion operator would favor alternative A with respect to alternative B. More generally, the multi-classifier fusion module 131 would produce values that populate a preference matrix with entry (i,j) indicating the preference relation between alternative i and j. For one form of the majority-voting rule used as the fusion operator (referred to as operator 1), the preference matrix for the present example would look as follows:

Operator 1 Alternative j

Alternative i A B C D

A

0 1 1 1

B 0 0 1 1

C 0 0 0 1

D 0 0 0 0
where an entry (i,j) is set to 1 if a majority of the decision engines prefer alternative i to alternative j. Using this matrix, the following global ordering G is obtained as the result of the fusion of the orderings expressed by the individual decision engines, which is the output generated from multi-classifier fusion module [0143] 131:
G→{A,B,C,D}
When the [0144] comparison module 150 compares the fused result to the production decision engine 145 ordering of {A,D,B,C}, it will see that the decision engine's choice of alternative A as being the best alternative is in agreement with the decisions of the decision engines using the majority rule.
The above example describes one instance of a convex combination and averaging operator that can be used by the [0145] multi-classifier fusion module 131, and one of ordinary skill in the art will recognize that similar examples can be provided for other types of convex combination and averaging operators. For instance, with the same example as above, one could use a fusion operator which produces a more informative preference matrix by using not just 0s and 1s as entries from a majority ruling but a score that indicates the fraction of experts who agree on a particular pair-wise ordering. The matrix for this type of fusion operator will look as follows:

Alternative

i A B C D

A

0 2/3 2/3 2/3

B 1/3 0 2/3 2/3

C 1/3 1/3 0 2/3

D 1/3 1/3 1/3 0
Another fusion operator available to [0146] multi-classifier fusion module 131 is described. Assume, for example, that there are four decision engines and four risk classes, A, B, C, D and that the decision engines provide the values summarized below.

Weight Classifier A B C D

w₁ S1 0.8 0.1 0.05 0.05

w₂ S2 0.7 0 0.2 0.1

w₃ S3 0 0 0.5 0.5

w₄ S4 0.9 0.1 0 0
Each decision engine produces a normalized preference vector described by the values in each risk class which is shown in the above rows. Also there is a weight that can be assigned to each decision engine depending on their experience or familiarity with the particular commercial segment considered, etc. (for a human underwriter) or the past performance of a particular automated decision engine for this particular segment. [0147]
In order to assess the overall preference for a risk class, Cj, the following fusion operation (a weighted average) for each class is performed[0148]
<Cj>=Σ _i w _i a _i/Σ_i w _i.
The sum is over the decision engines and the a[0149] _iare the values for each decision engine under rate class Cj. For simplicity, it is assumed that the weights for all the decision engines are the same. This gives the following fused preference vector for the decision engine. [A, C, D, B]:=>[0.6, 0.1875, 0.1625, 0.05]=[<C1>, <C2>, <C3>, <C4>]. Associated with this fused decision engine preference, a measure of confidence is also provided by using an entropy measure.
Confidence=1+Σi [<Ci>Log(<Ci>)+(1−<Ci>)Log(1−<Ci>)]
where the maximum confidence is 1 and the minimum confidence is 0. When this is applied to the fused preference vector above for this operator, the result obtained is Confidence=0.22. [0150]
FIG. 37 is a flowchart illustrating the operation of the system shown in FIG. 1. In FIG. 37, the results (e.g., I[0151] _C−I_D) from the decision engines are inputted to the multi-classifier fusion engine at 3700. The multi-classifier fusion modules fuse the decision engine results at 3710. In particular, multi-classifier fusion module 130 fuses the decision engine results using an associative function such as a triangular-norm operation, while multi-classifier fusion module 131 fuses the decision engine results using a convex combination and averaging operator. In addition, the multi-classifier fusion modules generate confidence measures at 3720. The comparison engine receives the fusion results and confidence measures at 3730. In addition, the comparison engine receives input from the production decision engine at 3740. The comparison engine chooses between the inputs from the two fusion modules at 3750 based on which result has the highest confidence. The comparison engine then compares the resultant choice with the production decision engine input at 3760. The output of this comparison is a risk class decision and a confidence measure that is then used to assign to one of several databases at 3770 for future use.
The foregoing flow charts, block diagrams and screen shots of this disclosure show the functionality and operation of the system shown in FIG. 1. In this regard, each block/component represents a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures or, for example, may in fact be executed substantially concurrently or in the reverse order, depending upon the functionality involved. Also, one of ordinary skill in the art will recognize that additional blocks may be added. [0152]
The various embodiments described above comprise an ordered listing of executable instructions for implementing logical functions. The ordered listing can be embodied in any computer-readable medium for use by or in connection with a computer-based system that can retrieve the instructions and execute them. In the context of this application, the computer-readable medium can be any means that can contain, store, communicate, propagate, transmit or transport the instructions. The computer readable medium can be an electronic, a magnetic, an optical, an electromagnetic, or an infrared system, apparatus, or device. An illustrative, but non-exhaustive list of computer-readable mediums can include an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM) (magnetic), a read-only memory (ROM) (magnetic), an erasable programmable read-only memory (EPROM or Flash memory) (magnetic), an optical fiber (optical), and a portable compact disc read-only memory (CDROM) (optical). [0153]
It is apparent that there has been provided in accordance with this invention, a system, process and computer program product for fusion classification for risk categorization in underwriting a financial risk instrument. While the invention has been particularly shown and described in conjunction with a preferred embodiment thereof, it will be appreciated that variations and modifications can be effected by a person of ordinary skill in the art without and departing from the scope of the invention. [0154]

Claims

1. A system for underwriting a financial risk instrument application represented by at least one risk attribute, comprising:

a plurality of decision engines that each examines the at least one risk attribute associated with the financial risk instrument application and assigns the application to one of a predetermined set of risk classes;

a fusion engine that compares the risk classes assigned by each of the plurality of decision engines and fuses the assigned risk classes into an aggregated result representative of the risk of the financial risk instrument application, wherein the fusion engine comprises a first multi-classifier fusion module that uses an associative function to fuse the assigned risk classes into a first aggregated result and a second multi-classifier fusion that uses a non-associative function to fuse the assigned risk classes into a second aggregated result;

a production decision engine that assigns the financial risk instrument application to one of a predetermined set of risk classes according to the at least one risk attribute associated with the application and generates a production result representative of the risk of the application; and

a comparison engine that selects one of the first aggregated result generated from the first multi-classifier fusion module and the second aggregated result generated from the second multi-classifier fusion module and compares with the production result generated from the production decision engine, the comparison engine generating an underwriting decision for the financial risk instrument application according to the comparison.

2. The system according to claim 1, further comprising a fusion confidence estimation engine that estimates a degree of confidence in the aggregated result of the fusion engine, wherein the fusion confidence estimation engine estimates a degree of confidence in the first aggregated result generated from the first multi-classifier fusion module and a degree of confidence in the second aggregated result generated from the second multi-classifier fusion module.

3. The system according to claim 2, wherein the comparison engine uses the estimated degree of confidences to select between the first aggregated result generated from the first multi-classifier fusion module and the second aggregated result generated from the second multi-classifier fusion module.

4. The system according to claim 1, further comprising a production confidence estimation engine that estimates a degree of confidence in the production result generated from the production decision engine.

5. The system according to claim 1, further comprising a comparison confidence estimation engine that estimates a degree of confidence in the underwriting decision made for the financial risk instrument application by the comparison engine.

6. The system according to claim 1, wherein the associative function used by the first multi-classifier fusion module comprises triangular norm operators.

7. The system according to claim 1, wherein the non-associative function used by the second multi-classifier fusion module comprises convex combination and averaging operators.

8. A system for underwriting a financial risk instrument application represented by at least one risk attribute, comprising:

a plurality of decision engines that each examines the at least one risk attribute associated with the financial risk instrument application and assigns a preference from a set of predetermined risk classes for the application, wherein the preference of risk classes provides a conviction in applicability of each risk class assigned to the financial risk instrument application;

a fusion engine that compares the preferences of risk classes generated by each of the plurality of decision engines and fuses the preferences of risk classes into an aggregated result representative of the risk of the financial risk instrument application, wherein the fusion engine comprises a first multi-classifier fusion module that uses an associative function to fuse the preferences of risk classes into a first aggregated result and a second multi-classifier fusion that uses a non-associative function to fuse the preferences of risk classes into a second aggregated result;

a production decision engine that assigns the financial risk instrument application to a preference from a set of predetermined risk classes according to the at least one risk attribute associated with the application and generates a production result representative of the risk of the application; and

9. The system according to claim 8, further comprising a fusion confidence estimation engine that estimates a degree of confidence in the aggregated result of the fusion engine, wherein the fusion confidence estimation engine estimates a degree of confidence in the first aggregated result generated from the first multi-classifier fusion module and a degree of confidence in the second aggregated result generated from the second multi-classifier fusion module.

10. The system according to claim 9, wherein the comparison engine uses the estimated degree of confidences to select between the first aggregated result generated from the first multi-classifier fusion module and the second aggregated result generated from the second multi-classifier fusion module.

11. The system according to claim 8, further comprising a production confidence estimation engine that estimates a degree of confidence in the production result generated from the production decision engine.

12. The system according to claim 8, further comprising a comparison confidence estimation engine that estimates a degree of confidence in the underwriting decision made for the financial risk instrument application made by the comparison engine.

13. A computer-implemented process for underwriting a financial risk instrument application represented by at least one risk attribute, comprising:

examining the at least one risk attribute associated with the financial risk instrument application with a plurality of decision engines;

using each decision engine to assign the application to one of a set of predetermined risk classes;

fusing the assigned risk classes into an aggregated result representative of the risk of the financial risk instrument application, wherein the fusing comprises:

applying an associative function to fuse the assigned risk classes into a first aggregated result; and

applying a non-associative function to fuse the assigned risk classes into a second aggregated result;

selecting one of the first aggregated result and the second aggregated result;

comparing the selected aggregated result with a production result generated from a production decision engine, and

generating an underwriting decision for the financial risk instrument application according to the comparison of the selected aggregated result with the production result.

14. The process according to claim 13, further comprising estimating a degree of confidence in the first aggregated result and in the second aggregated result.

15. The process according to claim 14, wherein the selecting comprises comparing the estimated degree of confidences associated with the first aggregated result and the second aggregated result.

16. The process according to claim 13, further comprising estimating a degree of confidence in the underwriting decision made for the financial risk instrument application.

17. The process according to claim 13, wherein the associative function comprises triangular norm operators.

18. The process according to claim 13, wherein the non-associative function comprises convex combination and averaging operators.

19. A computer-implemented process for underwriting a financial risk instrument application represented by at least one risk attribute, comprising:

using each decision engine to assign a preference from a set of predetermined risk classes for the application, wherein the preference of risk classes provides a conviction in applicability of each risk class assigned to the financial risk instrument application;

fusing the preferences of risk classes into an aggregated result representative of the risk of the financial risk instrument application, wherein the fusing comprises:

applying an associative function to fuse the preferences of risk classes into a first aggregated result; and

applying a non-associative function to fuse the preferences of risk classes into a second aggregated result;

selecting one of the first aggregated result and the second aggregated result;

20. The process according to claim 19, further comprising estimating a degree of confidence in the first aggregated result and in the second aggregated result.

21. The process according to claim 20, wherein the selecting comprises comparing the estimated degree of confidences associated with the first aggregated result and the second aggregated result.

22. The process according to claim 19, further comprising estimating a degree of confidence in the underwriting decision made for the financial risk instrument application.

23. A computer-readable medium storing computer instructions for instructing a computer system to underwrite a financial risk instrument application represented by at least one risk attribute, the instructions comprising:

using each decision engine to assign the application to a one of a set of predetermined risk classes;

selecting one of the first aggregated result and the second aggregated result;

24. The computer-readable medium according to claim 23 further comprising instructions for estimating a degree of confidence in the first aggregated result and in the second aggregated result.

25. The computer-readable medium according to claim 24, wherein the selecting comprises instructions for comparing the estimated degree of confidences associated with the first aggregated result and the second aggregated result.

26. The computer-readable medium according to claim 23, further comprising instructions for estimating a degree of confidence in the underwriting decision made for the financial risk instrument application.

27. The computer-readable medium according to claim 23, wherein the associative function comprises triangular norm operators.

28. The computer-readable medium according to claim 23, wherein the non-associative function comprises convex combination and averaging operators.

29. A computer-readable medium storing computer instructions for instructing a computer system to underwrite a financial risk instrument application represented by at least one risk attribute, the instructions comprising:

selecting one of the first aggregated result and the second aggregated result;

30. The computer-readable medium according to claim 29, further comprising instructions for estimating a degree of confidence in the first aggregated result and in the second aggregated result.

31. The computer-readable medium according to claim 30, wherein the selecting comprises instructions for comparing the estimated degree of confidences associated with the first aggregated result and the second aggregated result.

32. The computer-readable medium according to claim 29, further comprising instructions for estimating a degree of confidence in the underwriting decision made for the financial risk instrument application.

33. A system for underwriting a financial risk instrument application represented by at least one risk attribute, comprising:

a fusion engine that compares the risk classes assigned by each of the plurality of decision engines and fuses the assigned risk classes into an aggregated result representative of the risk of the financial risk instrument application, wherein the fusion engine comprises a first multi-classifier fusion module that uses an associative function to fuse the assigned risk classes into a first aggregated result and a second multi-classifier fusion that uses a non-associative function to fuse the assigned risk classes into a second aggregated result; and

a comparison engine that selects between the first aggregated result generated from the first multi-classifier fusion module and the second aggregated result generated from the second multi-classifier fusion module, the selected result being representative of an underwriting decision for the financial risk instrument application.

34. A computer-implemented process for underwriting a financial risk instrument application represented by at least one risk attribute, comprising:

selecting between the first aggregated result and the second aggregated result; and

generating an underwriting decision for the financial risk instrument application according to the selected aggregated result.

35. A computer-readable medium storing computer instructions for instructing a computer system to underwrite a financial risk instrument application represented by at least one risk attribute, the instructions comprising:

selecting one of the first aggregated result and the second aggregated result; and