|Publication number||US6937994 B1|
|Application number||US 09/571,917|
|Publication date||Aug 30, 2005|
|Filing date||May 16, 2000|
|Priority date||Feb 24, 2000|
|Publication number||09571917, 571917, US 6937994 B1, US 6937994B1, US-B1-6937994, US6937994 B1, US6937994B1|
|Inventors||Vijay Sourirajan Iyengar|
|Original Assignee||International Business Machines Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (13), Non-Patent Citations (9), Referenced by (26), Classifications (6), Legal Events (4)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This application claims the benefit of U.S. Provisional Patent Application Ser. No. 60/184,749, filed Feb. 24, 2000.
1. Field of the Invention
This invention relates generally to the fields of data processing and information storage and retrieval. More particularly, the invention relates to methods and apparatus for classification using a chosen set of labeled data and for generating a classification model for targeting products and promotions.
2. Discussion of the Prior Art
Targeting products and promotions to the appropriate set of customers is an important aspect of marketing. Typically, this is done with a classification model based on customer attributes for each product or promotion or for a group of these. For example, a model could indicate high interest for a sports car for customers having attributes such as age below 40 and high income, while low interest in the rest of the customers. Sometimes these models are developed by marketing personnel based on their experience or expertise. This expertise can also be enhanced by sampling the customers with product and promotion offerings or with questionnaires. Such live marketing experiments are relatively easier to do in the e-commerce domain (e.g., offer e-coupons, present product advertisements, present pop-up questionnaires). The response collected from these experiments are then used to develop formal or informal models for targeting the products or promotions in question.
The current approach to sampling is based on an “open loop” system for selecting samples from the customer population.
It would thus be highly desirable to provide a “closed loop”system methodology for selecting samples used for efficiently building models that may be used for targeting products and promotions.
In the context of generating models for targeting products and promotions, it would additionally be highly desirable to provide a closed loop system methodology for selecting samples used for efficiently building models, wherein the system implements a learning algorithm that achieves high classification accuracy by judiciously selecting and using a reduced labeled data training set.
As vast amounts of data in various forms are available for processing (for example, data in the form of natural language text (electronic mail, web page contents, news, technical and business reports, etc.); image data (satellite images, handwritten text, etc.); and, multiple attribute data on individuals and institutions (survey data, purchase histories, etc.)), there are ever increasing needs for extracting the maximum information out of such data. Various methods have been devised including classification—whereby a piece of data is classified into various categories. Classification applications typically implement supervised learning techniques since they require training data that contains examples for which the categories have been determined. The process of obtaining this training data is also called labeling (i.e., labeling each item in the training data with its category). The labeling process can be very expensive since, in most cases, it has to be done manually by persons with domain knowledge. For example, instances of electronic mail queries are examined manually and labeled as belonging to various categories. Such labeled data is then used by one of many methods for classification that can be used subsequently to automatically classify new data into categories. The accuracy of this classification depends on the quality and the quantity of the training data. Having higher quality and larger amounts of training data are two factors both of which usually result in higher accuracies for the classifiers. This has motivated work on methods of generating accurate classifiers that require reduced amounts of labeled training data.
Various methods have been attempted to reduce the amount of labeled training data for classification. Any method that creates artificial data for labeling is not useful since the artificially generated data may not have any meaning to the domain expert doing the labeling. Hence, the only relevant methods are those that choose a subset for labeling from the entire set of unlabeled data and then generate a classifier using the labeled subset.
Random sampling techniques, such as described in W. G. Cochran, Sampling Techniques, John Wiley & Sons, 1977, are clearly ineffective since the various categories can have very skewed distributions and instances of infrequent categories can get omitted from random samples. Stratified sampling techniques, such as described in the above-mentioned “Sampling Techniques” reference, is a method developed to address this problem with random samples. The unlabeled data is partitioned based on the attributes of each point in the data. Sampling is then done separately from each partition and can be biased based on the expected difficulty in classifying data in each partition. This approach is not very effective in high dimensional real life data sets where such partitions are difficult to generate.
Uncertainty sampling methods iteratively identify instances in the data that need to be labeled based on some measure that suggests that the labels for these instances are uncertain despite the already labeled training data. Various methods for measuring uncertainty have been proposed. In one scheme described in the reference to David D. Lewis and W. A. Gale entitled A Sequential Algorithm For Training Text Classifiers, SIGIR 94: Proceedings of Seventeenth Annual International ACM-SIGIR conference on Research and Development in Information Retrieval, pp. 3–12, 1994, a single classifier is used that produces an estimate of the degree of uncertainty in its prediction. The iterative process then selects some fixed number of instances with the maximum uncertainty for labeling. The newly labeled instances are added to the training set and the classifier is generated using this larger training set. This iterative process continues until some stopping criteria is satisfied. A more general version is described in U.S. Pat. No. 5,671,333 where two classifiers are used, the first one to determine the degree of uncertainty and, the second one to do the classification.
A general approach of using multiple classifiers is called “query by committee” (see Seung, H., et al. Query by Committee. In Proceedings of the Fifth Annual ACM Workshop of Computational Learning Theory, pp. 287–294, 1992 and Freund, Y., et al. Information, prediction and query by committee. In Advances in Neural Informations Processings Systems 4, San Mateo, Calif., 1992 Morgan Kaufmann). In this method, two classifiers consistent with the labeled training data are randomly chosen. Instances of the data for which two chosen classifiers disagree are chosen as candidates to be labeled. While “query by committee” has been studied theoretically, its effectiveness on real world tasks is not yet proven.
Another related area of the prior art is the use of an ensemble of classifiers to enhance the accuracy of the classification (see Sholom M. Weiss, et al., Maximizing Test-Mining Performance, IEEE Intelligent Systems & their application, July/August 1999, Vol. 14, No. 4 and U.S. Pat. No. 5,819,247). In these methods, multiple classifiers are generated from data obtained by resampling from the training set using weights for including each instance in the sample. The weights are generated using feedback from the generated classifiers biasing it towards including those instances in the labeled training data that were difficult to classify (i.e., had more errors). The term “adaptive resampling” has been used to refer to such methods. The final classification is arrived at by combining the ensemble of classifiers using some weighting scheme. The weighting scheme could range from a simple majority vote over the multiple classifiers to some more complicated function to combine the results from the ensemble. These techniques have been very successful in achieving high accuracy for practical classification problems in various domains.
It would be further desirable to provide a system and methodology for selecting samples, collecting responses, and building a model by implementing a learning algorithm that achieves a high classification accuracy and which can be applied to all domains.
It is an object of the invention to provide a “closed loop” system methodology for selecting samples used for efficiently building models that may be used for targeting products, promotions, surveys, and like marketing applications.
It is a further object of the invention to provide a “closed loop” system methodology for selecting samples used for efficiently building models, wherein the methodology implements a learning algorithm that achieves a high classification accuracy by judiciously selecting and using a reduced labeled data training set.
It is another object of the invention to provide a “closed loop” system methodology for iteratively selecting samples used for efficiently building models, wherein a computer implemented method is implemented for selecting data instances that need to be labeled to form a classification data training set.
It is yet still another object of the present invention to provide a “closed loop” system methodology for iteratively selecting samples used for efficiently building models where the resultant selected, labeled classification data training set may be used as for purposes including building other classifiers.
It is still another object of the present invention to employ a first classifier device to “guess” the labels of the unlabeled instances as a step in the process of selecting instances to be labeled and to use adaptive resampling methods with a second classifier to generate an ensemble of classification models that are combined to form the final classification solution.
Thus, according to the principles of the invention, there is provided a closed loop method of selecting samples used for the generation of classifiers. The sampling is broken up into phases where in each phase a subset of the samples are chosen using information from the collected responses up to that point and the model that has been generated using these collected responses. The total number of samples and the number of phases in which these samples are collected are quantities that can be chosen by the user. In a first phase, a subset of samples may be selected as before using random or stratified sampling. The collected response is used to build a model and, the collected response and the model built are then used along with customer attribute data to select another subset of samples for collecting responses. Many possible methods including those employing “active learning” may be implemented for selecting. The collected response from the second sample is added to the total set of collected responses and, the model is generated again based on this total set. The process of selecting samples, collecting responses and building a model is repeated until an adequate number of responses have been gathered and an accurate enough model has been generated.
The foregoing objectives of the invention are achieved by providing a system and methodology which comprises obtaining unlabeled data, selecting from this unlabeled data, labeling the selected data and moving it to a labeled data set, classifying the unlabeled data using a first classification method, generating an ensemble of classification models using adaptive resampling with a second classification method, calculating weights to be used for selecting based on said first classification results and second classification models, and combining the ensemble of classifiers into a final classification model.
Advantageously, the closed loop system achieves higher classification accuracy for marketing applications such as targeting products and promotions, surveys and the like, as compared to traditional methods. Furthermore, the resultant classifier device of the invention achieves higher classification accuracy as compared to traditional methods based on experiments with active learning in other domains, and further achieves like classification accuracy as traditional methods while requiring a smaller set of selected labeled data instances.
Further features, aspects and advantages of the apparatus and methods of the present invention will become better understood with regard to the following description, appended claims, and accompanying drawings where:
As will be described, the sample selection step 55 may implement one or more of the prior art “active learning” techniques, such as described in U.S. Pat. Nos. 5,819,247 and 5,671,333. However, a preferred active learning technique according to the principles of the invention will now be described in greater detail herein
Referring now to
The selected subset after labeling is added to the labeled data set 104. The labeled data set 104 thus comprises many instances with each instance 112 comprising the original set of attributes 120 that were in this instance as a part of the unlabeled data 101 and the label 121 that was added at step 103.
As further shown in
In the preferred embodiment, at block 105, the labeled data 104 is used to generate an ensemble of classifiers using adaptive re-sampling according to a second classification method C2. The ensemble of classifiers are generated in an iterative fashion where each iteration includes: 1) sampling from the labeled data using a set of probabilities for each instance of the labeled data, e.g., a function of error; 2) utilizing this sampled data with a second classification method to generate a classification model; and 3) utilizing the results of the previous classifications to compute new probabilities for each instance of the labeled data. This iterative process referred to as adaptive re-sampling is performed to generate an ensemble of classification models. The number of classification models in the ensemble may be determined by various methods.
The resulting ensemble of classification models at step 105 and guessed labels for the unlabeled data at block 106 are used to calculate weights 107 a for each instance of the unlabeled data for use in the selection process at 102. The weight 107 a for each instance reflects the importance of selecting it the next time the select block 102 is invoked. The weight calculation may be varied each time the weight calculation is performed at step 107.
The steps of selection 102, labeling 103, addition to the labeled data set 104, generation of an ensemble of classification models using adaptive re-sampling 105, classifying unlabeled data using a first classification method 106 and weight calculation 107 are repeated in an iterative fashion until the required number of instances are added to the labeled data set. Each iteration of these steps is called a phase and the number of iterations is referred to as the number of phases. The generation of an ensemble of classification models using adaptive re-sampling at step 105 is invoked after the last addition to the labeled data set in Block 104. It is understood that the ensemble of classification models generated in all the invocations of block 105 are combined to form the final resultant classifier 108 that is output.
According to another aspect of the invention, the labeled data set in 104 is another output of this invention and may be used as a selected, labeled training data for other purposes including use by other classifiers.
In a preferred embodiment, the selection performed at step 102 invokes a normalization process where the weights for each instance of the unlabeled data are normalized by dividing it by the sum of the weights of all instances and using the normalized weight as a probability measure of selecting that instance. After the selection of each instance, the weight of the selected instance is zeroed out and the weights re-normalized from the remaining unlabeled instances not yet selected. In another embodiment, the selection at 102 is performed by sorting the instances by weight and picking the largest weight instances. In one embodiment, the number of instances selected in each phase can be specified by the user. Preferably, the user specifies the number of phases and total number of instances to be added to the labeled set which are divided equally amongst all the phases.
In a preferred embodiment, the generation of classification models using adaptive re-sampling at block 105 in
where, for prob(i) for each instance i, the error(i) is the cumulative error for instance i over all the models M(r) created so far. The summation in the denominator is over all the labeled data j. Still another embodiment for performing step 206 in
Referring back to
In another embodiment, the guessed cumulative error itself may be used as the weight for each instance of the unlabeled data.
In a preferred embodiment of the invention, the output classifier is formed by combining the classifiers in step 108 which is accomplished by applying a chosen subset of the classification models generated in step 105 to a data instance and then picking the dominant class as predicted by the chosen subset of classification models as the predicted class for the data instance. In one embodiment, the entire set of classification models generated in block 105 may be chosen. Alternatively, the set of classification models that were generated in the last application of block 105 after all the labeled data selection was completed may be chosen. Still another system employed for step 108 may be the classification model described in U.S. Pat. No. 5,819,247.
Illustrating the advantages provided by the present invention, an example use of the present invention was applied to a benchmark called internet-ads available in a repository (UCI repository of machine learning databases, University of California, Irvine, Department of Information and Computer Science. In this benchmark, various attributes related to images being displayed on web pages are used to identify whether or not the corresponding images are internet advertisements or not. This task requires instances of the data to be labeled as being internet advertisements or not. The system of the present invention was used to perform this task with a smaller number of labeled instances. A standard evaluation criterion for such benchmarks is a measure of accuracy computed as the arithmetic mean of precision and recall. For this problem, recall is the percentage of internet-ads in the data that were correctly identified as being such and, precision is the fraction of instances of those identified by a classification method as being internet-ads that were correctly identified expressed as a percentage.
While the invention has been particularly shown and described with respect to illustrative and preformed embodiments thereof, it will be understood by those skilled in the art that the foregoing and other changes in form and details may be made therein without departing from the spirit and scope of the invention which should be limited only by the scope of the appended claims.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US5446919 *||Oct 9, 1991||Aug 29, 1995||Wilkins; Jeff K.||Communication system and method with demographically or psychographically defined audiences|
|US5671333||Apr 7, 1994||Sep 23, 1997||Lucent Technologies Inc.||Training apparatus and method|
|US5754938 *||Oct 31, 1995||May 19, 1998||Herz; Frederick S. M.||Pseudonymous server for system for customized electronic identification of desirable objects|
|US5819247||Jul 29, 1997||Oct 6, 1998||Lucent Technologies, Inc.||Apparatus and methods for machine learning hypotheses|
|US5930803 *||Apr 30, 1997||Jul 27, 1999||Silicon Graphics, Inc.||Method, system, and computer program product for visualizing an evidence classifier|
|US6061658 *||May 14, 1998||May 9, 2000||International Business Machines Corporation||Prospective customer selection using customer and market reference data|
|US6119101 *||Jan 17, 1997||Sep 12, 2000||Personal Agents, Inc.||Intelligent agents for electronic commerce|
|US6182050 *||May 28, 1998||Jan 30, 2001||Acceleration Software International Corporation||Advertisements distributed on-line using target criteria screening with method for maintaining end user privacy|
|US6192353 *||Feb 9, 1998||Feb 20, 2001||Motorola, Inc.||Multiresolutional classifier with training system and method|
|US6286005 *||Mar 11, 1998||Sep 4, 2001||Cannon Holdings, L.L.C.||Method and apparatus for analyzing data and advertising optimization|
|US6301579 *||Oct 20, 1998||Oct 9, 2001||Silicon Graphics, Inc.||Method, system, and computer program product for visualizing a data structure|
|US6370513 *||Jun 28, 1999||Apr 9, 2002||Parasoft Corporation||Method and apparatus for automated selection, organization, and recommendation of items|
|EP1049030A1 *||Apr 28, 1999||Nov 2, 2000||SER Systeme AG Produkte und Anwendungen der Datenverarbeitung||Classification method and apparatus|
|1||"Experiments with a New Boosting Algorithm", by Yoav Freund, et al., 1996, pp. 1-9.|
|2||"Heterogeneous Uncertainty Sample for Supervised Learning", by David D. Lewis, et al., pp. 148-156.|
|3||"Using EM to Classify Text from Labeled and Unlabeled Documents", by Kamal Nigam, et al., May 11, 1998, pp. 1-18.|
|4||Green, et al., "Adaptive Conjoint Analysis: Some Caveats and Suggestions", Journal of Marketing Research, vol. XXVIII, (May 1991), pp. 215-222.|
|5||Green, et al., "Conjoint Analysis in Marketing: New Developments With Implications for Research and Practice", Journal of Marketing, Oct. 1990, pp. 319.|
|6||*||Jain et al., Statistical Pattern Recognition: A Review, IEEE Transactions on Pattern Analysis and Machine Intelligence, Col. 22, No. 1, Jan. 2000 [GOOGLE].|
|7||Lewis, et al., "A Sequential Algorithm for Training Text Classifiers", AT & T Bell Laboratories; Murray Hill, NJ 07974; USA. In W. Bruce Croft and C.J. van Rijsbergen, eds., SIGIR 94: Proceedings of Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, Springer-Verlag, London, pp. 3-12|
|8||*||Weiss et al., Maximizing Text-Mining Performance, IEEE, Intelligent Systems, Jul./Aug. 1999.|
|9||Weiss, et al., "Maximizing Text-Mining Performance", Intelligent Information Retrieval, 1999 IEEE, Jul./Aug. 1999, pp. 63-69.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7590626 *||Oct 30, 2006||Sep 15, 2009||Microsoft Corporation||Distributional similarity-based models for query correction|
|US7721310 *||Dec 5, 2000||May 18, 2010||Koninklijke Philips Electronics N.V.||Method and apparatus for selective updating of a user profile|
|US7860813 *||Jul 31, 2008||Dec 28, 2010||Nec Laboratories America, Inc.||Monitoring driving safety using semi-supervised sequential learning|
|US8301584 *||Dec 16, 2003||Oct 30, 2012||International Business Machines Corporation||System and method for adaptive pruning|
|US8639692 *||Jun 28, 2010||Jan 28, 2014||Alcatel Lucent||High-dimensional stratified sampling|
|US8868402 *||Dec 30, 2009||Oct 21, 2014||Google Inc.||Construction of text classifiers|
|US9031897 *||Mar 23, 2012||May 12, 2015||Nuance Communications, Inc.||Techniques for evaluation, building and/or retraining of a classification model|
|US9047362 *||Oct 15, 2013||Jun 2, 2015||Alcatel Lucent||High-dimensional stratified sampling|
|US9311609||Apr 14, 2015||Apr 12, 2016||Nuance Communications, Inc.||Techniques for evaluation, building and/or retraining of a classification model|
|US9317564||Oct 20, 2014||Apr 19, 2016||Google Inc.||Construction of text classifiers|
|US9600779 *||Feb 3, 2016||Mar 21, 2017||Accenture Global Solutions Limited||Machine learning classifier that can determine classifications of high-risk items|
|US9619758 *||Dec 30, 2014||Apr 11, 2017||Baidu Online Network Technology (Beijing) Co., Ltd||Method and apparatus for labeling training samples|
|US20020104087 *||Dec 5, 2000||Aug 1, 2002||Philips Electronics North America Corp.||Method and apparatus for selective updating of a user profile|
|US20050131873 *||Dec 16, 2003||Jun 16, 2005||Wei Fan||System and method for adaptive pruning|
|US20080104056 *||Oct 30, 2006||May 1, 2008||Microsoft Corporation||Distributional similarity-based models for query correction|
|US20090083255 *||Sep 24, 2007||Mar 26, 2009||Microsoft Corporation||Query spelling correction|
|US20090191513 *||Jul 31, 2008||Jul 30, 2009||Nec Laboratories America, Inc.||Monitoring driving safety using semi-supervised sequential learning|
|US20110320447 *||Jun 28, 2010||Dec 29, 2011||Aiyou Chen||High-Dimensional Stratified Sampling|
|US20130138641 *||Dec 30, 2009||May 30, 2013||Google Inc.||Construction of text classifiers|
|US20130254153 *||Mar 23, 2012||Sep 26, 2013||Nuance Communications, Inc.||Techniques for evaluation, building and/or retraining of a classification model|
|US20140040268 *||Oct 15, 2013||Feb 6, 2014||Alcatel-Lucent Usa, Inc.||High-dimensional stratified sampling|
|US20140089086 *||Sep 24, 2012||Mar 27, 2014||Wal-Mart Stores, Inc.||Gift-recommendation apparatus and method|
|US20140279734 *||Mar 15, 2013||Sep 18, 2014||Hewlett-Packard Development Company, L.P.||Performing Cross-Validation Using Non-Randomly Selected Cases|
|US20150363822 *||Dec 16, 2013||Dec 17, 2015||Google Inc.||Splitting a purchase panel into sub-groups|
|US20160063395 *||Dec 30, 2014||Mar 3, 2016||Baidu Online Network Technology (Beijing) Co., Ltd||Method and apparatus for labeling training samples|
|US20160155069 *||Feb 3, 2016||Jun 2, 2016||Accenture Global Solutions Limited||Machine learning classifier|
|Cooperative Classification||G06Q30/0201, G06Q30/02|
|European Classification||G06Q30/02, G06Q30/0201|
|May 16, 2000||AS||Assignment|
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:IYENGAR, VIJAY S.;REEL/FRAME:010809/0344
Effective date: 20000515
|Mar 9, 2009||REMI||Maintenance fee reminder mailed|
|Aug 30, 2009||LAPS||Lapse for failure to pay maintenance fees|
|Oct 20, 2009||FP||Expired due to failure to pay maintenance fee|
Effective date: 20090830