WO2004003802A2 - Measurement of content ratings through vision and speech recognition - Google Patents

Measurement of content ratings through vision and speech recognition Download PDF

Info

Publication number
WO2004003802A2
WO2004003802A2 PCT/IB2003/002951 IB0302951W WO2004003802A2 WO 2004003802 A2 WO2004003802 A2 WO 2004003802A2 IB 0302951 W IB0302951 W IB 0302951W WO 2004003802 A2 WO2004003802 A2 WO 2004003802A2
Authority
WO
WIPO (PCT)
Prior art keywords
customer
detection
product
speech
image
Prior art date
Application number
PCT/IB2003/002951
Other languages
French (fr)
Inventor
Srinivas Gutta
Antonio Colmenarez
Miroslav Trajkovic
Original Assignee
Koninklijke Philips Electronics N.V.
U.S. Philips Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips Electronics N.V., U.S. Philips Corporation filed Critical Koninklijke Philips Electronics N.V.
Priority to AU2003247000A priority Critical patent/AU2003247000A1/en
Priority to EP03761741A priority patent/EP1520242A1/en
Priority to JP2004517151A priority patent/JP2005531080A/en
Publication of WO2004003802A2 publication Critical patent/WO2004003802A2/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04H—BROADCAST COMMUNICATION
    • H04H60/00—Arrangements for broadcast applications with a direct linking to broadcast information or broadcast space-time; Broadcast-related systems
    • H04H60/29—Arrangements for monitoring broadcast services or broadcast-related services
    • H04H60/33—Arrangements for monitoring the users' behaviour or opinions

Definitions

  • the present invention relates generally to vision and speech recognition, and more particularly, to methods and devices for measuring customer satisfaction through vision and/or speech recognition.
  • manufacturers and vendors of the displayed products often want information that they'd rather not reveal to the participants, such as characteristics like gender and ethnicity. This type of information can be very useful to manufacturers and vendors in marketing their products. However, because the manufacturers perceive the participants as not wanting to supply such information or be offended by such questioning, the manufacturers and vendors do not ask such questions on their product questionnaires. Therefore it is an object of the present invention to provide methods and apparatus for automatically measuring a customer's satisfaction of a product, service, or content.
  • a method for measuring customer satisfaction with at least one of a service, product, and content comprising: acquiring at least one of image and speech data for the customer; analyzing the acquired at least one of image and speech data for at least one of the following: (a) detection of a gaze of the customer; (b) detection of a facial expression of the customer; (c) detection of an emotion of the customer; (d) detection of a speech of the customer; and (e) detection of an interaction of the customer with at least one of the service, product, and content; and determining customer satisfaction based on at least one of (a) - (e).
  • the method further comprises determining at least one of a gender, ethnicity, and age of the customer from the at least one of image and speech data.
  • the acquiring preferably comprises identifying the customer in the image data.
  • the identifying preferably comprises detecting a face in the image data.
  • the identifying comprises classifying objects in the image data as people and non-people.
  • the detection of a gaze of the customer preferably comprises at least one of determining if a direction of the detected gaze is towards at least one of the service, product, and content and the duration of the gaze towards at least one of the service, product, and content.
  • the detection of a facial expression of the customer comprises determining whether the detected facial expression is one of satisfaction or dissatisfaction.
  • the method preferably further comprises detecting whether the gaze of the customer is towards at least one of the service, product, and content at a time when the facial expression is detected and wherein the determining of the customer satisfaction is at least partly based thereon.
  • the detection of an emotion of the customer is at least partly based on the detection of at least one of the speech and facial expression of the customer.
  • the detection of an emotion of the customer preferably comprises detecting an intensity of the emotion of the customer.
  • the detecting of an intensity of emotion is at least partly based on the detection of at least one of the speech and facial expression of the customer.
  • the detecting of a speech of the customer preferably comprises detecting specific phrases of the recognized speech.
  • the detecting of a speech of the customer comprises detecting emotion in the recognized speech.
  • the detection of an interaction of the customer with at least one of the service, product, and content preferably comprises detecting a physical interaction with at least one of the product, service, and content. Also provided is an apparatus for measuring customer satisfaction with at least one of a service, product, and content.
  • the apparatus comprising: at least one of a camera and microphone for acquiring at least one of image and speech data for the customer; and a processor having means for analyzing the acquired at least one of image and speech data for at least one of the following: (a) detection of a gaze of the customer; (b) detection of a facial expression of the customer; (c) detection of an emotion of the customer; (d) detection of a speech of the customer; and (e) detection of an interaction of the customer with at least one of the service, product, and content; wherein the processor further has means for determining customer satisfaction based on at least one of (a) - (e).
  • the processor further has means for determining at least one of a gender, ethnicity, and age of the customer from the at least one of image and speech data.
  • a computer program product for carrying out the methods of the present invention and a program storage device for the storage of the computer program product therein.
  • Figure 1 illustrates schematic of a preferred implementation of an apparatus for carrying out the methods of the present invention.
  • FIGS. 2a and 2b illustrate a flowchart showing a preferred implementation of a method of the present invention.
  • Apparatus 100 includes at least one, and preferably several cameras 102 having a field of view sufficient to capture image data within a predetermined area of a displayed product, service, or content 104.
  • the term camera is used in its generic sense to mean all image capturing devices.
  • the cameras 102 are preferably digital video cameras, however, they also may be analog video cameras, digital still image cameras and the like. If an analog camera is used, its output must be appropriately converted to a digital format.
  • the cameras 102 can be fixed or have a pan, tilt, and zoom capability.
  • the apparatus also includes at least one microphone 106 for capturing speech data from the predetermined area.
  • the microphone 106 is preferably a digital microphone, however, other types of microphones can also be utilized if the output signal thereof is appropriately converted to a digital format.
  • the term microphone is used in its generic sense to mean all sound capturing devices.
  • the cameras 102 and microphone 106 are useful in acquiring image and speech data for a customer 108a, 108b or other objects 109 within the predetermined area. Although, either a microphone 106 or at least one camera 102 is necessary for practicing the methods of the present invention, it is preferred that both are utilized.
  • the term "customer" refers to any person detected in the image and/or speech data within the field of view/sound of the cameras 102 and microphone 106.
  • Apparatus 100 also includes a processor 114, such as a personal computer.
  • the image and speech recognition means 110, 112, although shown in Figure 1 as separate modules, are preferably implemented in the processor 114 to carry out a set of instructions which analyze the input image and speech data from the cameras 102 and microphone 106.
  • the processor 114 further has means for determining at least one of a gender, ethnicity, and age of the customer 108a, 108b from the captured image and/or speech data.
  • the apparatus 100 also includes an output means 116 for outputting a result of the analysis by the processor 114.
  • the output means 116 can be a printer, monitor, or an electronic signal for use in a further method or apparatus.
  • Figures 2a and 2b illustrate a flowchart showing a preferred implementation of a method to be preferably carried out by apparatus 100, the method being generally referred to by reference numeral 200.
  • the method 200 measures customer satisfaction with at least one of a service, product, and content (collectively referred to herein as a "product").
  • the product can be displayed in a public area, such as a shopping area in which the product (e.g., a consumer product) is displayed within the predetermined area or in a private area in which the product (e.g., content such as a television program) is being viewed within the predetermined area.
  • a public area such as a shopping area in which the product (e.g., a consumer product) is displayed within the predetermined area or in a private area in which the product (e.g., content such as a television program) is being viewed within the predetermined area.
  • image and speech data are acquired for the predetermined area by the cameras 102 and/or microphone 106.
  • the customer(s) 108a, 108b are identified in the image and/or speech data at step 204.
  • the image data is so utilized using any method known in the art for recognizing humans in image data.
  • One such method is where faces are detected in the image data and each face is associated with a person. Once a face is found then it can be safely assumed that a human being exists.
  • An example of the recognition of people in image data by the detection of faces is disclosed in Gutta et al., Mixture of Experts for Classification of Gender, ethnic Origin, and Pose of Human Faces, IEEE Transactions on Neural Networks, Vol. 11, No. 4, July 200.
  • Another method is to classify objects in the image data as people and non-people.
  • Examples of some of the features that can be determined by an analysis of the image and/or speech data are: detection of a gaze of the customer 108a, 108b; detection of a facial expression of the customer 108a, 108b; detection of an emotion of the customer 108a, 108b; detection of a speech of the customer 108a, 108b; and detection of an interaction of the customer 108a, 108b with the product, one or more of which may be utilized to measure a customer's interest/satisfaction in a product.
  • detection of a gaze of the customer(s) 108a, 108b such is preferably carried out at step 206.
  • customer 108a in Figure 1 would be classified as having a gaze towards the product 104, while customer 108b would be classified as having a gaze away from the product 104.
  • the method 200 proceeds along path 208-NO and the customer 208b is not used in the analysis except for his or her apparent non-interest in the product 104 and the method loops back to step 204 where customers continue to be identified in the image data. If a customer 108a is found to have a gaze towards the product 104, the method continues along path 208-YES where other features are detected for that customer 108a.
  • the duration of the gaze can also be detected from the image data. It can be assumed that duration of gaze towards the product is indicative of interest in the product.
  • Methods for detecting gaze in image data are well known in the art, such as that disclosed in Rickert et al., Gaze Estimation using Morphable Models, Proceedings of the Third International Conference on Automatic Face and Gesture Recognition, Nara, Japan, April 14-16, 1998.
  • the detection of a facial expression of the customer is preferably carried out at step 210 only for those customers 108a that are found to be gazing towards the product 104.
  • the detection of a facial expression of the customer 108a comprises determining whether the detected facial expression is one of satisfaction or dissatisfaction. For instance, the detection of a smile or excited look would indicate satisfaction, while the detection of a frown or perplexed look would indicate dissatisfaction.
  • Methods for detecting facial expressions are well known in the art, such as that disclosed in Colmenarez et al., Modeling the Dynamics of Facial Expressions, CUES Workshop held in conjunction with the International Conference on Computer Vision and Pattern Recognition, Hawaii, USA, December 10 - 15, 2001.
  • the detection of speech is preferably carried out at step 212 and can be useful for not only identifying the customers 108a, 108b in the predetermined area but also in determining a measure of their satisfaction with the product.
  • the detecting of a speech of the customer 108a, 108b can detect specific phrases in the recognized speech. For instance, the recognition of terms “that's great” or “cool” would indicate a measure of satisfaction while the terms “stinks” or “terrible” would indicate a measure of dissatisfaction.
  • the emotion of a detected customer 108a, 108b can be detected. Since customer 108a is gazing at the product, only his or her emotion would be detected.
  • the detection of an emotion of the customer 108a is preferably based on (at least in part) the detection of the speech and/or facial expression of the customer 108a. Furthermore, an intensity of a detected emotion can also be detected. For instance, certain facial expressions, such as an excited look, have a greater emotional intensity than a smile. Similarly, an intensity of emotion can also be detected in the detected speech of the customer 108a, such as where the customer changes his speech pattern (e.g., speaks faster or louder) or uses expletives.
  • a determination that the customer 108a touched the product and possibly played with certain switches or other portions of the product can indicate a measure of satisfaction with the product, particularly when coupled with the detection of a favorable emotion, speech, and/or facial expression.
  • a determination of physical interaction can be made by analyzing the image data from the cameras 102 and/or from feedback from tactile sensors (not shown). Such methods for determining a physical interaction with products are well known in the art. As discussed above, the detection of other features such as gender, gender origin, and age of the customer 108a, 108b may also be made, preferably at step 218.
  • the method 200 can determine that most women are satisfied with a particular product, while most men are either dissatisfied or not interested with the product. Similar marketing strategies may be learned from an analysis of satisfaction and ethnic origin and/or age.
  • customer satisfaction is determined based on at least one of the above- discussed features, and preferably a combination of such features. A simple algorithm for such a determination would be to assign weights to each of the features and calculate a score therefrom which indicates a measure of satisfaction/dissatisfaction.
  • a score that is less than a predetermined number would indicate a dissatisfaction while a score above the predetermined number would indicate a satisfaction with the product 104.
  • Another example would be to assign a point for each feature where a possible satisfaction is indicated, where a cumulative score of the points for all of the features detected over a predetermined number would indicate a satisfaction while a cumulative score below the predetermined number would indicate a dissatisfaction with the product 104.
  • the algorithm may also be complicated and provide for a great number of scenarios and combinations of the detected features.
  • a customer 108a who is detected to be gazing at the product 104 for a long duration of time and whom there is detected a high intensity of emotion in his or her speech and facial expressions would indicate a great satisfaction with the product while a customer 108a who looks at a product with a dissatisfied facial expression and a dissatisfied emotion in his or her speech would indicate little or no interest in the product.
  • a customer 108a who only glances at a product 104 for a short tome and has little or no emotion in his or her speech and facial expression may indicate little or no interest in the product 104.
  • the results of the analysis are output for review, statistical analysis, or use in another method or apparatus.
  • the methods of the present invention are particularly suited to be carried out by a computer software program, such computer software program preferably containing modules corresponding to the individual steps of the methods.
  • a computer software program such computer software program preferably containing modules corresponding to the individual steps of the methods.
  • Such software can of course be embodied in a computer-readable medium, such as an integrated chip or a peripheral device.

Description

MEASUREMENT OF CONTENT RATINGS THROUGH VISION AND SPEECH RECOGNITION The present invention relates generally to vision and speech recognition, and more particularly, to methods and devices for measuring customer satisfaction through vision and/or speech recognition.
In the prior art there are known several ways to assess an interest in a displayed product, service, or content (collectively referred to herein as "product") by a customer. However, all of the known ways are manually carried out. For instance, questionnaire cards may be available near the product for passersby to take and fill-out. Alternatively, a store clerk or sales representative may solicit a customer's interest in the product by asking them a series of questions relating to the product. However, in either way, the persons must willingly participate in the questioning. If willing, the manual questioning takes time to complete, often much more time than people are willing to spend. Furthermore, the manual questioning depends on the truthfulness of the people participating. For content, such as television programming, one service, Nielson, automatically measures what content is currently being watched and by whom. However, they do not measure if the individual liked or disliked the content automatically.
Additionally, manufacturers and vendors of the displayed products often want information that they'd rather not reveal to the participants, such as characteristics like gender and ethnicity. This type of information can be very useful to manufacturers and vendors in marketing their products. However, because the manufacturers perceive the participants as not wanting to supply such information or be offended by such questioning, the manufacturers and vendors do not ask such questions on their product questionnaires. Therefore it is an object of the present invention to provide methods and apparatus for automatically measuring a customer's satisfaction of a product, service, or content.
Accordingly, a method for measuring customer satisfaction with at least one of a service, product, and content is provided. The method comprising: acquiring at least one of image and speech data for the customer; analyzing the acquired at least one of image and speech data for at least one of the following: (a) detection of a gaze of the customer; (b) detection of a facial expression of the customer; (c) detection of an emotion of the customer; (d) detection of a speech of the customer; and (e) detection of an interaction of the customer with at least one of the service, product, and content; and determining customer satisfaction based on at least one of (a) - (e).
Preferably, the method further comprises determining at least one of a gender, ethnicity, and age of the customer from the at least one of image and speech data. The acquiring preferably comprises identifying the customer in the image data. The identifying preferably comprises detecting a face in the image data. Alternatively, the identifying comprises classifying objects in the image data as people and non-people. The detection of a gaze of the customer preferably comprises at least one of determining if a direction of the detected gaze is towards at least one of the service, product, and content and the duration of the gaze towards at least one of the service, product, and content. Preferably, the detection of a facial expression of the customer comprises determining whether the detected facial expression is one of satisfaction or dissatisfaction. The method preferably further comprises detecting whether the gaze of the customer is towards at least one of the service, product, and content at a time when the facial expression is detected and wherein the determining of the customer satisfaction is at least partly based thereon.
Preferably, the detection of an emotion of the customer is at least partly based on the detection of at least one of the speech and facial expression of the customer. The detection of an emotion of the customer preferably comprises detecting an intensity of the emotion of the customer.
Preferably, the detecting of an intensity of emotion is at least partly based on the detection of at least one of the speech and facial expression of the customer. The detecting of a speech of the customer preferably comprises detecting specific phrases of the recognized speech. Preferably, the detecting of a speech of the customer comprises detecting emotion in the recognized speech.
The detection of an interaction of the customer with at least one of the service, product, and content preferably comprises detecting a physical interaction with at least one of the product, service, and content. Also provided is an apparatus for measuring customer satisfaction with at least one of a service, product, and content. The apparatus comprising: at least one of a camera and microphone for acquiring at least one of image and speech data for the customer; and a processor having means for analyzing the acquired at least one of image and speech data for at least one of the following: (a) detection of a gaze of the customer; (b) detection of a facial expression of the customer; (c) detection of an emotion of the customer; (d) detection of a speech of the customer; and (e) detection of an interaction of the customer with at least one of the service, product, and content; wherein the processor further has means for determining customer satisfaction based on at least one of (a) - (e).
Preferably, the processor further has means for determining at least one of a gender, ethnicity, and age of the customer from the at least one of image and speech data. Still yet provided are a computer program product for carrying out the methods of the present invention and a program storage device for the storage of the computer program product therein.
These and other features, aspects, and advantages of the apparatus and methods of the present invention will become better understood with regard to the following description, appended claims, and accompanying drawings where: Figure 1 illustrates schematic of a preferred implementation of an apparatus for carrying out the methods of the present invention.
Figures 2a and 2b illustrate a flowchart showing a preferred implementation of a method of the present invention.
Referring now to Figure 1, there is shown an apparatus for measuring customer satisfaction with at least one of a service, product, and content, the apparatus being generally referred to by reference numeral 100. Apparatus 100 includes at least one, and preferably several cameras 102 having a field of view sufficient to capture image data within a predetermined area of a displayed product, service, or content 104. The term camera is used in its generic sense to mean all image capturing devices. The cameras 102 are preferably digital video cameras, however, they also may be analog video cameras, digital still image cameras and the like. If an analog camera is used, its output must be appropriately converted to a digital format. The cameras 102 can be fixed or have a pan, tilt, and zoom capability. The apparatus also includes at least one microphone 106 for capturing speech data from the predetermined area. The microphone 106 is preferably a digital microphone, however, other types of microphones can also be utilized if the output signal thereof is appropriately converted to a digital format. The term microphone is used in its generic sense to mean all sound capturing devices. The cameras 102 and microphone 106 are useful in acquiring image and speech data for a customer 108a, 108b or other objects 109 within the predetermined area. Although, either a microphone 106 or at least one camera 102 is necessary for practicing the methods of the present invention, it is preferred that both are utilized. As used herein, the term "customer" refers to any person detected in the image and/or speech data within the field of view/sound of the cameras 102 and microphone 106. The customer may or may not be interested in the displayed products, services, and/or content, his or her presence in the predetermined area is cause enough to be classified as a "customer". The captured image and speech data is analyzed by respective image and speech recognition means 110, 112, respectively in a manner to be discussed below. Apparatus 100 also includes a processor 114, such as a personal computer. The image and speech recognition means 110, 112, although shown in Figure 1 as separate modules, are preferably implemented in the processor 114 to carry out a set of instructions which analyze the input image and speech data from the cameras 102 and microphone 106. Preferably, the processor 114 further has means for determining at least one of a gender, ethnicity, and age of the customer 108a, 108b from the captured image and/or speech data. The apparatus 100 also includes an output means 116 for outputting a result of the analysis by the processor 114. The output means 116 can be a printer, monitor, or an electronic signal for use in a further method or apparatus. A preferred implementation of a method of the present invention will now be described with regard to Figures 2a and 2b. Figures 2a and 2b illustrate a flowchart showing a preferred implementation of a method to be preferably carried out by apparatus 100, the method being generally referred to by reference numeral 200. The method 200 measures customer satisfaction with at least one of a service, product, and content (collectively referred to herein as a "product"). The product can be displayed in a public area, such as a shopping area in which the product (e.g., a consumer product) is displayed within the predetermined area or in a private area in which the product (e.g., content such as a television program) is being viewed within the predetermined area. At step 202, at least one, and preferably both, of image and speech data are acquired for the predetermined area by the cameras 102 and/or microphone 106. After acquisition of the image and/or speech data, the customer(s) 108a, 108b are identified in the image and/or speech data at step 204. Although, either or both of the image and speech data can be utilized to identify the cutomer(s) in the predetermined area, it is preferred that the image data is so utilized using any method known in the art for recognizing humans in image data.
One such method is where faces are detected in the image data and each face is associated with a person. Once a face is found then it can be safely assumed that a human being exists. An example of the recognition of people in image data by the detection of faces is disclosed in Gutta et al., Mixture of Experts for Classification of Gender, ethnic Origin, and Pose of Human Faces, IEEE Transactions on Neural Networks, Vol. 11, No. 4, July 200. Another method is to classify objects in the image data as people and non-people.
For instance, the people 108a, 108b in Figure 1 would be classified as customers while the dog 109 would be classified as a non-human and discarded for purposes of the analysis. An example of such a system is disclosed co-pending U.S. Patent Application Serial No. 09/794,443, to Gutta et al., entitled Classification of Objects through Model Ensembles, Filed February 27, 2001.
Once it is determined that a human being exists, other features may be determined like, gender, ethnic origin, facial pose, facial expressions, etc. As discussed below, these features may be used in determining a measure of the customer's interest in a displayed product. Methods for estimating a person's gender and ethnic origin are well known in the art, such as that disclosed in Gutta et al., Mixture of Experts for Classification of Gender, ethnic Origin, and Pose of Human Faces, IEEE Transactions on Neural Networks, Vol. 11, No. 4, July 200.
Examples of some of the features that can be determined by an analysis of the image and/or speech data are: detection of a gaze of the customer 108a, 108b; detection of a facial expression of the customer 108a, 108b; detection of an emotion of the customer 108a, 108b; detection of a speech of the customer 108a, 108b; and detection of an interaction of the customer 108a, 108b with the product, one or more of which may be utilized to measure a customer's interest/satisfaction in a product. With regard to the detection of a gaze of the customer(s) 108a, 108b, such is preferably carried out at step 206. At step 208 it is preferably determined whether the detected gaze is towards the product 104. For instance, customer 108a in Figure 1 would be classified as having a gaze towards the product 104, while customer 108b would be classified as having a gaze away from the product 104. If a detected customer 208b is found to have a gaze away from the product 104, the method 200 proceeds along path 208-NO and the customer 208b is not used in the analysis except for his or her apparent non-interest in the product 104 and the method loops back to step 204 where customers continue to be identified in the image data. If a customer 108a is found to have a gaze towards the product 104, the method continues along path 208-YES where other features are detected for that customer 108a.
Along with the direction of the gaze, the duration of the gaze, particularly the duration of the gaze towards the product can also be detected from the image data. It can be assumed that duration of gaze towards the product is indicative of interest in the product. Methods for detecting gaze in image data are well known in the art, such as that disclosed in Rickert et al., Gaze Estimation using Morphable Models, Proceedings of the Third International Conference on Automatic Face and Gesture Recognition, Nara, Japan, April 14-16, 1998. With regard to the detection of a facial expression of the customer, such is preferably carried out at step 210 only for those customers 108a that are found to be gazing towards the product 104. Preferably, the detection of a facial expression of the customer 108a comprises determining whether the detected facial expression is one of satisfaction or dissatisfaction. For instance, the detection of a smile or excited look would indicate satisfaction, while the detection of a frown or perplexed look would indicate dissatisfaction. Methods for detecting facial expressions are well known in the art, such as that disclosed in Colmenarez et al., Modeling the Dynamics of Facial Expressions, CUES Workshop held in conjunction with the International Conference on Computer Vision and Pattern Recognition, Hawaii, USA, December 10 - 15, 2001. With regard to the detection of speech, such is preferably carried out at step 212 and can be useful for not only identifying the customers 108a, 108b in the predetermined area but also in determining a measure of their satisfaction with the product. For instance, the detecting of a speech of the customer 108a, 108b can detect specific phrases in the recognized speech. For instance, the recognition of terms "that's great" or "cool" would indicate a measure of satisfaction while the terms "stinks" or "terrible" would indicate a measure of dissatisfaction. At step 214, the emotion of a detected customer 108a, 108b can be detected. Since customer 108a is gazing at the product, only his or her emotion would be detected. The detection of an emotion of the customer 108a is preferably based on (at least in part) the detection of the speech and/or facial expression of the customer 108a. Furthermore, an intensity of a detected emotion can also be detected. For instance, certain facial expressions, such as an excited look, have a greater emotional intensity than a smile. Similarly, an intensity of emotion can also be detected in the detected speech of the customer 108a, such as where the customer changes his speech pattern (e.g., speaks faster or louder) or uses expletives. Recognition of emotion in facial expressions and speech are well known in the art, such as that disclosed in Colmenarez et al., Modeling the Dynamics of Facial Expressions, CUES Workshop held in conjunction with the International Conference on Computer Vision and Pattern Recognition, Hawaii, USA, December 10 - 15, 2001; and Frank Dellaert et al., Recognizing Emotion in Speech, in Proc. of Int'l Conf. on Speech and Language Processing (1996); and Polzin et al., Detecting Emotions in Speech, Proceedings of the Cooperative Multimodal Communication Conference, 1998. At step 216, it is determined whether there is an interaction of the customer 108a with the product 104, such as a physical interaction with at the product. For instance, with regard to a product which is displayed (e.g., an automobile) a determination that the customer 108a touched the product and possibly played with certain switches or other portions of the product can indicate a measure of satisfaction with the product, particularly when coupled with the detection of a favorable emotion, speech, and/or facial expression. A determination of physical interaction can be made by analyzing the image data from the cameras 102 and/or from feedback from tactile sensors (not shown). Such methods for determining a physical interaction with products are well known in the art. As discussed above, the detection of other features such as gender, ethic origin, and age of the customer 108a, 108b may also be made, preferably at step 218. Although, such features may not be useful in determining a measure of satisfaction with a product, it can be very useful in terms of marketing. For instance, the method 200 can determine that most women are satisfied with a particular product, while most men are either dissatisfied or not interested with the product. Similar marketing strategies may be learned from an analysis of satisfaction and ethnic origin and/or age. At step 220, customer satisfaction is determined based on at least one of the above- discussed features, and preferably a combination of such features. A simple algorithm for such a determination would be to assign weights to each of the features and calculate a score therefrom which indicates a measure of satisfaction/dissatisfaction. That is, a score that is less than a predetermined number would indicate a dissatisfaction while a score above the predetermined number would indicate a satisfaction with the product 104. Another example would be to assign a point for each feature where a possible satisfaction is indicated, where a cumulative score of the points for all of the features detected over a predetermined number would indicate a satisfaction while a cumulative score below the predetermined number would indicate a dissatisfaction with the product 104. The algorithm may also be complicated and provide for a great number of scenarios and combinations of the detected features. For instance, as discussed above, a customer 108a who is detected to be gazing at the product 104 for a long duration of time and whom there is detected a high intensity of emotion in his or her speech and facial expressions would indicate a great satisfaction with the product while a customer 108a who looks at a product with a dissatisfied facial expression and a dissatisfied emotion in his or her speech would indicate little or no interest in the product. Similarly, a customer 108a who only glances at a product 104 for a short tome and has little or no emotion in his or her speech and facial expression may indicate little or no interest in the product 104. At step 222, the results of the analysis are output for review, statistical analysis, or use in another method or apparatus.
The methods of the present invention are particularly suited to be carried out by a computer software program, such computer software program preferably containing modules corresponding to the individual steps of the methods. Such software can of course be embodied in a computer-readable medium, such as an integrated chip or a peripheral device.
While there has been shown and described what is considered to be preferred embodiments of the invention, it will, of course, be understood that various modifications and changes in form or detail could readily be made without departing from the spirit of the invention. It is therefore intended that the invention be not limited to the exact forms described and illustrated, but should be constructed to cover all modifications that may fall within the scope of the appended claims.

Claims

CLAIMS:
1. A method for measuring customer satisfaction with at least one of a service, product, and content (104), the method comprising: acquiring at least one of image and speech data for the customer (108a); analyzing the acquired at least one of image and speech data for at least one of the following:
(a) detection of a gaze of the customer (108a);
(b) detection of a facial expression of the customer (108a); (c) detection of an emotion of the customer (108a);
(d) detection of a speech of the customer (108a); and
(e) detection of an interaction of the customer (108a) with at least one of the service, product, and content (104); and determining customer satisfaction based on at least one of (a) - (e).
2. The method of claim 1 , further comprising determining at least one of a gender, ethnicity, and age of the customer (108a) from the at least one of image and speech data.
3. The method of claim 1, wherein the acquiring comprises identifying the customer (108a) in the image data.
4. The method of claim 3, wherein the identifying comprises detecting a face in the image data.
5. The method of claim 3, wherein the identifying comprises classifying objects in the image data as people and non-people.
6. The method of claim 1, wherein the detection of a gaze of the customer (108a) comprises at least one of determining if a direction of the detected gaze is towards at least one of the service, product, and content (104) and the duration of the gaze towards at least one of the service, product, and content (104).
7. The method of claim 1, wherein the detection of a facial expression of the customer (108a) comprises determining whether the detected facial expression is one of satisfaction or dissatisfaction.
8. The method of claim 6, further comprising detecting whether the gaze of the customer (108a) is towards at least one of the service, product, and content (104) at a time when the facial expression is detected and wherein the determining of the customer satisfaction is at least partly based thereon.
9. The method of claim 1, wherein the detection of an emotion of the customer (108a) is at least partly based on the detection of at least one of the speech and facial expression of the customer (108a).
10. The method of claim 1, wherein the detection of an emotion of the customer (108a) comprises detecting an intensity of the emotion of the customer (108a).
11. The method of claim 10, wherein the detecting of an intensity of emotion is at least partly based on the detection of at least one of the speech and facial expression of the customer (108a).
12. The method of claim 1, wherein the detecting of a speech of the customer (108a) comprises detecting specific phrases of the recognized speech.
13. The method of claim 1, wherein the detecting of a speech of the customer (108a) comprises detecting emotion in the recognized speech.
14. The method of claim 1, wherein the detection of an interaction of the customer (108a) with at least one of the service, product, and content (104) comprises detecting a physical interaction with at least one of the product, service, and content (104).
15. A computer program product embodied in a computer-readable medium for measuring customer satisfaction with at least one of a service, product, and content (104), the computer program product comprising: computer readable program code means for acquiring at least one of image and speech data for the customer (108a); computer readable program code means for analyzing the acquired at least one of image and speech data for at least one of the following:
(a) detection of a gaze of the customer (108a);
(b) detection of a facial expression of the customer (108a); (c) detection of an emotion of the customer (108a);
(d) detection of a speech of the customer (108a); and
(e) detection of an interaction of the customer (108a) with at least one of the service, product, and content (104); and computer readable program code means for determining customer satisfaction based on at least one of (a) - (e).
16. The computer program product of claim 15, further comprising computer readable program code means for determining at least one of a gender, ethnicity, and age of the customer (108a) from the at least one of image and speech data.
17. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for measuring customer satisfaction with at least one of a service, product, and content (104), the method comprising: acquiring at least one of image and speech data for the customer (108a); analyzing the acquired at least one of image and speech data for at least one of the following:
(a) detection of a gaze of the customer (108a);
(b) detection of a facial expression of the customer (108a); (c) detection of an emotion of the customer (108a);
(d) detection of a speech of the customer (108a); and (e) detection of an interaction of the customer (108a) with at least one of the service, product, and content (104); and determining customer satisfaction based on at least one of (a) - (e).
18. The program storage device of claim 17, wherein the method further comprises determining at least one of a gender, ethnicity, and age of the customer (108a) from the at least one of image and speech data.
19. An apparatus (100) for measuring customer satisfaction with at least one of a service, product, and content (104), the apparatus comprising: at least one of a camera (102) and microphone (106) for acquiring at least one of image and speech data for the customer (108a); and a processor (114) having means (110, 112) for analyzing the acquired at least one of image and speech data for at least one of the following: (a) detection of a gaze of the customer (108a);
(b) detection of a facial expression of the customer (108a);
(c) detection of an emotion of the customer (108a);
(d) detection of a speech of the customer (108a); and
(e) detection of an interaction of the customer (108a) with at least one of the service, product, and content ( 104a) ; wherein the processor (114) further has means for determining customer satisfaction based on at least one of (a) - (e).
20. The apparatus of claim 19, wherein the processor (114) further has means for determining at least one of a gender, ethnicity, and age of the customer (108a) from the at least one of image and speech data.
PCT/IB2003/002951 2002-06-27 2003-06-13 Measurement of content ratings through vision and speech recognition WO2004003802A2 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
AU2003247000A AU2003247000A1 (en) 2002-06-27 2003-06-13 Measurement of content ratings through vision and speech recognition
EP03761741A EP1520242A1 (en) 2002-06-27 2003-06-13 Measurement of content ratings through vision and speech recognition
JP2004517151A JP2005531080A (en) 2002-06-27 2003-06-13 Content rating measurement via visual and speech recognition

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US10/183,759 US20040001616A1 (en) 2002-06-27 2002-06-27 Measurement of content ratings through vision and speech recognition
US10/183,759 2002-06-27

Publications (1)

Publication Number Publication Date
WO2004003802A2 true WO2004003802A2 (en) 2004-01-08

Family

ID=29779192

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2003/002951 WO2004003802A2 (en) 2002-06-27 2003-06-13 Measurement of content ratings through vision and speech recognition

Country Status (6)

Country Link
US (1) US20040001616A1 (en)
EP (1) EP1520242A1 (en)
JP (1) JP2005531080A (en)
CN (1) CN1662922A (en)
AU (1) AU2003247000A1 (en)
WO (1) WO2004003802A2 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11488181B2 (en) 2016-11-01 2022-11-01 International Business Machines Corporation User satisfaction in a service based industry using internet of things (IoT) devices in an IoT network

Families Citing this family (120)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7616233B2 (en) * 2003-06-26 2009-11-10 Fotonation Vision Limited Perfecting of digital image capture parameters within acquisition devices using face detection
US7269292B2 (en) * 2003-06-26 2007-09-11 Fotonation Vision Limited Digital image adjustable compression and resolution using face detection information
US7792970B2 (en) * 2005-06-17 2010-09-07 Fotonation Vision Limited Method for establishing a paired connection between media devices
US7792335B2 (en) * 2006-02-24 2010-09-07 Fotonation Vision Limited Method and apparatus for selective disqualification of digital images
US7574016B2 (en) * 2003-06-26 2009-08-11 Fotonation Vision Limited Digital image processing using face detection information
KR20070029794A (en) * 2004-07-08 2007-03-14 코닌클리케 필립스 일렉트로닉스 엔.브이. A method and a system for communication between a user and a system
US8488023B2 (en) * 2009-05-20 2013-07-16 DigitalOptics Corporation Europe Limited Identifying facial expressions in acquired digital images
US8235725B1 (en) 2005-02-20 2012-08-07 Sensory Logic, Inc. Computerized method of assessing consumer reaction to a business stimulus employing facial coding
JP5015926B2 (en) * 2005-08-04 2012-09-05 コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ Apparatus and method for monitoring individuals interested in property
JP2007041988A (en) * 2005-08-05 2007-02-15 Sony Corp Information processing device, method and program
US8542928B2 (en) * 2005-09-26 2013-09-24 Canon Kabushiki Kaisha Information processing apparatus and control method therefor
US8326775B2 (en) * 2005-10-26 2012-12-04 Cortica Ltd. Signature generation for multimedia deep-content-classification by a large-scale matching system and method thereof
US7804983B2 (en) 2006-02-24 2010-09-28 Fotonation Vision Limited Digital image acquisition control and correction method and apparatus
DE602007012246D1 (en) * 2006-06-12 2011-03-10 Tessera Tech Ireland Ltd PROGRESS IN EXTENDING THE AAM TECHNIQUES FROM GRAY CALENDAR TO COLOR PICTURES
US20080065468A1 (en) * 2006-09-07 2008-03-13 Charles John Berg Methods for Measuring Emotive Response and Selection Preference
US9167305B2 (en) 2007-01-03 2015-10-20 Tivo Inc. Authorable content rating system
US8295542B2 (en) * 2007-01-12 2012-10-23 International Business Machines Corporation Adjusting a consumer experience based on a 3D captured image stream of a consumer response
US8588464B2 (en) * 2007-01-12 2013-11-19 International Business Machines Corporation Assisting a vision-impaired user with navigation based on a 3D captured image stream
US8269834B2 (en) 2007-01-12 2012-09-18 International Business Machines Corporation Warning a user about adverse behaviors of others within an environment based on a 3D captured image stream
US8055067B2 (en) 2007-01-18 2011-11-08 DigitalOptics Corporation Europe Limited Color segmentation
CN101711388B (en) 2007-03-29 2016-04-27 神经焦点公司 The effect analysis of marketing and amusement
JP4904188B2 (en) * 2007-03-30 2012-03-28 三菱電機インフォメーションシステムズ株式会社 Distribution device, distribution program and distribution system
WO2008137581A1 (en) * 2007-05-01 2008-11-13 Neurofocus, Inc. Neuro-feedback based stimulus compression device
US20090328089A1 (en) * 2007-05-16 2009-12-31 Neurofocus Inc. Audience response measurement and tracking system
US8392253B2 (en) 2007-05-16 2013-03-05 The Nielsen Company (Us), Llc Neuro-physiology and neuro-behavioral based stimulus targeting system
US20090033622A1 (en) * 2007-05-30 2009-02-05 24/8 Llc Smartscope/smartshelf
KR20080110489A (en) * 2007-06-14 2008-12-18 소니 가부시끼 가이샤 Information processing apparatus and method and program
US8533042B2 (en) 2007-07-30 2013-09-10 The Nielsen Company (Us), Llc Neuro-response stimulus and stimulus attribute resonance estimator
US8386313B2 (en) 2007-08-28 2013-02-26 The Nielsen Company (Us), Llc Stimulus placement system using subject neuro-response measurements
US8392255B2 (en) 2007-08-29 2013-03-05 The Nielsen Company (Us), Llc Content based selection and meta tagging of advertisement breaks
US20090083129A1 (en) 2007-09-20 2009-03-26 Neurofocus, Inc. Personalized content delivery using neuro-response priming data
US8327395B2 (en) 2007-10-02 2012-12-04 The Nielsen Company (Us), Llc System providing actionable insights based on physiological responses from viewers of media
US9513699B2 (en) * 2007-10-24 2016-12-06 Invention Science Fund I, LL Method of selecting a second content based on a user's reaction to a first content
US20090112694A1 (en) * 2007-10-24 2009-04-30 Searete Llc, A Limited Liability Corporation Of The State Of Delaware Targeted-advertising based on a sensed physiological response by a person to a general advertisement
US20090112696A1 (en) * 2007-10-24 2009-04-30 Jung Edward K Y Method of space-available advertising in a mobile device
US20090112693A1 (en) * 2007-10-24 2009-04-30 Jung Edward K Y Providing personalized advertising
US20090113297A1 (en) * 2007-10-24 2009-04-30 Searete Llc, A Limited Liability Corporation Of The State Of Delaware Requesting a second content based on a user's reaction to a first content
US9582805B2 (en) * 2007-10-24 2017-02-28 Invention Science Fund I, Llc Returning a personalized advertisement
JP2011505175A (en) 2007-10-31 2011-02-24 エムセンス コーポレイション System and method for providing distributed collection and centralized processing of physiological responses from viewers
US8750578B2 (en) 2008-01-29 2014-06-10 DigitalOptics Corporation Europe Limited Detecting facial expressions in digital images
US8171407B2 (en) * 2008-02-21 2012-05-01 International Business Machines Corporation Rating virtual world merchandise by avatar visits
JP5159375B2 (en) 2008-03-07 2013-03-06 インターナショナル・ビジネス・マシーンズ・コーポレーション Object authenticity determination system and method in metaverse, and computer program thereof
US9710816B2 (en) * 2008-08-05 2017-07-18 Ford Motor Company Method and system of measuring customer satisfaction with purchased vehicle
US20100060713A1 (en) * 2008-09-10 2010-03-11 Eastman Kodak Company System and Method for Enhancing Noverbal Aspects of Communication
US20100185564A1 (en) * 2009-01-21 2010-07-22 Mccormick & Company, Inc. Method and questionnaire for measuring consumer emotions associated with products
IT1392812B1 (en) * 2009-02-06 2012-03-23 Gfk Eurisko S R L DEVICE FOR THE CONDUCT OF MARKET INVESTIGATIONS.
US20100250325A1 (en) 2009-03-24 2010-09-30 Neurofocus, Inc. Neurological profiles for market matching and stimulus presentation
US10987015B2 (en) 2009-08-24 2021-04-27 Nielsen Consumer Llc Dry electrodes for electroencephalography
US9560984B2 (en) * 2009-10-29 2017-02-07 The Nielsen Company (Us), Llc Analysis of controlled and automatic attention for introduction of stimulus material
US20110106750A1 (en) 2009-10-29 2011-05-05 Neurofocus, Inc. Generating ratings predictions using neuro-response data
KR101708682B1 (en) * 2010-03-03 2017-02-21 엘지전자 주식회사 Apparatus for displaying image and and method for operationg the same
KR20110066631A (en) * 2009-12-11 2011-06-17 한국전자통신연구원 Apparatus and method for game design evaluation
US8684742B2 (en) 2010-04-19 2014-04-01 Innerscope Research, Inc. Short imagery task (SIT) research method
US8655428B2 (en) 2010-05-12 2014-02-18 The Nielsen Company (Us), Llc Neuro-response data synchronization
CA2815273A1 (en) * 2010-10-21 2012-04-26 Holybrain Bvba Method and apparatus for neuropsychological modeling of human experience and purchasing behavior
US20120143693A1 (en) * 2010-12-02 2012-06-07 Microsoft Corporation Targeting Advertisements Based on Emotion
US8836777B2 (en) 2011-02-25 2014-09-16 DigitalOptics Corporation Europe Limited Automatic detection of vertical gaze using an embedded imaging device
US8620113B2 (en) 2011-04-25 2013-12-31 Microsoft Corporation Laser diode modes
US8760395B2 (en) 2011-05-31 2014-06-24 Microsoft Corporation Gesture recognition techniques
CN102298694A (en) * 2011-06-21 2011-12-28 广东爱科数字科技有限公司 Man-machine interaction identification system applied to remote information service
US8564684B2 (en) * 2011-08-17 2013-10-22 Digimarc Corporation Emotional illumination, and related arrangements
US8635637B2 (en) 2011-12-02 2014-01-21 Microsoft Corporation User interface presenting an animated avatar performing a media reaction
US9100685B2 (en) 2011-12-09 2015-08-04 Microsoft Technology Licensing, Llc Determining audience state or interest using passive sensor data
CN102541259A (en) * 2011-12-26 2012-07-04 鸿富锦精密工业（深圳）有限公司 Electronic equipment and method for same to provide mood service according to facial expression
US9569986B2 (en) 2012-02-27 2017-02-14 The Nielsen Company (Us), Llc System and method for gathering and analyzing biometric user feedback for use in social media and advertising applications
US8898687B2 (en) 2012-04-04 2014-11-25 Microsoft Corporation Controlling a media program based on a media reaction
US9451087B2 (en) * 2012-04-16 2016-09-20 Avaya Inc. Agent matching based on video analysis of customer presentation
CA2775700C (en) 2012-05-04 2013-07-23 Microsoft Corporation Determining a future portion of a currently presented media program
CN102930298B (en) * 2012-09-02 2015-04-29 北京理工大学 Audio visual emotion recognition method based on multi-layer boosted HMM
WO2014061015A1 (en) * 2012-10-16 2014-04-24 Sobol Shikler Tal Speech affect analyzing and training
US9299084B2 (en) * 2012-11-28 2016-03-29 Wal-Mart Stores, Inc. Detecting customer dissatisfaction using biometric data
JP2015111358A (en) * 2013-12-06 2015-06-18 株式会社ニコン Electronic apparatus
JP2015111357A (en) * 2013-12-06 2015-06-18 株式会社ニコン Electronic apparatus
JP2015130045A (en) * 2014-01-07 2015-07-16 日本放送協会 Charge presentation device and charge presentation system
JP6708122B2 (en) * 2014-06-30 2020-06-10 日本電気株式会社 Guidance processing device and guidance method
US9922350B2 (en) 2014-07-16 2018-03-20 Software Ag Dynamically adaptable real-time customer experience manager and/or associated method
US10380687B2 (en) 2014-08-12 2019-08-13 Software Ag Trade surveillance and monitoring systems and/or methods
EP3009979A1 (en) * 2014-10-15 2016-04-20 Wipro Limited System and method for recommending content to a user based on facial image analysis
US9449218B2 (en) * 2014-10-16 2016-09-20 Software Ag Usa, Inc. Large venue surveillance and reaction systems and methods using dynamically analyzed emotional input
US9269374B1 (en) * 2014-10-27 2016-02-23 Mattersight Corporation Predictive video analytics system and methods
US9576190B2 (en) * 2015-03-18 2017-02-21 Snap Inc. Emotion recognition in video conferencing
US9467718B1 (en) 2015-05-06 2016-10-11 Echostar Broadcasting Corporation Apparatus, systems and methods for a content commentary community
US9936250B2 (en) 2015-05-19 2018-04-03 The Nielsen Company (Us), Llc Methods and apparatus to adjust content presented to an individual
JP6561639B2 (en) * 2015-07-09 2019-08-21 富士通株式会社 Interest level determination device, interest level determination method, and interest level determination program
US10255487B2 (en) * 2015-12-24 2019-04-09 Casio Computer Co., Ltd. Emotion estimation apparatus using facial images of target individual, emotion estimation method, and non-transitory computer readable medium
US10268689B2 (en) 2016-01-28 2019-04-23 DISH Technologies L.L.C. Providing media content based on user state detection
US10984036B2 (en) 2016-05-03 2021-04-20 DISH Technologies L.L.C. Providing media content based on media element preferences
JP6219448B1 (en) * 2016-05-16 2017-10-25 Ｃｏｃｏｒｏ Ｓｂ株式会社 Customer service control system, customer service system and program
CN105959737A (en) * 2016-06-30 2016-09-21 乐视控股（北京）有限公司 Video evaluation method and device based on user emotion recognition
CN106303797A (en) * 2016-07-30 2017-01-04 杨超坤 A kind of automobile audio with control system
CN106570496B (en) * 2016-11-22 2019-10-01 上海智臻智能网络科技股份有限公司 Emotion identification method and apparatus and intelligent interactive method and equipment
US10888271B2 (en) 2016-12-08 2021-01-12 Louise M. Falevsky Systems, apparatus and methods for using biofeedback to facilitate a discussion
US9953650B1 (en) * 2016-12-08 2018-04-24 Louise M Falevsky Systems, apparatus and methods for using biofeedback for altering speech
US10390084B2 (en) 2016-12-23 2019-08-20 DISH Technologies L.L.C. Communications channels in media systems
US11196826B2 (en) 2016-12-23 2021-12-07 DISH Technologies L.L.C. Communications channels in media systems
US10764381B2 (en) 2016-12-23 2020-09-01 Echostar Technologies L.L.C. Communications channels in media systems
KR102520627B1 (en) * 2017-02-01 2023-04-12 삼성전자주식회사 Apparatus and method and for recommending products
FR3064097A1 (en) * 2017-03-14 2018-09-21 Orange METHOD FOR ENRICHING DIGITAL CONTENT BY SPONTANEOUS DATA
CN107463915B (en) * 2017-08-11 2018-04-24 胡佳威 A kind of restaurant's concocting method based on image recognition
CN107403288A (en) * 2017-08-11 2017-11-28 无锡北斗星通信息科技有限公司 A kind of adaptive rear kitchen dispatching method
CN107392799A (en) * 2017-08-11 2017-11-24 无锡北斗星通信息科技有限公司 Scheduling system in kitchen after adaptive
US10904615B2 (en) * 2017-09-07 2021-01-26 International Business Machines Corporation Accessing and analyzing data to select an optimal line-of-sight and determine how media content is distributed and displayed
EP3474533A1 (en) * 2017-10-20 2019-04-24 Checkout Technologies srl Device for detecting the interaction of users with products arranged on a stand with one or more shelves of a store
JP6708865B2 (en) * 2017-11-02 2020-06-10 株式会社ＵｓｉｄｅＵ Customer service system and customer service method
US10864443B2 (en) 2017-12-22 2020-12-15 Activision Publishing, Inc. Video game content aggregation, normalization, and publication systems and methods
JP6508367B2 (en) * 2018-02-02 2019-05-08 株式会社ニコン Electronic device system and notification method
JP6504279B2 (en) * 2018-02-02 2019-04-24 株式会社ニコン Electronic equipment system
CN108694372A (en) * 2018-03-23 2018-10-23 广东亿迅科技有限公司 A kind of net cast customer service attitude evaluation method and device
JP6964549B2 (en) * 2018-03-28 2021-11-10 東京瓦斯株式会社 Evaluation acquisition system
CN108694384A (en) * 2018-05-14 2018-10-23 芜湖岭上信息科技有限公司 A kind of viewer satisfaction investigation apparatus and method based on image and sound
CN109191178A (en) * 2018-08-03 2019-01-11 佛山市甜慕链客科技有限公司 A kind of method and system improved service quality by Internet of Things
US11037550B2 (en) 2018-11-30 2021-06-15 Dish Network L.L.C. Audio-based link generation
CN109858949A (en) * 2018-12-26 2019-06-07 秒针信息技术有限公司 A kind of customer satisfaction appraisal procedure and assessment system based on monitoring camera
CN109784678A (en) * 2018-12-26 2019-05-21 秒针信息技术有限公司 A kind of customer satisfaction appraisal procedure and assessment system based on audio
JP2019114293A (en) * 2019-03-26 2019-07-11 株式会社ニコン Electronic apparatus
CN110569714A (en) * 2019-07-23 2019-12-13 咪咕文化科技有限公司 Method for obtaining user satisfaction, server and computer readable storage medium
US11712627B2 (en) 2019-11-08 2023-08-01 Activision Publishing, Inc. System and method for providing conditional access to virtual gaming items
JP7354813B2 (en) * 2019-12-05 2023-10-03 富士通株式会社 Detection method, notification method, detection program and notification program
CN111507774A (en) * 2020-04-28 2020-08-07 上海依图网络科技有限公司 Data processing method and device
JP7063360B2 (en) * 2020-09-11 2022-05-09 株式会社ニコン Electronic device system and transmission method

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0546743A (en) * 1991-08-09 1993-02-26 Matsushita Electric Ind Co Ltd Personal identification device
IT1257073B (en) * 1992-08-11 1996-01-05 Ist Trentino Di Cultura RECOGNITION SYSTEM, ESPECIALLY FOR THE RECOGNITION OF PEOPLE.
US5619619A (en) * 1993-03-11 1997-04-08 Kabushiki Kaisha Toshiba Information recognition system and control system using same
US5774591A (en) * 1995-12-15 1998-06-30 Xerox Corporation Apparatus and method for recognizing facial expressions and facial gestures in a sequence of images

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11488181B2 (en) 2016-11-01 2022-11-01 International Business Machines Corporation User satisfaction in a service based industry using internet of things (IoT) devices in an IoT network

Also Published As

Publication number Publication date
CN1662922A (en) 2005-08-31
EP1520242A1 (en) 2005-04-06
JP2005531080A (en) 2005-10-13
US20040001616A1 (en) 2004-01-01
AU2003247000A1 (en) 2004-01-19

Similar Documents

Publication Publication Date Title
US20040001616A1 (en) Measurement of content ratings through vision and speech recognition
JP6260979B1 (en) Event evaluation support system, event evaluation support device, and event evaluation support program
US10019653B2 (en) Method and system for predicting personality traits, capabilities and suggested interactions from images of a person
US6873710B1 (en) Method and apparatus for tuning content of information presented to an audience
US20200134295A1 (en) Electronic display viewing verification
JP4165095B2 (en) Information providing apparatus and information providing method
US20090195392A1 (en) Laugh detector and system and method for tracking an emotional response to a media presentation
US20030039379A1 (en) Method and apparatus for automatically assessing interest in a displayed product
US20110208593A1 (en) Electronic advertisement apparatus, electronic advertisement method and recording medium
KR20130136557A (en) Personalized advertisement selection system and method
JP6807389B2 (en) Methods and equipment for immediate prediction of media content performance
JP7151959B2 (en) Image alignment method and apparatus
CN111353804A (en) Service evaluation method, device, terminal equipment and medium
CN109739354A (en) A kind of multimedia interaction method and device based on sound
CN110888997A (en) Content evaluation method and system and electronic equipment
CN113409822B (en) Object state determining method and device, storage medium and electronic device
KR20050024401A (en) Measurement of content ratings through vision and speech recognition
CN113887884A (en) Business-super service system
JP7003883B2 (en) A system for assessing the degree of similarity in psychological states between audiences
Lin et al. Face detection based on the use of eyes tracking
AU2021100211A4 (en) Predict Gender: Detect Faces and Predict their Gender, Age and Country Using Machine Learning Programming
AU2021103443A4 (en) Artificial intelligence based face recognition system for customer feedback analysis
KR102239015B1 (en) Image alignment method and apparatus thereof
EP4213105A1 (en) Gaze estimation system and method thereof
KR20240011324A (en) Customized Makeup Techniques Recommended Display System for Individuals' Daily Emotional Information and Facial Skin Conditions

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NI NO NZ OM PH PL PT RO RU SC SD SE SG SK SL TJ TM TN TR TT TZ UA UG UZ VC VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LU MC NL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
WWE Wipo information: entry into national phase

Ref document number: 2003761741

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2004517151

Country of ref document: JP

Ref document number: 20038147750

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 1020047021261

Country of ref document: KR

WWP Wipo information: published in national office

Ref document number: 1020047021261

Country of ref document: KR

WWP Wipo information: published in national office

Ref document number: 2003761741

Country of ref document: EP

WWW Wipo information: withdrawn in national office

Ref document number: 2003761741

Country of ref document: EP