Method and system for processing captured image information in an ...

Written Opinion of the International Searching Authority for PCT/ US2004/041319, filed Dec. 9, 2004.

International Preliminary Report on Patentability for PCT/US2004/ 041319, filed Dec. 9, 2004.

International Search Report for PCT/WO 02/100094, filed Dec. 9, 2004.

International Preliminary Examination Report for PCT/US2002/ 017843, filed Jun. 4, 2002.

International Search Report for PCT/US2002/017843, filed Jun. 4, 2002.

Written Opinion of the International Searching Authority for PCT/ US2004/035478, filed Oct. 25, 2004.

International Preliminary Report on Patentability for PCT/US2004/ 035478, filed Oct. 25, 2004.

Invitation to Pay Additional Fees and Partial International Search for PCT/US2004/035478, filed Oct. 25, 2004.

International Search Report for PCT/US2008/059900, filed Apr. 10, 2008.

International Search Report for PCT/US2004/035478, filed Oct. 25, 2004.

International Search Report for PCT/US2008/10750, filed Sep. 15, 2008.

International Search Report for PCT/US03/40321, filed Dec. 15, 2003.

Official Communication in Chinese Appln. No. 200480030951.8 dated Mar. 27, 2009.

Official Communication in Chinese Appln. No. 200480034551.4 dated May 19, 2008.

Official Communication in European Appln. No. 2002312346 dated Nov. 14, 2006.

Official Communication in Chinese Appln. No. 02815206.9 dated Dec. 12, 2008.

Notice of Opposition in European Appln. No. 02739710.8 dated May 14, 2007.

Summons to Attend Oral Proceedings in European Appln. No. 02739710.8.

Article 96(2) Communication dated Mar. 31, 2004 in European Appln. No. 02739710.8.

Article 96(2) Communication dated Feb. 25, 2005 in European Appln. No. 02739710.8.

Buxton, Bill, "Multi-Touch Systems That I Have Known and Loved," accessed Mar. 21, 2007, http://billbuxton.com/multitouchOverview. html.

Extended Search Report for European Application No. 06010825.5, filed Jun. 4, 2002, dated Jul. 10, 2006.

Haller, Michael et al., "Coeno-Storyboard: An Augmented Surface for Storyboard Presentations," Mensch & Computer 2005, Sep. 4-7, 2005, Linz, Austria.

Han, Jefferson Y., "Low-Cost Multi-Touch Sensing Through Frustrated Total Internal Reflection," 2005, ACM Symposium on User Interface Software and Technology (UIST).

International Preliminary Report on Patentability for PCT/US2008/ 059900, filed on Apr. 10, 2008.

Ivars Peterson, "Artificial reality; combining a person's live video

image with computer graphics suggests novel ways of working and

playing with computers" Science News, Jun. 22, 1985.

Katz, Itai et al., "A Multi-Touch Surface Using Multiple Cameras,"

2007, Advanced Concepts for Intelligent Vision Systems, vol. 4678/

2007.

Kjeldesn, Rick et al., "Dynamically Reconfigurable Vision-Based
User Interfaces," Apr. 2003, 3rd International Conference on Vision
Systems (ICVVS '03), Graz, Austria, pp. 6-12.
Levin, Golan "Computer VisionforArtistsandDesigners: Pedagogic
Tools and Techniques for Novice Programmers," Aug. 2006, Al &
Society, vol. 20, Issue 4, pp. 462-482.

Malik, Shahzad et al., "Visual Touchpad: A Two-Handed Gestural
Input Device," Oct. 13-15, 2004, International Conference on
Multimodal Interfaces (ICMI '04).

Microsoft Surface multi-touch interface table unveiled, May 30, 2007, downloaded from http://www.dancewithshadows.com/tech/ micro soft- surface. asp.

Microsoft Surface Web Page, downloaded from http://www. microsoft.com/surface/Pages/Product/Whatls.aspx on Sep. 24, 2009.

Experience Microsoft Surface, downloaded from http://www. microsoft.com/surface/Pages/Product/Specifications.aspx on Sep. 24, 2009.

Microsoft Surface, downloaded from http://en.wikipedia.org/wiki/ Microsoft surface on Sep. 24, 2009.

Mitsubishi DiamondTouch, http://www.merl.com/projects/ DiamondTouch/, visited Mar. 21, 2007.

Mo, Zhenyao "SmartCanvas: A Gesture-Driven Intelligent Drawing Desk System," Jan. 9-12,2005, Proceedings of Intelligent User Interfaces (IUI '05).

Morris, T. et al., "Real-Time Fingertip Detection for Hand Gesture Recognition," Sep. 9-11, 2002, Advanced Concepts for Intelligent Vision Systems (ACIVS '04), Ghent University, Belgium. Observation by third party Michael Saup dated Jan. 17, 2005, cited during opposition of European Application No. 02739710.8, filed Jun. 4, 2002.

Observation by third party Petra Trefzger dated Jan. 17, 2005, cited during opposition of European Application No. 02739710.8, filed Jun. 4, 2002.

Summons to Attend Oral Proceedings in European Application No. 02739710.8, dated Jun. 12, 2009.

The History of Microsoft Surface, downloaded from http://www. microsoft.com/presspass/presskits/surfacecomputing/docs/ SurfaceHistoryBG.doc on Sep. 24, 2009.

Torr, P.H.S. et al., "The Development and Comparison of Robust Methods for Estimating the Fundamental Matrix," 1997, International Journal of Computer Vision, vol. 24, No. 3, pp. 271-300. Wellner, Pierre, "Digital Desk Calculator:Tangible Manipulation on a Desktop Display" Proceedings of the Symposium on User Interface Software andTechnol (UIST), Hilton Head, S. Carolina, Nov. 11-13, 1991.

Wilson, Andrew, "PlayAnywhere: A Compact Interactive Tabletop Projection-Vision System," ACM Symposium on User Interface Software and Technology (UIST), 2005.

Written Opinion for PCT/US2002/017843, filed Jun. 4, 2002, dated Feb. 5,2003.

Written Opinion of the International Searching Authority for PCT/ US2008/059900, filed Apr. 10, 2008.

Letter of the opponent 02 dated May 28, 2010 in European Application No. 02739710.8, filed Jun. 4, 2002.

* cited by examiner

1

METHOD AND SYSTEM FOR PROCESSING
CAPTURED IMAGE INFORMATION IN AN
INTERACTIVE VIDEO DISPLAY SYSTEM

CROSS-REFERENCE TO RELATED 5
APPLICATIONS

This application is a continuation of U.S. patent application Ser. No. 10/974,044 filed on Oct. 25, 2004, which claims priority from U.S. Provisional Application No. 60/514,024 10 filed Oct. 24, 2003, both of which are hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention 15 The present invention relates to the field of visual electronic displays. Specifically, embodiments of the present invention relate to a method and system for processing captured image information in an interactive video display system.

2. Description of the Related Art

Recent technological advancements have led to the creation of a new interactive medium, the interactive video display system. Interactive video display systems allow realtime unencumbered human interactions with video displays. Natural physical motions by human users are captured by a computer vision system and used to drive visual effects. The computer vision system usually uses images captured by a video camera as input and has software processes that gather 3Q real-time information about people and other objects in the scene viewed by the camera.

One type of vision system creates a background model for distinguishing between foreground and background objects of captured images. The real world environment is very 35 dynamic. As a result, the background model developed from the camera input image may vary over time. For example, the overall lighting of the image viewed by the camera can change, scuff marks or other discolorations can appear, objects may be left on the screen, and specular reflected light from the sun or other sources can move or change, among other things. Consequently, the vision system needs to distinguish important changes in the image from other ones.

In certain situations, the vision system may be subjected to rapid environmental changes that effect the captured image. Such rapid changes can occur, for example, due to people turning lights on and off in a room as well as the sun going behind a cloud, a janitor cleaning the screen, or a change in the display surface material. Since most lighting changes affect the entire screen, the effect on the vision system as described so far would be catastrophic. For instance, with typical parameter settings, the entire screen would appear as foreground for several minutes, causing whatever software using the vision output image to behave in an unusable and erroneous manner.

In other situations, the vision system may treat an object as a foreground object when it should be treated as part of the background. For example, if a person sets an obj ect down, and the object does not move for several minutes, the vision system still considers the obj ect as part of the foreground until it is slowly learned into the background. As such, the vision system continues to react to the object is if it were in the foreground, distracting the viewers.

SUMMARY OF THE INVENTION

65

Various embodiments of the present invention, a method and system for processing captured image information in an

2

interactive video display system, are described herein. In one embodiment, a rapid learning condition of a captured camera image is detected. In one embodiment, the rapid learning condition is detected in response to determining that a predetermined percentage of pixels of a foreground/background distinction image (also referred to as a vision image) are foreground pixels. In another embodiment, the rapid learning condition is detected in response to determining that at least a portion of pixels of an image composed of the absolute value difference between the camera image and a background model have a value exceeding a threshold for a particular length of time.

More specifically, captured camera image is compared to a normal background model image and is compared to a second background model image, wherein the second background model is learned at a faster rate than the normal background model. In one embodiment, the second background model is generated by updating a history data structure of the second background model at a faster rate than a history data structure of the normal background model. In one embodiment, the comparison of the captured camera image to the normal background model image generates a first output image and the comparison of the captured camera image to the second background model image generates a second output image. In one embodiment, the first output image and the second output image and are black and white images identifying a foreground portion and a background portion of the captured camera image.

A vision output image is generated based on the comparison of captured camera image to the normal background model image and the comparison of captured camera image to the second background model image. In one embodiment, the vision image is generated by performing a logical AND operation on the first output image and the second output image.

In another embodiment, an object is detected from captured image information that does not move for a predetermined time period. In one embodiment, at least one pixel corresponding to the object is classified as a burn-in pixel if the pixel is a foreground pixel, as defined by the vision system, for the predetermined time period. In another embodiment, at least one pixel corresponding to the object is classified as a burn-in pixel if the pixel is a foreground pixel for a particular portion of the predetermined time period. In one embodiment, detecting the object includes updating a memory image for each foreground-background distinction image produced by the vision system, wherein a foreground pixel is stored as a non-zero value. An accumulation image is updated with the memory image, wherein the memory image is added to the accumulation image. A pixel is identified as a burn-in pixel if a value of the pixel exceeds a threshold.

A burn-in image comprising the object is generated, wherein the burn-in image is operable to allow a vision system of the interactive video display system to classify the object as background. In one embodiment, burned-in pixels of the burn-in image are represented with a "1" mask, wherein the burn-in pixels correspond to the object, and the remaining pixels of the burn-in image are represented with a "0" mask. It is appreciated that the selected binary mask values can be swapped.

In one embodiment, a modified vision system output image is generated by setting all pixels in the vision output image that are defined as burned-in in the burn-in image to background. In one embodiment, a logical AND operation is performed on the burn-in image and the foreground-background distinction image to generate the vision system's output image.