|Publication number||USRE43447 E1|
|Application number||US 13/048,945|
|Publication date||Jun 5, 2012|
|Filing date||Mar 16, 2011|
|Priority date||Jul 22, 1997|
|Also published as||US6160899, USRE43184|
|Publication number||048945, 13048945, US RE43447 E1, US RE43447E1, US-E1-RE43447, USRE43447 E1, USRE43447E1|
|Inventors||Jin Soo Lee, Hyeon-June Kim|
|Original Assignee||Lg Electronics Inc.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (17), Non-Patent Citations (3), Referenced by (1), Classifications (9)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This application is a Reissue of U.S. Pat. No. 6,160,899. More than one reissue application has been filed for the reissue of U.S. Pat. No. 6,160,899. The Reissue application numbers are Ser. Nos. 13/027,619 and 13/048,945 (the present application).
1. Field of the Invention
The present invention relates to a method of selecting and activating an application menu, and more particularly, to an improved method of application menu selection and activation through image cognition, wherein a menu is selected and activated in correspondence to a user's motion while the motion image of the user is recognized at real time by an image-capturing device such as a camera.
2. Description of the Background Art
In order to select and activate a particular item from a list of application menu being displayed on a monitor screen, a computer generally adopts an input device, such as keyboard, mouse and touchpad.
Under a touch-screen method, the moment a user touches directly by hand a desired menu item among the menu list displayed on the monitor screen, the menu item becomes activated.
As another example, a pointer type wireless control device is employed to select and activate a menu list using an infrared transmission device. Such a pointer type wireless control device is provided with a plurality of sensors at corner portions of a monitor and it calculates a phase difference using an infrared signal being generated from a transmission unit, and accordingly coordinate values are obtained so that a transmitter may move the pointer to a desired position, thereby selecting and activating the required menu item.
However, such a conventional technology requires an additional, external device for the menu selection and activation.
Further, in case of a touch-screen and a pointer type wireless control device, there should be disadvantageously provided a plurality of sensors at corner portions of the monitor.
The present invention is directed to solving the conventional disadvantages.
Accordingly, it is an object of the present invention to provide a method of application menu selection and activation using image cognition which is capable of selecting and activating a menu list in response to a user's motion or a particular device movement while recognizing a user's image at real time by use of an image-capturing device such as a camera.
According to an embodiment of the present invention, using an image-capturing device such as a camera attached to a system, a user's image is recognized at real time and displayed on an initial screen of a monitor. The user makes a direct hand motion while viewing his own image displayed on the initial screen, and when a desired menu icon is designated among a variety of menu icons arrayed on the initial screen, the system guides the user's hand image to the corresponding menu icon for its selection. When the user makes a particular body motion to activate the selected menu, the system recognizes the motion for thereby activating the selected menu.
In the above-described embodiment, a pattern wearable on a finger may be employed so as to accurately recognize a user's specific motion. When the user indicates a desired menu icon wearing the pattern on his finger, the system guides the user's hand image on the screen to move toward the corresponding menu icon for the menu selection. As described in the above-described embodiment, when the user makes a particular body motion to activate the selected menu, the system recognizes the motion for thereby activating the selected menu.
According to another embodiment of the present invention, a particular pattern grabbable by a user is employed. When the user indicates a desired menu icon, the system guides the user's hand image displayed on the screen to move to the corresponding menu icon for its selection, and when the user operates a menu activating member provided in the pattern itself, the system responds, whereby the selected menu becomes activated.
The object and advantages of the present invention will become more readily apparent from the detailed description given hereinafter. However, it should be understood that the detailed description and specific example, while indicating preferred embodiments of the invention, are given by way of illustration only, since various changes and modifications within the spirit and scope of the invention will become apparent to those skilled in the art from this detailed description.
The present invention will become better understood with reference to the accompanying drawings which are given only by way of illustration and thus are not limitative of the present invention, wherein:
On the initial screen serving as a client's window region, there are displayed a plurality of menu lists in type form of icons 11, 12, 13, . . . , 16. A user's image is displayed on the entire initial screen together with the menu lists.
The menu icons 11, 12, 13, . . . , 16 are displayed on the left of the screen, and the dotted squares B1, B2, B3, . . . , B6 enclosing the icons 11, 12, 13, . . . , 16, respectively, are pattern regions for pattern cognition and they do not appear on the real screen.
When the system 2 starts operation, the user's image captured by the camera 1 is displayed on the monitor screen. Accordingly, the user can view his own image being displayed on the screen as shown in
Likewise, as the user's own image is displayed on the screen, the user can easily notice his hand's location while feeling he stands in front of a mirror.
Then, following the hand's movement of the user, a menu icon will be selected and the selected menu icon will be activated and the relevant steps will now be described.
When the user moves his hand toward the region of menu icons 11, 12, 13, . . . , 16, the user's hand image on the screen also moves toward the menu icons.
In the meantime, the system 2 continuously checks up the screen color within the plurality of patterns regions B1, B2, B3, . . . , B6 (Step S22). Since the user's hand is flesh color and the screen background color is not so, when the user moves his hand to a certain pattern region B2, the color in the pattern region B2 changes to flesh color. The system 2 checks up whether the screen color within the pattern regions B1, B2, B3, . . . , B6 is converted to flesh color, thereby determining that the user's hand is positioned on a particular menu icon (Step S23).
In the next step, if the user nods his head, the system 2 recognizes the nodding through a gesture cognition device provided within the system 2 and accordingly activates the selected menu icon 12 (Steps S25, S26).
Meanwhile, in order for the system to recognize the user's gesture, there should be provided a pattern cognition using a moving image. That is, the system continuously captures the user's image and the captured moving image is preprocessed, and the previously captured image is compared with the presently captured image so as to extract characteristics of the two images, whereby the nodding of the user's head can be determined on the ground of the extracted characteristics.
The method in the above-described embodiment is to activate menu by recognizing the user's gesture. Here, the menu activation can be also performed when a particular pattern stays within a certain pattern region for a certain time period. Here, by adding a function to the system, the stationed time period of the particular pattern may be counted so that if a predetermined time lapses the menu becomes activated.
In the selection mode of the menu using the hand motion recognition of the user, there may occur an error operation in the result of erroneous recognition in which a hand motion of the user is mistaken for an arm motion due to the inaccurate recognition of the system. In order to overcome such an erroneous operation of the system, a simple type of pattern can be worn on a user's finger.
As shown in
As further shown in
The indication rod as shown in
With reference to
First, the data with regard to the first pattern portion 12 and the second pattern portion 13 are set in the system.
Step S31 is identical to Step S21 in
Next, when the button 14 of the indication rod is pressed by the user, the second pattern portion 13 is externally exposed from the first pattern portion 12. When the exposure of the second pattern portion 13 is detected by the system, the selected menu icon 12 becomes activated. Likewise, if there is employed such an indication rod having the first and second pattern portions, the system does not require such a gesture cognition function as described in the first embodiment of the present invention.
As shown in
When the user moves his hand leftward to select a menu, the system 52 recognizes the leftward motion so that the hand in the user's image displayed in the image block 53 makes a leftward movement, and accordingly the user's desired menu is selected by checking up the screen color of the pattern region 54.
When the user moves his hand, the user's image is displayed inside the image block 63 and the system causes the pointer 64 to move in response to the user's hand movement. Here, the pointer serves as a mouse pointer mainly employed in the window's operating system in a computer.
The method of menu selection and activation using image cognition according to the preferred embodiments of the present invention may also replace the mouse-oriented menu selection and activation in prevalent window system computers.
As the present invention may be embodied in several forms without departing from the spirit of essential characteristics thereof, it should also be understood that the above-described embodiments are not limited by any of the details of the foregoing description, unless otherwise specified, but rather should be construed broadly within its spirit and scope as defined in the appended claims, and therefore all changes and modifications that fall within meets and bounds of the claims, or equivalences of such meets and bounds are therefore intended to embrace the appended claims.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4565999 *||Apr 1, 1983||Jan 21, 1986||Prime Computer, Inc.||Light pencil|
|US5252951 *||Oct 21, 1991||Oct 12, 1993||International Business Machines Corporation||Graphical user interface with gesture recognition in a multiapplication environment|
|US5319747||May 14, 1993||Jun 7, 1994||U.S. Philips Corporation||Data processing system using gesture-based input data|
|US5454043||Jul 30, 1993||Sep 26, 1995||Mitsubishi Electric Research Laboratories, Inc.||Dynamic and static hand gesture recognition through low-level image analysis|
|US5511148 *||Apr 29, 1994||Apr 23, 1996||Xerox Corporation||Interactive copying system|
|US5528263||Jun 15, 1994||Jun 18, 1996||Daniel M. Platzker||Interactive projected video image display system|
|US5553277||Dec 29, 1992||Sep 3, 1996||Fujitsu Limited||Image search method for searching and retrieving desired image from memory device|
|US5617312 *||Nov 18, 1994||Apr 1, 1997||Hitachi, Ltd.||Computer system that enters control information by means of video camera|
|US5732227 *||Jul 5, 1995||Mar 24, 1998||Hitachi, Ltd.||Interactive information processing system responsive to user manipulation of physical objects and displayed images|
|US5802220 *||Dec 15, 1995||Sep 1, 1998||Xerox Corporation||Apparatus and method for tracking facial motion through a sequence of images|
|US5898434 *||Aug 22, 1994||Apr 27, 1999||Apple Computer, Inc.||User interface system having programmable user interface elements|
|US5900863 *||Mar 13, 1996||May 4, 1999||Kabushiki Kaisha Toshiba||Method and apparatus for controlling computer without touching input device|
|US5926168 *||Sep 5, 1995||Jul 20, 1999||Fan; Nong-Qiang||Remote pointers for interactive televisions|
|US5926264 *||Oct 10, 1995||Jul 20, 1999||The Secretary Of State For Defence In Her Britannic Majesty's Government Of The United Kingdom Of Great Britain And Northern Ireland||Position sensing of a remote target|
|US6094197 *||May 17, 1995||Jul 25, 2000||Xerox Corporation||Graphical keyboard|
|US6525749 *||Oct 25, 1996||Feb 25, 2003||Xerox Corporation||Apparatus and method for supporting the implicit structure of freeform lists, outlines, text, tables and diagrams in a gesture-based input system and editing system|
|JPH05324181A||Title not available|
|1||Korean Office Action dated Sep. 22, 1999. (Application No. 10-1997-0034165).|
|2||Notice of Allowance issued in U.S. Appl. No. 13/027,619 dated Dec. 13, 2011.|
|3||Office Action issued in U.S. Appl. No. 13/027,619 dated Jul. 1, 2011.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US20150177843 *||Jun 27, 2014||Jun 25, 2015||Samsung Electronics Co., Ltd.||Device and method for displaying user interface of virtual input device based on motion recognition|
|U.S. Classification||382/103, 715/863, 348/77, 382/100|
|International Classification||G06K9/00, G06F3/00, G06F3/01|