WO2006136696A1

WO2006136696A1 - Method and device for rendering interactive a volume or surface

Info

Publication number: WO2006136696A1
Application number: PCT/FR2006/001395
Authority: WO
Inventors: Frédéric Jacques GUERAULT; Christophe Jean-Louis Chesnaud; Frédéric OMNES; Luis Pedro Gomes
Original assignee: 3Dfeel
Priority date: 2005-06-22
Filing date: 2006-06-21
Publication date: 2006-12-28

Abstract

The inventive device comprises an elongated bar (100) provided with two matrix image sensors (120, 160) each of which delivers a two-dimensional image comprising a plurality of lines or a plurality of columns positioned on the bar ends and whose optical fields are superimposed and encompass said surface, a storage means (125) for each matrix image sensor, at least one area of interest consisting of one part comprising at least one plurality of line segments and one plurality of column segments of an image delivered by said matrix image sensor, wherein each area of interest comprises the cross-sectional image of said surface and at least one eventual pointer intersecting said surface and means for determining (125) the position of at least one said pointer intersecting said surface according to the part of the pointer images placed in said areas of interest.

Description

METHOD AND DEVICE FOR MAKING AN INTERACTIVE VOLUME OR SURFACE

The present invention relates to a method and a device for rendering a volume or an interactive surface. It applies, in particular, to allow a user, with gestures made in this area or this volume, to control actions, for example computer applications such as reading media files and / or consulting data. .

Many devices are known to make interactive a volume or a surface.

US 4,746,770, EP 0279652 and JP 63223819 describe a method for isolating and manipulating graphic objects. It implements a frame that supports a plurality of optical sensors and a plurality of opaque objects, for example fingers, that obstruct a portion of the light normally received by the sensors. This method and the associated device, based on the optical occultation carried out with a finger, therefore require the use of a complex, bulky and expensive frame since it must bear, on at least two sides facing each other, light sources and sensors. In addition, its accuracy is limited to the product of the number of sensors by the number of light sources, generally only a few hundred points that can be discriminated.

The touch screen described in US 5,162,783 has the same disadvantages. WO 99/40562 and US 19980020812 disclose a video camera touch system which takes the form of a bar placed above a computer screen. This system implements complex periscope equipment to capture two images of the scene right in front of the computer screen, from different angles. Finally, this system does not identify pointers outside the surface of the screen.

US 6,061,177 discloses a touch data input device for a back projection system. It has a transparent surface and a diffuser on which an image is projected and captured from the back by a camera. When the screen is touched, the light reflection is interrupted by the absorption on the surface of the finger touching the screen and captured by the camera. This system imposes the release of a large volume corresponding to the sum of the optical field of the camera and the optical field of the overhead projector, which affects its use on large surfaces. In addition, this system can not be implemented outdoors where the incident light incident can be very dazzling for the image sensor of the camera. Finally, this system does not allow pointer detection outside the image projection area by the overhead projector. The device described in JP 2003 312123 has the same drawbacks. Similarly, the device described in DE 1995 1322 and WO 01/31424 has the same disadvantages. Similarly, the device described in GB 2315859 has the same disadvantages.

The present invention aims to remedy these drawbacks and in particular to provide a device operating outdoors or indoors, comprising modules comprising image sensors and lighting sources that can be positioned in the plane of the interactive surface, without respecting binding positioning criteria while ensuring a high accuracy of position discrimination of one or more pointers, for example fingers.

It is another object of the present invention to provide a device for making interactive a surface or a volume, of which all the electrical components are incorporated in a linear bar easy to transport, to implant on site or to integrate into a piece of furniture. and to operate opposite said surface, regardless of the position of frame members, these having no electrical component.

For this purpose, the present invention aims, in a first aspect, a device for rendering an interactive surface or volume, characterized in that it comprises: an elongated bar which comprises two matrix image sensors providing each a two-dimensional image, having a plurality of lines and a plurality of columns, placed at the ends of the bar and whose optical fields are superimposed and encompassing said surface, a storage means, for each of said matrix image sensors, at least one region of interest consisting of a portion comprising, at least, a plurality of line segments and a plurality of column segments of the image provided by said array image sensor, each said region of interest representing the image, in section, of said surface and at least one possible pointer intersecting said surface and position determining means of at least one said pointer cutting said surf ace, depending on the portion of the images of said pointer in said regions of interest.

Thanks to these arrangements, the matrix image sensors, for example consumer video cameras and consumer image acquisition cards, which are less expensive than linear or specific sensors and cards, can be implemented and reduce the number of image sensors. complexity and cost of the device. In addition, the device, because of its elongated bar shape, can be easily transported, for example by a postal service. An installer may position this bar in any position provided that the area of superposition of the optical fields of the matrix image sensors covers the surface or the volume to be made interactive and that the illumination, ambient or provided by at least one light source integrated in the bar allows the detection of pointers, by each matrix image sensor in the whole of this surface. The implementation of a bar also allows many stages of parameterization, in manufacture, such as that of the distance between the cameras, the focal lengths of the lenses of the cameras and the angles between the optical axis of the cameras and the camera. axis passing through the optical centers of the lenses of the cameras. On-site installation is therefore greatly facilitated. It is therefore possible to provide installers or untrained operators with a bar and software implementing the storage means of regions of interest and the position determination means. The device object of the present invention can thus become a simple component, similar to a large touch screen.

According to particular features, the device as briefly described above further comprises means for controlling the display of at least one reference point opposite said surface and means for storing coordinates of a pointer. placed next to each reference point, said coordinates from the processing of the pointer images provided by the image sensors, for determining a correspondence between the position of the pointer and the coordinates, in the plane of the matrix display, of the point of the display which is opposite said pointer.

According to particular features, the device as briefly described above comprises, in addition: means for memorizing coordinates of a pointer successively placed at extreme points of at least one active zone in said surface, said coordinates coming from the processing of the pointer images provided by the image sensors, means for associating, with each active zone, at least one action to be performed and, - an action triggering means, adapted, when the determination means position provides a position of a pointer corresponding to one of the active areas, to trigger an action associated with said active area.

Thanks to these provisions, each time a user positions a pointer opposite the interior of one of the active zones, the processing of the images of this pointer in the regions of interest of the two sensors makes it possible to determine that it is inside the active zone considered and launch an action assigned to said active zone.

According to particular features, the device as briefly described above comprises, in addition: a matrix display positioned parallel to and facing a portion of said surface; - A coordinate storage means of a pointer placed successively opposite the end points of said matrix display, said coordinates defining an active area, said active area being associated with at least one action to be performed.

With these provisions, a user can interact with an animated image or a computer screen, as with a mouse. According to particular features, the device as briefly described above further comprises: means for displaying, on said matrix display, at least one reference point; coordinate storage means of a pointer placed opposite each said reference point, said coordinates from the processing of the pointer images provided by the image sensors, means for storing data representative of optical distortion.

Thanks to these arrangements, the coordinate determination accuracy of a point of the display pointed by a pointer can be increased and allows operation in computer pointing device similar to that of a mouse. According to particular features, the device as briefly described above further comprises: means for determining successive positions of a pointer in the plane of the interactive rendered surface, by processing the last two images received from the sensors of image, each position having two coordinates and a position correction means of the pointer, during which, as corrected position, a position is assigned a function of at least two successive positions.

Thanks to these arrangements, even if the image pickups of the two matrix sensors are not synchronized, at least partially compensates for the position errors resulting from the displacement of the pointer between the times of taking of the cameras. This compensation is effective that we use permanently the last image provided by each camera or waiting for a new image of the two cameras before performing the treatment.

According to particular features, said correction means is adapted so that the corrected position is the weighted barycentre of the last, the penultimate and the antépénultième positions, the last and the antepenultimate position each having an equal weight at half the weight of penultimate position. Thus, the compensated position is very close to the actual position of the pointer while adding a response time less than the duration of the time interval between two successive shots of views by the same camera, typically 20 milliseconds.

According to particular features, the device as briefly described above further comprises: a movable light source positioned successively along a frame placed in the plane of the surface to be rendered interactive, outside said surface, at least two positions within the optical field of each array image sensor and directing said light source toward said array image sensor; means for memorizing the coordinates in the plane of the image supplied by the matrix image sensor, of the point corresponding to said moving light source, for each position of said light source, and means for determining the region of interest, in the plane of the image provided by said sensor, said region of interest comprising each line segment drawn between two points corresponding to the successive positions of said moving light source in the images provided by each image sensor matrix.

Thanks to these provisions, the determination of the region of interest of each image provided by an image sensor is easy and accurate.

According to particular features, the device as briefly described above further comprises means for determining the region of interest, in the plane of the image supplied by said sensor, said region of interest comprising each segment. of a straight line drawn between two successive pointer positions in the region of interest, in an image taken by each matrix sensor.

Thanks to these arrangements, it is not necessary to have a light source in the field of the cameras and the determination of the regions of interest can be carried out remotely from an image supplied by each camera, for example in Ia. carrying on the Internet.

According to particular features, the device as briefly described above comprises a detection means adapted, for detecting a pointer, to determine, for each column of picture elements of the region of interest of each sensor, the minimum of brightness for the points of this column and to compare this minimum to a brightness threshold value and the number of consecutive minimums exceeding this threshold value to at least one threshold value of apparent width. It is thus determined whether the pointer meets width criteria distinguishing it from a reflection or any object.

According to particular features, the device as briefly described above comprises: a means for allocating a computer application, for at least one movement of the pointer and for at least one active zone or the entire interactive surface; , and a suitable control means, when said movement is detected in said active zone or in the interactive surface, respectively, to start a step of operation of said computer application. Thanks to these provisions, it is possible to match horizontal or vertical hand movements or holds in position of different durations, different computer applications such as screen changes, launches or exits of an application or application. A file.

According to particular features, the allocation means is adapted so that at least one said movement is defined by its amplitude, duration or speed, spatial tolerance, direction and direction.

According to particular features, the device as briefly described above comprises a Kalman prediction determination means and a suitable position assignment means, when at least two pointers are detected, to attribute to at least one of said pointers a possible positions whose distance to a Kalman prediction is minimal.

According to particular features, the device as briefly described above comprises a Kalman prediction determination means and a suitable position assignment means, when at least two pointers are detected, to assign auditors pointers possible positions compatible with Pointer images that minimize the sum of their distance to Kalman's predictions.

According to particular features, the device as briefly described above further comprises: means for determining at least one active zone in said surface and means for associating, with each active zone, at least one action to be performed at least one gesture defined by at least one relationship between successive positions, means for determining successive positions of at least one pointer according to the images captured by the image sensors and triggering means of an action associated with said active zone as a function of the successive positions of said pointer, when, in an operating phase, the position of a pointer corresponds to an active zone.

Thanks to these provisions, it is possible to associate a gesture with an action, for example, launching a computer application or an audiovisual program, opening a file, changing a page or an image of a presentation, the user has only to reproduce the gesture in the area of interest of the sensors and in the associated active zone, to trigger this action. According to a second aspect, the present invention is directed to a method for rendering an interactive surface or volume, characterized in that it comprises: a step of installing an elongated shape bar which comprises two matrix image sensors providing, each, a two-dimensional image, having a plurality of lines and a plurality of columns, placed at the ends of the bar and whose optical fields are superimposed and encompassing said surface, a storage step, for each of said matrix image sensors of at least one region of interest consisting of a portion comprising at least a plurality of line segments and a plurality of column segments of the image provided by said array image sensor, each said region of interest representing the image, seen in section, of said surface and of at least one possible pointer intersecting said surface and a step of determining the position of at least one said pointer c orpicking said surface, depending on the portion of the images of said pointer in said regions of interest. According to particular features, the method as briefly described above further comprises a step of controlling the display of at least one reference point opposite said surface and a step of memorizing coordinates of a pointer. placed next to each reference point, said coordinates from the processing of the pointer images provided by the image sensors, to determine a correspondence between the position of the pointer and the coordinates, in the plane of the matrix display, of the point the display that is next to the pointer.

According to particular features, the method as briefly described above comprises, in addition: a step of memorizing coordinates of a pointer successively placed at extreme points of at least one active zone in said surface, said coordinates coming from the processing of the pointer images provided by the image sensors, a step of associating, with each active zone, at least one action to be performed and, an action triggering step, when the position determining means provides a position of a pointer corresponding to one of the active zones, an action associated with said active zone. The advantages of this method being similar to those of the device, as succinctly described above, they are not recalled here.

The inventors have discovered that a problem that arises when one wishes to use a finger or the hand to interact with a large surface is that of triggering actions, in addition to pointing an object, as would be done by double clicking with a mouse or using the right and, to create new ways to trigger actions, impossible with mice.

For this purpose, according to a third aspect, the present invention aims a method for rendering an interactive surface, characterized in that it comprises, on the one hand, an initialization step which comprises: a positioning step of at least two image sensors in the plane of said surface, the optical field of each image sensor covering the whole of said surface, a step of determining at least one active area in said surface, an association step, at each active zone, of at least one action to be performed at least one gesture defined by at least one relationship between successive positions and, secondly, an exploitation step which comprises: a step of determining successive positions of at least one pointer according to the images captured by the image sensors and when the position of a pointer corresponds to an active zone, a step of triggering an action associated with said active zone according to successive positions of said pointer.

Thanks to these provisions, we can associate a gesture with an action, for example, the launch of a computer application or an audiovisual program, the opening of a file, a change of page or image of a presentation, the user has only to reproduce the gesture in the area of interest of the sensors and in the associated active zone, to trigger this action.

According to particular features, the step of determining at least one active zone comprises a step of memorizing coordinates of a pointer successively placed at extreme points of said active zone in said surface, said coordinates coming from the image processing of the image. pointer provided by the image sensors.

Thanks to these provisions, to declare an active area, for example opposite a poster on a window or access to a property, the user only has to indicate the extreme points, for example the corners of this active zone. The programming is thus fast and intuitive.

According to particular features, during the positioning step, each image sensor is a matrix image sensor, the method further comprising a step of storing a region of interest in the image supplied. by each of said matrix sensors, said regions of interest representing the image, in sectional view, of said surface and of each pointer intersecting said surface.

With these features, the particular advantages and features of the first aspect of the present invention, briefly set forth above, are found to be associated with those of the third aspect of the present invention.

According to particular features, the method as briefly described above comprises, in addition: a step of positioning at least one matrix display parallel to and facing a portion of said surface; a step of memorizing coordinates of a pointer placed successively opposite the end points of said matrix display, said coordinates defining an active zone, said active zone being associated with at least one action to be performed.

With these provisions, an active area can be easily created next to a display screen, for example a computer screen or TV and actions associated with the active area can influence the content displayed on this screen.

According to particular features, the method as briefly described above further comprises: a step of displaying, on said matrix display, at least one reference point; a step of memorizing coordinates of a pointer placed opposite each said reference point, said coordinates coming from the processing of the pointer images provided by the image sensors and a step of processing said coordinates to determine a correction to be applied to the images.

Thanks to these arrangements, a fine calibration of the pointer marking can be carried out. According to particular features, the method as briefly described above comprises, in addition: a step of determining successive positions of a pointer in the plane of the interactive rendered surface, by processing the last two images received from the sensors of image, each position having two coordinates and a position correction step of the pointer, during which, a corrected position is assigned a position function of at least two successive positions. According to particular features, during the position correction step, the corrected position is the weighted barycentre of the last, the penultimate and the third last positions, the last and the third position respectively having a weight equal to half the weight of the penultimate position.

With each of these arrangements, the matrix image sensors do not need to be synchronized to provide precise positioning of the pointer: the effects of triangulation errors due to the possible movements of the pointer between images taken by the two sensors are compensated.

According to particular features, the method as briefly described above comprises, in addition: a step of successive positioning along a frame placed in the plane of the surface to be interactive, outside said surface, in at least two positions in the optical field of each image sensor of a moving light source facing said image sensor; for each position of said light source, a step of storing the coordinates, in the plane of the image supplied by the image sensor, of the point corresponding to said moving light source and a step of determining the region of interest, in the image plane provided by said sensor, said region of interest comprising each line segment drawn between two points corresponding to the successive positions of said moving light source. Thanks to these provisions, the determination of the region of interest of each image provided by an image sensor is easy and accurate.

According to particular features, the method as briefly described above further comprises: a step of pointing, in an image taken by each matrix sensor, of at least two positions corresponding to points of a region of interest ; for each position, a storage of the coordinates, in the plane of the image supplied by the image sensor, of the corresponding point and a step of determining the region of interest, in the plane of the image supplied by said sensor image, said region of interest comprising each line segment drawn between two successive points whose coordinates have been stored.

According to particular characteristics, in order to detect a pointer, the minimum brightness for the points of this column is determined for each column of picture elements of the region of interest of each sensor, and this minimum is compared with a value brightness threshold and the number of consecutive minimums exceeding this threshold value to at least one threshold value of apparent width.

Thanks to these arrangements, the determination of the position of the pointer is easy and accurate and avoids at least part of the possible problems of stray light.

According to particular features, the method as briefly described above comprises: a step of assigning a computer application, for at least one movement of the pointer and for at least one active zone or the entire interactive surface, and when said movement is detected in said active area or in the interactive surface, respectively, a step of operating said computer application. Thanks to these provisions, computer applications can be launched with a simple gesture.

According to particular features, during the step of assigning a computer application, at least one said movement is defined by its amplitude, duration or speed, spatial tolerance, direction and direction. Thanks to these provisions, the gestures are easily memorized by the user and can be parameterized.

According to particular characteristics, when at least two pointers are detected, a step of assigning to at least one of said pointers one of the possible positions whose distance to a Kalman prediction is minimal. According to particular characteristics, when at least two pointers are detected, a step of assigning pointers audits of the possible positions compatible with the images of the pointers which minimize the sum of their distance to the Kalman predictions.

With each of these provisions, the user can use two pointers, for example two fingers or two hands, to interact with the interactive surface, the Kalman prediction then solving the multitude of potentially valid triangulations.

According to a fourth aspect, the present invention aims at a device for rendering an interactive surface, characterized in that it comprises: two image sensors in the plane of said surface, the optical field of each image sensor covering the whole of said surface; means for determining at least one active zone in said surface and means for associating, with each active zone, at least one action to be performed on at least one gesture defined by at least one relationship between successive positions, means for determining successive positions of at least one pointer based on images captured by the image sensors and means for triggering an action associated with said active zone as a function of the successive positions of said pointer, when, in an operating phase, the position of a pointer corresponds to a zone active.

Since the advantages, aims and particular characteristics of this device are similar to those of the process that is the subject of the third aspect of the present invention, they are not recalled here. The aims, essential or particular characteristics and advantages of each of the methods or devices that are the subject of the various aspects of the present invention are preferably combined to form a method and a device making an interactive volume or surface particularly easy to transport, install and parameterize. and enabling rich interactions with the user, including the triggering of actions as gestures performed by the user in an active area.

Other advantages, aims and features of the present invention will emerge from the description which follows, made for an explanatory and non-limiting purpose with reference to the accompanying drawings, in which: FIG. 1 represents, schematically, a section parallel to the surface; interactive, part of a first embodiment of the device object of the present invention; Figure 2 shows schematically in section perpendicular to the interactive surface, a portion of the first embodiment of the device object of the present invention; FIG. 3 schematically represents, in section parallel to the interactive surface, part of a second embodiment of the device that is the subject of the present invention; - Figure 4 shows schematically in section parallel to the interactive surface, a portion of a third embodiment of the device object of the present invention; Figure 5 shows schematically in section parallel to the interactive surface, a part of a fourth embodiment of the device object of the present invention; FIGS. 6 and 7 represent a light box used for the installation of particular embodiments of the device that is the subject of the present invention; Figure 8 shows schematically four positions taken by the light box illustrated in Figures 6 and 7; FIG. 9 represents points, pointer positions and corrected point positions implemented in particular embodiments of the method that is the subject of the present invention; FIG. 10 represents possible positions of two pointers and Kalman predictions of two pointers, implemented in particular embodiments of the method that is the subject of the present invention; Fig. 11 shows positions of a pointer obscured by another pointer, Kalman estimates and predictions of the obscured pointer; FIG. 12 represents measured positions and corrected positions of a moving pointer according to particular embodiments of the method that is the subject of the present invention; FIGS. 13A to 13C represent, in the form of a logic diagram, steps implemented in particular embodiments of the method that is the subject of the present invention, and FIGS. 14 to 16 represent frame profiles comprising concave portions.

Throughout the description, we speak of a surface to make interactive. However, the present invention is not limited to this particular type of plane volume but extends, quite the contrary to the volumes defined as the intersection of the cones having as a base frame and as the top optical center of the cameras. Thus, the larger the framework, the larger it becomes interactive.

FIG. 1 shows a bar 100 comprising two parts 110 and 150 situated longitudinally at the left and right ends, respectively, of the bar 100. Each part 110 and 150 has a window, respectively 112 and 152, facing the surface. to render interactive and provided with a filter filter visible light and letting the rays in the near infrared, respectively 115 and 155, a monochrome camera whose sensitivity spectrum covers the visible and the near infrared, respectively 120 and 160 with a lens , respectively 122 and 162 and a matrix image sensor, respectively 124 and 164. Each camera provides a signal representative of a two-dimensional image having a plurality of lines and a plurality of columns. Preferably, each matrix image sensor has more than one hundred lines and more than one hundred columns of photosensitive zones, or pixels, and the camera provides a signal representative of the light intensity received by each of these photosensitive zones. .

Each camera is oriented, through the corresponding window, to the interactive surface and a longitudinal opening for ventilation. The bar 100, whose length is, for example, at least equal to five times the other dimensions, comprises, between the parts 110 and 150, a longitudinal cylindrical Fresnel lens 130 oriented towards the surface to make interactive, one or two lines electroluminescent diodes 135 emitting in the near infrared and oriented towards the lens 130, a longitudinal fan 140 and a set of links 145 providing power to the cameras and light-emitting diodes and transporting the signals from the cameras to a computer 125. The lens 130 converges the light rays from the light-emitting diodes 135 towards the interactive surface.

FIG. 2 shows a sectional view of an installation comprising the bar 100, above an interactive rendering surface 170, and a lower frame member 175. Side frame members 180, not shown in FIG. Figure 2, bound the optical field of the cameras, around the surface to make interactive. The frame elements have very little reflectivity in the wavelengths that the filters 115 and 155 pass through. For example, the frame may be formed of a black strip 2 to 4 centimeters wide which is painted with paint. Krylon ultra flat black, reference

K1602 for the spray (registered trademarks) and placed all around the desired interactive area. The further the lower part of the frame is moved away from the bar (the HM1 value of documentations becoming larger), the wider this black band will be. The profile of the frame may be plane, possibly bevelled or chamfered on the side opposite the support. The profile of the frame may also take a concave shape with at least two plane walls forming between them an angle greater than or equal to 90 °, at least one of said walls being able to be vertical to limit the parasitic light incident on the inside of the profile , as illustrated by the exemplary profiles 1400, 1500 and 1600, in FIGS. 14 to 16.

In the installation mode illustrated in FIG. 2, a wall 185 carries the bar 100, a window 190 placed opposite the surface to be made interactive and the frame elements 175 and 180. It is observed that the presence of a window or any object, wall, linear or furniture shelving, is in no way necessary for the operation of the device. The steps performed with the computer 125, installation, parameterization and operation of the particular embodiment of the device object of the presentation illustrated in Figures 1 and 2 are described later, with reference to Figures 9 and following.

In a particular embodiment, the bar is made elongable. In this embodiment illustrated in FIG. 3, a bar comprises the parts 110 and 150 and intermediate elements 305 carrying, in preferred embodiments, light-emitting diodes 310 and longitudinal cylindrical Fresnel lenses, these elements 305 being mounted on a deformable structure 320 which keeps the elements 305 equidistant from each other during the elongation of the bar.

It is observed that the method and the device which are the subject of the present invention do not impose the use of a light source, here the light-emitting diodes 310 and the Fresnel lenses, in the case where the ambient light, for example solar , is sufficient.

In a particular embodiment, illustrated in FIG. 4, a bar 400 is positioned at a corner with respect to the surface to be rendered interactive and comprises the same elements as the bar 100, the windows and the cameras 420 and 460 then being positioned in parallel and an "L" frame whose horizontal 475 and vertical 480 elements are positioned around the opposite corner of the interactive surface.

In a particular embodiment, illustrated in FIG. 5, each camera module 500 comprises a window 505, a filter 510, a camera 520 comprising a lens 522 and a matrix image sensor 524, two lenses 530, two or four lines of light-emitting diodes 535 placed on either side of the camera 520, two ventilation openings 540 and a fan 550. Two of these camera modules can be positioned anywhere in the plane of the surface to be interactive, subject to that the intersection of their optical fields covers all of this interactive surface and that the light from light-emitting diodes 535 is sufficient, over the entire interactive surface, to detect the light reflected by a pointer.

FIGS. 6 and 7 show a pointing box 600 comprising a light-emitting diode 605 emitting in the near infrared and having an emission angle greater than 60 degrees. At the rear of this diode 605, two planes 610 and 615 are formed, oblique with respect to the optical axis of the diode 605, and present with this axis an angle of approximately 30 degrees. Thus, when the housing 600 is placed against the side frame 180 at the maximum height of the surface to be interactive with an orientation of the optical axis of the diode 605 upwards, each camera in the opposite corner is illuminated by the diode 605, as illustrated in FIG. when the housing 600 is placed in a corner, resting on the lower frame member 175, the two cameras are always in the emission field of the diode 605, as illustrated in FIG. 7.

In alternative embodiments, the diode 605 is replaced by a set of diodes forming a one or two dimensional array to increase the number of image points corresponding to the region of interest. In variants, the pointing case 600 has, on one of its lateral faces, a switch so that the installer can trigger the lighting of the diode or diodes. In variants having said switch and at least two diodes positioned laterally, with respect to the plane perpendicular to the representations of the box 600 in FIGS. 6 to 8, the diodes are turned on in phase-shifted sequences so that, by processing the images taken by the cameras, the coordinates of the image points corresponding to the direct vision of the diodes are differentiated and the coordinates of the image points corresponding to the reflection of the light emitted by these diodes.

To aid the identification of the light-emitting diode in the optical field of the cameras, this diode is preferably controlled to flash, for example with a period of one second. The installation procedure described below applies to the case of the bar 100. The person skilled in the art knows how to extend it to the other embodiments of the device illustrated in FIGS. 3 to 5, taking into account the angle between the optical axes of the lenses of the cameras.

To automatically adjust the parameters of each camera and, in particular, the illumination or gain factor, this factor is set to the lowest value and is gradually increased, while the housing illustrated in FIG. the field of the camera, until the level of perception of the light-emitting diode of this housing corresponds to a predetermined value, for example the level of white.

In a variant, a permanent automatic gain control is implemented to adapt the sensitivity of the camera to the ambient brightness. The cameras used in the devices of the present invention being preferably matrix cameras, they provide images which represent, successively along their smallest dimension, on the one hand the environment close to a first side of the camera. surface to make interactive, for example a showcase, on the other hand, the volume defined by the frame and the lens of the camera, narrow volume that surrounds the plane parallel to the surface to make interactive that goes through the center the lens of the camera, called region of interest or "ROI" (for, in English "region of interest") and finally the environment close to this volume on the other side of the surface to make interactive or, if this surface is reflective, the reflection of the near environment lying on the first side.

It is observed that the region of interest RO1 is not necessarily parallel to the larger side of this image, because of the mechanical positioning tolerances of the camera and the optical components and the sensor inside the camera considered.

Likewise, this region of interest ROI consists of two diamonds, in the image supplied by the camera, when, as is generally the case, the part of the frame seen by each camera consists of two rectangles. To determine which part of the image constitutes the region of interest ROI, preferential procedure is carried out according to one of the procedures indicated below.

It is observed that a region of interest ROI is defined for each of the matrix image sensors, by coordinates in the image plane associated with the image sensor. At least one region of interest consists of a part of the pixels of the matrix image sensor and therefore of the image provided by said matrix image sensor, this part comprising, at least, a plurality of line segments and a plurality of column segments. Because of the lateral position of each camera, each region of interest associated with an image sensor represents the image, seen in section, of the surface or volume rendered interactive and of each possible pointer, for example a finger or a hand, cutting the surface or volume rendered interactive.

A region of interest covers between 1/1000 and 1/10 of the pixels of the image sensor and preferably, between 5/1000 and 2/100 of the pixels of the image sensor when it is desired to make a viewed surface interactive. in section, in this region of interest or a narrow volume surrounding this surface. In a first procedure, it implements, as illustrated in FIG. 8, the pointing box illustrated in FIGS. 6 and 7 by positioning it successively along the frame in at least three positions 811 to 814 located in the optical field of each camera, directing the axis of the light source towards the cameras. In the present case, when the housing is located in positions 811 to 813, it is in the optical field of the camera 820 and, when the housing is in positions 812 to 814, it is in the optical field of the camera 860.

The coordinates in the image plane of each light point indicate points of the region of interest ROI. From these at least three points, two half-lines are determined in the image plane of the camera and, as a region of interest ROI, the points of these half-lines and the points lying at a distance from these half-lines are considered. half-lines less than a predetermined limit distance, for example the points of the five half-lines shifted, upward or downward from the image, by at most five pixels or pixels.

In a second procedure, the installer moves a pointer, finger, fist or hand over the entire surface to be rendered interactive, and for each camera is determined the narrow diamond-shaped dark areas surrounding a luminous area in movement corresponding to the pointer.

In particular embodiments, once the determination of the region of interest has been made, the sensitivity of each camera is determined or, which amounts to the same, apart from the glare phenomena extending over the surface of the sensor. image (blooming and smearing), the amplification of the signal coming out and the digitized range of values that will be taken into account, so that the region of interest has, over its entire surface a value considered black. It is observed that this normalization of the signal can be carried out point by point (each point corresponding to a pixel) and dynamically, considering, for each image point of the region of interest, the average value over a long period, for example five minutes, as black value. In particular embodiments, the brightness limit value, point by point or globally, is determined beyond which it will be considered that the pointer is potentially opposite a point of the region of interest.

In particular embodiments, for detecting a pointer, the minimum brightness for the points of this column is determined for each column of the region of interest ROI.

Then a threshold value is determined, for example equal to twice the average of these minimums. It is observed that the use of this minimum brightness has many advantages. On the one hand, small movements of the finger perpendicular to the interactive surface can be detected since as soon as the pointer leaves a dark line, the minimum takes this low light. On the other hand, the reflections and artifacts on the frame do not disturb the detection because they cause luminosities greater than the minimum brightness. Finally, even if the region of interest covers more than the dark frame, the detection of the pointer is not disturbed.

The number of successive columns for which the considered minimum is greater than the threshold value is then determined, which provides a central point of each pointer and a width of each pointer seen by each camera.

In embodiments, the detection of a real pointer is performed when a sufficient number of image points representing a rectangle in the image (e.g., three columns and eight lines in the image), exhibit a higher brightness to the brightness limit value.

The lenses of the cameras create a radial distortion. During the calibration phase, the parameters of this distortion are measured and modeled by a polynomial of the third degree which makes it possible to correct the effects thereof (for example a polynomial of the form r _c = C ₀ + C ₁ J + c ₂ .r ² + c ₃ .r ³ , where r _c is the corrected radius, r the distance between the current point and the intersection of the optical axis of the camera with the image sensor and C ₀ to C ₃ are correction coefficients).

The calibration phase made in the factory makes it possible to calculate the focal length of the objectives of each camera and the angle between the optical axis of each camera and the horizontal.

The two image coordinates provided by the cameras for each pointer make it possible to determine the position of the pointer in the user plane, that is to say the interactive surface. The current coordinate system used for the computation of the coordinates is a reference mark whose origin is the optical center of the camera placed on the left in figure 9. The axis of the abscissae is the horizontal axis oriented towards the camera represented on the right in figure 9 The distance between the optical centers of the lenses of the cameras defines the unit of length. The y-axis is downward, the unit length being equal to that of the x-axis.

The coordinates in the plane thus provided with an orthogonal coordinate system are determined as a function of the angles μ1 and μ2 formed between the lines starting from each of the optical centers of the lenses and passing through the center of the pointer on the one hand and the horizontal axis on the other hand as follows:

Tx = sin μ1 x cos μ2 / sin (μ1 + μ2)

Ty = sin μ1 x sin μ2 / sin (μ1 + μ2)

But the angles μ1 and μ2 are functions of the angle between the optical axis of the corresponding camera and the horizontal and the abscissa of the center of the image of the pointer formed on the image sensor. From the coordinates in the plane, screen coordinates are defined which are positions on an image formed near the interactive surface, for example by projection on a frosted part of a window, for example between 0 and 65535 in each direction. . To obtain these screen coordinates, the positions of the four corners of this image represented at 820 in FIG. 8 are memorized, and a bilinear transformation of the coordinates in the plane is carried out.

From the coordinates in the plane, metric coordinates (on the interactive surface) are defined by multiplying the coordinates in the plane by the metric distance between the optical centers of the lenses of the cameras.

Since the measurements are tainted with errors, an optimization phase is preferentially carried out during which, in the image projected on the screen area 820, successive points of intersection of a regular grid, represented by crosses in figure

9, and the installer successively points each of these points in the interactive surface, with the physical pointer, for example his finger, as illustrated in FIG. 9. By processing the data thus collected, it is illustrated by black dots in FIG. 9, a finer correction than during the initial calibration step.

The processing of several targets or pointers detected in the region of interest of a camera is different depending on whether one of the pointers masks another in the image of one of the cameras, or not.

It is assumed that the pointers are successively detected by the cameras, several new pointers not being detected simultaneously.

In the case where none of the pointers is masked, the possible positions of the pointers are first determined. For example, there are four possible positions when two pointers are detected by each of the two cameras.

For each old pointer, that is to say for each pointer that has already been detected, the Kalman prediction of the position of said pointer is then determined.

The distance between the Kalman prediction and each of the possible positions is calculated.

If there is only one old pointer, it is assigned the nearest possible position of the Kalman prediction and the remaining position is associated with the new pointer.

If there are two old pointers, they are assigned the two compatible positions that minimize the sum of the distances between these compatible positions and the Kalman predictions. If there is a new pointer, it is assigned the position compatible with the positions of the other pointers.

And so on.

Figure 10 shows the case where there are two old pointers whose Kalman predictions are represented at 1005 and 1010. There are four possible positions 1015 to 1030 for each of these pointers, defining a quadrilateral. We then define the four sums of the distances of the predictions with the pairs of opposite vertices of the quadrilateral.

D1 ~ dist (1005, 1015) + dist (1010, 1025)

D2 = dist (1005, 1020) + dist (1010, 1030)

D3 = dist (1005, 1025) + dist (1010, 1015) D4 = dist (1005, 1030) + dist (1010, 1020) We select the smallest of these sums of distances and assign the corresponding positions to the pointers in this smaller sum. In the case of FIG. 10, the pointer whose Kalman prediction is represented at 1005 is assigned the position 1020 and the pointer whose Kalman prediction is represented at 1010 is assigned the position 1030. model when the minimum of the distances between a Kalman prediction and the four possible positions is much smaller, for example at least four times lower, at least the distance between the other Kalman prediction and the four possible positions. In this case, the pointer whose Kalman prediction is closest to one of the possible positions is assigned this position and, at the other pointer, the opposite position in the quadrilateral. In the case where one of the pointers is obscured by another, there are half as many possible positions and we retain as the position of each pointer, the Kalman prediction, without updating the Kalman filter, as shown in Figure 11.

As a variant, in the case where one of the pointers is occultated by another, there are half as many possible positions and the position of each pointer is retained, that which minimizes the sum of the distances between the predictions of Kalman and compatible positions, with or without updating the Kalman filter.

In the case where the cameras are not synchronized, the processing of the last image from each camera causes an irregularity of trajectory around the real trajectory of the pointer, finger, fist or hand, as soon as it is moving. In fact, during triangulation, the direction of the pointer determined from the image taken earlier will be erroneous as a function of increasing the speed of movement of the pointer and the duration of the time interval separating the two shots. 'picture.

Figure 12 illustrates two treatments applied to the erroneous positions thus determined. In particular embodiments of the present invention, to overcome these disadvantages, after triangulation taking into account the last image taken by each camera to determine a new position of the pointer, called "primary position", we determine the middle position of the last two positions and is considered the actual position of the pointer. This middle position is called "secondary" and is therefore the isobarycenter of the last two determined primary positions. In particular embodiments of the present invention, the so-called "tertiary" middle position of the two last secondary medium positions thus determined is determined and is considered as the actual position of the pointer. The tertiary position is therefore the isobarycentre of the last two secondary positions and also the center of gravity of the last three primary positions, assigned respective weights 1, 2 and 1. It is observed that by modifying the mode of detection of the pointer, for example by determining the minimum per column for two halves of region of interest representing two halves of the dark frame, one can exploit a third dimension, perpendicular to the interactive surface. Two different behaviors can be associated with the two actions of touching a window or just approaching it. You can also perform a zoom function depending on the distance between the pointer and the screen area. Figures 13A to 13C summarize the steps performed during the installation of a device object of the present invention and during operation of said device. Although it is described, for purposes of explanation, a one-shot processing carried out by the computer 125, in addition to its display processing of a screen page, preferably the various treatments described below are carried out in multitasking.

During a step 1300, the installer positions the device, including the matrix image sensors and the light sources so that the interactive surface to be covered by each of the optical fields of each image sensor and the image sensor. all the light sources, as well as the frame, so that the frame is in the union of the optical fields of the cameras. During a step 1302, the installer positions the housing illustrated in FIG. 6, successively along the frame in at least three positions in the optical field of each camera, by orienting the axis of the light source towards cameras. The coordinates in the image plane of each light point indicate points of the region of interest ROI. During this same step 1302, the region of interest is defined in the images captured by each image sensor and the coordinates are stored in these images.

During a step 1304, to automatically adjust the parameters of each camera and, in particular, the illumination or gain factor, this factor is set to the lowest value and is gradually increased, while the housing illustrated in FIG. 6 is in the field of the camera, until the level of perception of the light-emitting diode of this housing corresponds to a predetermined value, for example the level of white. In a variant, starting from step 1304, a permanent automatic gain control is implemented to adapt the sensitivity of the camera to the ambient brightness.

From the at least three points per camera, during a step 1306, two half-lines are determined in the image plane of the camera and the points of these half-lines are considered as the region of interest ROI. the points lying at a distance from these half-lines less than a predetermined limit distance, for example the points of the five half-lines shifted upward or downward from the image by at most five elements of image or pixels.

During a step 1308, the sensitivity of each camera is determined or, which amounts to the same, apart from the glare phenomena extending over the surface of the image sensor (blooming and smearing), the amplification of the signal out and the digitized range of values that will be taken into account, so that the region of interest has, over its entire surface a value considered black. It is observed that this normalization of the signal can be carried out point by point and dynamically, during the operation of the device, by considering, for each image point of the region of interest, the average value over a long period, by example five minutes, as black value.

From step 1316, to detect a pointer, the minimum brightness for the points of this column is determined for each column of the region of interest ROI. Then a threshold value is determined, for example equal to twice the average of these minimums. The number of successive columns for which the minimum considered is greater than the threshold value is then determined. which provides a center point of each pointer and a width of each pointer seen by each camera.

During a step 1316, an angle registration phase is carried out, by pointing, through the interactive surface, at least two points lying on a horizontal line and treating their coordinates in the image planes. During this step 1316, an optimization phase is also carried out during which, in the projected image, successive points of intersection of a regular grid are displayed successively, and the installer successively points each of these points. points in the interactive surface, with the physical pointer, for example his finger, as shown in Figure 9. The computer 125 performs, by data processing thus collected, a finer correction than in the initial calibration step.

During a step 1320, the active areas, represented in 820 to 828 in FIG. 8, that is to say the screen areas, here 820 associated with a video projector and 821, are defined in the interactive surface. associated with a flat computer screen placed near the interactive surface, and the sensitive areas, 822 to 828, pointing, in turn, their corners. Then, step 1322, the installer assigns, with each gesture and each sensitive zone, an action to be performed. It is observed that a zone can be reserved for the detection of gestures, outside the screen areas and the areas associated with graphics corresponding to launches of applications. Similarly, a single zone, called "keyboard" placed next to a poster representing, by matrix, alphanumeric characters, identified by its four corners, can allow the input of these characters, without the user having to point each of the corners of each of the subfields carrying a character, the device automatically decomposing the "keyboard" zone into sub-zones assigned to the different characters, the disposition of the characters on the poster being known to the device, moreover.

For the screen areas, the computer 125 defines the actions that will be successively launched when icons, graphics, texts or images will be selected with the pointer then acting as a mouse or with the horizontal and vertical gestures detected in these screen areas during a step 1322.

During a step 1324, the operation of the device is started.

During a step 1326, it is determined whether a pointer is in the field of the cameras. If no, step 1326 is repeated. If yes, during a step 1328, the coordinates in the plane of the detected pointer are determined.

The two image coordinates provided by the cameras for a pointer make it possible to determine the position of the pointer in the user plane, that is to say the interactive surface. The current coordinate system used for the computation of the coordinates is a reference mark whose origin is the optical center of the camera placed on the left in figure 9. The axis of the abscissae is the horizontal axis oriented towards the camera represented on the right in figure 9 The distance between the optical centers of the lenses of the cameras defines the unit of length. The y-axis is downward, the unit length being equal to that of the x-axis.

The coordinates in the plane thus provided with an orthogonal coordinate system are determined as a function of the angles μ1 and μ2 formed between the lines starting from each of the optical centers of the lenses and passing through the center of the pointer on the one hand and the horizontal axis on the other hand as follows: Tx = sin μ1 x cos μ2 / sin (μ1 + μ2) Ty = sin μ1 x sin μ2 / sin (μ1 + μ2)

But the angles μ1 and μ2 are functions of the angle between the optical axis of the corresponding camera and the horizontal and the abscissa of the center of the image of the pointer formed on the image sensor.

During a step 1330, it is determined whether at least one new image has been supplied by one of the cameras (only one new image is received at a time if the cameras are not synchronized and two images are simultaneously received if the cameras are synchronized). Otherwise, step 1330 is repeated. If yes, during a step 1332, it is determined whether at least one new image has been provided by one of the cameras (only one new image is received at a time if the cameras are not synchronized and two images are simultaneously received if the cameras are synchronized). Otherwise, step 1332 is repeated. If yes, during a step 1334, the coordinates in the plane of the pointer are determined for each section of images successively received from one or the other of the cameras and, during a step 1336, the first pointer is assigned, as corrected position, the center of gravity of the last three positions determined from these successive pairs, assigned respective weights 1, 2 and 1.

Then, during a step 1338, it is determined whether the pointer is a finger or a hand, by measuring the actual diameters of the pointer as a function of the apparent diameters and the real coordinates of the center of the pointer and they are compared with a threshold value. for example 3 centimeters. Still during step 1338, it is determined whether a gesture (the gesture term also covering the immobility) can be identified from the last corrected positions of each pointer and the parameters defining each gesture. If so, during a step 1340, the action corresponding to said gesture and to the active zone in which this gesture is performed is started and step 1348 is taken. Otherwise, step 1342 is carried out. During a step 1342, it is determined whether at least one new image has been provided by one of the cameras (only one new image is received at a time if the cameras are not synchronized and two images are simultaneously received if the cameras are synchronized). Otherwise, step 1342 is repeated. If yes, during a step 1344, it is determined whether at least two pointers have been detected. Otherwise, during a step 1346, the coordinates in the plane of the pointer are determined for each section of images received successively from one or the other of the cameras and, during a step 1348, one assigns at the first pointer, as the corrected position, the centroid of the last three positions determined from these successive pairs, assigned respective weights 1,

If the result of step 1344 is positive, that is to say if there are at least two detected pointers, it is determined during a step 1352, if one of the pointers hides another in the optical field of at least one camera. Otherwise, during a step 1354, the possible positions of the pointers are determined. For example, there are four possible positions when two pointers are detected by each of the two cameras. Then, during a step 1356, for each old pointer, that is to say for each pointer that has already been detected, the Kalman prediction of the position of said pointer is then determined. Then, during a step 1358, for each old pointer, the distance between the Kalman prediction and each of the possible positions is calculated.

During a step 1360, it is determined whether a model break occurs, that is to say if the minimum of the distances between a Kalman prediction and the possible positions is much smaller (for example at least four times lower ), at least distances between another prediction of

Kalman and the possible positions. Otherwise, proceed to step 1364. If yes, during a step 1362, this position is assigned to the pointer whose Kalman prediction is closest to one of the possible positions and this step is repeated for other pointers by eliminating the possible positions corresponding to each position already allocated. Then proceed to step 1366. In step 1364, the old pointers are assigned the compatible positions that minimize the sum of the distances between these compatible positions and the Kalman predictions. Then we go to step 1366.

If there is a new pointer, during the step 1366, it is assigned the position compatible with the positions of the old pointers determined during the step 1362 or 1364. Then, during a step 1368, each pointer detected in at least three successive images is attributed, as corrected position, the barycentre of the last three positions determined from these successive pairs, assigned weights 1, 2 and 1. Then, we go to step 1338 .

If the result of step 1352 is positive, that is to say if one of the pointers hides another in the optical field of at least one camera, during a step 1370, it is retained as position of each pointer, that which minimizes the sum of the distances between the Kalman predictions and the compatible positions, without updating the Kalman filter.

Then, during a step 1372, each detected pointer is assigned in at least three successive images, as corrected position, the center of gravity of the last three positions determined from these successive pairs, assigned respective weights 1, 2 and 1. Then, we return to step 1338.

Below, we describe a particular embodiment of the present invention implementing several software associated with the device illustrated in Figures 1 and 2.

To be able to operate, the system must have been set up logically with the "Installation Kit". The installer is responsible for defining the location of the interactive zone as well as the settings adapted to the system environment. In other words, we define the field of action that must be considered by the system.

This field of action can never go beyond the previously installed black frame. If the frame exceeds in size the measurements indicated in the installation guide, it is these measurements that will correspond to the limit of the interactive zone.

Then, we logically set the location of the frame relative to the bar and adjust the system to take into account distances and the light environment. It also allows you to check and improve the accuracy of cursor tracking.

We then define the screen area (video projection, plasma screen,.,.) And the sensitive areas. These areas will be included in the previously defined field of action. Software "DK" (for "Development kit") and "Reader" allow, for their part, to exploit these detections, to organize the interactive zone by dividing it into zones and to parameterize the behavior of the system during the execution of a gesture by the user.

The file "config.roi" is the file that gives the system the location of the frame relative to the bar carrying the cameras to read the region of interest.

The file "val.lum" is the storage file of the system parameters in terms of sensitivity and operating mode (automatic or fixed). It is generated by the "Installation Kit" but can be modified under the "DK" software and the "Reader" software.

The file "DebugFile.smg" is the file that will be sent to the contact, that is to say to the installer, for example on the Internet, for remote assistance. It will be used to retrieve information needed for remote help and remote config.roi file generation. The "DebugFile.smg" file is reset each time an "Installation Kit" software is launched. Under the "DK" or "Reader" software, this file is reinitialized when requesting the display of the images taken by the cameras, in particular for checking and correcting the regions of interest with a score of the points corresponding to the ends and at the corners of the frame.

The file extension "3df" is the file where is stored the setting of the window. This file contains the location of the different active zones, the applications associated with these active zones and the behavior setting, in terms of association of gestures and actions in each of these zones. This is the file to open explicitly to have the setting active. All other files are implicitly loaded by the software; they are searched in the installation directory. This is charged by the operator. Thus, for the same environment in terms of bar, frame, light, one can have various behaviors.

The "Development Kit" defines the interactive environment of the installation and its applications. The "Development Kit" will therefore make it possible to define: • the locations of the specific zones on the showcase,

• the applications associated with the different zones and

• the gestures / actions associations of the different zones.

The "Development Kit" allows to prepare the software deployment in advance, before the installation on site. The file "3df" can be generated without being connected to a physical system. This makes it possible to prepare the set of applications that will be used, to test their parameterization under the software "DK" by simulating a real activation.

The "Development Kit" is there for all the preparatory work for the final deployment of the system: prior configuration, online or off-line.

The "Reader" software allows you to open a pre-recorded document by the "DK" version. This version will be launched when the user double clicks on a document of type "3df".

The "Reader" software corresponds to a daily use of the final installation. The possible settings of the installation are minimal and correspond to the adaptation of the work done under the "DK" during the final deployment. The main screen area is the main area of the system. It is quadrilateral zone. This zone corresponds to the display zone, ie a video projection zone or an area opposite which a flat screen is placed behind the interactive surface.

Within this zone are: • cursor movements and

• the detections of gestures.

The main zone is defined first because it is used to define other types of zones. We note that, preferably, if the main area is not exploited by the end users, it can nevertheless be defined, for example in a corner, in small, to have a physical existence and exploitable without interacting with end users. Sensitive areas are usually simple detection zones (that is to say without association of gestures and actions), with regard to a graphic, an object, a poster, an opening, a text, a photo, by example.

When enabled, they can:

• trigger a list of actions (launching applications, sending keyboard shortcuts) and • change the setting of the screen area in terms of combination of gestures and actions.

A sensitive zone is an area that corresponds, in terms of automation, to a simple button. This is a generally quadrilateral zone.

To activate a simple sensitive zone of detection, it is enough that the end user touches it. Typically, on a shop window, a simple sensitive detection zone is opposite a sticker on the periphery of the main screen area. Each self-adhesive corresponding to a theme, we can associate different applications and therefore a different mode of operation.

The applications that will be used by the device must have been chosen to match the message that is to be delivered.

As far as gesture and action associations are concerned, the installer can make the most of the possibilities of the device in terms of visual effects: at the detection of a horizontal gesture, open a "bonus" window, change the appearance of objects ...

The ".3df" file contains:

• the default configuration of the screen,

• the list of posters and their configuration (program, resource, option, keypad messages, and configuration of the post-detection screen area in the sensitive zone),

• the application to launch at the opening of the document and

• the level of protection of the "3DF" file under the "Reader" software: to set this up, a dialog box appears on the first backup.

Use this command to save and name the active document displays the "Save As" dialog box that allows you to name the document.

To save a document under its current name and in the current directory, use the "save" command.

The "DK" software is used to define a main screen area where users can interact with their hand or finger. This software can also create "sensitive areas". To delimit a screen area, the installer successively points the four corners of the screen according to the instructions displayed on this screen. Preferably, each corner requires a duration of immobility of the finger from 1 to 2 seconds to take the position of the stabilized finger and thus provide better accuracy. Once the screen area is delimited, the hotspots can be created.

The device makes it possible to set the action to be performed when the device is launched and the action to be performed when an active zone is touched. There is the possibility to choose an executable to launch. A resource can also be associated with it.

To set the operation in action / gesture associated with an application associated with an active zone, this operation in command of actions from the discrimination of gestures by the device, is used a menu "configure active zone".

The device has, for example, as possible actions:

1. Click action

A mouse click is generated at the current position. As part of a click or double-click gesture, the click is generated where the gesture was detected.

2. The Double Click action

A double mouse click is generated at the current position. As part of a click or double-click gesture, the click is generated where the gesture was detected.

3. The Keyboard Shortcut Action A keyboard message is sent to the application with the focus:

• 0 to 9.

• a to z

. F1 to F12 • Entry, Space, Escape, Ait, Control, Shift, Windows (registered trademark) and

• Left, right, up, down arrows.

All these keys can be combined with the keys Ait, Control, Shift, Windows.

4. The program launch action

Launching a program that can have a resource and an associated option with a wait of at least 500 milliseconds before any action is allowed. This waiting time avoids several launches of the same application if the gesture is detected very quickly several times

(point with a short duration).

The management of the paths relating to the position of the program is taken into account.

This application will not be closed if a hotspot is enabled. 5. Clic action maintained

Everything happens as if the left mouse button was pressed. The click is released when no more finger is detected in the interactive area.

6. The action back to the default configuration

This action closes the last application launched by the system that can be destroyed and restart the default application with its associated setting. This application will be closed if a hotspot is enabled. The device has a list of detectable gestures.

1. The Pointer gesture

Definition: A stationary finger points at the screen area for a certain period of time. The notion of immobility is concretely expressed by a small area around the finger. If the finger remains during the required time in the same zone, it is considered as immobile. So, in the context of a single gesture "pointer" parametrized, the detection of gesture "point", as the finger does not come out of this area, no gesture is set.

Use: The pointer must be made straight (perpendicular to the interactive plane) or you will detect a hand.

For example, if a duration of 3 seconds is set with the launch of an application, you can move your finger without anything happening. If one stabilizes his finger (immobility) for 3 seconds, the application starts. It does not restart until the finger remains stationary in the same area.

On the other hand, as soon as the finger leaves the zone and then re-stabilizes (elsewhere or in the same zone), the application can be restarted.

Several "pointer" gestures set

If two "point" gestures are set with two different duration parameters A and B; with A <B.

Let us note (A) the detection of the gesture "point" of parameter of duration A ms and (B) the detection of the gesture

"Pointer" parameter of duration B ms. If (A) is done and one moves his finger - before the duration B has elapsed - to get him out of the immobility zone, then (B) does not have places. If the finger remains stationary for the duration B, the action (A) is executed after A ms, then the action (B) after B ms.

2. The Double Pointer gesture

While a first immobile finger is pointing to the screen area, a second (remote from the previous) enters and exits the same screen area.

The constraints of using the double pointer are:

• Two detections fairly far apart, otherwise they may be detected as a hand.

• Moving the cursor between the two points. 3. The hand gesture

A hand is present in the screen area. The constraints of using the hand gesture are:

• If the hand is too close to the bar, it may pose a problem of location and

• little precision for cursor selection and tracking: this gesture must be associated with active areas of sufficient size to allow detection.

4. The gesture Vertical Movement: Vertical movement of the finger characterized by a duration; a given amplitude and meaning.

An amplitude of 25 cm, for example, is sufficient. To differentiate between cursor tracking on vertical movement and vertical gesture, a time of 700 ms is usually appropriate. The longer the duration, the more the gesture of the user can be slow to cover the distance. 5. The Horizontal Movement gesture

Vertical movement of the finger characterized by a given duration, amplitude and direction.

6. The gesture Nothing

This gesture is detected when the user no longer uses the interactivity of the window. No more gestures are made or finger detected. This gesture has a duration in ms.

Typically, it is used to reset the showcase after a user's departure.

7. Click gesture

The finger taps the showcase with a dry and sharp gesture and immediately comes out. There are no parameters associated with this gesture. 8. The Double-Click gesture

The finger taps the showcase twice in a row as it comes in and out with a dry, crisp gesture. There are no parameters associated with this gesture.

The double click gesture corresponds to two "click" gestures very quickly and in the same place. A double-click made by the user thus generates the detection of a click and a double click. 9. Other gestures: The circular movements of the hand, of a diameter greater than a predetermined diameter, in one direction or the other, may provide two gestures interpreted by the device and associated, alone or in combination with other gestures, actions, for example speed and direction of an audiovisual program, a slide show or a page. Similarly, the crosses "X" or "+" and the zigzags "N" and "Z" may provide gestures interpreted by the device and associated, alone or in combination with other gestures, to actions, for example "Move to next offer" and "put the offer in a selection", pause and resume.

Simultaneous setting of gestures: the system can manage simultaneously:

• Three gesture detections point at once if three different durations.

• one double-point detection • one hand detection

• two horizontal movements: depending on the direction. The other two numeric parameters (duration and amplitude) can only be set for the first direction. For the second, they must be identical to the first.

• two vertical movements: depending on the direction. The other two numeric parameters (duration and amplitude) can only be set for the first direction. For the second, they must be identical to the first.

• A click

• Double-click

• A gesture nothing Hand and point: the combination of the pointer and the hand is possible on the same application but requires a precise pointer. Indeed, if the device does not see the finger sufficiently fine, he will see the hand. This is why we must be vigilant in the combination of these two actions in the same application.

Click and point: if the two gestures "click" and "point" with a very short duration are set at the same time, it should be noted that a click generates two detections of gestures: the point and the click. So a single user gesture is the detection of two system gestures. If they have the effect of making a mouse click both, two mouse clicks will be generated very quickly which will generate a double-click mouse. This behavior can select texts for example, or prevent the operation of a "drop-down list" by opening it (first click), then closing it immediately (second click).

Click, double click and hand: detection of the click gesture is difficult if the hand is also detected because the position of the finger must then be straight.

If you tap on the glass with a double-click but with the hand, the double-click gesture will be detected. Hand and double-pointer: the hand and the double-pointer are the gestures most likely to be undesirably detected. If used together, there is a risk of common detections.

Hand and horizontal / vertical gestures: If the hand gesture is done dynamically (the user puts his hand and moves it in the interactive zone), we can detect the horizontal or vertical gestures. If the gestures are set together, the following detections can occur: hand, horizontal, hand.

Point and double point: if the user makes a point and enters another finger and then removes it, we can have the detection series point-second target-point. If there are two "pointer" gestures set with the double-pointer gesture, it is necessary to wait for the time necessary for the pointeri (point it with the shortest duration) before the re-detection, on the other hand if the pointer2 (with the longest duration) is configured there will not be pointeri -second target-pointer2, the gestures being reset.

For the setting of sensitive areas, a hotspot can be viewed as a single zone (a button) that can be activated. It is a rectangular area. It is associated with a list of actions to perform when it is activated (launching applications, keyboard events ...) and a modification of the screen space settings.

Activating this zone corresponds to touching the physical part that defines the zone. Typically, on a showcase, this would be a sticker on the periphery of the screen area. Each self-adhesive corresponding to a theme, we can associate different applications and therefore a different mode of operation. The system has a list of hotspots. We can thus, for each zone of the list:

• Delete an existing hotspot: "Delete" button

• Define the actions to be carried out just after activating the sensitive zone: "Set" button

• Define the setting of the screen area after activating the sensitive zone: "Advanced" button

• Change the location of the selected hotspot: "Repointed" button • Create a new hotspot that is empty and inactive but has the other parameters (actions post detection and gesture / action settings) of the copied area "Create a copy "This setting can be done:

• By selecting the item of the window that lists the sensitive areas and activating the corresponding buttons • Right-click on the showcase representation in the existing hotspot and activate desired features

Note 1: Inactive hotspots are grayed out in the hotspot list and are placed to the right of the in-line showcase representation. Note 2: In the absence of a config.cfg file, all the sensitive zones (active or not) are placed to the right of the window representation window because it can not be done without a "config.cfg" file.

In the software DK ₁ the representation of the showcase is an approximate representation of the location of the different areas that form a showcase. This makes it possible to visualize the location of the zones relative to each other, but the positions are not real.

First example of use of horizontal gestures. A PowerPoint slideshow (trademark) will scroll when you make a horizontal gesture to the right and return to the previous page when you make a horizontal gesture to the left.

To set such a use, the installer or the operator goes into the "configure / sensitive areas" menu of the "DK" software. It adds a new empty hotspot. For its definition, it puts in the program the software "PowerPoint Viewer" (registered trademarks) and as a resource file extension ".ppt" (trademark). It does not put an option because PowerPoint Viewer has the characteristic of launching files in viewer mode.

As parameterization in terms of gesture / action associations, it empties everything that exists and then sets the horizontal gesture associated with a keyboard shortcut.

For the direction "left to right", it puts a gesture of amplitude 15 cm, tolerance 10 cm and duration 2 s. He puts the keyboard shortcut "Page Down" and clicks on "Add".

For the "right to left" direction, the parameter values are, by default, identical to those in the other direction. The installer puts the keyboard shortcut "Page Up" and clicks on "Add". Second example of use. The installer or the operator has an HTML application in which he wants to switch from link to link. In addition, this application contains a specific area where a double click of the mouse on a position - within this area - causes the displacement of a graphic object under the double click position.

This application is associated with an object. This object will be accessible by the end user. He can grab it.

On the representation of the window, it is the circle that pierces the interactive surface to grab an object that is on a shelf behind the surface. On the representation of the window, a rectangle represents the screen area, management area of the cursor tracking. Small areas represent sensitive areas for which stickers have been placed. The round is a hollow through which an object can be grasped. When the user enters the object, he goes through the defined area and thus activates the application.

The sensitive area to be defined is a rectangular area that encompasses the circle. By default, entering and resting the object would cause the application that is enabled for detection to be restarted when the user rests the object. To avoid this, we change the activation mode of the sensitive zone. We will put mode 2 with infinite duration to prevent the application from being reset. In addition, for gesture / action settings, click and double-click gestures - finger gesture - click and double click actions - simulation of mouse button use - must be selected.

This example of use makes it possible, among other things, to make a withdrawal point for objects bought or rented on the Internet, on a site whose pages are programmed in HTML, the object being accessible only after payment or identification of the user.

Other applications of the present invention include the following. To make an interactive window, once the device and the corresponding frame set up along the window, a computer screen image is formed behind the window (for example on a computer screen), or on the showcase (for example by video-projection on a frosted surface of the window). The device is then calibrated and the actions associated with the different actions that can be performed in relation to the different active zones defined on the showcase, possibly with the help of the displayed computer screen image.

In the case where the interactive surface covers openings, for example lockers, the object taken through these openings may cause actions such as the addition of an amount corresponding to the contents of the locker to a total amount to be adjusted.

In a particular application, the positions, shapes and trajectories of the pointers, fingers or hands, control musical parameters, such as the sound volume, the balance between several loudspeakers, the speed of play of a piece, etc. In variants , the cameras mentioned in the description are provided with a polarizer filter to reduce the influence of reflections on the support or on the frame. Indeed, these reflections have a particular polarization. In these variants, the polarizing filters preferentially have a direction of polarization perpendicular to that of these reflections.

Claims

1 - Device for rendering an interactive surface (170) or volume, characterized in that it comprises: a rod (100) of elongate shape which comprises two matrix image sensors (120, 160) each providing an image in two dimensions, having a plurality of rows and a plurality of columns, placed at the ends of the bar and whose optical fields are superimposed and encompassing said surface, a storage means (125), for each of said matrix image sensors, at least one region of interest consisting of a portion comprising, at least, a plurality of line segments and a plurality of column segments of the image provided by said array image sensor, each said region of interest representing the image, in section, of said surface and at least one possible pointer intersecting said surface and a position determining means (125) of at least one said pointer intersecting said surface, depending on the part of s images of said pointer in said regions of interest. 2 - Device according to claim 1, characterized in that it further comprises ₍ display control means of at least one reference point (820) opposite said surface and a storage means (125) of coordinates of a pointer placed opposite each reference point, said coordinates coming from the processing of the pointer images provided by the image sensors, to determine a correspondence between the position of the pointer and the coordinates, in the plane of the image. matrix display, the point of the display that is next to said pointer.

3 - Device according to any one of claims 1 or 2, characterized in that it further comprises: means for storing (125) coordinates of a pointer placed successively at the end points of at least one zone active (820 to 828) in said surface, said coordinates from the processing of the pointer images provided by the image sensors, means for associating (125), with each active area, at least one action to be performed and an action triggering means (125), adapted, when the position determining means provides a position of a pointer corresponding to one of the active areas, triggering an action associated with said active area.

4 - Device according to any one of claims 1 to 3, characterized in that it further comprises: a matrix display positioned parallel to and facing a portion of said surface; a coordinate storage means of a pointer placed successively opposite the end points of said matrix display (820), said coordinates defining an active zone, said active zone being associated with at least one action to be performed.

5 - Device according to any one of claims 1 to 4, characterized in that it further comprises: means for displaying, on said matrix display, at least one reference point (820); coordinate storage means (125) of a pointer placed opposite each said reference point, said coordinates from the processing of the pointer images provided by the image sensors and means for processing (125) said coordinates for determine a correction to apply to the images.

6 - Device according to any one of claims 1 to 5, characterized in that it further comprises: means for determining (125) successive positions of a pointer in the plane of the interactive rendered surface, by treatment of the last two images received from the image sensors, each position comprising two coordinates and a means (125) for correcting the position of the pointer adapted to assign, as corrected position, a position which is a function of at least two successive positions.

7 - Device according to claim 6, characterized in that said correction means (125) is adapted so that the corrected position is the weighted centroid of the last, the penultimate and the antépénultième positions, the last and the antepenultimate position each having a weight equal to half the weight of the penultimate position.

8 - Device according to any one of claims 1 to 7, characterized in that it further comprises: a movable light source (600) positioned successively along a frame placed in the plane of the surface to be rendered interactive, outside said surface, at least two positions in the optical field of each array image sensor and directing said light source to said array image sensor; means for storing (125) the coordinates in the plane of the image provided by the array image sensor, the point corresponding to said moving light source, for each position of said light source, and determining means ( 125) of the region of interest, in the image plane provided by said array image sensor, said region of interest comprising each line segment drawn between two points corresponding to the successive positions of said moving light source in the images provided by each image sensor. 9 - Device according to any one of claims 1 to 8, characterized in that it comprises a detection means (125) adapted, for detecting a pointer, to determine, for each column of image elements of the region of interest of each sensor, the minimum brightness for the points of this column and to compare this minimum to a brightness threshold value and the number of consecutive minimums exceeding this threshold value to at least a threshold value of apparent width. 10 - Device according to any one of claims 1 to 9, characterized in that it comprises: means for allocating (125) a computer application, for at least one movement of the pointer and for at least one active area or the entire interactive surface, and a control means (125) adapted, when said movement is detected in said active area or in the interactive surface, respectively, to initiate a step of operation of said computer application. 11 - Device according to claim 10, characterized in that the allocation means (125) is adapted so that at least one said movement is defined by its amplitude, duration or speed, its spatial tolerance its direction and its meaning.

12 - Device according to any one of claims 1 to 11, characterized in that it comprises a Kalman prediction determination means (125) and a suitable position assignment means (125), when at least two pointers are detected, to assign, to at least one of said pointers, one of the possible positions whose distance to a Kalman prediction is minimal.

13 - Device according to any one of claims 1 to 12, characterized in that it comprises a Kalman prediction determination means (125) and a suitable position assignment means (125), when at least two pointers are detected, to assign pointers audits of possible positions compatible with the images of pointers that minimize the sum of their distance to Kalman predictions.

14 - Device according to any one of claims 1 to 13, characterized in that it comprises a frame (1400, 1500, 1600) placed in the plane of the surface (170) to make interactive, outside said surface, said frame having a concave profile.

15 - Device according to any one of claims 1 to 14, characterized in that it further comprises: means for determining (125) at least one active area in said surface and means of association (125 ), at each active zone, at least one action to be performed at least one gesture defined by at least one relationship between successive positions, means for determining (125) successive positions of at least one pointer according to the images captured by the image sensors and means for triggering an action associated with said active zone as a function of the successive positions of said pointer, when, in an operating phase, the position of a pointer corresponds to an active zone .

16 - Method for rendering an interactive surface or volume, characterized in that it comprises: a step of installing an elongated shape bar which comprises two matrix image sensors each providing a two-dimensional image, comprising a plurality of rows and a plurality of columns, placed at the ends of the bar and whose optical fields are superimposed and encompassing said surface, a storage step, for each of said matrix image sensors, of at least one region of an interest consisting of a part comprising, at least, a plurality of line segments and a plurality of column segments of the image provided by said array image sensor, each said region of interest representing the image, seen in section, of said surface and at least one possible pointer intersecting said surface and a step of determining position of at least one said pointer intersecting said surface, as a function of the part of i said pointer being in said regions of interest.

17 - Method according to claim 16, characterized in that it further comprises a step of controlling the display of at least one reference point opposite said surface and a step of memorizing coordinates of a pointer placed next to each reference point, the said coordinates from the processing of the pointer images provided by the image sensors, to determine a correspondence between the position of the pointer and the coordinates, in the plane of the matrix display, of the point of the display which is opposite said pointer.

18 - Process according to any one of claims 16 or 17, characterized in that it further comprises: a step of memorizing coordinates of a pointer successively placed at extreme points of at least one active zone in said surface, said coordinates from the processing of the pointer images provided by the image sensors, a step of associating, with each active zone, at least one action to be performed and, a step of triggering action, when the position determining means provides a position of a pointer corresponding to one of the active areas, an action associated with said active area.

19 - Method according to any one of claims 16 to 18, characterized in that it comprises, on the one hand, an initialization step which comprises: - a positioning step (1300) of at least two sensors d in the plane of said surface, the optical field of each image sensor covering the whole of said surface, a step of determining (1320) at least one active area in said surface, an association step ( 1322), to each active zone, at least one action to be performed at least one gesture defined by at least one relationship between successive positions and, secondly, an operating step which comprises: a determination step successive positions of at least one pointer (1326) as a function of the images captured by the image sensors and when the position of a pointer corresponds to an active zone, a step of triggering an action associated with said active zone (1340) according to successive positions of said po inteur.

20 - Process according to claim 19, characterized in that the step of determining at least one active zone comprises a step of memorizing (1320) coordinates of a pointer successively placed at extreme points of said active zone in said surface, said coordinates from the processing of the pointer images provided by the image sensors. 21 - Process according to any one of claims 16 to 20, characterized in that it further comprises: a step of positioning at least one matrix display parallel to and facing a portion of said surface; a step of storing (1320) coordinates of a pointer placed successively opposite the end points of said matrix display, said coordinates defining an active area, said active area being associated with at least one action to be performed.

22 - Process according to any one of claims 16 to 21, characterized in that it further comprises: a display step (1316) on said matrix display of at least one reference point; a step of storing (1316) coordinates of a pointer placed opposite each said reference point, said coordinates coming from the processing of the pointer images provided by the image sensors and a processing step (1316) of said coordinates for determine a correction to apply to the images.

23 - Process according to any one of claims 16 to 22, characterized in that it further comprises: a step of determining (1328, 1334) successive positions of a pointer in the plane of the interactive rendered surface, by processing the last two images received from the image sensors, each position comprising two coordinates and a position correction step (1336) of the pointer, during which, as corrected position, a position according to at least two successive positions.

24 - Method according to claim 23, characterized in that, during the position correction step (1336), the corrected position is the weighted barycentre of the last, the penultimate and the antépénultième positions , the last and the third penultimate position each having a weight equal to half the weight of the penultimate position.

25 - Process according to any one of claims 16 to 24, characterized in that it further comprises: a step of positioning (1302) sequentially along a frame placed in the plane of the surface to be interactive, outside said surface, at least two positions in the optical field of each image sensor, of a moving light source facing said image sensor; for each position of said light source, a step of storing (1302) the coordinates, in the plane of the image supplied by the image sensor, of the point corresponding to said moving light source and a step of determining the region of interest (1302), in the image plane provided by said sensor, said region of interest comprising each line segment drawn between two points corresponding to successive positions of said moving light source.

26 - Process according to any one of claims 16 to 24, characterized in that it further comprises: a step of pointing, in an image taken by each matrix sensor, at least two positions corresponding to points d a region of interest; for each position, a memorization of the coordinates, in the plane of the image supplied by the image sensor, of the corresponding point and a step of determining the region of interest, in the plane of the image provided by said image sensor, said region of interest comprising each line segment drawn between two successive points whose coordinates have been stored.

27 - Method according to any one of claims 16 to 26, characterized in that, for detecting a pointer, is determined for each column of pixels of the region of interest of each sensor, the minimum brightness for the points in this column and we compare this minimum to a brightness threshold value and the number of consecutive minimums exceeding this threshold value to at least one apparent width threshold value.

28 - Method according to any one of claims 16 to 27, characterized in that it comprises: a step of assigning (1322) a computer application, for at least one movement of the pointer and for at least one active area or the entire interactive surface, and when said movement is detected (1338) in said active area or in the interactive surface, respectively, a step of operation of said computer application.

29 - Method according to claim 28, characterized in that, during the step of assigning a computer application (1322), at least one said movement is defined by its amplitude, duration or speed, its tolerance spatial direction and meaning.

30 - Method according to any one of claims 16 to 29, characterized in that, when at least two pointers are detected, a step is performed for assigning to at least one of said pointers one of the possible positions whose distance to a prediction Kalman is minimal (1356 to 1366).

31 - Method according to any one of claims 16 to 30, characterized in that, when at least two pointers are detected, a step of assigning pointers auditors possible positions compatible with the images pointers that minimize the sum of their distance from Kalman's predictions (1356 to 1366).