|Publication number||US6496598 B1|
|Application number||US 09/516,326|
|Publication date||Dec 17, 2002|
|Filing date||Mar 1, 2000|
|Priority date||Sep 2, 1997|
|Also published as||CA2302199A1, CN1278349A, EP1019873A1, US20020191841, WO1999012127A1|
|Publication number||09516326, 516326, US 6496598 B1, US 6496598B1, US-B1-6496598, US6496598 B1, US6496598B1|
|Inventors||Philip Victor Harman|
|Original Assignee||Dynamic Digital Depth Research Pty. Ltd.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (14), Referenced by (546), Classifications (47), Legal Events (5)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This application is a continuation of International Application Serial No. PCT/AU98/00716 filed Sep. 2, 1998, the teachings of which are incorporated herein by reference, which in turn claims priority from Australian Serial No. PO 8944 filed Sep. 2, 1997.
The present invention relates generally to stereoscopic image systems, and in particular to the synthesis of stereoscopic image pairs from monoscopic images for stereoscopic display. The present invention may also be directed towards a five module method for producing stereoscopic images, that digitises a monoscopic source, analyses it for motion, generates the stereoscopic image pairs, optimises the stereoscopic effect, transmits or stores them and then enables them to be displayed on a stereoscopic display device.
The advent of stereoscopic or three dimensional (3D) display systems which create a more realistic image for the viewer than conventional monoscopic or two dimensional (2D) display systems, requires that stereoscopic images be available to be seen on the 3D display systems. In this regard there exists many monoscopic image sources, for example existing 2D films or videos, which could be manipulated to product stereoscopic images for viewing on a stereoscopic display device.
Preexisting methods to convert such monoscopic images for stereoscopic viewing do not product acceptable results. Other attempts in film and video have used techniques to duplicate the stereoscopic depth cue of “Motion Parallax”. These involved producing a delay for the images presented to the trailing eye when laterals, left or right, motion is present in the images. Other attempts have used ‘Lateral Shifting’ of the images to the left and right eyes to provide depth perception.
However, these two techniques are limited and generally only suit specific applications. For example, the Motion Parallax technique is only good for scenes with left or right motion and is of limited value for the stereoscopic enhancement of still scenes. The Lateral Shifting technique will only give an overall depth effect to a scene and not allow different objects at varying depths to be perceived at the depths where they occur. Even the combination of these two techniques will only give a limited stereoscope effect for most 2D films or videos.
Some existing approaches demonstrate limitations of these techniques. When an image has vertical motion and some lateral motion and a delay is provided to the image presented to the trailing eye then the result is often a large vertical disparity between the left and right views such that the images are uncomfortable to view. Scenes with contra motion, such as objects moving left and right in the same scene are also uncomfortable to view. Certain embodiments of these methods define that when objects of varying depths are present in an image there is a distinct ‘card board cut-out’ appearance of the objects with distinct depth modules rather than a smooth transition of objects from foreground to background.
In all these approaches no successful attempt has been made to develop a system or method to suit all image sequences or to resolve the problem of viewer discomfort or to optimise the stereoscopic effect for each viewer or display device.
There is therefore a need for a system with improved methods of converting monoscopic images into stereoscopic image pairs and a system for providing inproved stereoscopic images to a viewer.
An object of the present invention is to provide such a system with improved methods.
In order to address the problems noted above the present invention provides in one aspect a method for converting monoscopic images for viewing in three dimensions including the steps of:
receiving said monoscopic images;
analysing said monoscopic images to determine characteristics of the images;
processing said monoscopic images based on the determined image characteristics;
outputting the processed images to suitable storage and/or stereoscopic display systems.
wherein analysing of said monoscopic images to determine the motion includes the steps of:
dividing each image into a plurality of blocks, wherein corresponding blocks on an adjacent image are offset horizontally and/or vertically; and
comparing each said block with said corresponding blocks to find the minimum mean square error and thereby the motion of the block.
An image conversion system for converting monoscopic images for viewing in three dimensions including:
an input means adapted to receive monoscopic images;
a preliminary analysis means to determine if there is any continuity between a first image and a second image of the monoscopic image sequence;
a secondary analysis means for receiving monoscopic images which have a continuity, and analysing the images to determine at least one of the speed and direction of motion, or the depth, size and position of objects, wherein analysing of said monoscopic images to determine the motion includes the steps of: dividing each image into a plurality of blocks, wherein corresponding blocks on an adjacent image are offset horizontally and/or vertically, and comparing each said block with said corresponding blocks to find the minimum mean square error and thereby the motion of the block;
a first processing means for processing the monoscopic images based on data received from the preliminary analysis means and/or the secondary analysis means.
Ideally, the input means also includes a means to capture and digitise the monoscopic images.
Preferably the image analysis means is capable of determining the speed and direction of motion, the depth, size and position of objects and background within an image.
In a further aspect the present invention provides a method of optimising the stereoscopic image to further improve the stereoscopic effect and this process is generally applied prior to transmission, storage and display.
In yet a further aspect the present invention provides a method of improving stereoscopic image pairs by adding a viewer reference point to the image.
In still yet a further aspect the present invention provides a method of analysing monoscopic images for conversion to stereoscopic image pairs including the steps of: scaling each image into a plurality of regions; comparing each region of a first image with corresponding and adjacent regions of a second image to determine the nature of movement between said first image and said second image.
Preferably a motion vector is defined for each image based on a comparison of the nature of motion detected with predefined motion categories ranging from no motion to a complete scene change.
In yet a further aspect the present invention provides a system for converting monoscopic images for viewing in three dimensions including:
a first module adapted to receive a monoscopic image;
a second module adapted to receive the monoscopic image and analyse the monoscopic image to create image date, wherein analysing of said monoscopic image to determine the motion includes the steps of: dividing each image into a plurality of blocks, wherein corresponding blocks on an adjacent image are offset horizontally and/or vertically, and comparing each said block with said corresponding blocks to find the minimum mean square error and thereby the motion of the block;
a third module adapted to create stereoscopic image pairs from the monoscopic image using at least one predetermined technique selected as a function of the image data;
a fourth module adapted to transfer the stereoscopic image pairs to a stereoscopic display means;
a fifth module consisting of a stereoscopic display means.
Preferably the first module is further adapted to convert any analogue images into a digital image. Also, the second module is preferably adapted to detect any objects in a scene and make a determination as to the speed and direction of any such motion. Conveniently, the image may be compressed prior to any such analysis.
Preferably the third module further includes an optimisation stage to further enhance the stereoscopic image pairs prior to transmitting the stereoscopic image pairs to the stereoscopic display means. Conveniently, the fourth module may also include a storage means for storing the stereoscopic image pairs for display on the stereoscopic display means at a later time.
It will be appreciated that the process of the present invention can be suspended at any stage and stored for continuation at a later time or transmitted for continuation at another location if required.
The present invention provides a conversion technology with a number of unique advantages including:
The ability to convert monoscopic images to stereoscopic image pairs can be performed in realtime or non-realtime. Operator intervention may be applied to manually manipulate the images. An example of this is in the conversion of films or videos where every sequence may be tested and optimised for its stereoscopic effect by an operator.
The present invention utilises a plurality of techniques to further enhance the basic techniques of motion parallax and lateral shifting (forced parallax) to generate stereoscopic image pairs. These techniques include but are not limited to the use of object analysis, tagging, tracking and morphing, parallax zones, reference points, movement synthesis and parallax modulation techniques.
Reverse 3D is ideally detected as part of the 3D Generation process by analysing the motion characteristics of an image. Correction techniques may then employed to minimise Reverse 3D so as to minimise viewer discomfort.
The present invention discloses a technique applicable to a broad range of applications and describes a complete process for applying the stereoscopic conversion process to monoscopic applications. The present invention
Humans see by a complex combination of physiological and psychological processes involving the eyes and the brain. Visual perception involves the use of short and long term memory to be able to interpret visual information with known and experienced reality as defined by our senses. For instance, according to the Cartesian laws on space and perspective the further an object moves away from the viewer the smaller it gets. In other words, the brain expects that if an object is large it is close to the viewer and if it is small it is some distance off. This is a learned process based on knowing the size of the object in the first place. Other monoscopic or minor depth cues that can be represented in visual information are for example shadows, defocussing, texture, light, atmosphere.
These depth cues are used to great advantage in the production of ‘Perspective 3D’ video games and computer graphics. However, the problem with these techniques in achieving a stereoscopic effect is that the perceived depth cannot be quantified: it is an illusion of displaying 2D objects in a 2D environment. Such displays do not look real as they do not show a stereoscopic image because the views to both eyes are identical.
Stereoscopic images are an attempt to recreate real world visuals, and require much more visual information than ‘Perspective 3D’ images so that depth can be quantified. The stereoscopic or major depth cues provide this additional data so that a person's visual perception can be stimulated in three dimensions. These major depth cues are described as follows:
Retinal Disparity—refers to the fact that both eyes see a slightly different view. This can easily be demonstrated by holding an object in front of a person's face and focussing on the background. Once the eyes have focused on the background it will appear as though there are actually two objects in front of the face. Disparity is the horizontal distance between the corresponding lefts and right image points of superimposed retinal images. While Parallax is the actual spatial displacement between the viewed images.
2) Motion Parallax—Those objects that are closer to the viewer will describes on the one hand techniques for 3D Generation where both the image processing equipment and stereoscopic display equipment are located substantially at the same location. While on the other hand techniques are defined for generation of the stereoscopic image pairs at one location and their transmission, storage and subsequent display at a remote location.
The present invention accommodates any stereoscopic display device and ideally has built in adjustment facilities. The 3D Generation process can also take into account the type of display device in order to optimise the stereoscopic effect.
The invention will be more fully understood from the following description of a preferred embodiment of the conversion method and integrated system and as illustrated in the accompanying figures. It is, however, to be appreciated that the present invention is not limited to the described embodiment.
FIG. 1 shows the breakdown into modules of a complete system utilising the present invention.
FIG. 2 shows a possible use of multiple processors with a complete system utilising the present invention.
FIG. 3 shows a flow diagram of Module 1 (Video Digitising) and the first part of Module 2 (Image Analysis).
FIG. 4 shows the second part of a flow diagram of Module 2.
FIG. 5 shows the third part of a flow diagram of Module 2.
FIG. 6 shows the fourth part of a flow diagram of Module 2.
FIG. 7 shows a flow diagram of the first part of Module 3 (3D Generation).
FIG. 8 shows the second part of a flow diagram of Module 3 and Module 4 (3D Media—Transmission & Storage) and Module 5 (3D Display).
The present invention aims to provide a viewer with a stereoscopic image that uses the full visual perception capabilities of an individual. Therefore it is necessary to provide the depth cues the brain requires to interpret such images. appear to move faster even if they are travelling at the same speed as more distant objects. Therefore relative motion is a minor depth cue. But the major stereoscopic depth cue of lateral motion is the creation of motion parallax. With motion in an image moving from right to left, the right eye is the leading eye while the left eye becomes the trailing eye with its image being delayed. This delay is a normal function of our visual perception mechanism. For left to right motion the right eye becomes the trailing eye. The effect of this delay is to create retinal disparity (two different views to the eyes), which is perceived as binocular parallax thus providing the stereoscopic cue known as Motion Parallax.
3) Accommodation—The eye brings an object into sharp focus by either compressing the eye lens (more convex shape for close object) or expanding the eye lens (less convex shape for far object) through neuromotor activity. The amount and type of neuromotor activity is a stereoscopic cue for depth in an image.
4) Convergence—Is the response of the eye's neuromotor system that brings images of an object into alignment with the central visual area of the eyes such that only one object is seen. For example, when a finger held at arms length is viewed by both eyes and slowly brought towards the face, the eyes turn inwards (converge) indicating that the finger is getting closer. That is, the depth to the finger is decreasing.
The eyes convergence response is physiologically linked to the accommodation mechanism in normal vision. In stereoscopic viewing, when viewers are not accommodated to the ‘Fixation Plane’ (that to which the eyes are converged), they may experience discomfort. The ‘Plane of Fixation’ is normally the screen plane.
The present invention describes a system that is capable of taking any monoscopic input and converting it to an improved stereoscopic output. For ease of description this complete system can be broken down into a number of independent modules or processes, namely:
MODULE 1—Monoscopic Image Input (typically video input)
MODULE 2—Image Analysis
MODULE 3—3D Generation
MODULE 4—3D Media (Transmission or Storage)
MODULE 5—3D Display
FIG. 1 shows this top down approach to the stereoscopic conversion process, where video or some other monoscopic image source is input, images are analysed, stereoscopic image pairs are generated, transmitted and/or stored and then displayed on a stereoscopic display. Each Module describes an independent process of the complete system from monoscopic image input to stereoscopic display. However, it will be appreciated that the various modules may be operated independently.
Generally, all five modules are used, from monoscopic image input to display for a particular application. For example, this system may be used in theatres or cinemas. In such an application the 2D video input can take the form of analogue or digital to the video sources. These sources would then be analysed to determine speed and direction of any motion. The processes would then work in either real-time or non real-time in order to create the 3D images. This can be further optimised through the use of borders, parallax modification, reverse 3D analysis, shading, and/or texturing. The 3D images may then be stored or transmitted to a 3D display, including shutterglasses, polarising glasses or an autostereoscopic display.
This system may also be adapted for use with cable or pay-TV systems. In this application the 2D video input could be video from a VTR, a laser disc, or some other digital source. Again the 3D Generation and/or optimisation can proceed in either real time or non real time. The 3D media module would conveniently take the form of transmission via cable or satellite to enable 3D display on TV, video projector, or an auto stereoscopic display.
The system may also be used with video arcade games, in multimedia, or with terrestrial or network TV. Depending on the application the 2D video input module may obtain source monoscopic images from a games processor, video from a laser disc, video from VTR, video from a network, or some other digital storage device or digital source or telecine process. The 3D Generation can take place in real time or non real time, and be generated by computer at a central conversion site, in a user's computer, on a central processor, or some other image processor. The stereoscopic images can then be stored on video or other digital storage device, prior to distribution to cinemas or transmission by a local network. These stereoscopic images may also be transmitted to video projectors via a local transmission, or alternatively via VHF/UHF facilities or satellite.
The 3D display is dependent on the application required, and can take the form of an auto stereoscopic display device, a video projector with polarising glasses, a local monitor with shutter-glasses, or a set-top box with suitable viewing glasses.
Single & Multiple Processors
The complete system can be operated on a single processor with all five modules being processed together or individually in realtime or non-realtime (Modules 2, 3 and 4). Modules 2 and 3 can be further segmented to suit a multitasking or multiprocessor environment, as can be seen in FIG. 2 for example.
The use of multiple processors can also be configured to the application on hand. For example, modules 1 and 2 could be handled by a first processor, and modules 3 to 5 by a second processor. If desired, the first processor of this arrangement could be used as a look-ahead processor, and the second processor could generate the stereoscopic images after a delay. Alternatively, a first processor could be used to receive realtime video, digitise the video and transfer the digitised video to a suitable digital storage device. A second processor, either on site or remotely, could then analyse the digitised image and perform the necessary tasks to display a stereoscopic image on a suitable display device.
Look-ahead processing techniques may be employed to predict trends in sequences of film or video so that the image processing modes may be more efficiently selected to optimise the overall stereoscopic effect.
The present invention is primarily concerned with the analysis of monoscopic images and conversion of the monoscopic images into stereoscopic image pairs together with the optimisation of the stereoscopic effect. In this regard the present invention is applicable to a broad range of monoscopic inputs, transmission means and viewing means. However, for completeness all five defined modules will be described herein:
Module 1 requires that a monoscopic image source or video input is provided. This source may be provided as either a digital image source or an analogue image source which may then be digitised. These image sources may include:
1) Analogue Source
a) Tape based—VCR/VTR or Film.
b) Disk based—Laser Disk.
c) Video Camera or other realtime image capture device.
d) Computer generated images or graphics.
2) Digital Source
a) Tape based—Typical examples are DAT, AMPEX's DCT, SONY's Digital Betacam, Panasonic's digital video formats or the new Digital Video Cassette (DVC) format using 6.5 mm tape.
b) Disk based storage—Magneto Optical (MO) hard disk (HD), compact disk (CD), Laser Disk, CD-ROM, DAT, Digital Video Cassette (DVC) or Digital Video Disk (DVD) based data storage devices—uses JPEG, MPEG or other digital formats.
c) Video Camera or other realtime image capture device.
d) Computer generated images or graphics.
What is important for the conversion process of the present invention is that a monoscopic image source be provided. It is noted that a stereoscopic image source may be provided which would generally obviate the need for modules 1 to 3, however, any such stereoscopic image may be passed through an optimisation stage prior to display.
Referring now to FIGS. 3 to 8 which show flow diagrams demonstrating a preferred arrangement of the present invention.
Following reception of 2D images, digitised video or digital image data is processed on a field by field or image by image basis in realtime or non-realtime by hardware, software or by a combination of both. Firstly, the image analysis process occurs including the steps of:
1) Image compression.
2) Motion detection.
3) Object detection.
4) Motion analysis.
1) Image Compression
Compression of the image is not essential, however, for many processes and applications, compression is a practical option particularly, where the processor is not powerful enough to process a full resolution image in the time required.
Preferably the images are scaled to smaller dimensions. The scaling factor is dependent on the digital video resolution used for each image, and is usually defined by the type of image capture facility used in the digitising process.
2) Motion Detection
In a preferred embodiment each image may be analysed in blocks of pixels. A motion vector is calculated for each block by comparing blocks from one image with corresponding blocks from an adjacent image that are offset horizontally and/or vertically by up to a predetermined number of pixels, for example ±9, and recording the position that gives the minimum Mean Squared Error.
For each block, the vector and minimum and maximum Mean Square Error are recorded for later processing.
To save on processing time, vectors need not be calculated if there is no detail in the block, for example, when the block is a homogeneous colour.
Other methods for calculating the motion can be utilised, for example image subtraction. The present embodiment uses the Mean Squared Error method.
3) Object Detection
An Object is defined as a group of pixels or image elements that identify a part of an image that has common features. Those characteristics may relate to regions of similar luminance value (similar brightness), chrominance value (similar colour), motion vector (similar speed and direction of motion) or similar picture detail (similar pattern or edge).
For example a car driving past a house. The car is a region of pixels or pixel blocks that is moving at a different rate to the background. If the car stopped in front of the house then the car would be difficult to detect, and other methods may be used.
A connectivity algorithm may be used to combine the motion vectors into regions of similar motion vectors. An Object may be comprised of one or more of such regions. Other image processing algorithms, such as edge detection etc, may be used in the detection of Objects.
Once Objects are identified in an image they are preferably tagged or given an identification number. These Objects and their relevant details (for example position, size, motion vector, type, depth ) are then stored in a database so that further processing may occur. If an Object is followed over a sequence of images then this is known as Object Tracking. By tracking Objects and analysing their characteristics they can be identified as being foreground or background Objects and therefore enhanced to emphasise their depth position in an image.
4) Motion Analysis
Once Objects have been detected, the Objects can be analysed to determine the overall speed and direction of motion in the image. In the preferred embodiment, this stage determines the type of motion in the image, and also provides an overall vector.
By using the Object Detection information and comparing the data to several image motion models a primary determination can be made as to the best method to convert monoscopic images to stereoscopic image pairs.
The image motion models as used in the preferred embodiment of the present invention are:
a) Scene Change.
b) Simple Pan.
c) Complex Pan.
d) Moving Object over stationary background.
e) Foreground Object over moving background.
f) No Motion.
Other motion models may be used as required.
a) Scene Change
A scene change as the name suggests is when one image has little or no commonality to a previous image or scene. It may be detected as a very large absolute difference in luminance between the two images, or a large difference in the colours of the two images.
In a preferred arrangement a scene change may be determined when the median of the differences of luminance values (0-255) between previous and current images is typically above 30. This value may vary with application but trial and error has determined that this value is appropriate for determining most scene changes.
A secondary test to determine a scene change can be when there are too many regions of motion vectors, which appears like random noise on the image and is likely due to a scene change. This may occur if there is a very large amount of motion in the image.
A third technique to detect a scene change is to analyse the top few lines of each image to detect a scene change. The top of each image changes the least.
Alternatively, when the majority of motion vector blocks have large error values the difference between the two images is too great and will therefore be considered as a scene change.
Scene Change and Field Delay
In the preferred embodiment when there is lateral motion detected in a scene the image to the trailing eye is delayed by an amount of time that is inversely proportional to the speed of the motion. For an image moving right to left the trailing eye is the left eye and for an image moving left to right the trailing eye is the right eye.
The image sequence delay (or Field Delay) to the trailing eye, may be created by temporally delaying the sequence of video fields to the trailing eye by storing them in digital form in memory. The current video field is shown to the leading eye and the delayed image to the trailing eye is selected from the stored video fields depending on the speed of the lateral motion.
Over a number of fields displayed, a history as to the change in motion and change in Field Delays to the trailing eye can be maintained. This helps in smoothing the stereoscopic effect by enabling the image processor to predict any motion trends and to react accordingly by modifying the delay so that there are no sudden changes.
If a scene change is detected the Field Delay for the preferred embodiment of the present invention is set to zero to prevent the image breaking apart and the Field Delay history is also reset. Field Delay history is preferably reset on each scene change.
b) Simple Pan
A simple pan describes a lateral motion trend over a series of images whereby the majority of analysed motion is in one direction. This will preferably also cover the situation where the majority of the scene has a consistent motion, and no stationary objects are detected in the foreground.
A simple pan can be detected as the major Object having a non zero motion vector.
The result of a simple pan is that a positive motion vector is generated if the scene is moving to the right (or panning left). In this case, the image to the right eye will be delayed. Similarly, a negative motion vector is generated if the scene is moving to the left (or panning right). In this case, the image to the left eye will be delayed.
c) Complex Pan
A complex pan differs from a simple pan in that there is significant vertical motion in the image. Therefore, in the preferred embodiment, to minimise vertical disparity between the stereoscopic image pair sequences, Field Delay is not applied and only Object Processing is used to create a stereoscopic effect. Field Delay history is stored to maintain continuity with new lateral motion.
d) Moving Object over Stationary Background
A moving object over a stationary background is simply the situation whereby the majority of a scene has no motion, and one or more moving Objects of medium size are in scene. This situation also results in a positive motion vector if the majority of Objects are moving to the right, and a negative motion vector if the majority of Objects are moving to the left. A positive motion vector produces a delay to the right eye, and a negative motion vector produces a delay to the left eye.
In the case where the motion vectors of the Objects in the scene are not consistent, for example, objects moving to the left and right in the same scene, then Contra Motion exists and Reverse 3D correction techniques may be applied.
e) Foreground Object over Moving Background
A Foreground Object over a moving background refers to the situation where a majority of the scene has motion, and an Object having a different motion is in the scene, for example a camera following a person walking. A Background Object is detected as a major Object of non-zero motion vector (That is, a panning background) behind an Object of medium size with zero or opposite motion vector to the main Object, or a major Object of zero vector in front of minor Objects of non zero vector that are spread over the entire field (That is, a large stationary object filling most of the field, but a pan is still visible behind it).
A decision should be made as to whether the foreground Object should be given priority in the generation of Motion Parallax, or whether the background should be given priority. If the background contains a large variation in depth (for example, trees), then motion vectors are assigned as if a Simple pan was occurring. If the background contains little variation in depth (for example, a wall) then a motion vector is assigned that is antiparallel or negative.
When the background contains a large variation in depth, and a motion vector is assigned to the scene as per Simple Pan methods, then the foreground object will be in Reverse 3D, and suitable correction methods should be applied.
f) No Motion
If no motion is detected such that the motion vectors are entirely zero, or alternatively the largest moving Object is considered too small, then the Field Delay will be set to zero. This situation can occur where only random or noise motion vectors are determined, or where no motion information is available, for example during a pan across a blue sky.
Once the images are analysed they can then be processed to create the stereoscopic image pairs.
When viewing a real world scene both eyes see a slightly different image. This is called retinal disparity. This in turn produces stereopsis or depth perception. In other words we see stereoscopically by having each eye see a slightly different image of the same scene.
Parallax on the other hand is defined as the amount of horizontal or lateral shift between the images which is perceived by the viewer as retinal disparity. When a stereoscopic image pair is created, a three-dimensional scene is observed from two horizontally-shifted viewpoints.
The present invention utilises a number of image and object processing techniques to generate stereoscopic image pairs from monoscopic images.
These techniques include:
1) Motion Parallax.
2) Forced Parallax (Lateral Shifting).
3) Parallax Zones.
4) Image Rotation about the Y-Axis.
5) Object Processing.
1) Motion Parallax
When a scene is moving from right to left, the right eye will observe the scene first while the left eye will receive a delayed image and visa versa for a scene moving in the opposite direction. The faster the motion the less delay between the images to both eyes. This is known as motion parallax and is a major depth cue. Therefore, if there is lateral motion in a scene, by creating a delay between the images to the eyes a stereoscopic effect will be perceived.
a) Field Delay Calculation
Once the nature of the motion in an image has been analysed and an overall motion vector determined, the required Field Delay can then be calculated. Preferably, the calculated Field Delay is averaged with previous delays to filter out ‘noisy’ values and also to prevent the Field Delay changing too quickly.
As stated above, the faster the motion the less delay between the image to each eye. Accordingly, smaller values of Field Delay are used in scenes with large motion vectors, whereas larger delays are used in scenes with little lateral motion. That is, an inverse relationship exists in the preferred embodiment between the delay and amount of motion.
When a scene change is determined, the history of Field Delays should be reset to zero, as if no motion had occurred previously. At the first detection of motion when a non zero Field Delay is calculated whilst the history of Field Delays is still zero, the entire history of Field Delay is set to the calculated Field Delay. This enables the system to immediately display the correct Field Delay when motion is detected.
b) Field Delay Implementation
Motion Parallax can be generated in hardware and software by storing digitised images in memory. Preferably, the digitised images could be stored in a buffer and a single input pointer used with two output pointers, one for the left eye image and one for the right eye image. The leading eye's image memory pointer is maintained at or near the current input image memory pointer while the delayed eyes image memory pointer is set further down the buffer to produce a delayed output. Many images may be stored, up to 8-10 video fields is typical in video applications. The delay is dependent on the speed of the motion analysed in the image. Maximum field delay is when there is minimum motion.
2) Forced Parallax (Lateral Shifting)
Forced parallax can be created by introducing a lateral shift between:
i) An exact copy of an image and itself
ii) The two fields of a video frame
iii) Two frames of a film sequence
iv) A transformed copy of an image and its original
A Negative lateral shift is produced by displacing the left image to the right and the right image to the left by the same amount (establishes a depth of field commencing from the screen plane and proceeding in front of it) and a Positive lateral shift by displacing the left image to the left and the right image to the right by the same amount (establishes a depth of field commencing from the screen plane and receding behind it).
Forced Parallax may be reduced to enhance the stereoscopic effect for a stationary object in front of a pan, where the object is ‘placed’ closer to the screen plane and the background is ‘pushed back’ from the defined object plane.
3) Parallax Zones
Because most scenes are viewed with the background at the top and the foreground at the bottom it is possible to accentuate a scene's depth by ‘Veeing’ the Forced Parallax. This is done by laterally shifting the top of the image more than the bottom of an image thus accentuating the front to back depth observed in a scene.
Another technique is to use a combination of Motion Parallax and Forced Parallax on different parts of the image. For example, by splitting the image vertically in half and applying different parallax shifts to each side, a scene such as looking forwards from a moving train down a railway track has the correct stereoscopic effect. Otherwise one side would always appear in Reverse 3D.
4) Image Rotation about the Y-Axis
When an object is moving towards the viewer in a real world scene, the object is rotated slightly in the view for each eye. The rotation effect is more pronounced as the object moves closer. Translating this rotation into the stereoscopic image pairs defines the effect as follows:
i) Moving towards the viewer—The left image is rotated vertically about its central axis in an anti-clockwise direction and the right image in a clockwise direction.
ii) Moving away from the viewer—The left image is rotated vertically about its central axis in a clockwise direction and the right image in an anti-clockwise direction.
Therefore, by image rotation, the perspective of objects in the image is changed slightly so that depth is perceived. When this technique is combined with Forced Parallax for certain scenes the combined effect provides very powerful stereoscopic depth cues.
5) Object Processing
Object processing is performed to further enhance the stereoscopic effect, particularly in still images, by separating the Objects and background so that these items can be processed independently. It is most effective when the objects are large in size, few in number and occupy distinct depth levels throughout the depth of field.
A database for Object Tagging and Object Tracking can be used to establish trends so that an Object can be digitally ‘cut out’ from its background and appropriate measures taken to enhance the stereoscopic effect. Once processing has taken place the Object is ‘Pasted’ back in the same position on to the background again. This can be termed the ‘Cut and Paste’ technique and is useful in the conversion process.
By integrating the processes of Object Tagging, Tracking, Cutting and Pasting a powerful tool is available for enabling Object Processing and Background Processing.
Another Object Processing technique is Object Layering which defines an independent depth module for each moving Object. The Object can then be placed anywhere on an image because the background fill detail has been defined when the Object was not in that position. This is not generally possible with a still Object unless the background fill-in is interpolated.
A most important issue in stereoscopic conversion is the correction of Reverse 3D and Accommodation/Convergence imbalances that cause viewer discomfort. Object Processing in the preferred embodiment allows corrections to this problem too.
a) Mesh Distortion and Morphing
This Object processing technique allows an Object to be cut and pasted onto a distorted mesh to enhance depth perception. By distorting an Object in the left eye image to the right and by distorting the same object in the right eye image to the left, thus creating Object Parallax, the Object can be made to appear much closer to a viewer when using a stereoscopic display device.
b) Object Barrelling
This technique is a specific form of Mesh Distortion and refers to a technique of cutting an Object from the image and wrapping onto a vertically positioned half barrel. This makes the Object appear to have depth by making the centre portion of the Object appear closer than the Object edges.
c) Object Edge Enhancement
By emphasising the edges of an Object there is greater differentiation between the background or other Objects in an image. The stereoscopic effect is enhanced in many applications by this technique.
d) Object Brightness Enhancement
In any image the eye is always drawn to the largest and brightest objects. By modifying an Object's luminance the Object can be emphasised more over the background, enhancing the stereoscopic effect.
e) Object rotation about Y-axis
Object rotation about the Y-axis refers to a similar process to that of image rotation about the Y-axis, except that this time the rotation occurs to the Object only. If the Object in the stereoscopic image pair is ‘Cut’ from its background and rotated slightly the change in perspective generated by the rotation is perceived as depth.
1) Reference Points or Borders
When using a normal TV or video monitor to display stereoscopic images the eye continually observes the edge of the monitor or screen and this is perceived as a point of reference or fixation point for all depth perception. That is, all objects are perceived at a depth behind or in front of this reference point.
If the edge of the monitor is not easily seen because of poor ambient lighting or due to its dark colour then this reference point may be lost and the eyes may continually search for a fixation point in the 3D domain. Under prolonged stereoscopic viewing this can cause eye fatigue and decreased depth perception. A front or rear projection screen display system may also suffer from the same problems.
The present invention therefore preferably also defines a common border or reference point within a viewed image. Ideally the reference plane is set at the screen level and all depth is perceived behind this level. This has the advantage of enhancing the stereoscopic effect in many scenes.
This reference point can be a simple video border or reference graphic and, for example, may be of the following types:
i) A simple coloured video border around the perimeter of the image.
ii) A complex coloured video border consisting of two or more concentric borders that may have opaque or transparent sections between them. For example, a 2-3 cm wide mesh border or a wide outer border with two thin inner borders.
iii) A partial border that may occupy any one edge, or any two horizontal or vertical edges.
iv) A LOGO or other graphic located at some point within the image.
v) A picture within a picture.
vi) A combination of any of the above.
What is essential in this embodiment is that the eyes of the viewer be provided with a reference point by which the depth of the objects in the image can be perceived.
If a border or graphic is added at the 3D Generation level then it may be specified to provide a reference point at a particular depth by creating left and right borders that are laterally shifted from each other. This enables the reference or fixation point to be shifted in space to a point somewhere behind or in front of the screen level. Borders or graphics defined with no parallax for the left and right eyes will be perceived at the screen level. This is the preferred mode of the present invention.
A image border or reference graphic may be inserted at the 3D Generation point or it may be defined externally and genlocked onto the stereoscopic image output for display. Such an image border or reference graphic may be black, white or coloured, plain or patterned, opaque, translucent or transparent to the image background, or it may be static or dynamic. Whilst a static border is appropriate in most instances, in some circumstances a moving or dynamic border may be used for motion enhancement.
2) Parallax Adjustment—Depth Sensitivity Control
Stereoscopic images viewed through a stereoscopic display device automatically define a depth range (called depth acuity) which can be increased or decreased by modifying the type and amount of parallax applied to the image or objects. It has been found that different viewers have varying stereoscopic viewing comfort levels based on the depth range or amount of stereopsis defined by stereoscopic image pairs. That is, while some viewers prefer a pronounced stereoscopic effect with a greater depth range, others prefer an image with minimal depth.
To adjust the level of depth sensitivity and viewing comfort many techniques may be used, namely:
i) Varying the amount of Motion Parallax by varying the Field Delay
ii) Varying the amount of Forced Parallax to an image
iii) Varying the amount of Parallax applied to objects
By reducing the maximum level of Parallax the depth range can be reduced, improving the viewing comfort for those with perception faculties having greater sensitivity to sterescopy.
3) Parallax Smoothing
Parallax Smoothing is the process of maintaining the total amount of Parallax (Motion Parallax plus Forced Parallax) as a continuous function. Changes in Field Delay for specific motion types, that is, Simple Pan and Foreground Object Motion, cause discontinuities in the amount of Motion Parallax produced, which are seen as “jumps” in the stereoscopic images by the viewer. Discontinuities only occur in the image produced for the trailing eye, as the leading eye is presented with an undelayed image. These discontinuities can be compensated for by adjusting the Forced Parallax or Object Parallax in an equal and opposite direction for the trailing eye, thus maintaining a continuous total parallax.
The Forced Parallax or Object Parallax is then adjusted smoothly back to its normal value, ready for the next change in Field Delay. The adjustments made to Forced Parallax by Parallax Smoothing are a function of Field Delay change, motion type and motion vector. To implement Parallax Smoothing, the Forced Parallax for the left and right eye images should be independently set.
4) Parallax Modulation
The Forced Parallax technique of creating a stereoscopic effect can also be used to moderate the amount of stereopsis detected by the viewer. This is done by varying the Forced Parallax setting between a minimum and maximum limit over a short time such that the perceived depth of an object or image varies over time. Ideally the Forced Parallax is modulated between its minimum and maximum settings every 0.5 to 1 second. This enables a viewer to accommodate to their level of stereoscopic sensitivity.
5) Movement Synthesis
By creating pseudo movement, by randomly moving the background in small undetectable increments, the perceived depth of foreground objects is emphasised. Foreground objects are ‘Cut’ from the background, the background is altered pseudo-randomly by one of the techniques below and then the foreground object is ‘Pasted’ back on to the background ready for display. Any of the following techniques may be used:
i) Luminance values varied on a pseudo-random basis
ii) Chrominance values varied on a pseudo-random basis
iii) Adding pseudo-random noise to the background to create movement
6) Reverse 3D Analysis and Correction
Reverse 3D occurs, when the depth order of Objects created by Parallax is perceived to be different to that corresponding to the depth order in the real world. This generally leads to viewer discomfort and should be corrected. When converting monoscopic images to stereoscopic image pairs Reverse 3D may be produced by:
i) Contra motion, objects moving left and right in the same image.
ii) Objects and background moving in different directions.
iii) Many objects moving at varying speeds
Reverse 3D is corrected by analysing the nature of the motion of the objects in an image and then manipulating each Object individually using mesh distortion techniques so that the Object Parallax matches with the expected visual perception norms.
7) Miscellaneous Techniques
By modifying the perspective of an object within an image and by enhancing many of the minor depth cues the stereoscopic effect can be emphasised. The techniques below all operate using the ‘Cut and Paste’ technique. That is, a foreground object is ‘Cut’, enhanced and then ‘Pasted’ back on to the background.
a) Shadows—Shading gives an object perspective.
b) Foreground/Background—By defocussing the background, through blurring or fogging, a foreground object may be emphasised, while defocussing the foreground object the background depth may be emphasised
c) Edge Enhancement—Edges help to differentiate an object from its background.
d) Texture Mapping—Helps to differentiate the object from the background.
As for module 1, modules 4 and 5 are not essential to the present invention. Module 4 provides for the transmission and/or storage of the stereoscopic images. The transmission means can be adapted for a particular application. For example the following can be employed:
1) Local Transmission—can be via coax cable
2) Network TV Transmission—can be via
3) Digital Network—INTERNET, etc
4) Stereoscopic (3D) Image Storage
An image storage means may be used for storage of the image data for later transmission or display and may include:
i) Analogue Storage—Video Tape, Film, etc
ii) Digital Storage—Laser Disk, Hard Disk, CD-ROM, Magneto Optical Disk, DAT, Digital Video Cassette (DVC), DVD.
As for the transmission means the display means can be dependent on the application requirements and can include:
1) Set-top Box
A set-top box by definition is a small box of electronics that receives, decodes, provides accessories interfaces and finally has outputs to suit the application. It may incorporate the following:
a) Video or RF receiver.
b) Stereoscopic (3D) decoder to provide separate left and right image outputs to Head Mounted Devices or other stereoscopic displays where separate video channels are required.
c) Resolution Enhancement—Line Doubling/Pixel Interpolation.
d) Shutter or Sequential Glasses Synchronisation.
e) Stereoscopic depth sensitivity control circuitry.
f) Accessories interface—remote control with features such as a 2D/3D switch and Depth control.
g) Audio interface—audio output, headphone connection.
h) Access channel decoding—cable and pay TV applications.
i) Video or RF outputs.
2) Stereoscopic Displays
Use special glasses or head gear to provide separate images to the left and right eyes including:
a) Polarising glasses—Linear and Circular polarisers.
b) Anaglyphic glasses—Coloured lenses—red/green, etc.
c) LCD Shutter glasses.
d) Colour Sequential Glasses.
e) Head Mounted Devices (HMD)—Head gear fitted with two miniature video monitors (one for each eye), VR headsets.
3) Autostereoscopic Displays
a) Video Projector/Retroreflective screen based display systems.
b) Volumetric display systems.
c) Lenticular lens based display systems.
d) Holographic Optical Element (HOE) based display systems.
In summary, the present invention provides in a preferred embodiment a system that is capable of inputting monoscopic image sequences in a digital format, or in an analogue format in which case an analogue to digital conversion process is involved. This image data is then subjected to a method of image analysis whereby the monoscopic images are compressed, if this is required for the particular application.
By comparing blocks of pixels in an image, with corresponding blocks in an adjacent image, and by obtaining the minimum Mean Square Error for each block, motion within the image can be determined.
Following motion detection, regions of an image are identified for similar characteristics, such as, image brightness, colour, motion, pattern and edge continuity. The data is then subjected to motion analysis in order to determine the nature of the motion in the image. This motion analysis takes the form of determining the direction, speed, type, depth and position of any motion in the image. This motion is then categorised into a number of categories including whether the motion is a complete scene change, a simple pan, a complex pan, an object moving on a stationary background, a stationary object in front of a moving background, or whether there is no motion at all. Further actions are then determined based on these categories to convert the monoscopic images into stereoscopic image pairs suitable for viewing on an appropriate stereoscopic display device.
In the preferred embodiment, once the monoscopic images are analysed, if a scene change or a complex pan is detected then no further analysis of that particular scene is required, rather the Field Delay and Field Delay history are both reset to zero. An object detection process is then applied to the new scene in order to try and identify objects within that scene. Once these objects are identified, then object processing takes place. If no objects are identified, then the image is passed on for further processing using forced parallax and 3D optimisation.
If the motion categorised during the image analysis is not a scene change, then further analysis of that scene is required. If further analysis of that scene results in the motion being categorised as a simple pan, then it is necessary to apply a Field Delay in accordance with the principles of motion parallax. It is then passed on for further processing. If the motion is not categorised as a simple pan, but rather as an object in motion on a stationary background, then again we have to apply a Field Delay in accordance with the principles of motion parallax. In this regard, once the motion parallax has been applied, it is necessary to consider whether the objects all have a uniform direction. If the objects do move in a uniform direction, then it is passed on for further processing at a later stage. If the objects do not have a uniform direction, then it is necessary to perform further object processing on selected objects within that scene to correct for the Reverse 3D effect. This can be achieved through using mesh distortion and morphing techniques.
If the motion is categorised as being a stationary object on a moving background, it is then necessary to consider whether the background has a large variation in depth. If it does not, then we apply a Field Delay with the object having priority using the principles of motion parallax. However, if the background does have large variation in depth, then we apply a Field Delay with the background having priority as opposed to the object, again using the principles of motion parallax. In this case, it is then also necessary to perform further object processing on the foreground object to correct for the Reverse 3D effect prior to being passed on for further processing.
If no motion is detected, then we next consider whether an object in the scene was known from any previous motion. If this is so, then we perform object processing on that selected object. If not, then we apply an object detection process to that particular scene in order to attempt to identify any objects in it. If an object is identified, then we perform object processing on that particular object, if not, Forced Parallax and 3D Optimisation is performed.
Where object processing is required, objects are identified, tagged and tracked, and then processed by using techniques of mesh distortion and morphing, object baralleling, edge enhancement, brightness modification and object rotation.
In all cases, once the motion has been categorised and the primary techniques to convert to stereoscopic images have been applied, then a further amount of parallax or lateral shifting called forced parallax is applied to the image. It is noted that in the preferred embodiment, forced parallax is applied to every image, not just for depth smoothing purposes but to provide an underlying stereoscopic effect that all images are seen as having depth behind or in front of the stereoscopic display device's reference plane, generally the front of the monitor screen. The advantages of applying forced parallax are that the system is better able to cope with changes in the category of the motion detected without causing sudden changes in the viewers depth perception.
Once the forced parallax has been applied to the image, the image is then passed for 3D Optimisation. Again, this is not necessary in order to see a stereoscopic image, however the optimisation does enhance the image's depth perception by the viewer. The 3D Optimisation can take in a number of forms including the addition of reference points or borders, parallax modulation, parallax smoothing and parallax adjustment for altering the depth sensitivity of any particular viewer. The image can also be optimised by modifying luminance or chrominance values pseudo randomly so that background motion behind foreground objects can be observed so that the depth perception is enhanced. It is also possible to analyse for Reverse 3D so that a viewers eyestrain is minimised. Further techniques such as shadowing, foreground and background fogging or blurring and edge enhancement of the image can also be carried out in this stage.
Once the image has been optimised it is then transmitted to the appropriate display device. This transmission can take a number of forms including cable, co-axial, satellite or any other form of transmitting the signal from one point to another. It is also possible that,the image could be stored prior to being sent to a display device. The display device can take on a number of forms, and only need be appropriate for the application in hand, for example, it is possible to use existing video monitors with a set top device in order to separate the left and right images, increase the scan rate and to synchronise viewing glasses. Alternatively, dedicated stereoscopic displays can be used which incorporate the use of glasses or head gear to provide the stereoscopic images or alternatively, an auto-stereoscopic display device can be used. It is envisaged that the present invention will have application in theatres, cinemas, video arcades, cable or network TV, in the education area, particularly in the multimedia industry and in many other areas such as theme parks and other entertainment applications.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4925294||Dec 17, 1986||May 15, 1990||Geshwind David M||Method to convert two dimensional motion pictures for three-dimensional systems|
|US5510832||Dec 1, 1993||Apr 23, 1996||Medi-Vision Technologies, Inc.||Synthesized stereoscopic imaging system and method|
|US5682437||Feb 12, 1996||Oct 28, 1997||Sanyo Electric Co., Ltd.||Method of converting two-dimensional images into three-dimensional images|
|US5689312 *||Dec 26, 1995||Nov 18, 1997||Daewoo Electronic Co., Ltd.||Block matching motion estimation method|
|US5717415||Jul 10, 1996||Feb 10, 1998||Sanyo Electric Co., Ltd.||Display system with 2D/3D image conversion where left and right eye images have a delay and luminance difference base upon a horizontal component of a motion vector|
|US5748199 *||Dec 20, 1995||May 5, 1998||Synthonics Incorporated||Method and apparatus for converting a two dimensional motion picture into a three dimensional motion picture|
|US5760933||Feb 26, 1997||Jun 2, 1998||Fujitsu Limited||Stereoscopic display apparatus and method|
|US5777666||Apr 16, 1996||Jul 7, 1998||Sanyo Electric Co., Ltd.||Method of converting two-dimensional images into three-dimensional images|
|US5969766 *||Aug 25, 1997||Oct 19, 1999||Daewoo Electronics Co., Ltd||Method and apparatus for contour motion estimating a binary image by using a weighted block match algorithm|
|EP0264964A2||Oct 23, 1987||Apr 27, 1988||The Grass Valley Group, Inc.||Method and apparatus for processing a video signal|
|EP0704823A2||Sep 22, 1995||Apr 3, 1996||Sanyo Electric Co. Ltd||Method of judging back-and-forth positional relationship between subjects and method of converting two-dimensional images into three-dimensional images|
|EP0717373A2||Dec 8, 1995||Jun 19, 1996||Sanyo Electric Co. Ltd||Method of converting two-dimensional images into three-dimensional images in video game set|
|WO1996041311A2||Jun 7, 1996||Dec 19, 1996||Geshwind David M||Stereoscopic interactive painting|
|WO1997024000A1||Dec 20, 1996||Jul 3, 1997||Xenotech Research Pty. Ltd.||Image conversion and encoding techniques|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US6760469 *||Oct 5, 2000||Jul 6, 2004||Canon Kabushiki Kaisha||Flipping stereoscopic radiographs|
|US6765568 *||Jun 12, 2001||Jul 20, 2004||Vrex, Inc.||Electronic stereoscopic media delivery system|
|US6771799 *||May 15, 2001||Aug 3, 2004||Koninklijke Philips Electronics N.V.||Foreground/background detector|
|US6816629 *||Sep 7, 2001||Nov 9, 2004||Realty Mapping Llc||Method and system for 3-D content creation|
|US6893128 *||Oct 16, 2002||May 17, 2005||Kowa Company, Ltd.||Unit for obtaining and displaying fundus image|
|US6983013 *||Jul 31, 2000||Jan 3, 2006||Koninklijke Philips Electronics N.V.||Method and device for encoding video signal|
|US6985604 *||May 7, 2001||Jan 10, 2006||Koninklijke Philips Electronics N.V.||Method, system and apparatus for block matching based on visible pixels|
|US7027665 *||Sep 29, 2000||Apr 11, 2006||Microsoft Corporation||Method and apparatus for reducing image acquisition time in a digital imaging device|
|US7043084 *||Jul 30, 2002||May 9, 2006||Mitsubishi Electric Research Laboratories, Inc.||Wheelchair detection using stereo vision|
|US7113193 *||Aug 26, 2004||Sep 26, 2006||Sony Computer Entertainment Inc.||Method for color transition detection|
|US7116323||Dec 10, 2002||Oct 3, 2006||In-Three, Inc.||Method of hidden surface reconstruction for creating accurate three-dimensional images converted from two-dimensional images|
|US7180536 *||May 26, 2003||Feb 20, 2007||Raphael-Armament Development Authority Ltd.||Method for producing stereoscopic images from monoscopic images|
|US7254265||Jan 8, 2003||Aug 7, 2007||Newsight Corporation||Methods and systems for 2D/3D image conversion and optimization|
|US7254280||Apr 5, 2006||Aug 7, 2007||Microsoft Corporation||Method and apparatus for reducing image acquisition time in a digital imaging device|
|US7269285 *||Jul 6, 2001||Sep 11, 2007||Mitsubishi Electric Information Technology Centre Europe Bv||Method and apparatus for representing and searching for an object in an image|
|US7286168 *||Oct 11, 2002||Oct 23, 2007||Canon Kabushiki Kaisha||Image processing apparatus and method for adding blur to an image|
|US7321374||May 29, 2003||Jan 22, 2008||Newsight Corporation||Method and device for the generation of 3-D images|
|US7379120 *||May 17, 2005||May 27, 2008||Sony Corporation||Image processing device and image processing method|
|US7388986||Jul 12, 2007||Jun 17, 2008||Mitsubishi Denki Kabushiki Kaisha||Method and apparatus for representing and searching for an object in an image|
|US7419617||Oct 20, 2005||Sep 2, 2008||Basf Aktiengesellschaft||Coolant comprising azole derivatives for cooling systems in fuel-cell drives|
|US7419618||Jun 8, 2006||Sep 2, 2008||Basf Aktiengesellschaft||Coolant comprising azole derivatives for cooling systems in fuel-cell drives|
|US7474766 *||Aug 28, 2001||Jan 6, 2009||Minolta Co., Ltd.||Motion image processor, motion image processing method and recording medium|
|US7593577||Apr 3, 2008||Sep 22, 2009||Mitsubishi Denki Kabushiki Kaisha||Method and apparatus for representing and searching for an object in an image|
|US7643554 *||Nov 30, 1999||Jan 5, 2010||Sharp Kabushiki Kaisha||Image retrieving apparatus performing retrieval based on coding information utilized for feature frame extraction or feature values of frames|
|US7643672 *||Aug 23, 2005||Jan 5, 2010||Kazunari Era||Image processing apparatus, image pickup device and program therefor|
|US7666096 *||Jun 19, 2006||Feb 23, 2010||Tdvision Corporation S.A. De C.V.||Method for generating the left and right perspectives in a 3D videogame|
|US7679616 *||Apr 24, 2003||Mar 16, 2010||Sharp Kabushiki Kaisha||Image data generation apparatus for adding attribute information regarding image pickup conditions to image data, image data reproduction apparatus for reproducing image data according to added attribute information, and image data recording medium related thereto|
|US7689041||Apr 3, 2008||Mar 30, 2010||Mitsubishi Denki Kabushiki Kaisha||Method and apparatus for representing and searching for an object in an image|
|US7705876||Sep 23, 2004||Apr 27, 2010||Microsoft Corporation||Stereoscopic image display|
|US7715643 *||Jun 13, 2002||May 11, 2010||Sony Corporation||Image processing apparatus and method, and image pickup apparatus|
|US7756339||Apr 3, 2008||Jul 13, 2010||Mitsubishi Denki Kabushiki Kaisha||Method and apparatus for representing and searching for an object in an image|
|US7777764 *||Sep 9, 2004||Aug 17, 2010||Sharp Kabushiki Kaisha||Portable display device|
|US7856055||Mar 13, 2003||Dec 21, 2010||Imax Corporation||Systems and methods for digitally re-mastering or otherwise modifying motion pictures or other image sequences data|
|US7856125||Jan 30, 2007||Dec 21, 2010||University Of Southern California||3D face reconstruction from 2D images|
|US7869621 *||Jun 7, 2007||Jan 11, 2011||Aydin Arpa||Method and apparatus for interpreting images in temporal or spatial domains|
|US7884823||Jun 12, 2007||Feb 8, 2011||Microsoft Corporation||Three dimensional rendering of display information using viewer eye coordinates|
|US7907793||Aug 17, 2009||Mar 15, 2011||Legend Films Inc.||Image sequence depth enhancement system and method|
|US7940987||Jun 2, 2010||May 10, 2011||Mitsubishi Denki Kabushiki Kaisha||Method and apparatus for representing and searching for an object in an image|
|US8004515 *||Mar 15, 2005||Aug 23, 2011||Nvidia Corporation||Stereoscopic vertex shader override|
|US8063930 *||Jan 12, 2006||Nov 22, 2011||Rafael-Armament Development Authority Ltd.||Automatic conversion from monoscopic video to stereoscopic video|
|US8073247||Nov 1, 2010||Dec 6, 2011||Legend3D, Inc.||Minimal artifact image sequence depth enhancement system and method|
|US8078006||Nov 1, 2010||Dec 13, 2011||Legend3D, Inc.||Minimal artifact image sequence depth enhancement system and method|
|US8086004||Aug 27, 2008||Dec 27, 2011||Accuray Incorporated||Use of a single X-ray image for quality assurance of tracking|
|US8126261 *||Jul 25, 2007||Feb 28, 2012||University Of Southern California||3D face reconstruction from 2D images|
|US8133115||Oct 22, 2003||Mar 13, 2012||Sony Computer Entertainment America Llc||System and method for recording and displaying a graphical path in a video game|
|US8150217 *||Apr 15, 2008||Apr 3, 2012||Fujifilm Corporation||Image processing apparatus, method and program|
|US8160390||Nov 1, 2010||Apr 17, 2012||Legend3D, Inc.||Minimal artifact image sequence depth enhancement system and method|
|US8184068 *||Aug 23, 2011||May 22, 2012||Google Inc.||Processing objects for separate eye displays|
|US8194118 *||Feb 18, 2006||Jun 5, 2012||Dennis J Solomon||Performance display system|
|US8199186||Mar 5, 2009||Jun 12, 2012||Microsoft Corporation||Three-dimensional (3D) imaging based on motionparallax|
|US8204272||Jun 17, 2011||Jun 19, 2012||Sony Computer Entertainment Inc.||Lighting control of a user environment via a display device|
|US8206218||Feb 22, 2010||Jun 26, 2012||Tdvision Corporation S.A. De C.V.||3D videogame system|
|US8213680||Mar 19, 2010||Jul 3, 2012||Microsoft Corporation||Proxy training data for human body tracking|
|US8243089||Feb 1, 2011||Aug 14, 2012||Sony Computer Entertainment Inc.||Implementing lighting control of a user environment|
|US8253746||May 1, 2009||Aug 28, 2012||Microsoft Corporation||Determine intended motions|
|US8264536||Aug 25, 2009||Sep 11, 2012||Microsoft Corporation||Depth-sensitive imaging via polarization-state mapping|
|US8265341||Jan 25, 2010||Sep 11, 2012||Microsoft Corporation||Voice-body identity correlation|
|US8267781||Jan 30, 2009||Sep 18, 2012||Microsoft Corporation||Visual target tracking|
|US8274552||May 25, 2011||Sep 25, 2012||3Dmedia Corporation||Primary and auxiliary image capture devices for image processing and related methods|
|US8279418||Mar 17, 2010||Oct 2, 2012||Microsoft Corporation||Raster scanning for depth detection|
|US8284310||Apr 5, 2011||Oct 9, 2012||Sony Computer Entertainment America Llc||Delay matching in audio/video systems|
|US8284847||May 3, 2010||Oct 9, 2012||Microsoft Corporation||Detecting motion for a multifunction sensor device|
|US8289325||Oct 7, 2008||Oct 16, 2012||Sony Computer Entertainment America Llc||Multi-pass shading|
|US8294767||Jan 30, 2009||Oct 23, 2012||Microsoft Corporation||Body scan|
|US8295546||Oct 21, 2009||Oct 23, 2012||Microsoft Corporation||Pose tracking pipeline|
|US8296151||Jun 18, 2010||Oct 23, 2012||Microsoft Corporation||Compound gesture-speech commands|
|US8320619||Jun 15, 2009||Nov 27, 2012||Microsoft Corporation||Systems and methods for tracking a model|
|US8320621||Dec 21, 2009||Nov 27, 2012||Microsoft Corporation||Depth projector system with integrated VCSEL array|
|US8325909||Jun 25, 2008||Dec 4, 2012||Microsoft Corporation||Acoustic echo suppression|
|US8325984||Jun 9, 2011||Dec 4, 2012||Microsoft Corporation||Systems and methods for tracking a model|
|US8330134||Sep 14, 2009||Dec 11, 2012||Microsoft Corporation||Optical fault monitoring|
|US8330822||Jun 9, 2010||Dec 11, 2012||Microsoft Corporation||Thermally-tuned depth camera light source|
|US8340432||Jun 16, 2009||Dec 25, 2012||Microsoft Corporation||Systems and methods for detecting a tilt angle from a depth image|
|US8351651||Apr 26, 2010||Jan 8, 2013||Microsoft Corporation||Hand-location post-process refinement in a tracking system|
|US8351652||Feb 2, 2012||Jan 8, 2013||Microsoft Corporation||Systems and methods for tracking a model|
|US8363212||Apr 2, 2012||Jan 29, 2013||Microsoft Corporation||System architecture design for time-of-flight system having reduced differential pixel size, and time-of-flight systems so designed|
|US8374423||Mar 2, 2012||Feb 12, 2013||Microsoft Corporation||Motion detection using depth images|
|US8379101||May 29, 2009||Feb 19, 2013||Microsoft Corporation||Environment and/or target segmentation|
|US8379919||Apr 29, 2010||Feb 19, 2013||Microsoft Corporation||Multiple centroid condensation of probability distribution clouds|
|US8381108||Jun 21, 2010||Feb 19, 2013||Microsoft Corporation||Natural user input for driving interactive stories|
|US8385557||Jun 19, 2008||Feb 26, 2013||Microsoft Corporation||Multichannel acoustic echo reduction|
|US8385596||Dec 21, 2010||Feb 26, 2013||Microsoft Corporation||First person shooter control with virtual skeleton|
|US8385684||Feb 17, 2011||Feb 26, 2013||Legend3D, Inc.||System and method for minimal iteration workflow for image sequence depth enhancement|
|US8390680||Jul 9, 2009||Mar 5, 2013||Microsoft Corporation||Visual representation expression based on player expression|
|US8396328||Oct 27, 2010||Mar 12, 2013||Legend3D, Inc.||Minimal artifact image sequence depth enhancement system and method|
|US8401225||Jan 31, 2011||Mar 19, 2013||Microsoft Corporation||Moving object segmentation using depth images|
|US8401242||Jan 31, 2011||Mar 19, 2013||Microsoft Corporation||Real-time camera tracking using depth maps|
|US8401336||Dec 22, 2010||Mar 19, 2013||Legend3D, Inc.||System and method for rapid image sequence depth enhancement with augmented computer-generated elements|
|US8408706||Dec 13, 2010||Apr 2, 2013||Microsoft Corporation||3D gaze tracker|
|US8411931 *||Jun 25, 2007||Apr 2, 2013||Imax Corporation||Methods and systems for converting 2D motion pictures for stereoscopic 3D exhibition|
|US8411948||Mar 5, 2010||Apr 2, 2013||Microsoft Corporation||Up-sampling binary images for segmentation|
|US8416187||Jun 22, 2010||Apr 9, 2013||Microsoft Corporation||Item navigation using motion-capture data|
|US8418085||May 29, 2009||Apr 9, 2013||Microsoft Corporation||Gesture coach|
|US8422769||Mar 5, 2010||Apr 16, 2013||Microsoft Corporation||Image segmentation using reduced foreground training data|
|US8428340||Sep 21, 2009||Apr 23, 2013||Microsoft Corporation||Screen space plane identification|
|US8436893||Jul 23, 2010||May 7, 2013||3Dmedia Corporation||Methods, systems, and computer-readable storage media for selecting image capture positions to generate three-dimensional (3D) images|
|US8437506||Sep 7, 2010||May 7, 2013||Microsoft Corporation||System for fast, probabilistic skeletal tracking|
|US8441520||Aug 13, 2012||May 14, 2013||3Dmedia Corporation||Primary and auxiliary image capture devcies for image processing and related methods|
|US8448056||Dec 17, 2010||May 21, 2013||Microsoft Corporation||Validation analysis of human target|
|US8448094||Mar 25, 2009||May 21, 2013||Microsoft Corporation||Mapping a natural input device to a legacy system|
|US8451278||Aug 3, 2012||May 28, 2013||Microsoft Corporation||Determine intended motions|
|US8452051||Dec 18, 2012||May 28, 2013||Microsoft Corporation||Hand-location post-process refinement in a tracking system|
|US8452087||Sep 30, 2009||May 28, 2013||Microsoft Corporation||Image selection techniques|
|US8456419||Apr 18, 2008||Jun 4, 2013||Microsoft Corporation||Determining a position of a pointing device|
|US8457353||May 18, 2010||Jun 4, 2013||Microsoft Corporation||Gestures and gesture modifiers for manipulating a user-interface|
|US8467133||Apr 6, 2012||Jun 18, 2013||Osterhout Group, Inc.||See-through display with an optical assembly including a wedge-shaped illumination system|
|US8467574||Oct 28, 2010||Jun 18, 2013||Microsoft Corporation||Body scan|
|US8472120||Mar 25, 2012||Jun 25, 2013||Osterhout Group, Inc.||See-through near-eye display glasses with a small scale image source|
|US8477425||Mar 25, 2012||Jul 2, 2013||Osterhout Group, Inc.||See-through near-eye display glasses including a partially reflective, partially transmitting optical element|
|US8482859||Mar 26, 2012||Jul 9, 2013||Osterhout Group, Inc.||See-through near-eye display glasses wherein image light is transmitted to and reflected from an optically flat film|
|US8483436||Nov 4, 2011||Jul 9, 2013||Microsoft Corporation||Systems and methods for tracking a model|
|US8487871||Jun 1, 2009||Jul 16, 2013||Microsoft Corporation||Virtual desktop coordinate transformation|
|US8487938||Feb 23, 2009||Jul 16, 2013||Microsoft Corporation||Standard Gestures|
|US8487963||May 30, 2008||Jul 16, 2013||Adobe Systems Incorporated||Preview representation of pixels effected by a brush tip area|
|US8488246||Mar 26, 2012||Jul 16, 2013||Osterhout Group, Inc.||See-through near-eye display glasses including a curved polarizing film in the image source, a partially reflective, partially transmitting optical element and an optically flat film|
|US8488888||Dec 28, 2010||Jul 16, 2013||Microsoft Corporation||Classification of posture states|
|US8497838||Feb 16, 2011||Jul 30, 2013||Microsoft Corporation||Push actuation of interface controls|
|US8498481||May 7, 2010||Jul 30, 2013||Microsoft Corporation||Image segmentation using star-convexity constraints|
|US8499257||Feb 9, 2010||Jul 30, 2013||Microsoft Corporation||Handles interactions for human—computer interface|
|US8503494||Apr 5, 2011||Aug 6, 2013||Microsoft Corporation||Thermal management system|
|US8503766||Dec 13, 2012||Aug 6, 2013||Microsoft Corporation||Systems and methods for detecting a tilt angle from a depth image|
|US8508580 *||Jul 23, 2010||Aug 13, 2013||3Dmedia Corporation||Methods, systems, and computer-readable storage media for creating three-dimensional (3D) images of a scene|
|US8508919||Sep 14, 2009||Aug 13, 2013||Microsoft Corporation||Separation of electrical and optical components|
|US8509479||Jun 16, 2009||Aug 13, 2013||Microsoft Corporation||Virtual object|
|US8509545||Nov 29, 2011||Aug 13, 2013||Microsoft Corporation||Foreground subject detection|
|US8514269||Mar 26, 2010||Aug 20, 2013||Microsoft Corporation||De-aliasing depth images|
|US8523667||Mar 29, 2010||Sep 3, 2013||Microsoft Corporation||Parental control settings based on body dimensions|
|US8526734||Jun 1, 2011||Sep 3, 2013||Microsoft Corporation||Three-dimensional background removal for vision system|
|US8537200 *||Jan 29, 2010||Sep 17, 2013||Qualcomm Incorporated||Depth map generation techniques for conversion of 2D video data to 3D video data|
|US8542252||May 29, 2009||Sep 24, 2013||Microsoft Corporation||Target digitization, extraction, and tracking|
|US8542910||Feb 2, 2012||Sep 24, 2013||Microsoft Corporation||Human tracking system|
|US8542929||Jul 31, 2007||Sep 24, 2013||Samsung Electronics Co., Ltd.||Image processing method and apparatus|
|US8548270||Oct 4, 2010||Oct 1, 2013||Microsoft Corporation||Time-of-flight depth imaging|
|US8553934||Dec 8, 2010||Oct 8, 2013||Microsoft Corporation||Orienting the position of a sensor|
|US8553939||Feb 29, 2012||Oct 8, 2013||Microsoft Corporation||Pose tracking pipeline|
|US8558830 *||Dec 18, 2009||Oct 15, 2013||3D Fusion, Inc.||System and method for adaptive scalable dynamic conversion, quality and processing optimization, enhancement, correction, mastering, and other advantageous processing of three dimensional media content|
|US8558873||Jun 16, 2010||Oct 15, 2013||Microsoft Corporation||Use of wavefront coding to create a depth image|
|US8564534||Oct 7, 2009||Oct 22, 2013||Microsoft Corporation||Human tracking system|
|US8565476||Dec 7, 2009||Oct 22, 2013||Microsoft Corporation||Visual target tracking|
|US8565477||Dec 7, 2009||Oct 22, 2013||Microsoft Corporation||Visual target tracking|
|US8565485||Sep 13, 2012||Oct 22, 2013||Microsoft Corporation||Pose tracking pipeline|
|US8571263||Mar 17, 2011||Oct 29, 2013||Microsoft Corporation||Predicting joint positions|
|US8571305 *||Sep 21, 2009||Oct 29, 2013||Chunghwa Picture Tubes, Ltd.||Image processing device for enhancing stereoscopic sensation of an image using a depth image and method thereof|
|US8577084||Dec 7, 2009||Nov 5, 2013||Microsoft Corporation||Visual target tracking|
|US8577085||Dec 7, 2009||Nov 5, 2013||Microsoft Corporation||Visual target tracking|
|US8578302||Jun 6, 2011||Nov 5, 2013||Microsoft Corporation||Predictive determination|
|US8587583||Jan 31, 2011||Nov 19, 2013||Microsoft Corporation||Three-dimensional environment reconstruction|
|US8587773||Dec 13, 2012||Nov 19, 2013||Microsoft Corporation||System architecture design for time-of-flight system having reduced differential pixel size, and time-of-flight systems so designed|
|US8588465||Dec 7, 2009||Nov 19, 2013||Microsoft Corporation||Visual target tracking|
|US8588517||Jan 15, 2013||Nov 19, 2013||Microsoft Corporation||Motion detection using depth images|
|US8592739||Nov 2, 2010||Nov 26, 2013||Microsoft Corporation||Detection of configuration changes of an optical element in an illumination system|
|US8597142||Sep 13, 2011||Dec 3, 2013||Microsoft Corporation||Dynamic camera based practice mode|
|US8605763||Mar 31, 2010||Dec 10, 2013||Microsoft Corporation||Temperature measurement and control for laser and light-emitting diodes|
|US8610665||Apr 26, 2013||Dec 17, 2013||Microsoft Corporation||Pose tracking pipeline|
|US8611607||Feb 19, 2013||Dec 17, 2013||Microsoft Corporation||Multiple centroid condensation of probability distribution clouds|
|US8613666||Aug 31, 2010||Dec 24, 2013||Microsoft Corporation||User selection and navigation based on looped motions|
|US8618405||Dec 9, 2010||Dec 31, 2013||Microsoft Corp.||Free-space gesture musical instrument digital interface (MIDI) controller|
|US8619122||Feb 2, 2010||Dec 31, 2013||Microsoft Corporation||Depth camera compatibility|
|US8620113||Apr 25, 2011||Dec 31, 2013||Microsoft Corporation||Laser diode modes|
|US8625837||Jun 16, 2009||Jan 7, 2014||Microsoft Corporation||Protocol and format for communicating an image from a camera to a computing environment|
|US8629976||Feb 4, 2011||Jan 14, 2014||Microsoft Corporation||Methods and systems for hierarchical de-aliasing time-of-flight (TOF) systems|
|US8630457||Dec 15, 2011||Jan 14, 2014||Microsoft Corporation||Problem states for pose tracking pipeline|
|US8631355||Jan 8, 2010||Jan 14, 2014||Microsoft Corporation||Assigning gesture dictionaries|
|US8633890||Feb 16, 2010||Jan 21, 2014||Microsoft Corporation||Gesture detection based on joint skipping|
|US8635637||Dec 2, 2011||Jan 21, 2014||Microsoft Corporation||User interface presenting an animated avatar performing a media reaction|
|US8638985||Mar 3, 2011||Jan 28, 2014||Microsoft Corporation||Human body pose estimation|
|US8643788 *||Apr 25, 2003||Feb 4, 2014||Sony Corporation||Image processing apparatus, image processing method, and image processing program|
|US8644609||Mar 19, 2013||Feb 4, 2014||Microsoft Corporation||Up-sampling binary images for segmentation|
|US8649554||May 29, 2009||Feb 11, 2014||Microsoft Corporation||Method to control perspective for a camera-controlled computer|
|US8655069||Mar 5, 2010||Feb 18, 2014||Microsoft Corporation||Updating image segmentation following user input|
|US8657439 *||Oct 1, 2012||Feb 25, 2014||Kenneth Martin Jacobs||Continuous adjustable 3Deeps filter spectacles for optimized 3Deeps stereoscopic viewing and its control method and means|
|US8659658||Feb 9, 2010||Feb 25, 2014||Microsoft Corporation||Physical interaction zone for gesture-based user interfaces|
|US8660303||Dec 20, 2010||Feb 25, 2014||Microsoft Corporation||Detection of body and props|
|US8660310||Dec 13, 2012||Feb 25, 2014||Microsoft Corporation||Systems and methods for tracking a model|
|US8667519||Nov 12, 2010||Mar 4, 2014||Microsoft Corporation||Automatic passive and anonymous feedback system|
|US8670029||Jun 16, 2010||Mar 11, 2014||Microsoft Corporation||Depth camera illuminator with superluminescent light-emitting diode|
|US8675981||Jun 11, 2010||Mar 18, 2014||Microsoft Corporation||Multi-modal gender recognition including depth data|
|US8676581||Jan 22, 2010||Mar 18, 2014||Microsoft Corporation||Speech recognition analysis via identification information|
|US8681255||Sep 28, 2010||Mar 25, 2014||Microsoft Corporation||Integrated low power depth camera and projection device|
|US8681321||Dec 31, 2009||Mar 25, 2014||Microsoft International Holdings B.V.||Gated 3D camera|
|US8682028||Dec 7, 2009||Mar 25, 2014||Microsoft Corporation||Visual target tracking|
|US8687044||Feb 2, 2010||Apr 1, 2014||Microsoft Corporation||Depth camera compatibility|
|US8693724||May 28, 2010||Apr 8, 2014||Microsoft Corporation||Method and system implementing user-centric gesture control|
|US8702507||Sep 20, 2011||Apr 22, 2014||Microsoft Corporation||Manual and camera-based avatar control|
|US8704843 *||Dec 15, 2010||Apr 22, 2014||Canon Kabushiki Kaisha||Image processing apparatus and image processing method|
|US8707216||Feb 26, 2009||Apr 22, 2014||Microsoft Corporation||Controlling objects via gesturing|
|US8717469||Feb 3, 2010||May 6, 2014||Microsoft Corporation||Fast gating photosurface|
|US8723118||Oct 1, 2009||May 13, 2014||Microsoft Corporation||Imager for constructing color and depth images|
|US8724887||Feb 3, 2011||May 13, 2014||Microsoft Corporation||Environmental modifications to mitigate environmental factors|
|US8724906||Nov 18, 2011||May 13, 2014||Microsoft Corporation||Computing pose and/or shape of modifiable entities|
|US8730231||Nov 19, 2008||May 20, 2014||Image Metrics, Inc.||Systems and methods for creating personalized media content having multiple content layers|
|US8730232||Feb 1, 2011||May 20, 2014||Legend3D, Inc.||Director-style based 2D to 3D movie conversion system and method|
|US8743187||Jun 6, 2012||Jun 3, 2014||Microsoft Corporation||Three-dimensional (3D) imaging based on MotionParallax|
|US8744121||May 29, 2009||Jun 3, 2014||Microsoft Corporation||Device for identifying and tracking multiple humans over time|
|US8745541||Dec 1, 2003||Jun 3, 2014||Microsoft Corporation||Architecture for controlling a computer using hand gestures|
|US8749557||Jun 11, 2010||Jun 10, 2014||Microsoft Corporation||Interacting with user interface via avatar|
|US8751215||Jun 4, 2010||Jun 10, 2014||Microsoft Corporation||Machine based sign language interpreter|
|US8760395||May 31, 2011||Jun 24, 2014||Microsoft Corporation||Gesture recognition techniques|
|US8760571||Sep 21, 2009||Jun 24, 2014||Microsoft Corporation||Alignment of lens and image sensor|
|US8762894||Feb 10, 2012||Jun 24, 2014||Microsoft Corporation||Managing virtual ports|
|US8773355||Mar 16, 2009||Jul 8, 2014||Microsoft Corporation||Adaptive cursor sizing|
|US8775916||May 17, 2013||Jul 8, 2014||Microsoft Corporation||Validation analysis of human target|
|US8781156||Sep 10, 2012||Jul 15, 2014||Microsoft Corporation||Voice-body identity correlation|
|US8782567||Nov 4, 2011||Jul 15, 2014||Microsoft Corporation||Gesture recognizer system architecture|
|US8786730||Aug 18, 2011||Jul 22, 2014||Microsoft Corporation||Image exposure using exclusion regions|
|US8787658||Mar 19, 2013||Jul 22, 2014||Microsoft Corporation||Image segmentation using reduced foreground training data|
|US8788973||May 23, 2011||Jul 22, 2014||Microsoft Corporation||Three-dimensional gesture controlled avatar configuration interface|
|US8803800||Dec 2, 2011||Aug 12, 2014||Microsoft Corporation||User interface control based on head orientation|
|US8803888||Jun 2, 2010||Aug 12, 2014||Microsoft Corporation||Recognition system for sharing information|
|US8803946 *||Nov 20, 2009||Aug 12, 2014||Sony Corporation||Image signal processing device, method and projecting device for stereoscopic images|
|US8803952||Dec 20, 2010||Aug 12, 2014||Microsoft Corporation||Plural detector time-of-flight depth mapping|
|US8810635||Apr 18, 2013||Aug 19, 2014||3Dmedia Corporation||Methods, systems, and computer-readable storage media for selecting image capture positions to generate three-dimensional images|
|US8811938||Dec 16, 2011||Aug 19, 2014||Microsoft Corporation||Providing a user interface experience based on inferred vehicle state|
|US8814691||Mar 16, 2011||Aug 26, 2014||Microsoft Corporation||System and method for social networking gaming with an augmented reality|
|US8818002||Jul 21, 2011||Aug 26, 2014||Microsoft Corp.||Robust adaptive beamforming with enhanced noise suppression|
|US8824630||Oct 25, 2011||Sep 2, 2014||Accuray Incorporated||Method and apparatus for treating a target's partial motion range|
|US8824749||Apr 5, 2011||Sep 2, 2014||Microsoft Corporation||Biometric recognition|
|US8831273||Sep 12, 2011||Sep 9, 2014||Reald Inc.||Methods and systems for pre-processing two-dimensional image files to be converted to three-dimensional image files|
|US8831285||Jul 26, 2012||Sep 9, 2014||Hewlett-Packard Development Company, L.P.||Detecting objects with a depth sensor|
|US8842730||Jan 29, 2007||Sep 23, 2014||Imax Corporation||Methods and systems for digitally re-mastering of 2D and 3D motion pictures for exhibition with enhanced visual quality|
|US8843857||Nov 19, 2009||Sep 23, 2014||Microsoft Corporation||Distance scalable no touch computing|
|US8849633||Oct 25, 2011||Sep 30, 2014||Accuray Incorporated||Method and apparatus for selecting a tracking method to use in image guided treatment|
|US8854426||Nov 7, 2011||Oct 7, 2014||Microsoft Corporation||Time-of-flight camera with guided light|
|US8856691||May 29, 2009||Oct 7, 2014||Microsoft Corporation||Gesture tool|
|US8860663||Nov 22, 2013||Oct 14, 2014||Microsoft Corporation||Pose tracking pipeline|
|US8861839||Sep 23, 2013||Oct 14, 2014||Microsoft Corporation||Human tracking system|
|US8864581||Jan 29, 2010||Oct 21, 2014||Microsoft Corporation||Visual based identitiy tracking|
|US8866889||Nov 3, 2010||Oct 21, 2014||Microsoft Corporation||In-home depth camera calibration|
|US8867820||Oct 7, 2009||Oct 21, 2014||Microsoft Corporation||Systems and methods for removing a background of an image|
|US8869072||Aug 2, 2011||Oct 21, 2014||Microsoft Corporation||Gesture recognizer system architecture|
|US8878842 *||May 24, 2012||Nov 4, 2014||Lg Display Co., Ltd.||Stereoscopic image display device and method for driving the same|
|US8879831||Dec 15, 2011||Nov 4, 2014||Microsoft Corporation||Using high-level attributes to guide image processing|
|US8882310||Dec 10, 2012||Nov 11, 2014||Microsoft Corporation||Laser die light source module with low inductance|
|US8884968||Dec 15, 2010||Nov 11, 2014||Microsoft Corporation||Modeling an object from image data|
|US8885208 *||Jul 21, 2006||Nov 11, 2014||Adobe Systems Incorporated||Progressive refinement of an edited image using secondary high resolution image processing|
|US8885890||May 7, 2010||Nov 11, 2014||Microsoft Corporation||Depth map confidence filtering|
|US8888331||May 9, 2011||Nov 18, 2014||Microsoft Corporation||Low inductance light source module|
|US8891067||Jan 31, 2011||Nov 18, 2014||Microsoft Corporation||Multiple synchronized optical sources for time-of-flight range finding systems|
|US8891827||Nov 15, 2012||Nov 18, 2014||Microsoft Corporation||Systems and methods for tracking a model|
|US8892495||Jan 8, 2013||Nov 18, 2014||Blanding Hovenweep, Llc||Adaptive pattern recognition based controller apparatus and method and human-interface therefore|
|US8896721||Jan 11, 2013||Nov 25, 2014||Microsoft Corporation||Environment and/or target segmentation|
|US8897491||Oct 19, 2011||Nov 25, 2014||Microsoft Corporation||System for finger recognition and tracking|
|US8897493||Jan 4, 2013||Nov 25, 2014||Microsoft Corporation||Body scan|
|US8897495||May 8, 2013||Nov 25, 2014||Microsoft Corporation||Systems and methods for tracking a model|
|US8898687||Apr 4, 2012||Nov 25, 2014||Microsoft Corporation||Controlling a media program based on a media reaction|
|US8902319 *||May 28, 2009||Dec 2, 2014||Sharp Kabushiki Kaisha||Image signal processing apparatus, image signal processing method, image display apparatus, television receiver, and electronic device|
|US8908091||Jun 11, 2014||Dec 9, 2014||Microsoft Corporation||Alignment of lens and image sensor|
|US8908959 *||Mar 23, 2011||Dec 9, 2014||Kabushiki Kaisha Toshiba||Image processing apparatus|
|US8913319 *||Jan 7, 2014||Dec 16, 2014||Kenneth Martin Jacobs||Continuous adjustable pulfrich filter spectacles for optimized 3DEEPS stereoscopic viewing and its control method and means|
|US8917240||Jun 28, 2013||Dec 23, 2014||Microsoft Corporation||Virtual desktop coordinate transformation|
|US8917954||Apr 9, 2013||Dec 23, 2014||Electronics And Telecommunications Research Institute||Method and apparatus for converting 2D video image into 3D video image|
|US8920241||Dec 15, 2010||Dec 30, 2014||Microsoft Corporation||Gesture controlled persistent handles for interface guides|
|US8926431||Mar 2, 2012||Jan 6, 2015||Microsoft Corporation||Visual based identity tracking|
|US8928579||Feb 22, 2010||Jan 6, 2015||Andrew David Wilson||Interacting with an omni-directionally projected display|
|US8929612||Nov 18, 2011||Jan 6, 2015||Microsoft Corporation||System for recognizing an open or closed hand|
|US8929668||Jun 28, 2013||Jan 6, 2015||Microsoft Corporation||Foreground subject detection|
|US8933884||Jan 15, 2010||Jan 13, 2015||Microsoft Corporation||Tracking groups of users in motion capture system|
|US8941919 *||Jul 16, 2014||Jan 27, 2015||Kenneth Martin Jacobs||Continuous adjustable 3Deeps filter spectacles for optimized 3Deeps stereoscopic viewing, control method and means therefor, and system and method of generating and displaying a modified video|
|US8942428||May 29, 2009||Jan 27, 2015||Microsoft Corporation||Isolate extraneous motions|
|US8942917||Feb 14, 2011||Jan 27, 2015||Microsoft Corporation||Change invariant scene recognition by an agent|
|US8953844||May 6, 2013||Feb 10, 2015||Microsoft Technology Licensing, Llc||System for fast, probabilistic skeletal tracking|
|US8959541||May 29, 2012||Feb 17, 2015||Microsoft Technology Licensing, Llc||Determining a future portion of a currently presented media program|
|US8963829||Nov 11, 2009||Feb 24, 2015||Microsoft Corporation||Methods and systems for determining and tracking extremities of a target|
|US8968091||Mar 2, 2012||Mar 3, 2015||Microsoft Technology Licensing, Llc||Scalable real-time motion recognition|
|US8970487||Oct 21, 2013||Mar 3, 2015||Microsoft Technology Licensing, Llc||Human tracking system|
|US8971612||Dec 15, 2011||Mar 3, 2015||Microsoft Corporation||Learning image processing tasks from scene reconstructions|
|US8976986||Sep 21, 2009||Mar 10, 2015||Microsoft Technology Licensing, Llc||Volume adjustment based on listener position|
|US8977037 *||Aug 31, 2012||Mar 10, 2015||Google Inc.||Methods and systems of creating a stereoscopic image|
|US8982151||Jun 14, 2010||Mar 17, 2015||Microsoft Technology Licensing, Llc||Independently processing planes of display data|
|US8983173||Jan 24, 2014||Mar 17, 2015||Cybernet Systems Corporation||Portable composable machine vision system for identifying projectiles|
|US8983233||Aug 30, 2013||Mar 17, 2015||Microsoft Technology Licensing, Llc||Time-of-flight depth imaging|
|US8988432||Nov 5, 2009||Mar 24, 2015||Microsoft Technology Licensing, Llc||Systems and methods for processing an image for target tracking|
|US8988437||Mar 20, 2009||Mar 24, 2015||Microsoft Technology Licensing, Llc||Chaining animations|
|US8988508||Sep 24, 2010||Mar 24, 2015||Microsoft Technology Licensing, Llc.||Wide angle field of view active illumination imaging system|
|US8994718||Dec 21, 2010||Mar 31, 2015||Microsoft Technology Licensing, Llc||Skeletal control of three-dimensional virtual world|
|US9001118||Aug 14, 2012||Apr 7, 2015||Microsoft Technology Licensing, Llc||Avatar construction using depth camera|
|US9007365||Nov 27, 2012||Apr 14, 2015||Legend3D, Inc.||Line depth augmentation system and method for conversion of 2D images to 3D images|
|US9007404||Mar 15, 2013||Apr 14, 2015||Legend3D, Inc.||Tilt-based look around effect image enhancement method|
|US9007417||Jul 18, 2012||Apr 14, 2015||Microsoft Technology Licensing, Llc||Body scan|
|US9008355||Jun 4, 2010||Apr 14, 2015||Microsoft Technology Licensing, Llc||Automatic depth camera aiming|
|US9013489||Nov 16, 2011||Apr 21, 2015||Microsoft Technology Licensing, Llc||Generation of avatar reflecting player appearance|
|US9015638||May 1, 2009||Apr 21, 2015||Microsoft Technology Licensing, Llc||Binding users to a gesture based system and providing feedback to the users|
|US9019201||Jan 8, 2010||Apr 28, 2015||Microsoft Technology Licensing, Llc||Evolving universal gesture sets|
|US9030532||Aug 19, 2004||May 12, 2015||Microsoft Technology Licensing, Llc||Stereoscopic image display|
|US9031103||Nov 5, 2013||May 12, 2015||Microsoft Technology Licensing, Llc||Temperature measurement and control for laser and light-emitting diodes|
|US9039528||Dec 1, 2011||May 26, 2015||Microsoft Technology Licensing, Llc||Visual target tracking|
|US9052382||Oct 18, 2013||Jun 9, 2015||Microsoft Technology Licensing, Llc||System architecture design for time-of-flight system having reduced differential pixel size, and time-of-flight systems so designed|
|US9052746||Feb 15, 2013||Jun 9, 2015||Microsoft Technology Licensing, Llc||User center-of-mass and mass distribution extraction using depth images|
|US9053562 *||Jun 16, 2011||Jun 9, 2015||Gregory S. Rabin||Two dimensional to three dimensional moving image converter|
|US9054764||Jul 20, 2011||Jun 9, 2015||Microsoft Technology Licensing, Llc||Sensor array beamformer post-processor|
|US9056254||Oct 6, 2014||Jun 16, 2015||Microsoft Technology Licensing, Llc||Time-of-flight camera with guided light|
|US9063001||Nov 2, 2012||Jun 23, 2015||Microsoft Technology Licensing, Llc||Optical fault monitoring|
|US9067136||Mar 10, 2011||Jun 30, 2015||Microsoft Technology Licensing, Llc||Push personalization of interface controls|
|US9069381||Mar 2, 2012||Jun 30, 2015||Microsoft Technology Licensing, Llc||Interacting with a computer based application|
|US9070222||Mar 13, 2013||Jun 30, 2015||Dreamworks Animation Llc||Techniques for automating stereo settings for stereoscopic computer animation|
|US9075434||Aug 20, 2010||Jul 7, 2015||Microsoft Technology Licensing, Llc||Translating user motion into multiple object responses|
|US9076262 *||Mar 13, 2013||Jul 7, 2015||Dreamworks Animation Llc||Scripted stereo curves for stereoscopic computer animation|
|US9082018 *||Oct 8, 2014||Jul 14, 2015||Google Inc.||Method and system for retroactively changing a display characteristic of event indicators on an event timeline|
|US9087406||Mar 13, 2013||Jul 21, 2015||Dreamworks Animation Llc||Automated stereoscopic computer-animation techniques for determining scaled stereo parameters|
|US9091851||Jan 25, 2012||Jul 28, 2015||Microsoft Technology Licensing, Llc||Light control in head mounted displays|
|US9092657||Mar 13, 2013||Jul 28, 2015||Microsoft Technology Licensing, Llc||Depth image processing|
|US9097890||Mar 25, 2012||Aug 4, 2015||Microsoft Technology Licensing, Llc||Grating in a light transmissive illumination system for see-through near-eye display glasses|
|US9097891||Mar 26, 2012||Aug 4, 2015||Microsoft Technology Licensing, Llc||See-through near-eye display glasses including an auto-brightness control for the display brightness based on the brightness in the environment|
|US9098110||Aug 18, 2011||Aug 4, 2015||Microsoft Technology Licensing, Llc||Head rotation tracking from depth-based center of mass|
|US9098493||Apr 24, 2014||Aug 4, 2015||Microsoft Technology Licensing, Llc||Machine based sign language interpreter|
|US9098873||Apr 1, 2010||Aug 4, 2015||Microsoft Technology Licensing, Llc||Motion-based interactive shopping environment|
|US9100685||Dec 9, 2011||Aug 4, 2015||Microsoft Technology Licensing, Llc||Determining audience state or interest using passive sensor data|
|US9106900||Mar 8, 2013||Aug 11, 2015||Fujifilm Corporation||Stereoscopic imaging device and stereoscopic imaging method|
|US9117281||Nov 2, 2011||Aug 25, 2015||Microsoft Corporation||Surface segmentation from RGB and depth images|
|US9123316||Dec 27, 2010||Sep 1, 2015||Microsoft Technology Licensing, Llc||Interactive content creation|
|US9128281||Sep 14, 2011||Sep 8, 2015||Microsoft Technology Licensing, Llc||Eyepiece with uniformly illuminated reflective display|
|US9129295||Mar 26, 2012||Sep 8, 2015||Microsoft Technology Licensing, Llc||See-through near-eye display glasses with a fast response photochromic film system for quick transition from dark to clear|
|US9129436||Mar 13, 2013||Sep 8, 2015||Dreamworks Animation Llc||Techniques for smoothing scripted stereo curves for stereoscopic computer animation|
|US9134534||Mar 26, 2012||Sep 15, 2015||Microsoft Technology Licensing, Llc||See-through near-eye display glasses including a modular image source|
|US9135516||Mar 8, 2013||Sep 15, 2015||Microsoft Technology Licensing, Llc||User body angle, curvature and average extremity positions extraction using depth images|
|US9135715 *||Mar 22, 2012||Sep 15, 2015||Google Inc.||Local feature cameras for structure from motion (SFM) problems with generalized cameras|
|US9137463||May 12, 2011||Sep 15, 2015||Microsoft Technology Licensing, Llc||Adaptive high dynamic range camera|
|US9141193||Aug 31, 2009||Sep 22, 2015||Microsoft Technology Licensing, Llc||Techniques for using human gestures to control gesture unaware programs|
|US9147253||Jun 19, 2012||Sep 29, 2015||Microsoft Technology Licensing, Llc||Raster scanning for depth detection|
|US9154837||Dec 16, 2013||Oct 6, 2015||Microsoft Technology Licensing, Llc||User interface presenting an animated avatar performing a media reaction|
|US9158974||Oct 8, 2014||Oct 13, 2015||Google Inc.||Method and system for motion vector-based video monitoring and event categorization|
|US9159151||Jul 13, 2009||Oct 13, 2015||Microsoft Technology Licensing, Llc||Bringing a visual representation to life via learned input from the user|
|US9170707||Oct 8, 2014||Oct 27, 2015||Google Inc.||Method and system for generating a smart time-lapse video clip|
|US9171264||Dec 15, 2010||Oct 27, 2015||Microsoft Technology Licensing, Llc||Parallel processing machine learning decision tree training|
|US9182596||Mar 26, 2012||Nov 10, 2015||Microsoft Technology Licensing, Llc||See-through near-eye display glasses with the optical assembly including absorptive polarizers or anti-reflective coatings to reduce stray light|
|US9182814||Jun 26, 2009||Nov 10, 2015||Microsoft Technology Licensing, Llc||Systems and methods for estimating a non-visible or occluded body part|
|US9185388||Nov 3, 2011||Nov 10, 2015||3Dmedia Corporation||Methods, systems, and computer program products for creating three-dimensional video sequences|
|US9191570||Aug 5, 2013||Nov 17, 2015||Microsoft Technology Licensing, Llc||Systems and methods for detecting a tilt angle from a depth image|
|US9195305||Nov 8, 2012||Nov 24, 2015||Microsoft Technology Licensing, Llc||Recognizing user intent in motion capture system|
|US9208571||Mar 2, 2012||Dec 8, 2015||Microsoft Technology Licensing, Llc||Object digitization|
|US9210401||May 3, 2012||Dec 8, 2015||Microsoft Technology Licensing, Llc||Projected visual cues for guiding physical movement|
|US9213903||Oct 8, 2014||Dec 15, 2015||Google Inc.||Method and system for cluster-based video monitoring and event categorization|
|US9215478||Nov 27, 2013||Dec 15, 2015||Microsoft Technology Licensing, Llc||Protocol and format for communicating an image from a camera to a computing environment|
|US9223134||Mar 25, 2012||Dec 29, 2015||Microsoft Technology Licensing, Llc||Optical imperfections in a light transmissive illumination system for see-through near-eye display glasses|
|US9224044||Oct 8, 2014||Dec 29, 2015||Google Inc.||Method and system for video zone monitoring|
|US9229227||Mar 25, 2012||Jan 5, 2016||Microsoft Technology Licensing, Llc||See-through near-eye display glasses with a light transmissive wedge shaped illumination system|
|US9230315 *||Dec 8, 2010||Jan 5, 2016||Thomson Licensing||Complexity estimation of a 2D/3D conversion|
|US9241147||May 1, 2013||Jan 19, 2016||Legend3D, Inc.||External depth map transformation method for conversion of two-dimensional images to stereoscopic images|
|US9242171||Feb 23, 2013||Jan 26, 2016||Microsoft Technology Licensing, Llc||Real-time camera tracking using depth maps|
|US9244533||Dec 17, 2009||Jan 26, 2016||Microsoft Technology Licensing, Llc||Camera navigation for presentations|
|US9247238||Jan 31, 2011||Jan 26, 2016||Microsoft Technology Licensing, Llc||Reducing interference between multiple infra-red depth cameras|
|US9251590||Jan 24, 2013||Feb 2, 2016||Microsoft Technology Licensing, Llc||Camera pose estimation for 3D reconstruction|
|US9256282||Mar 20, 2009||Feb 9, 2016||Microsoft Technology Licensing, Llc||Virtual object manipulation|
|US9259643||Sep 20, 2011||Feb 16, 2016||Microsoft Technology Licensing, Llc||Control of separate computer game elements|
|US9262673||May 24, 2013||Feb 16, 2016||Microsoft Technology Licensing, Llc||Human body pose estimation|
|US9264807||Jan 23, 2013||Feb 16, 2016||Microsoft Technology Licensing, Llc||Multichannel acoustic echo reduction|
|US9268404||Jan 8, 2010||Feb 23, 2016||Microsoft Technology Licensing, Llc||Application gesture interpretation|
|US9270965||Aug 17, 2015||Feb 23, 2016||Legend 3D, Inc.||Multi-stage production pipeline system|
|US9274606||Mar 14, 2013||Mar 1, 2016||Microsoft Technology Licensing, Llc||NUI video conference controls|
|US9274747||Feb 19, 2013||Mar 1, 2016||Microsoft Technology Licensing, Llc||Natural user input for driving interactive stories|
|US9278287||Oct 20, 2014||Mar 8, 2016||Microsoft Technology Licensing, Llc||Visual based identity tracking|
|US9280203||Aug 2, 2011||Mar 8, 2016||Microsoft Technology Licensing, Llc||Gesture recognizer system architecture|
|US9282313||Dec 14, 2012||Mar 8, 2016||Imax Corporation||Methods and systems for converting 2D motion pictures for stereoscopic 3D exhibition|
|US9282316 *||Mar 7, 2013||Mar 8, 2016||Fujifilm Corporation||Stereoscopic imaging device and stereoscopic imaging method|
|US9282321||Aug 17, 2015||Mar 8, 2016||Legend3D, Inc.||3D model multi-reviewer system|
|US9285589||Jan 3, 2012||Mar 15, 2016||Microsoft Technology Licensing, Llc||AR glasses with event and sensor triggered control of AR eyepiece applications|
|US9286658 *||Mar 22, 2012||Mar 15, 2016||Qualcomm Incorporated||Image enhancement|
|US9286941||May 11, 2015||Mar 15, 2016||Legend3D, Inc.||Image sequence enhancement and motion picture project management system|
|US9288476||Aug 17, 2015||Mar 15, 2016||Legend3D, Inc.||System and method for real-time depth modification of stereo images of a virtual reality environment|
|US9291449||Nov 25, 2013||Mar 22, 2016||Microsoft Technology Licensing, Llc||Detection of configuration changes among optical elements of illumination system|
|US9292083||May 29, 2014||Mar 22, 2016||Microsoft Technology Licensing, Llc||Interacting with user interface via avatar|
|US9298263||Oct 27, 2010||Mar 29, 2016||Microsoft Technology Licensing, Llc||Show body position|
|US9298287||Mar 31, 2011||Mar 29, 2016||Microsoft Technology Licensing, Llc||Combined activation for natural user interface systems|
|US9305398 *||Oct 8, 2010||Apr 5, 2016||City University Of Hong Kong||Methods for creating and displaying two and three dimensional images on a digital canvas|
|US9311560||Aug 12, 2015||Apr 12, 2016||Microsoft Technology Licensing, Llc||Extraction of user behavior from depth images|
|US9313376||Apr 1, 2009||Apr 12, 2016||Microsoft Technology Licensing, Llc||Dynamic depth power equalization|
|US9329689||Mar 16, 2011||May 3, 2016||Microsoft Technology Licensing, Llc||Method and apparatus for biometric data capture|
|US9338426 *||Oct 26, 2011||May 10, 2016||Panasonic Intellectual Property Management Co., Ltd.||Three-dimensional image processing apparatus, three-dimensional imaging apparatus, and three-dimensional image processing method|
|US9341843||Mar 26, 2012||May 17, 2016||Microsoft Technology Licensing, Llc||See-through near-eye display glasses with a small scale image source|
|US9342139||Dec 19, 2011||May 17, 2016||Microsoft Technology Licensing, Llc||Pairing a computing device to a user|
|US9342817||Aug 29, 2011||May 17, 2016||Sony Interactive Entertainment LLC||Auto-creating groups for sharing photos|
|US9344701||Dec 27, 2011||May 17, 2016||3Dmedia Corporation||Methods, systems, and computer-readable storage media for identifying a rough depth map in a scene and for determining a stereo-base distance for three-dimensional (3D) content creation|
|US9349040||Nov 19, 2010||May 24, 2016||Microsoft Technology Licensing, Llc||Bi-modal depth-image analysis|
|US9354794||Oct 8, 2014||May 31, 2016||Google Inc.||Method and system for performing client-side zooming of a remote video feed|
|US9366862||Mar 26, 2012||Jun 14, 2016||Microsoft Technology Licensing, Llc||System and method for delivering content to a group of see-through near eye display eyepieces|
|US9372544||May 16, 2014||Jun 21, 2016||Microsoft Technology Licensing, Llc||Gesture recognition techniques|
|US9377857||May 1, 2009||Jun 28, 2016||Microsoft Technology Licensing, Llc||Show body position|
|US9380292||May 25, 2011||Jun 28, 2016||3Dmedia Corporation||Methods, systems, and computer-readable storage media for generating three-dimensional (3D) images of a scene|
|US9383823||May 29, 2009||Jul 5, 2016||Microsoft Technology Licensing, Llc||Combining gestures beyond skeletal|
|US9384329||Jun 11, 2010||Jul 5, 2016||Microsoft Technology Licensing, Llc||Caloric burn determination from body movement|
|US9386294||Jan 4, 2012||Jul 5, 2016||Google Technology Holdings LLC||Method and apparatus for 3DTV image adjustment|
|US9400548||Oct 19, 2009||Jul 26, 2016||Microsoft Technology Licensing, Llc||Gesture personalization and profile roaming|
|US9400559||May 29, 2009||Jul 26, 2016||Microsoft Technology Licensing, Llc||Gesture shortcuts|
|US9407904||May 11, 2015||Aug 2, 2016||Legend3D, Inc.||Method for creating 3D virtual reality from 2D images|
|US9420331||Oct 8, 2014||Aug 16, 2016||Google Inc.||Method and system for categorizing detected motion events|
|US9424634||Mar 15, 2013||Aug 23, 2016||Cybernet Systems Corporation||Machine vision system for identifying and sorting projectiles and other objects|
|US9438878||Sep 17, 2015||Sep 6, 2016||Legend3D, Inc.||Method of converting 2D video to 3D video using 3D object models|
|US9442186||Oct 16, 2013||Sep 13, 2016||Microsoft Technology Licensing, Llc||Interference reduction for TOF systems|
|US9443310||Oct 9, 2013||Sep 13, 2016||Microsoft Technology Licensing, Llc||Illumination modules that emit structured light|
|US9443338||Mar 14, 2013||Sep 13, 2016||Dreamworks Animation Llc||Techniques for producing baseline stereo parameters for stereoscopic computer animation|
|US9443555||Feb 22, 2016||Sep 13, 2016||Legend3D, Inc.||Multi-stage production pipeline system|
|US9449229||Jun 12, 2015||Sep 20, 2016||Google Inc.||Systems and methods for categorizing motion event candidates|
|US9454244||May 7, 2008||Sep 27, 2016||Microsoft Technology Licensing, Llc||Recognizing a movement of a pointing device|
|US9462253||Sep 23, 2013||Oct 4, 2016||Microsoft Technology Licensing, Llc||Optical modules that reduce speckle contrast and diffraction artifacts|
|US9465980||Sep 5, 2014||Oct 11, 2016||Microsoft Technology Licensing, Llc||Pose tracking pipeline|
|US9468848||Dec 12, 2013||Oct 18, 2016||Microsoft Technology Licensing, Llc||Assigning gesture dictionaries|
|US9470778||Mar 29, 2011||Oct 18, 2016||Microsoft Technology Licensing, Llc||Learning from high quality depth measurements|
|US9478057||Feb 9, 2015||Oct 25, 2016||Microsoft Technology Licensing, Llc||Chaining animations|
|US9479822||Jun 3, 2016||Oct 25, 2016||Google Inc.||Method and system for categorizing detected motion events|
|US9483836||Jun 6, 2013||Nov 1, 2016||Sony Corporation||Method and apparatus for real-time conversion of 2-dimensional content to 3-dimensional content|
|US9484065||Oct 15, 2010||Nov 1, 2016||Microsoft Technology Licensing, Llc||Intelligent determination of replays based on event identification|
|US9485497||Sep 12, 2011||Nov 1, 2016||Reald Inc.||Systems and methods for converting two-dimensional images into three-dimensional images|
|US9489053||Feb 26, 2015||Nov 8, 2016||Microsoft Technology Licensing, Llc||Skeletal control of three-dimensional virtual world|
|US9489580||Sep 24, 2015||Nov 8, 2016||Google Inc.||Method and system for cluster-based video monitoring and event categorization|
|US9491226||Aug 1, 2014||Nov 8, 2016||Microsoft Technology Licensing, Llc||Recognition system for sharing information|
|US9498718||May 29, 2009||Nov 22, 2016||Microsoft Technology Licensing, Llc||Altering a view perspective within a display environment|
|US9501915||Jun 12, 2015||Nov 22, 2016||Google Inc.||Systems and methods for analyzing a video stream|
|US9508385||Nov 21, 2013||Nov 29, 2016||Microsoft Technology Licensing, Llc||Audio-visual project generator|
|US9519828||Dec 22, 2014||Dec 13, 2016||Microsoft Technology Licensing, Llc||Isolate extraneous motions|
|US9519970||Oct 9, 2015||Dec 13, 2016||Microsoft Technology Licensing, Llc||Systems and methods for detecting a tilt angle from a depth image|
|US9519989||Mar 4, 2013||Dec 13, 2016||Microsoft Technology Licensing, Llc||Visual representation expression based on player expression|
|US9522328||Sep 4, 2014||Dec 20, 2016||Microsoft Technology Licensing, Llc||Human tracking system|
|US9524024||Jan 21, 2014||Dec 20, 2016||Microsoft Technology Licensing, Llc||Method to control perspective for a camera-controlled computer|
|US9529566||Aug 31, 2015||Dec 27, 2016||Microsoft Technology Licensing, Llc||Interactive content creation|
|US9535563||Nov 12, 2013||Jan 3, 2017||Blanding Hovenweep, Llc||Internet appliance system and method|
|US9539500||Aug 5, 2014||Jan 10, 2017||Microsoft Technology Licensing, Llc||Biometric recognition|
|US9544636||Oct 8, 2014||Jan 10, 2017||Google Inc.||Method and system for editing event categories|
|US9547937||Nov 30, 2012||Jan 17, 2017||Legend3D, Inc.||Three-dimensional annotation system and method|
|US9551914||Mar 7, 2011||Jan 24, 2017||Microsoft Technology Licensing, Llc||Illuminator with refractive optical element|
|US9552514 *||Aug 18, 2014||Jan 24, 2017||Ricoh Company, Ltd.||Moving object detection method and system|
|US9557574||Jun 8, 2010||Jan 31, 2017||Microsoft Technology Licensing, Llc||Depth illumination and detection optics|
|US9557836||Nov 1, 2011||Jan 31, 2017||Microsoft Technology Licensing, Llc||Depth image compression|
|US9560274||Nov 7, 2011||Jan 31, 2017||Sony Corporation||Image generation apparatus and image generation method|
|US9569005||Apr 3, 2014||Feb 14, 2017||Microsoft Technology Licensing, Llc||Method and system implementing user-centric gesture control|
|US9582717||Oct 27, 2014||Feb 28, 2017||Microsoft Technology Licensing, Llc||Systems and methods for tracking a model|
|US9582918||Mar 14, 2013||Feb 28, 2017||Dreamworks Animation Llc||Techniques for producing creative stereo parameters for stereoscopic computer animation|
|US9594430||Jun 1, 2011||Mar 14, 2017||Microsoft Technology Licensing, Llc||Three-dimensional foreground selection for vision system|
|US9595296||Sep 12, 2016||Mar 14, 2017||Legend3D, Inc.||Multi-stage production pipeline system|
|US9596643||Jul 15, 2014||Mar 14, 2017||Microsoft Technology Licensing, Llc||Providing a user interface experience based on inferred vehicle state|
|US9597587||Jun 8, 2011||Mar 21, 2017||Microsoft Technology Licensing, Llc||Locational node device|
|US9602860||Jul 5, 2016||Mar 21, 2017||Google Inc.||Method and system for displaying recorded and live video feeds|
|US9607213||Mar 16, 2015||Mar 28, 2017||Microsoft Technology Licensing, Llc||Body scan|
|US9609307||Dec 14, 2015||Mar 28, 2017||Legend3D, Inc.||Method of converting 2D video to 3D video using machine learning|
|US9609380||Jul 5, 2016||Mar 28, 2017||Google Inc.||Method and system for detecting and presenting a new event in a video feed|
|US9615082||Mar 14, 2016||Apr 4, 2017||Legend3D, Inc.||Image sequence enhancement and motion picture project management system and method|
|US9619561||Nov 10, 2014||Apr 11, 2017||Microsoft Technology Licensing, Llc||Change invariant scene recognition by an agent|
|US9628844||Jul 31, 2015||Apr 18, 2017||Microsoft Technology Licensing, Llc||Determining audience state or interest using passive sensor data|
|US9641825||Feb 12, 2014||May 2, 2017||Microsoft International Holdings B.V.||Gated 3D camera|
|US9646340||Aug 2, 2012||May 9, 2017||Microsoft Technology Licensing, Llc||Avatar-based virtual dressing room|
|US9652042||Feb 12, 2010||May 16, 2017||Microsoft Technology Licensing, Llc||Architecture for controlling a computer using hand gestures|
|US9656162||Apr 14, 2014||May 23, 2017||Microsoft Technology Licensing, Llc||Device for identifying and tracking multiple humans over time|
|US9659377||Dec 15, 2014||May 23, 2017||Microsoft Technology Licensing, Llc||Methods and systems for determining and tracking extremities of a target|
|US9672427||Oct 25, 2016||Jun 6, 2017||Google Inc.||Systems and methods for categorizing motion events|
|US9674563||Nov 4, 2013||Jun 6, 2017||Rovi Guides, Inc.||Systems and methods for recommending content|
|US9674570||Jul 6, 2016||Jun 6, 2017||Google Inc.||Method and system for detecting and presenting video feed|
|US20020009211 *||May 7, 2001||Jan 24, 2002||Braspenning Ralph Antonius Cornelius||Method, system and apparatus|
|US20020048389 *||Aug 28, 2001||Apr 25, 2002||Yoshio Komaki||Motion image processor, motion image processing method and recording medium|
|US20020085096 *||May 15, 2001||Jul 4, 2002||Mertens Mark Jozef Willem||Foreground/background detector|
|US20030058242 *||Sep 7, 2001||Mar 27, 2003||Redlich Arthur Norman||Method and system for 3-D content creation|
|US20030071905 *||Oct 11, 2002||Apr 17, 2003||Ryo Yamasaki||Image processing apparatus and method, control program, and storage medium|
|US20030098907 *||Apr 1, 2001||May 29, 2003||Rolf-Dieter Naske||Method and device for generating 3d images|
|US20030103136 *||Dec 5, 2001||Jun 5, 2003||Koninklijke Philips Electronics N.V.||Method and system for 2D/3D illusion generation|
|US20030128871 *||Jan 8, 2003||Jul 10, 2003||Rolf-Dieter Naske||Methods and systems for 2D/3D image conversion and optimization|
|US20030147555 *||Jul 6, 2001||Aug 7, 2003||Miroslaw Bober||Method and apparatus for representing and searching for an object in an image|
|US20030202585 *||Nov 30, 1999||Oct 30, 2003||Shuichi Watanabe||Image retrieving apparatus performing retrieval based on coding information utilized for feature frame extraction or feature values of frames|
|US20040022439 *||Jul 30, 2002||Feb 5, 2004||Paul Beardsley||Wheelchair detection using stereo vision|
|US20040027346 *||May 29, 2003||Feb 12, 2004||Rolf-Deiter Naske||Method and device for the generation of 3-D images|
|US20040032489 *||Aug 13, 2002||Feb 19, 2004||Tyra Donald Wayne||Method for displaying a visual element of a scene|
|US20040047513 *||Jun 13, 2002||Mar 11, 2004||Tetsujiro Kondo||Image processing apparatus and method, and image pickup apparatus|
|US20040075813 *||Oct 16, 2002||Apr 22, 2004||Yutaka Mizukusa||Unit for obtaining and displaying fundus image|
|US20040130680 *||Mar 13, 2003||Jul 8, 2004||Samuel Zhou||Systems and methods for digitally re-mastering or otherwise modifying motion pictures or other image sequences data|
|US20050024379 *||Aug 26, 2004||Feb 3, 2005||Marks Richard L.||Method for color transition detection|
|US20050041156 *||Apr 25, 2003||Feb 24, 2005||Tetsujiro Kondo||Image processing apparatus, image processing method, and image processing program|
|US20050062678 *||Sep 24, 2004||Mar 24, 2005||Mark Resources, Llc||Autostereoscopic display system|
|US20050131659 *||Dec 16, 2003||Jun 16, 2005||Mei Hsaio L.S.||Systems and methods for 3D modeling and asset management|
|US20050146521 *||Mar 2, 2004||Jul 7, 2005||Kaye Michael C.||Method for creating and presenting an accurate reproduction of three-dimensional images converted from two-dimensional images|
|US20050226489 *||Mar 4, 2005||Oct 13, 2005||Glenn Beach||Machine vision system for identifying and sorting projectiles and other objects|
|US20050231505 *||Jun 30, 2004||Oct 20, 2005||Kaye Michael C||Method for creating artifact free three-dimensional images converted from two-dimensional images|
|US20050244050 *||Apr 24, 2003||Nov 3, 2005||Toshio Nomura||Image data creation device, image data reproduction device, and image data recording medium|
|US20050270418 *||May 17, 2005||Dec 8, 2005||Makoto Kondo||Image processing device and image processing method|
|US20060033074 *||Oct 20, 2005||Feb 16, 2006||Basf Aktiengesellschaft||Coolant comprising azole derivatives for cooling systems in fuel-cell drives|
|US20060038880 *||Aug 19, 2004||Feb 23, 2006||Microsoft Corporation||Stereoscopic image display|
|US20060038881 *||Sep 23, 2004||Feb 23, 2006||Microsoft Corporation||Stereoscopic image display|
|US20060088206 *||Aug 23, 2005||Apr 27, 2006||Kazunari Era||Image processing apparatus, image pickup device and program therefor|
|US20060092272 *||May 26, 2003||May 4, 2006||Karni Wolowelsky||Method for producing stereoscopic images from monoscopic images|
|US20060139750 *||Feb 18, 2006||Jun 29, 2006||Solomon Dennis J||Performance display system|
|US20060192174 *||Mar 30, 2006||Aug 31, 2006||Basf Aktiengesellschaft||Coolant comprising azole derivatives for cooling systems in fuel-cell drives|
|US20060210194 *||Apr 5, 2006||Sep 21, 2006||Microsoft Corporation||Method and apparatus for reducing image acquisition time in a digital imaging device|
|US20060219975 *||Jun 8, 2006||Oct 5, 2006||Basf Aktiengesellschaft||Coolant comprising azole derivatives for cooling systems in fuel-cell drives|
|US20070035831 *||Jun 19, 2006||Feb 15, 2007||Gutierrez Novelo Manuel R||3D Videogame system|
|US20070097103 *||Sep 9, 2004||May 3, 2007||Shoji Yoshioka||Portable display device|
|US20070269136 *||Aug 7, 2007||Nov 22, 2007||Newsight Corp.||Method and device for generating 3d images|
|US20080013834 *||Jul 12, 2007||Jan 17, 2008||Mitsubishi Denki Kabushiki Kaisha||Method and apparatus for representing and searching for an object in an image|
|US20080085049 *||Aug 7, 2007||Apr 10, 2008||Rolf-Dieter Naske||Methods and systems for 2d/3d image conversion and optimization|
|US20080106546 *||Dec 13, 2007||May 8, 2008||Newsight Corporation||Method and device for generating 3d images|
|US20080129819 *||Feb 4, 2008||Jun 5, 2008||Mark Resources, Llc||Autostereoscopic display system|
|US20080152213 *||Jul 25, 2007||Jun 26, 2008||Clone Interactive||3d face reconstruction from 2d images|
|US20080170067 *||Jul 31, 2007||Jul 17, 2008||Samsung Electronics Co., Ltd.||Image processing method and apparatus|
|US20080187177 *||Apr 3, 2008||Aug 7, 2008||Miroslaw Bober||Method and apparatus for representing and searching for an object in an image|
|US20080187226 *||Apr 3, 2008||Aug 7, 2008||Miroslaw Bober||Method and apparatus for representing and searching for an object in an image|
|US20080187227 *||Apr 3, 2008||Aug 7, 2008||Miroslaw Bober||Method and apparatus for representing and searching for an object in an image|
|US20080253683 *||Apr 15, 2008||Oct 16, 2008||Satoshi Nakamura||Image processing apparatus, method and program|
|US20080278569 *||Jan 12, 2006||Nov 13, 2008||Efrat Rotem||Automatic Conversion from Monoscopic Video to Stereoscopic Video|
|US20080309660 *||Jun 12, 2007||Dec 18, 2008||Microsoft Corporation||Three dimensional rendering of display information|
|US20090073170 *||Oct 21, 2005||Mar 19, 2009||Koninklijke Philips Electronics, N.V.||Disparity map|
|US20090116732 *||Jun 25, 2007||May 7, 2009||Samuel Zhou||Methods and systems for converting 2d motion pictures for stereoscopic 3d exhibition|
|US20090132371 *||Nov 19, 2008||May 21, 2009||Big Stage Entertainment, Inc.||Systems and methods for interactive advertising using personalized head models|
|US20090135177 *||Nov 19, 2008||May 28, 2009||Big Stage Entertainment, Inc.||Systems and methods for voice personalization of video content|
|US20090180678 *||Aug 27, 2008||Jul 16, 2009||Gopinath Kuduvalli||Use of a single x-ray image in a stereo imaging pair for quality assurance of tracking|
|US20090316923 *||Jun 19, 2008||Dec 24, 2009||Microsoft Corporation||Multichannel acoustic echo reduction|
|US20100020160 *||Jul 5, 2007||Jan 28, 2010||James Amachi Ashbey||Stereoscopic Motion Picture|
|US20100021145 *||Jun 22, 2009||Jan 28, 2010||Panasonic Corporation||Recording medium, playback apparatus, integrated circuit, playback method, and program|
|US20100128106 *||Nov 20, 2009||May 27, 2010||Hideo Tomita||Image signal processing device, image signal processing method and image projection device|
|US20100151944 *||Feb 22, 2010||Jun 17, 2010||Manuel Rafael Gutierrez Novelo||3d videogame system|
|US20100156897 *||Dec 18, 2009||Jun 24, 2010||3D Fusion Inc.||System and Method For Adaptive Scalable Dynamic Conversion, Quality and Processing Optimization, Enhancement, Correction, Mastering, And Other Advantageous Processing of Three Dimensional Media Content|
|US20100188584 *||Jul 28, 2009||Jul 29, 2010||Industrial Technology Research Institute||Depth calculating method for two dimensional video and apparatus thereof|
|US20100225743 *||Mar 5, 2009||Sep 9, 2010||Microsoft Corporation||Three-Dimensional (3D) Imaging Based on MotionParallax|
|US20100239175 *||Jun 2, 2010||Sep 23, 2010||Mitsubushi Denki Kabushiki Kaisha||Method and apparatus for representing and searching for an object in an image|
|US20100284606 *||Sep 21, 2009||Nov 11, 2010||Chunghwa Picture Tubes, Ltd.||Image processing device and method thereof|
|US20100302145 *||Jun 1, 2009||Dec 2, 2010||Microsoft Corporation||Virtual desktop coordinate transformation|
|US20110025825 *||Jul 23, 2010||Feb 3, 2011||3Dmedia Corporation||Methods, systems, and computer-readable storage media for creating three-dimensional (3d) images of a scene|
|US20110025830 *||Jul 23, 2010||Feb 3, 2011||3Dmedia Corporation||Methods, systems, and computer-readable storage media for generating stereoscopic content via depth map creation|
|US20110096832 *||Jan 29, 2010||Apr 28, 2011||Qualcomm Incorporated||Depth map generation techniques for conversion of 2d video data to 3d video data|
|US20110128449 *||May 28, 2009||Jun 2, 2011||Sharp Kabushiki Kaisha||IMAGE SIGNAL PROCESSING APPARATUS, IMAGE SIGNAL PROCESSING METHOD, IMAGE DISPLAY APPARATUS, TELEVISION RECEIVER, AND ELECTRONIC DEVICE (amended|
|US20110157209 *||Dec 15, 2010||Jun 30, 2011||Canon Kabushiki Kaisha||Image processing apparatus and image processing method|
|US20110188027 *||Jan 31, 2011||Aug 4, 2011||Microsoft Corporation||Multiple synchronized optical sources for time-of-flight range finding systems|
|US20110188028 *||Feb 4, 2011||Aug 4, 2011||Microsoft Corporation||Methods and systems for hierarchical de-aliasing time-of-flight (tof) systems|
|US20110197161 *||Feb 9, 2010||Aug 11, 2011||Microsoft Corporation||Handles interactions for human-computer interface|
|US20110221755 *||Mar 12, 2010||Sep 15, 2011||Kevin Geisner||Bionic motion|
|US20110228251 *||Mar 17, 2010||Sep 22, 2011||Microsoft Corporation||Raster scanning for depth detection|
|US20110228976 *||Mar 19, 2010||Sep 22, 2011||Microsoft Corporation||Proxy training data for human body tracking|
|US20120068996 *||Sep 21, 2010||Mar 22, 2012||Sony Corporation||Safe mode transition in 3d content rendering|
|US20120069143 *||Sep 20, 2010||Mar 22, 2012||Joseph Yao Hua Chu||Object tracking and highlighting in stereoscopic images|
|US20120070069 *||Mar 23, 2011||Mar 22, 2012||Kabushiki Kaisha Toshiba||Image processing apparatus|
|US20120086705 *||Oct 8, 2010||Apr 12, 2012||City University Of Hong Kong||Methods for creating and displaying two and three dimensional images on a digital canvas|
|US20120264515 *||Jun 21, 2012||Oct 18, 2012||Tdvision Corporation S.A. De C.V.||3d videogame system|
|US20130027390 *||May 24, 2012||Jan 31, 2013||Suhyung Kim||Stereoscopic image display device and method for driving the same|
|US20130038688 *||Oct 1, 2012||Feb 14, 2013||Kenneth Martin Jacobs||Continuous adjustable 3deeps filter spectacles for optimized 3deeps stereoscopic viewing and its control method and means|
|US20130083993 *||Sep 11, 2012||Apr 4, 2013||Sony Corporation||Image processing device, image processing method, and program|
|US20130120370 *||Nov 14, 2012||May 16, 2013||Samsung Electronics Co., Ltd.||Method and apparatus for measuring asthenopia of three dimensional image|
|US20130162678 *||Jul 21, 2006||Jun 27, 2013||Jerry G. Harris||Progressive refinement of an edited image using secondary high resolution image processing|
|US20130182082 *||Mar 7, 2013||Jul 18, 2013||Fujifilm Corporation||Stereoscopic imaging device and stereoscopic imaging method|
|US20130208088 *||Oct 26, 2011||Aug 15, 2013||Yasunori Ishii||Three-dimensional image processing apparatus, three-dimensional imaging apparatus, and three-dimensional image processing method|
|US20130251281 *||Mar 22, 2012||Sep 26, 2013||Qualcomm Incorporated||Image enhancement|
|US20130259318 *||Dec 8, 2010||Oct 3, 2013||Thomson Licensing||Complexity estimation of a 2d/3d conversion|
|US20140036036 *||Mar 13, 2013||Feb 6, 2014||Dreamworks Animation Llc||Scripted stereo curves for stereoscopic computer animation|
|US20150055828 *||Aug 18, 2014||Feb 26, 2015||Ricoh Company, Ltd.||Moving object detection method and system|
|US20160092737 *||May 28, 2015||Mar 31, 2016||Google Inc.||Method and System for Adding Event Indicators to an Event Timeline|
|USD782495||Oct 7, 2014||Mar 28, 2017||Google Inc.||Display screen or portion thereof with graphical user interface|
|CN101952837B||Dec 11, 2008||Apr 23, 2014||艾可瑞公司||Use of a single x-ray image in a stereo imaging pair for quality assurance of tracking|
|EP1501317A1 *||Apr 24, 2003||Jan 26, 2005||Sharp Corporation||Image data creation device, image data reproduction device, and image data recording medium|
|WO2005084298A2 *||Feb 28, 2005||Sep 15, 2005||In-Three, Inc.||Method for creating and presenting an accurate reproduction of three-dimensional images converted from two-dimensional images|
|WO2005084298A3 *||Feb 28, 2005||Jul 27, 2006||In Three Inc||Method for creating and presenting an accurate reproduction of three-dimensional images converted from two-dimensional images|
|WO2008013575A3 *||Jan 31, 2007||Jun 19, 2008||Fidaleo Douglas||3d face reconstruction from 2d images|
|WO2009091382A1 *||Dec 11, 2008||Jul 23, 2009||Accuray Incorporated||Use of a single x-ray image in a stereo imaging pair for quality assurance of tracking|
|WO2012109102A3 *||Feb 3, 2012||Nov 15, 2012||Microsoft Corporation||Three-dimensional display with motion parallax|
|U.S. Classification||382/154, 348/E13.04, 348/E13.071, 348/E13.022, 348/E13.067, 348/E13.004, 348/E13.037, 348/E13.029, 382/103, 348/E13.041, 382/107, 348/E13.02, 348/E14.054, 348/E13.062, 348/E13.038, 382/232, 348/E13.058|
|International Classification||G06T17/40, H04N13/00|
|Cooperative Classification||H04N19/597, G06T7/20, H04N13/0203, H04N13/0033, H04N13/0018, H04N13/0055, H04N13/0488, H04N2013/0092, H04N13/0404, H04N13/0459, H04N13/0059, H04N2013/0085, H04N13/026, H04N13/0438, H04N13/0285, H04N2013/0081, H04N13/0022, H04N13/044, H04N13/0434, H04N2213/002, H04N13/0431|
|European Classification||H04N13/02C, H04N13/00P17, H04N13/00P1R, H04N13/00P15, H04N13/00P1D, G06T7/00D, H04N19/00P5|
|Aug 7, 2000||AS||Assignment|
Owner name: DYNAMIC DIGITAL DEPTH RESEARCH PTD. LTD., AUSTRIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HARMAN, PHILIP VICTOR;REEL/FRAME:010989/0008
Effective date: 20000307
|Aug 6, 2002||AS||Assignment|
Owner name: DYNAMIC DIGITAL DEPTH RESEARCH PTY. LTD., AUSTRALI
Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE S NAME PTD TO PTY, PREVIOUSLY RECORDED AT REEL 010989FRAME 0008;ASSIGNOR:HARMAN, PHILIP VICTOR;REEL/FRAME:013173/0562
Effective date: 20000307
|May 26, 2006||FPAY||Fee payment|
Year of fee payment: 4
|Jun 4, 2010||FPAY||Fee payment|
Year of fee payment: 8
|May 18, 2014||FPAY||Fee payment|
Year of fee payment: 12