A system and method are disclosed for digitally compositing an object from an input image onto a destination image. The object is composited from an image having an arbitrary or non-uniform colored background containing some non-static elements onto a destination image with reduced effects from shadows cast by the object and with reduced gaps or holes within the object. Various improvements in the compositing procedure such as shadow reduction and hole filling, and less restrictive requirements regarding the object's surroundings are disclosed. A background model is created and a frame of an input image containing the object is obtained. An alpha image is created in which each pixel is either a zero, indicating it is not part of the object, or a one, indicating that it is part of the object. The effect of shadows emanating from the object is reduced so that the composited object in the destination image contains only the object clearly outlined by the object's physical boundaries... |
Citations|
| US4589013 | Jun 13, 1983 | May 13, 1986 | 501 Ultimatte Corporation | Automated encoded signal color image compositing | | US4641349 | Feb 20, 1985 | Feb 3, 1987 | | Iris recognition system | | US4843568 | Apr 11, 1986 | Jun 27, 1989 | | Real time perception of and response to the actions of an unencumbered participant/user | | US5148477 | Aug 24, 1990 | Sep 15, 1992 | Board of Regents of the University of Oklahoma | Method and apparatus for detecting and quantifying motion of a body part | | US5278921 | May 21, 1992 | Jan 11, 1994 | Fuji Photo Film Co., Ltd. | Method of determining exposure | | US5321764 | Jan 10, 1992 | Jun 14, 1994 | Kansas State University Research Foundation | Identification of wheat cultivars by visual imaging | | US5343252 | Apr 15, 1993 | Aug 30, 1994 | Ultimatte Corporation | Method and apparatus for compositing video images | | US5384912 | May 29, 1992 | Jan 24, 1995 | New Microtime Inc. | Real time video image processing system | | US5454043 | Jul 30, 1993 | Sep 26, 1995 | Mitsubishi Electric Research Laboratories, Inc. | Dynamic and static hand gesture recognition through low-level image analysis | | US5469536 | May 21, 1993 | Nov 21, 1995 | ImageWare Software, Inc. | Image editing system including masking capability | | US5548659 | Sep 23, 1994 | Aug 20, 1996 | Kabushiki Kaisha Toshiba | Method and apparatus for detecting changes in dynamic images | | US5570113 | Jun 29, 1994 | Oct 29, 1996 | International Business Machines Corporation | Computer based pen system and method for automatically cancelling unwanted gestures and preventing anomalous signals as inputs to such system | | US5577179 | Jul 23, 1992 | Nov 19, 1996 | ImageWare Software, Inc. | Image editing system | | US5581276 | Sep 8, 1993 | Dec 3, 1996 | Kabushiki Kaisha Toshiba | 3D human interface apparatus using motion recognition based on dynamic image processing | | US5623587 | Jun 12, 1995 | Apr 22, 1997 | Kideo Productions, Inc. | Method and apparatus for producing an electronic image | | US5631697 | Jun 2, 1995 | May 20, 1997 | Hitachi, Ltd. | Video camera capable of automatic target tracking | | US5767867 | Nov 27, 1995 | Jun 16, 1998 | Sun Microsystems, Inc. | Method for alpha blending images utilizing a visual instruction set | | US5781198 | Dec 22, 1995 | Jul 14, 1998 | Intel Corporation | Method and apparatus for replacing a background portion of an image | | US5790124 | Nov 20, 1995 | Aug 4, 1998 | Silicon Graphics, Inc. | System and method for allowing a performer to control and interact with an on-stage display device | | US5956419 | Apr 28, 1995 | Sep 21, 1999 | Xerox Corporation | Unsupervised training of character templates using unsegmented samples | | US6011595 | Sep 19, 1997 | Jan 4, 2000 | Eastman Kodak Company | Method for segmenting a digital image into a foreground region and a key color region | | US6048313 | Sep 17, 1998 | Apr 11, 2000 | | Method and apparatus for fractal-based enhancement of ultrasound imaging | | US6121981 | May 19, 1997 | Sep 19, 2000 | Microsoft Corporation | Method and system for generating arbitrary-shaped animation in the user interface of a computer | | US6141063 | Sep 22, 1997 | Oct 31, 2000 | U.S. Philips Corporation | Chroma key method and arrangement | | US6631240 | Jul 21, 1998 | Oct 7, 2003 | University of Washington | Multiresolution video | | US20010028731 | Feb 13, 2001 | | | Canonical correlation analysis of image/control-point location coupling for the automatic location of control points | | US20020090114 | Dec 13, 2001 | | | Watermark enabled video objects |
Referenced by|
| US7324166 | Nov 15, 2004 | Jan 29, 2008 | Contour Entertainment Inc | Live actor integration in pre-recorded well known video |
Claims1. A computer implemented method of filling holes in a digital video object being composited comprising: - assigning to each column in an alpha image an index, wherein the columns in the alpha image are comprised of alpha pixels having values, and wherein the alpha image corresponds to an image that includes the digital video object being composited;
- determining the values of the alpha pixels;
- calculating for each index an index value that is based at least in part on the values of the alpha pixels comprising the column with which the index is associated;
- determining indices of boundaries and sections of the object, thereby identifying the shape of the object;
- computing a set of a plurality of templates using the indices of the boundaries and sections of the object such that the set of templates comprises the shape of the object wherein the set of templates lies entirely within the portion of the object it represents including all boundary pixels; and
- adjusting one or more portions of the alpha image based at least in part on the set of templates such that the one or more portions of the alpha image that correspond to image locations that lie within the any template of the set of templates are within the set of templates are processed as being part of the object so that no holes appear in the image in image locations that correspond to alpha image locations that lie within any template of the set of templates.
2. A computer implemented method as recited in claim 1 further including computing a set of bounding rectangles such that portions of the object fit within the set of bounding rectangles and such that the set of bounding rectangles is computed using the indices of the boundaries and sections of the object. 3. A computer program product for filling holes in a digital video object being composited, the computer program product being embodied in a computer readable medium and comprising computer instructions for: - assigning to each column in an alpha image an index, wherein the columns in the alpha image are comprised of alpha pixels having values, and wherein the alpha image corresponds to an image that includes the digital video object being composited;
- determining the values of the alpha pixels;
- calculating for each index an index value that is based at least in part on the values of the alpha pixels comprising the column with which the index is associated;
- determining indices of boundaries and sections of the object, thereby identifying the shape of the object;
- computing a set of a plurality of templates using the indices of the boundaries and sections of the object such that the set of templates comprises the shape of the object wherein the set of templates lies entirely within the portion of the object it represents including all boundary pixels; and
- adjusting one or more portions of the alpha image based at least in part on the set of templates such that the one or more portions of the alpha image that correspond to image locations that lie within the any template of the set of templates are processed as being part of the object so that no holes appear in the image in image locations that correspond to alpha image locations that lie within any template of the set of templates.
4. A computer implemented system for filling holes in an object digitally composited onto a destination image comprising: - an image indexer for assigning to each column in an alpha image an index, wherein the columns in the alpha image are comprised of alpha pixels having values, and wherein the alpha image corresponds to an image that includes the digital video object being composited;
- a pixel evaluator for determining the values of the alpha pixels;
- an index calculator for calculating for each index an index value that is based at least in part on the values of the alpha pixels comprising the column with which the index is associated;
- an index locator for determining indices of boundaries and sections of the object, thereby identifying the shape of the object;
- a template configurator for computing a set of a plurality of templates using the indices of the boundaries and sections of the object such that the set of templates comprises the shape of the object wherein the set of templates lies entirely within the portion of the object it represents including all boundary pixels; and
- an image modifier for adjusting one or more portions of the alpha image based at least in part on the set of templates such that the one or more portions of the alpha image that correspond to image locations that lie within any template of the set of templates are processed as being part of the object so that no holes appear in the image in image locations that correspond to alpha image locations that lie within any template of the set of templates.
5. A system as recited in claim 4 further comprising an object configurator for computing a set of bounding rectangles such that portions of the object fit within the set of bounding rectangles and the set of bounding rectangles is derived using the indices of the boundaries and sections of the object. 6. A system for filling holes in an object digitally composited onto a destination image, the system comprising: - means for assigning to each column in an alpha image an index, wherein the columns in the alpha image are comprised of alpha pixels having values, and wherein the alpha image corresponds to an image that includes the digital video object being composited;
- means for determining the values of the alpha pixels;
- means for calculating for each index an index value that is based at least in part on the values of the alpha pixels comprising the column with which the index is associated;
- means for determining indices of boundaries and sections of the object, thereby identifying the shape of the object;
- means for computing a set of a plurality of templates using the indices of the boundaries and sections of the object such that the set of templates comprises the shape of the object wherein the set of templates lies entirely within the portion of the object it represents including all boundary pixels; and
- means for adjusting one or more portions of the alpha image based at least in part on the set of templates such that the one or more portions of the alpha image that correspond to image locations that lie within the any template of the set of templates are processed as being part of the object so that no holes appear in the image in image locations that correspond to alpha image locations that lie within any template of the set of templates.
7. A system as recited in claim 6 further comprising means for computing a set of bounding rectangles such that portions of the object fit within the set of bounding rectangles and the set of bounding rectangles is derived using the indices of the boundaries and sections of the object. 8. A computer program product as recited in claim 3, the computer program product further comprising computer instructions for computing a set of bounding rectangles such that portions of the object fit within the set of bounding rectangles and such that the set of bounding rectangles is computed using the indices of the boundaries and sections of the object. |
