Method for editing multi-channel images

IIIIIIH

US007860306B2

United States Patent

Mallick et al.

(io) Patent No.: (45) Date of Patent:

US 7,860,306 B2 Dec. 28, 2010

METHOD FOR EDITING MULTI-CHANNEL IMAGES

Inventors: Satya P. Mallick, La Jolla, CA (US);

David J. Kriegman, San Diego, CA
(US); Todd E. Zickler, Cambridge, MA
(US); Peter N. Belhumeur, New York,
NY (US)

Assignee: The Regents of the University of
California, Oakland, CA (US)

Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 0 days.

Appl.No.: 12/705,554

Filed: Feb. 12, 2010

Prior Publication Data

US 2010/0142804 Al Jun. 10, 2010

Related U.S. Application Data

Continuation of application No. 11/471,091, filed on Jun. 19, 2006, now Pat. No. 7,689,035.

Provisional application No. 60/797,947, filed on May 5, 2006, provisional application No. 60/691,251, filed on Jun. 17, 2005.

Int. CI.

G06K9/00 (2006.01)

U.S. CI 382/163

Field of Classification Search 382/162,

382/167, 190; 345/581, 589 See application file for complete search history.

References Cited

U.S. PATENT DOCUMENTS 5,850,472 A 12/1998 Alston etal.

The present invention presents a framework for separating specular and diffuse reflection components in images and videos. Each pixel of the an M-channel input image illuminated by N light sources is linearly transformed into a new color space having (M-N) channels. For an RGB image with one light source, the new color space has two color channels (U, V) that are free of specularities and a third channel (S) that contains both specular and diffuse components. When used with multiple light sources, the transformation may be used to produce a specular invariant image. A diffuse RGB image can be obtained by applying a non-linear partial differential equation to an RGB image to iteratively erode the specular component at each pixel. An optional third dimension of time may be added for processing video images. After the specular and diffuse components are separated, dichromatic editing may be used to independently process the diffuse and the specular components to add or suppress visual effects. The (U,V) channels of images can be used as input to 3-D shape estimation algorithms including shape-from-shading, photometric stereo, binocular and multinocular stereopsis, and structurefrom-motion.

61 Claims, 6 Drawing Sheets

U.S. PATENT DOCUMENTS

7,062,419 B2 6/2006 Grzeszczuk et al.

7,689,035 B2* 3/2010 Mallicketal 382/163

2007/0132759 Al 6/2007 Mallicketal.

OTHER PUBLICATIONS

Lin, "Diffuse-Specular Separation and Depth Recovery from Image
Sequences", European Conf. on Comp. Vision (ECCV 2002)
Copenhagen, May 28-31, 2002, Part III, 210-224.
Mallick, "Beyond Lambert: Reconstructing Specular Surfaces Using
Color", Proc. IEEE Conf. Comp. Vision & Patt. Recog., Jun. 2005.
Mallick, "Specularity Removal in Images and Videos: A PDE
Approach", Proc. Eur. Conf. Comp. Vision, Berlin, May 2006, Part I,
550-563.

Nayar, "Separation of Reflection Components Using Color and
Polarization", Int. J. Comp. Vision, 1997, 163-186, vol. 21(3),
Kluwer Academic Publishers.

Tan, "Separating Reflection Components Based on Chromaticity and Noise Analysis", IEEE Trans. Patt. Analysis and Mach. Intel., Oct.

2004, 1373-1381, vol. 26(10).

Tan, "Separating Reflection Components of Textured Surfaces using a Single Image", IEEE Trans. Patt. Analysis and Mach. Intel., Feb.

2005, 178-193, 27(2).

Tsumura, "Medical Vision: measurement of skin absolute spectralreflectance-image ... ", Proc. Int. Conf. Multispect. Color Sci., 2001, 25-28.

Zickler, "Color Subspaces as Photometric Invariants", Proc. IEEE
Conf. Comp. Vision and Patt. Recog., Jun. 2006, pp. 1-8.
Zickler, "Helmholtz Stereopsis: Exploiting Reciprocity for Surface
Reconstruction", Proc. Eur. Conf. Comp. Vision (ECCV 2002)
Copenhagen, May 28-31, 2002, Part III, 869-883.

* cited by examiner