|Publication number||US7742624 B2|
|Application number||US 11/380,196|
|Publication date||Jun 22, 2010|
|Filing date||Apr 25, 2006|
|Priority date||Apr 25, 2006|
|Also published as||US8494224, US20070248281, US20100208943, US20110169986|
|Publication number||11380196, 380196, US 7742624 B2, US 7742624B2, US-B2-7742624, US7742624 B2, US7742624B2|
|Inventors||Boaz J. Super, Bruce A. Augustine, James E. Crenshaw, Evan A. Groat, Michael S. Thiems|
|Original Assignee||Motorola, Inc.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (7), Referenced by (11), Classifications (13), Legal Events (5)|
|External Links: USPTO, USPTO Assignment, Espacenet|
Many camera based products, such as videophones, webcams and mobile devices, require wide angle lenses. In order to avoid having their faces look too small, users tend to move their faces close to the camera. As a result, perspective distance effects become large and their faces appear distorted in the image, with large noses and pinched cheeks. Users find this unflattering. This distance-based perspective effect is an undesirable feature of videophone and video email products.
There are other perspective effects such as wide-angle (fish-eye) distortion and oblique distortion (which causes keystoning). Compensation techniques have been proposed for mitigating these effects, but these techniques are not applicable to the type of distortion described above.
The novel features believed characteristic of the invention are set forth in the appended claims. The invention itself, however, as well as the preferred mode of use, and further objects and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawing(s), wherein:
While this invention is susceptible of embodiment in many different forms, there is shown in the drawings and will herein be described in detail one or more specific embodiments, with the understanding that the present disclosure is to be considered as exemplary of the principles of the invention and not intended to limit the invention to the specific embodiments shown and described. In the description below, like reference numerals are used to describe the same, similar or corresponding parts in the several views of the drawings.
The distance-based perspective effect is illustrated in the diagram in
In some video applications, such as video-telephony, the user is able to monitor the image of his or her face that is being transmitted. If the image does not fill the image frame the user is likely to move closer to the camera. This is especially true if the camera uses a wide angle lens. The result is that the image of the user's face, while filling more of the image frame, becomes distorted due to the distance-based perspective effect. The resulting image is unflattering, showing an enlarged nose and pinched cheeks.
In the description below, an embodiment is described in which the object is a face. However, the invention has application to the reduction of distance-based distortion in images of other objects and images of multiple objects.
In this way, the image size is adjusted so the user is not prompted to move closer to the camera. If the user is too close to the camera, so that distance based distortion is unacceptably larger, the zoom is increased so that image of the user's face appears too large for the image frame, prompting the user to move farther from the camera, thereby reducing the distance-based distortion.
The region of the image corresponding to the user's face may be identified so that the camera can zoom in on this region. Methods of identifying a user's face are well known in the art. The region may be identified and tracked as the user moves, using techniques well known in the art.
If the distance from the lens 104 to a disc is known, the image of the disc can be modified in size to compensate for the distance-based distortion. This same approach may be used with an image such as a face.
For a given object surface, the distance d from the camera lens along the optical axis is a function of the x and y coordinates in a plane perpendicular to the optical axis, so d=d(x,y). Light at position (x, y, d) meets the image sensor array at a position
where α is a parameter dependent upon the optical characteristics (such as the magnification) of the camera. The parameter α may be determined by calibration. The image at position (x′,y′) on the sensor array corresponds to an object point at a distance
from the camera. If the distance di(x′,y′) is known, a pixel value pcorrected in a corrected image may be calculated from a pixel value pimage in the captured image as
where β is a scale factor. In one embodiment, the scale factor β is a constant. The constant may be proportional to the mean value of d(x,y) or the root mean square value of d(x,y), for example. In another embodiment, the scale factor β=γ[d0+di(x′,y′)] where d0 is a constant distance and γ is a scale factor. In this embodiment, a reprojected or virtual image in obtained for which the face appears a distance d0 farther from the camera. In other words, a reprojected image of the object is generated that is dependent upon the estimated distances and upon a virtual viewpoint that is more distant than the camera from the object. A flow chart of an exemplary method for estimating the distance d(x,y) is shown in
In an alternative embodiment, the position of the image of a person's head is detected using matching to templates or other means known in the art. Then a stored, template-independent, warping transformation is applied to the image of the head. The transformation may be parameterized by the size and orientation of the detected image of the head. For example, a geometric model of head, such as a spherical, cylindrical or ellipsoidal model, may be used. The size of the image of the head is used to estimate the distance of the head from the camera and then the geometric model is used to estimate the variation of distance across the image of the head.
The methods described above with reference to
Examples of video cameras include a video camera of a videoconferencing system, a videophone, a video email product, a door camera and a security camera. The approach may also be used in still cameras, such as cameras in cellular telephone and other portable electronic devices.
In one embodiment of the invention, the camera 100 is a video camera that captures images of the face of a user in a video conferencing or video telephony system. The display unit 604 displays images of the user and also images received from a remote camera. The processor 602 controls the zoom of camera 100. If the user is so close to the camera that a large amount of distance-based distortion is present in the image, the zoom is controlled such that the image of the user on the display is too large for the display frame, thereby encouraging the user to move farther from the display.
Additionally, the processor 602 may process images captured by the camera to reduce distance-based distortion using the methods described above. This may be done by estimating distances from the camera to the user's face or by matching templates to the image of the user's face. On one embodiment, the distance is estimated using a distance measuring system 608, such an infrared or ultrasonic system. In another embodiment, the distance is estimated using a stereo vision system.
The system shown in
The present invention, as described in embodiments herein, may be implemented using a programmed processor executing programming instructions that are broadly described above in flow chart form that can be stored on any suitable electronic storage medium. However, those skilled in the art will appreciate that the processes described above can be implemented in any number of variations and in many suitable programming languages without departing from the present invention. For example, the order of certain operations carried out can often be varied, additional operations can be added or operations can be deleted without departing from the invention. Error trapping can be added and/or enhanced and variations can be made in user interface and information presentation without departing from the present invention. Such variations are contemplated and considered equivalent.
Alternatively, the invention may be implemented in hardware.
The programmed processor or hardware may be integral to the camera or may be a separate processor operating on images received from the camera.
While the invention has been described in conjunction with specific embodiments, it is evident that many alternatives, modifications, permutations and variations will become apparent to those of ordinary skill in the art in light of the foregoing description. Accordingly, it is intended that the present invention embrace all such alternatives, modifications and variations as fall within the scope of the appended claims.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US20020130961 *||Mar 14, 2002||Sep 19, 2002||Lg Electronics Inc.||Display device of focal angle and focal distance in iris recognition system|
|US20030117675||Aug 29, 2001||Jun 26, 2003||Norimitu Shirato||Curved image conversion method and record medium where this method for converting curved image is recorded|
|US20040001146||Jun 28, 2002||Jan 1, 2004||Zicheng Liu||Real-time wide-angle image correction system and method for computer image viewing|
|US20040201587||Feb 25, 2003||Oct 14, 2004||Kazufumi Mizusawa||Image combination/conversion apparatus|
|US20040201768||Jan 17, 2002||Oct 14, 2004||Eastman Kodak Company||Electronic imaging system having a sensor for correcting perspective projection distortion|
|US20040207743 *||Apr 1, 2004||Oct 21, 2004||Nikon Corporation||Digital camera system|
|US20050251015 *||Apr 18, 2005||Nov 10, 2005||Omron Corporation||Magnified display apparatus and magnified image control apparatus|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US8095991||Feb 26, 2008||Jan 10, 2012||International Business Machines Corporation||Digital rights management of streaming captured content based on criteria regulating a sequence of elements|
|US8154626 *||Aug 11, 2008||Apr 10, 2012||Canon Kabushiki Kaisha||Imaging apparatus having enhanced zoom operation|
|US8185959 *||Feb 26, 2008||May 22, 2012||International Business Machines Corporation||Digital rights management of captured content based on capture associated locations|
|US8266716||Jul 28, 2011||Sep 11, 2012||International Business Machines Corporation||Digital rights management of streaming captured content based on criteria regulating a sequence of elements|
|US8805110 *||Jan 21, 2011||Aug 12, 2014||Digimarc Corporation||Methods and systems for content processing|
|US8850594||Mar 23, 2012||Sep 30, 2014||International Business Machines Corporation||Digital rights management of captured content based on capture associated locations|
|US20090051789 *||Aug 11, 2008||Feb 26, 2009||Canon Kabushiki Kaisha||Imaging apparatus|
|US20090217343 *||Feb 26, 2008||Aug 27, 2009||Bellwood Thomas A||Digital Rights Management of Streaming Captured Content Based on Criteria Regulating a Sequence of Elements|
|US20090217344 *||Feb 26, 2008||Aug 27, 2009||Bellwood Thomas A||Digital Rights Management of Captured Content Based on Capture Associated Locations|
|US20110212717 *||Jan 21, 2011||Sep 1, 2011||Rhoads Geoffrey B||Methods and Systems for Content Processing|
|WO2012125242A1||Feb 10, 2012||Sep 20, 2012||Motorola Mobility Llc||Method and apparatus for morphing a user during a video call|
|U.S. Classification||382/106, 348/208.15, 382/100, 356/3, 348/333.04|
|International Classification||H04N5/222, H04N5/228, G01C3/00, G01C5/00, G06K9/00|
|Cooperative Classification||G06K2009/363, G06K9/00228|
|Apr 25, 2006||AS||Assignment|
Owner name: MOTOROLA, INC., ILLINOIS
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SUPER, BOAZ J.;AUGUSTINE, BRUCE A.;CRENSHAW, JAMES E.;AND OTHERS;REEL/FRAME:017526/0952;SIGNING DATES FROM 20060417 TO 20060420
Owner name: MOTOROLA, INC.,ILLINOIS
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SUPER, BOAZ J.;AUGUSTINE, BRUCE A.;CRENSHAW, JAMES E.;AND OTHERS;SIGNING DATES FROM 20060417 TO 20060420;REEL/FRAME:017526/0952
|Dec 13, 2010||AS||Assignment|
Owner name: MOTOROLA MOBILITY, INC, ILLINOIS
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MOTOROLA, INC;REEL/FRAME:025673/0558
Effective date: 20100731
|Oct 2, 2012||AS||Assignment|
Owner name: MOTOROLA MOBILITY LLC, ILLINOIS
Free format text: CHANGE OF NAME;ASSIGNOR:MOTOROLA MOBILITY, INC.;REEL/FRAME:029216/0282
Effective date: 20120622
|Nov 26, 2013||FPAY||Fee payment|
Year of fee payment: 4
|Nov 24, 2014||AS||Assignment|
Owner name: GOOGLE TECHNOLOGY HOLDINGS LLC, CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MOTOROLA MOBILITY LLC;REEL/FRAME:034419/0001
Effective date: 20141028