Publication number | USRE43206 E1 |
Publication type | Grant |
Application number | US 11/541,517 |
Publication date | Feb 21, 2012 |
Filing date | Sep 28, 2006 |
Priority date | Feb 4, 2000 |
Fee status | Paid |
Also published as | US6798923 |
Publication number | 11541517, 541517, US RE43206 E1, US RE43206E1, US-E1-RE43206, USRE43206 E1, USRE43206E1 |
Inventors | Jun-Wei Hsieh, Cheng-Chin Chiang, Der-Lor Way |
Original Assignee | Transpacific Ip Ltd. |
Export Citation | BiBTeX, EndNote, RefMan |
Patent Citations (26), Non-Patent Citations (2), Referenced by (7), Classifications (16), Legal Events (3) | |
External Links: USPTO, USPTO Assignment, Espacenet | |
The present invention relates to an apparatus, algorithm, and method for stitching different pieces of images of a scene into a panoramic environment map.
The most common way to electronically represent the real world is with image data. Unlike traditional graph-based systems, there are systems which use panoramic images to construct a virtual world. The major advantage of a system which uses panoramic images is that very vivid and photo-realistic rendering results can be obtained even when using PCs. In addition, the cost of constructing the virtual world is independent of scene complexity. In such systems, panoramic images are stitched together into a panoramic map from several individual images which are acquired by rotating a camera horizontally or vertically. This panoramic map can be used in different applications such as movie special effects, the creation of virtual reality, or games. A typical problem is how to stitch the different pieces of a scene into a larger picture or map. One approach to address this problem is to manually establish correspondences between images to solve unknown parameters of their relative transformation. Because manual methods are tedious for large applications, automatic schemes are preferably used for generating a seamless panoramic image from different pieces of images.
One proposed approach uses a nonlinear minimization algorithm for automatically stitching panoramic images by minimizing the discrepancy in intensities between images. This approach has the advantage of not requiring easily identifiable features. However, this technique does not guarantee finding the global minimum if the selection of starting points is not proper. Further because the optimization process is time-consuming, the approach is inefficient. In this invention, the domain of images under consideration is panoramic images.
The invention allows users to generate panoramic images from a sequence of images acquired by a camera rotated about its optical center. In general, the invention combines feature extraction, correlation, and relaxation techniques to get a number of reliable and robust matching pairs used to derive registration parameters. Based on the obtained registration parameters, different pieces of consecutive images can-be stitched together to obtain a seamless panoramic image.
In a first aspect, a method of merging a pair of images to form a seamless panoramic image includes the following steps. A set of feature points along the edges of the images is extracted, each feature point defining an edge orientation. A set of registration parameters is obtained by determining an initial set of feature points from a first one of the images which matches a set of feature points of a second one of the images. A seamless panoramic image is rendered using the first and second images with the set of registration parameters.
The invention provides a feature-based approach for automatically stitching panoramic images acquired by a rotated camera and obtaining a set of matching pairs from a set of feature points for registration. Since the feature points are extracted along the edges, each feature point specifies an edge orientation. Because the orientation difference between two panoramic images is relatively small, the difference between edge orientations of two feature points is also small if they are good matches. Based on this assumption, edge information of feature points can be used to eliminate in advance many false matches by checking their orientation difference. Moreover, many unnecessary calculations involving cross-correlation can be screened in advance, thereby significantly reducing the search time needed for obtaining correct matching pairs. After checking, by calculating the value of correlation of the remaining matching pairs, a set of possible matches can be selected with a predefined threshold. The set of possible matches are further verified through a relaxation scheme by calculating the quality of their matches. Once all of the correct matching pairs are found, they are then used to derive registration parameters. In this invention, an iterative scheme is applied to increase the reliability in providing matching results. Since only three iteration or fewer are needed and only a few feature points are involved in the matching pairs, the whole procedure can be accomplished very efficiently. Also, as discussed above, because the orientation difference of two feature points is checked in advance (before matching). Many calculations involving cross-correlation are not required and the efficiency of stitching is significantly improved. Compared with conventional algorithms, the proposed scheme offers improved efficiency and reliability for stitching images.
Embodiments of this aspect of the invention may include one or more of the following features. In one embodiment, a set of feature points are first extracted through wavelet formations. Among other advantages, the invention uses wavelet transforms to obtain a number of feature points with edge orientations. Such edge information can speed up the entire registration process by eliminating many impossible matches in advance and avoiding many unnecessary calculations of correlation. The method determines a number of reliable and robust matching pairs through relaxation. The method also measures the quality of a matching pair, imposes angle consistency constraint for improving the robustness of registration, and uses a voting concept to get the desired solution from the set of final matching results.
In other embodiments, the method forms the final panoramic image with the help of the registration results. In particular, the method adjusts the intensity differences between consecutive input images and blends the intensities of adjacent images to obtain a seamless panoramic image. The final panoramic images can then be used to build a virtual world.
Still other embodiments may include one or more of the following features:
For example, the method selects a number of feature points through wavelet transforms. Each feature point is associated with an edge orientation so that the speed of the registration process is increased.
The method uses an angle constraint to construct a set of matching pairs, which are used to obtain reliable matching results through a relaxation and a voting technique. The set of matching results are then used to form the final seamless panoramic image.
Constructing an initial set of matching pairs for registration includes comparing the edge orientation differences of feature points in one image and its corresponding feature points in another, calculating the values of correlation of each possible matching pair, and thresholding them with a predefined threshold.
Getting reliable matching results through relaxation and a voting technique includes calculating the quality of a matching pair, imposing angle consistency constraint to filter out impossible matching pairs, updating matching results through relaxation, and using the voting technique to obtain the reliable registration parameters. In addition, it refines the final registration results by using the correlation technique with a proper starting point.
Forming the final panoramic images includes dynamically adjusting and properly blending the intensity differences between adjacent images.
In another aspect, the invention features a system for merging pairs of images to form a panoramic image. The system includes an imaging device which, in operation, acquires a series of images, a storage for storing a series of images, a memory which stores computer code, and at least one processor which executes computer code to extract a set of feature points along the edges of the images, each feature point defining an edge orientation and to obtain a set of registration parameters by determining an initial set of feature points from a first one of the images which matches a set of feature points of a second one of the images, and to render a seamless panoramic image using the first and second images with the set of registration parameters.
Other advantages and features of the invention will become apparent from the following description, including the claims and the drawings.
Referring first to the flow diagram of
In general, it is difficult to seamlessly stitch two adjacent images together to form a panoramic image due to perspective distortion introduced by a camera. To remove the effects of this distortion, these images are preferably reprojected onto a simple geometry, e.g., a cube, a cylinder, or a sphere. In many applications, a cylindrical geometry is preferable since its associated geometrical transformation is simple. In this example, the cylindrical geometry is used.
and
Moreover, since the radius r is equal to f, Equations (1) and (2) can be rewritten as follows:
u=f tan^{−1}x/f
and
Based on Eqs. (3) and (4), input images are provided and then (step 10) warped into a cylindrical map for further registration to construct a complete panoramic scene (step 11).
Referring again to
and
Let
and
At each scale 2^{j}, the 2-D wavelet transform of a function I(x,y) in L^{2}(R^{2}) can be decomposed into two independent directions as follows:
W_{2} _{ j } ^{1}I(x, y)=I*ψ_{2} _{ j } ^{1}(x, y) and W_{2} _{ j } ^{2}I(x, y)=I*ψ_{2} _{ j } ^{2}(x, y).
Basically, these two components are equivalent to the gradients of I(x,y) smoothed by S(x,y) at scale 2^{j }in the x and y directions. At a specific scale s=2^{i}, the modulus of the gradient vector of f(x,y) can be calculated as follows:
M_{2} _{ j }I(x, y)=√{square root over (|W^{1} _{2} _{ j }I(x, y)|^{2}+|W^{2} _{2} _{ j }I(x,y)|^{2})}{square root over (|W^{1} _{2} _{ j }I(x, y)|^{2}+|W^{2} _{2} _{ j }I(x,y)|^{2})}.
If the local maximum of M_{2} _{ j }I(x, y) are located and thresholded with a preset value, then the edge points of I(x,y) at scale 2^{j }can be detected. Since we are interested in specific feature points for scene registration, additional constraints have to be introduced. In general, noise is the main cause of false detection of edge points. In order to suppress the effect of noise, a criterion called edge correlation is introduced (step 32):
where n is a positive integer indicating the number of scales involved in the multiplication, and j represents the initial scale for edge correlation. R_{n }reveals a peak whenever a true edge exists and is suppressed by the multiplication process if a point at location (x,y) is not a true edge. Thus, using the relationship for R_{n}(j, x, y) above, the noise in an image can be suppressed while the true edges can be retained. In one embodiment of the invention, the number of scales for multiplication is chosen to be 2. In order to conserve the energy level, R_{n}(j, x, y) should be normalized as follows:
where
and
During the feature point selection process, an edge point is recognized as a candidate if its corresponding normalized edge correlation R_{2}(j, x, y) is larger than its corresponding modulus value. Basically, the above mentioned process is equivalent to detecting an edge point with the strongest edge response in a local area. The three conditions which will be used to judge whether a point P(x,y) is a feature point or not are as follows:
where N_{p }is the neighborhood of P(x,y) (step 34).
Referring to
Arg(W_{2} _{ j } ^{2}I(x, y)+W_{2} _{ j } ^{2}I(x, y)).
However, the above representation can be very sensitive to noise. Therefore, an edge tracking technique plus a line-fitting model is used to solve the noise problem (step 41).
Let P be a feature point and N_{e }be its neighborhood. Since P is an edge point, there should exist an edge line passing through it. By considering P as a bridge point, an edge line passing through P can be determined by searching in all the directions from P. All the edge points on the edge are then used as candidates for determining the orientation of the edge line. During the searching process, the edge connection constraint and the direction consistency constraint are used to restrict the searching domain. The edge connection constraint means that if N_{e }contains another edge l but l does not pass P, all edge points in l will not be included in estimating the orientation of P. In certain cases, however, there will exist more than one edge line passing through P. In these cases, the first line detected is adopted to estimate the orientation. Let l_{1 }denote this line. The direction consistency constraint means all the edge points along other edge lines whose orientations are inconsistent with l_{1 }are not included to estimate the orientation of P. In this way, a set of edge points can be selected and then used to estimate the orientation of P using a line-fitting model. In other embodiments, other edge tracking technique can also be applied to provide a better estimation.
After the orientation is estimated, all feature points u will associate with an edge orientation A(u). For a feature point p_{i }in the set of points FP_{I} _{ a }and q_{j }in the set of points FP_{I} _{ b }, the orientation difference between them is calculated as follows (step 42):
θ_{ij}=A(q_{j})−A(p_{i}).
In fact, if p_{i }and q_{i }provide a good match, the value of θ_{ij }will be small since the orientation of image I_{a }is similar to that of I_{b}. Assuming this is the case, step 14 (
For a feature point p_{i }in FP_{I} _{ a }and a feature point q_{j }in FP_{I} _{ b }, if they form a good match (step 44), the following condition will be satisfied:
|A(p_{i})−A(q_{i})|<10°. Condition 1:
Adding this criterion will significantly speed up the search time. On the other hand, if p_{i }and q_{i }form a good match, the similarity degree between p_{i }and q_{i }should be larger. A cross-correlation which can be used to measure the similarity degree between p_{i }and q_{j }(step 46) and is defined as follows (step 32):
where u_{i }and u_{i}u_{j }are the local means of p_{i }and q_{j}, respectively, σ_{i }and σ_{i }are the local variances of p_{i }and q_{j}, respectively; and (2M+1)^{2 }represents the area of matching window. Based on this correlation measure, a pair {p_{i}
_{q} _{j}} is qualified as a possible matching pair if the following conditions are satisfied (step 33):
and
C_{i} _{ a } _{I} _{ b }(p_{i};q_{j})>T_{e}, where T_{c}=0.65 (step 35). Condition 4:
Condition 2 means that given a feature point p_{i }it is desired to find a point q_{i}εFP_{I} _{ b }such that a the value of C_{I} _{ a } _{I} _{ b }(p_{i};q_{j}) is maximized for all points q_{k}εFP_{I} _{ b }. Condition 3 means that given a feature points q_{i}, it is desired to find a point p_{i}εFP_{I} _{ a }such that the value of C_{I} _{ a } _{I} _{ b }(p_{i};q_{j}) is maximized. If only Condition 2 is used, it is possible that several points p_{i }match with single point q_{j}. Conversely, if only Condition 3 is used, several points q_{j }possibly will match with single point p_{i}. As for Condition 3, it forces the value of C_{I} _{ a } _{I} _{ b }of a matching pair to be larger than a threshold. In a preferred implementation, the orientation constraint will be checked first. If the constraint is not satisfied, it is not necessary to check Conditions 2, 3, and 4. In this way, only a few pairs are needed to calculate the cross-correlation measure C_{I} _{ a } _{I} _{ b }, which is considered a time bottleneck of the whole process.
Referring to
where
with the two predefined thresholds
The contribution of a pair {n_{k} ^{1}
Referring to
then the relaxation procedure can be formulated as follow:
Iterate { | |
-Compute the quality for each candidate match | |
-Choose the best possible candidates for minimizing F | |
according to the quality value G_{I} _{ a } _{I} _{ b } (i). | |
} until F converges. | |
On the other hand, in order to make the matching results more reliable, the method includes an “angle consistency” constraint (step 53) within the first iteration of the relaxation process to further eliminate impossible matching pairs. That is, if {p_{i}
q_{i}} and {p_{j} q_{j}} are well matched, the angle between {right arrow over (p_{i}p_{j})} and {right arrow over (q_{i}q_{j})} must be close to zero. In this case, during this first iteration, for each element {p_{i} q_{i}} in MP_{I} _{ a } _{,I} _{ b }, a counter CA_{i }is used to record the number of matches {p_{k} q_{k}} in MP_{I} _{ a } _{,I} _{ b }where the angle between {right arrow over (p_{i}p_{k})} and {right arrow over (q_{i}q_{k})} is less than a predefined threshold θ. According to the value of CA_{i}, the elements of MP_{I} _{ a } _{,I} _{ b }are sorted in increasing order. The first Q% of matches in MP_{I} _{ a } _{,I} _{ b }are considered to be impossible matches.After applying the relaxation process, a set of reliable matches is obtained. Referring to
Note that due to noise and image quality, the positions of feature points will not be precisely located and the accuracy of T is affected. In practical implementation, the y-component
be the horizontal gradient of the point (u_{x},k) in I_{h}, where u_{x }is the x-coordinate of u. Instead of using the starting point u directly, we use another starting point ū for refining the desired offset
In general, when stitching two adjacent images, discontinuities of intensity exist between their common areas. Therefore, step 17 of
of I_{a }and I_{b}, that is,
where A is the overlapping area of I_{a }and I_{b}, |A| is the number of pixels in A, p(i) is a pixel in I_{a}, and q(i) is its corresponding pixel in I_{b}.
In particular, referring to
and
After this adjusting step, the intensities of I_{a }and I_{b }in
The second stage uses a ray-casting method to blend different pixel intensities together. Referring to
Where d_{a }is the distance between p_{i }and I_{a}, d_{h }the distance between q_{i }and I_{h}, and t is an adjustable parameter. Using Equation (10), the intensities in I_{a }are gradually changed to approach the intensities of pixels in I_{h }such that the final composite image I looks very smooth. In fact, if the blending area is chosen too large, a “ghostlike” effect will occur, particularly when moving objects in the common overlapping area exist between I_{a }and I_{b}. However, since the intensities of I_{a }and I_{b }have been adjusted, the blending width can be chosen small such that the so-called ghostlike effect is significantly reduced. In one preferred embodiment, the blending width is chosen as one-third of the original width of the overlapping area.
Referring to
Referring to
Referring to
Cited Patent | Filing date | Publication date | Applicant | Title |
---|---|---|---|---|
US5613013 * | May 13, 1994 | Mar 18, 1997 | Reticula Corporation | Glass patterns in image alignment and analysis |
US5625408 * | Dec 20, 1995 | Apr 29, 1997 | Canon Kabushiki Kaisha | Three-dimensional image recording/reconstructing method and apparatus therefor |
US5850352 | Nov 6, 1995 | Dec 15, 1998 | The Regents Of The University Of California | Immersive video, including video hypermosaicing to generate from multiple video views of a scene a three-dimensional video mosaic from which diverse virtual video scene images are synthesized, including panoramic, scene interactive and stereoscopic images |
US5953076 | Jun 12, 1996 | Sep 14, 1999 | Princeton Video Image, Inc. | System and method of real time insertions into video using adaptive occlusion with a synthetic reference image |
US5963664 * | Jun 22, 1995 | Oct 5, 1999 | Sarnoff Corporation | Method and system for image combination using a parallax-based technique |
US5987164 * | Aug 1, 1997 | Nov 16, 1999 | Microsoft Corporation | Block adjustment method and apparatus for construction of image mosaics |
US6009190 * | Aug 1, 1997 | Dec 28, 1999 | Microsoft Corporation | Texture map construction method and apparatus for displaying panoramic image mosaics |
US6011581 | Jan 20, 1995 | Jan 4, 2000 | Reveo, Inc. | Intelligent method and system for producing and displaying stereoscopically-multiplexed images of three-dimensional objects for use in realistic stereoscopic viewing thereof in interactive virtual reality display environments |
US6044168 | Nov 14, 1997 | Mar 28, 2000 | Texas Instruments Incorporated | Model based faced coding and decoding using feature detection and eigenface coding |
US6075905 * | Jul 16, 1997 | Jun 13, 2000 | Sarnoff Corporation | Method and apparatus for mosaic image construction |
US6078701 * | May 29, 1998 | Jun 20, 2000 | Sarnoff Corporation | Method and apparatus for performing local to global multiframe alignment to construct mosaic images |
US6349153 * | Apr 4, 2000 | Feb 19, 2002 | Mgi Software Corporation | Method and system for composition images |
US6393162 * | Dec 30, 1998 | May 21, 2002 | Olympus Optical Co., Ltd. | Image synthesizing apparatus |
US6393163 * | Dec 2, 1999 | May 21, 2002 | Sarnoff Corporation | Mosaic based image processing system |
US6411339 * | Jun 4, 1998 | Jun 25, 2002 | Nippon Telegraph And Telephone Corporation | Method of spatio-temporally integrating/managing a plurality of videos and system for embodying the same, and recording medium for recording a program for the method |
US6434276 * | Sep 28, 1998 | Aug 13, 2002 | Sharp Kabushiki Kaisha | Image synthesis and communication apparatus |
US6466262 * | Jun 9, 1998 | Oct 15, 2002 | Hitachi, Ltd. | Digital wide camera |
US6473536 * | Sep 13, 1999 | Oct 29, 2002 | Sanyo Electric Co., Ltd. | Image synthesis method, image synthesizer, and recording medium on which image synthesis program is recorded |
US6486908 * | May 27, 1998 | Nov 26, 2002 | Industrial Technology Research Institute | Image-based method and system for building spherical panoramas |
US6516099 * | Aug 5, 1998 | Feb 4, 2003 | Canon Kabushiki Kaisha | Image processing apparatus |
EP0249644A1 | Jun 14, 1986 | Dec 23, 1987 | ANT Nachrichtentechnik GmbH | Method for transmitting television signals with an improved picture quality |
EP0257129A1 | Aug 29, 1986 | Mar 2, 1988 | ANT Nachrichtentechnik GmbH | Process for the reproduction of television signals with improved image quality |
EP0415648A2 | Aug 23, 1990 | Mar 6, 1991 | Canon Kabushiki Kaisha | Image processing apparatus |
TW300369B | Title not available | |||
TW350183B | Title not available | |||
TW376670B | Title not available |
Reference | ||
---|---|---|
1 | * | Jacques Fayolle et al. "Application of Multiscale Charaterization of Edges to Motion Determination" IEEE-1998, pp. 1174-1179. |
2 | * | Mingu Sun et al. "Measurement of Signal Similarity Using the Maxima of the Wavelet Transform" IEEE-1993, pp. 583-586. |
Citing Patent | Filing date | Publication date | Applicant | Title |
---|---|---|---|---|
US9047692 * | Dec 20, 2012 | Jun 2, 2015 | Google Inc. | Scene scan |
US9076238 * | Aug 21, 2013 | Jul 7, 2015 | Seiko Epson Corporation | Intelligent weighted blending for ultrasound image stitching |
US9530076 * | Feb 16, 2015 | Dec 27, 2016 | A9.Com, Inc. | Method and system for matching an image using normalized feature vectors |
US20120120099 * | Oct 25, 2011 | May 17, 2012 | Canon Kabushiki Kaisha | Image processing apparatus, image processing method, and storage medium storing a program thereof |
US20150055839 * | Aug 21, 2013 | Feb 26, 2015 | Seiko Epson Corporation | Intelligent Weighted Blending for Ultrasound Image Stitching |
US20150154761 * | Dec 20, 2012 | Jun 4, 2015 | Google Inc. | Scene scan |
US20150161480 * | Feb 16, 2015 | Jun 11, 2015 | A9.Com, Inc. | Method and system for matching an image using normalized feature vectors |
U.S. Classification | 382/284, 345/629, 382/240, 348/36, 382/190 |
International Classification | G06K9/36, H04N5/262, H04N7/00, G06T3/40, G09G5/00 |
Cooperative Classification | H04N5/2624, H04N5/23238, G06T3/403 |
European Classification | G06T3/40E, H04N5/262M, H04N5/232M |
Date | Code | Event | Description |
---|---|---|---|
Dec 28, 2010 | AS | Assignment | Owner name: TRANSPACIFIC IP LTD., TAIWAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:INDUSTRIAL TECHNOLOGY RESEARCH INSTITUTE;REEL/FRAME:025542/0601 Effective date: 20061124 |
Aug 28, 2012 | CC | Certificate of correction | |
Feb 23, 2016 | FPAY | Fee payment | Year of fee payment: 12 |