US RE43206 E1 Abstract A method and system of merging a pair of images to form a seamless panoramic image including the following steps. A set of feature points are extracted along the edges of the images, each feature point defining an edge orientation. A set of registration parameters is obtained by determining an initial set of feature points from a first one of the images which matches a set of feature points of a second one of the images. A seamless panoramic image is rendered using the first and second images with the set of registration parameters.
Claims(20) 1. A method of merging a different pair of images to form a seamless panoramic image comprising:
extracting a set of feature points along the edges of the images, each feature point defining an edge orientationsorientation, wherein extracting the set of feature points along the edges of the images includes applying wavelet transforms to the images;
obtaining a set of registration parameters by determining an initial set of feature points from a first one of the images which matches a set of feature points of a second one of the images; and
rendering a seamless panoramic image using the first and second images with the set of registration parameters.
2. The method of
calculating the edge orientation of each feature point;
comparing thean orientation difference between the matching pair;
calculating thea value of correlation of the matching pair; and
comparing the value of correlation with a predefined threshold.
3. The method of
calculating an edge correlation for each image;
locating thea feature point whose edge response is thea maxima within a window;
comparing the maxima with a predefined threshold.
4. The method of
determining an initial set of matching pairs for registration;
calculating a quality value for the initial set of matching pairs;
updating the matching result according to the quality value of the match;
imposing an angle consistency constraint to filter out impossible matches; and
using a voting technique to obtain the registration parameters.
5. The method of
dynamically adjusting the intensity differences between adjacent images; and
properly blending thean intensity difference between consecutive images.
6. A system for merging a pair of images to form a panoramic image comprising:
an image device which, in operation, acquires a series ofthe images;
a storage for storing the series of images:
a memory which stores computer code; and
at least one processor which executes the computer code to:
extract different sets of feature points along the edges of each input image, each feature point defining an edge orientation;
extract a set of feature points along the edges of the images by applying wavelet formations to the images, each feature point defining an edge of orientation;
obtain a set of registration parameters by determining an initial set of feature points from a first one of the images which matches a set of feature points of a second one of the images; and
render a seamless panoramic image using the first and second images with the set of registration parameters.
7. The system of
calculating an edge correlation for each image;
locating thea feature point whose edge response is thea maxima within a window;
comparing the maxima with a predefined threshold.
8. An apparatus for merging a different pair of images to form a seamless panoramic image comprising:
means for extracting a set of feature points along the edges of the images, one or more feature point defining an edge orientations, wherein said means for extracting the set of feature points along the edges of the images includes means for applying wavelet transforms to the images; means for obtaining a set of registration parameters by determining an initial set of feature points from a first one of the images which matches a set of feature points of a second one of the images; and means for rendering a seamless panoramic image using the first and second images with the set of registration parameters. 9. An apparatus as claimed in claim 8, wherein said obtaining means for obtaining a set of registration parameters by determining an initial set of feature points comprises:
means for calculating the edge orientation of one or more feature points; means for comparing the orientation difference between the matching pair; means for calculating the value of correlation of the matching pair; and means for comparing the value of correlation with a predefined threshold. 10. An apparatus as claimed in claim 8, wherein said obtaining means comprises:
means for calculating an edge correlation for one or more images; means for locating the feature point whose edge response is a maxima within a window; and means for comparing the maxima with a predefined threshold. 11. An apparatus as claimed in claim 8, wherein said obtaining means comprises:
means for determining an initial set of matching pairs for registration; means for calculating a quality value for the initial set of matching pairs; means for updating the matching result according to the quality value of the match; means for imposing an angle consistency constraint to filter out impossible matches; and means for using a voting technique to obtain the registration parameters. 12. An apparatus as claimed in claims 8, wherein said rendering comprises:
means for dynamically adjusting the intensity differences between adjacent images; and means for properly blending the intensity difference between consecutive images. 13. An article of manufacture comprising a storage medium having instructions stored thereon that, if executed, result in merging of a different pair of images to form a seamless panoramic image by:
extracting a set of feature points along the edges of the images, each feature point defining an edge orientations, wherein extracting the set of feature points along the edges of the images includes applying wavelet transforms to the images; obtaining a set of registration parameters by determining an initial set of feature points from a first one of the images which matches a set of feature points of a second one of the images; and rendering a seamless panoramic image using the first and second images with the set of registration parameters. 14. An article of manufacture as claimed in claim 13, wherein the instructions, if executed, further result in merging of a different pair of images to form a seamless panoramic image by:
calculating the edge orientation of one or more feature points; comparing the orientation difference between the matching pair; calculating the value of correlation of the matching pair; and comparing the value of correlation with a predefined threshold. 15. An article of manufacture as claimed in claim 13, wherein the instructions, if executed, further result in merging of a different pair of images to form a seamless panoramic image by:
calculating an edge correlation for each image; locating the feature point whose edge response is a maxima within a window; and comparing the maxima with a predefined threshold. 16. An article of manufacture as claimed in claim 13, wherein the instructions, if executed, further result in merging of a different pair of images to form a seamless panoramic image by:
determining an initial set of matching pairs for registration; calculating a quality value for the initial set of matching pairs; updating the matching result according to the quality value of the match; imposing an angle consistency constraint to filter out impossible matches; and using a voting technique to obtain the registration parameters. 17. An article of manufacture as claimed in claims 13, wherein the instructions, if executed, further result in merging of a different pair of images to form a seamless panoramic image by:
dynamically adjusting the intensity differences between adjacent images; and properly blending the intensity difference between consecutive images. 18. A system for merging a pair of images to form a panoramic image comprising:
a memory capable of storing one or more images acquired from an image device; and at least one processor coupled to said memory, said processor being capable of:
extracting a set of feature points along the edges of the images by applying wavelet formations to the images, one or more feature point defining an edge of orientation;
obtaining a set of registration parameters by determining an initial set of feature points from a first one of the images which matches a set of feature points of a second one of the images; and
rendering a seamless panoramic image using the first and second images with the set of registration parameters.
19. The system as claimed in claim 18, wherein said processor is further capable of:
calculating an edge correlation for each image; locating the feature point whose edge response is a maxima within a window; and comparing the maxima with a predefined threshold. 20. The system as claimed in claim 18, wherein said processor is further capable of:
determining an initial set of matching pairs for registration; calculating a quality value for the initial set of matching pairs; updating the matching result according to the quality value of the match; imposing an angle consistency constraint to filter out impossible matches; and using a voting technique to obtain the registration parameters. Description The present invention relates to an apparatus, algorithm, and method for stitching different pieces of images of a scene into a panoramic environment map. The most common way to electronically represent the real world is with image data. Unlike traditional graph-based systems, there are systems which use panoramic images to construct a virtual world. The major advantage of a system which uses panoramic images is that very vivid and photo-realistic rendering results can be obtained even when using PCs. In addition, the cost of constructing the virtual world is independent of scene complexity. In such systems, panoramic images are stitched together into a panoramic map from several individual images which are acquired by rotating a camera horizontally or vertically. This panoramic map can be used in different applications such as movie special effects, the creation of virtual reality, or games. A typical problem is how to stitch the different pieces of a scene into a larger picture or map. One approach to address this problem is to manually establish correspondences between images to solve unknown parameters of their relative transformation. Because manual methods are tedious for large applications, automatic schemes are preferably used for generating a seamless panoramic image from different pieces of images. One proposed approach uses a nonlinear minimization algorithm for automatically stitching panoramic images by minimizing the discrepancy in intensities between images. This approach has the advantage of not requiring easily identifiable features. However, this technique does not guarantee finding the global minimum if the selection of starting points is not proper. Further because the optimization process is time-consuming, the approach is inefficient. In this invention, the domain of images under consideration is panoramic images. The invention allows users to generate panoramic images from a sequence of images acquired by a camera rotated about its optical center. In general, the invention combines feature extraction, correlation, and relaxation techniques to get a number of reliable and robust matching pairs used to derive registration parameters. Based on the obtained registration parameters, different pieces of consecutive images can-be stitched together to obtain a seamless panoramic image. In a first aspect, a method of merging a pair of images to form a seamless panoramic image includes the following steps. A set of feature points along the edges of the images is extracted, each feature point defining an edge orientation. A set of registration parameters is obtained by determining an initial set of feature points from a first one of the images which matches a set of feature points of a second one of the images. A seamless panoramic image is rendered using the first and second images with the set of registration parameters. The invention provides a feature-based approach for automatically stitching panoramic images acquired by a rotated camera and obtaining a set of matching pairs from a set of feature points for registration. Since the feature points are extracted along the edges, each feature point specifies an edge orientation. Because the orientation difference between two panoramic images is relatively small, the difference between edge orientations of two feature points is also small if they are good matches. Based on this assumption, edge information of feature points can be used to eliminate in advance many false matches by checking their orientation difference. Moreover, many unnecessary calculations involving cross-correlation can be screened in advance, thereby significantly reducing the search time needed for obtaining correct matching pairs. After checking, by calculating the value of correlation of the remaining matching pairs, a set of possible matches can be selected with a predefined threshold. The set of possible matches are further verified through a relaxation scheme by calculating the quality of their matches. Once all of the correct matching pairs are found, they are then used to derive registration parameters. In this invention, an iterative scheme is applied to increase the reliability in providing matching results. Since only three iteration or fewer are needed and only a few feature points are involved in the matching pairs, the whole procedure can be accomplished very efficiently. Also, as discussed above, because the orientation difference of two feature points is checked in advance (before matching). Many calculations involving cross-correlation are not required and the efficiency of stitching is significantly improved. Compared with conventional algorithms, the proposed scheme offers improved efficiency and reliability for stitching images. Embodiments of this aspect of the invention may include one or more of the following features. In one embodiment, a set of feature points are first extracted through wavelet formations. Among other advantages, the invention uses wavelet transforms to obtain a number of feature points with edge orientations. Such edge information can speed up the entire registration process by eliminating many impossible matches in advance and avoiding many unnecessary calculations of correlation. The method determines a number of reliable and robust matching pairs through relaxation. The method also measures the quality of a matching pair, imposes angle consistency constraint for improving the robustness of registration, and uses a voting concept to get the desired solution from the set of final matching results. In other embodiments, the method forms the final panoramic image with the help of the registration results. In particular, the method adjusts the intensity differences between consecutive input images and blends the intensities of adjacent images to obtain a seamless panoramic image. The final panoramic images can then be used to build a virtual world. Still other embodiments may include one or more of the following features: For example, the method selects a number of feature points through wavelet transforms. Each feature point is associated with an edge orientation so that the speed of the registration process is increased. The method uses an angle constraint to construct a set of matching pairs, which are used to obtain reliable matching results through a relaxation and a voting technique. The set of matching results are then used to form the final seamless panoramic image. Constructing an initial set of matching pairs for registration includes comparing the edge orientation differences of feature points in one image and its corresponding feature points in another, calculating the values of correlation of each possible matching pair, and thresholding them with a predefined threshold. Getting reliable matching results through relaxation and a voting technique includes calculating the quality of a matching pair, imposing angle consistency constraint to filter out impossible matching pairs, updating matching results through relaxation, and using the voting technique to obtain the reliable registration parameters. In addition, it refines the final registration results by using the correlation technique with a proper starting point. Forming the final panoramic images includes dynamically adjusting and properly blending the intensity differences between adjacent images. In another aspect, the invention features a system for merging pairs of images to form a panoramic image. The system includes an imaging device which, in operation, acquires a series of images, a storage for storing a series of images, a memory which stores computer code, and at least one processor which executes computer code to extract a set of feature points along the edges of the images, each feature point defining an edge orientation and to obtain a set of registration parameters by determining an initial set of feature points from a first one of the images which matches a set of feature points of a second one of the images, and to render a seamless panoramic image using the first and second images with the set of registration parameters. Other advantages and features of the invention will become apparent from the following description, including the claims and the drawings. Referring first to the flow diagram of In general, it is difficult to seamlessly stitch two adjacent images together to form a panoramic image due to perspective distortion introduced by a camera. To remove the effects of this distortion, these images are preferably reprojected onto a simple geometry, e.g., a cube, a cylinder, or a sphere. In many applications, a cylindrical geometry is preferable since its associated geometrical transformation is simple. In this example, the cylindrical geometry is used. Referring again to -
- Condition 1: P(x,y) must be an edge point of the image I(x,y). This means that P(x,y) is a local maxima of M
_{2}_{ j }I(x, y), and M_{2}_{ j }I(x, y)>a threshold (step**34**), - Condition 2:
R _{2}(1, x, y)>M_{2}_{ j }I(x, y) (step**36**), - Condition 3:
- Condition 1: P(x,y) must be an edge point of the image I(x,y). This means that P(x,y) is a local maxima of M
Referring to Let P be a feature point and N After the orientation is estimated, all feature points u will associate with an edge orientation A(u). For a feature point p For a feature point p
where u _{q} _{j}} is qualified as a possible matching pair if the following conditions are satisfied (step 33):
Referring to _{i}}_{i=1, 2 . . . }be the set of matching pairs which satisfy Conditions 1, 2, 3, and 4 above, where p_{i }is a point in image I_{a}(x, y) and q_{i }is another point in image I_{b}(x, y). Let Ne_{I} _{ a }(p_{k}) and Ne_{I} _{ b }(q_{k}) denote the neighbors of p_{k }and q_{k }within an area of radius R, respectively. Assume that NP_{p} _{ i } _{q} _{ j }={n_{k} ^{1} n_{k} ^{2}}_{k=1, 2 . . . }is the set of matching pairs, where n_{k} ^{1}εNe_{I} _{ a }(p_{i}), n_{k} ^{2}εNe_{I} _{ b }(q_{j}), and all elements of NP_{p} _{ i } _{q} _{ j }belong to MP_{I} _{ a } _{I} _{ b }. The proposed method is based on a concept that if {p_{i} q_{i}} and {p_{j} q_{j}} provide a pair of good matches, the distance between p_{i }and p_{j }should be similar to the one between q_{i }and q_{j}. Based on this assumption, we can measure the quality of a matching pair {p_{i} q_{i}} according to how many matches {n_{j} ^{1} n_{j} ^{2}} in NP_{P} _{ i } _{q} _{j }whose distance d(p_{i}, n_{j} ^{1}) is similar to the distance d(q_{i}, n_{j} ^{2}), where d(u_{i}, u_{j})=|u_{i}−u_{j}∥, the Euclidean distance between two points u_{i }and u_{j}. With this concept, the measure of the quality of a match {p_{i} q_{i}} is defined as follows:
_{k} ^{2}} in NP_{p} _{ i } _{q} _{ i }monotonically decreases based on the value of dist(i,k). Further, if the value of u(i,k) is larger than the threshold T_{2}, the contribution of {n_{k} ^{1} n_{k} ^{2}} is set to zero.
Referring to _{i}} in MP_{I} _{ a } ^{,I} _{ b }(step 51), relative quality value of each pair is obtained to G_{I} _{ a } _{I} _{ b }(i) to further eliminate false matches (step 51). Now, based on the quality value of each candidate match, a relaxation technique is used to eliminate false candidates for further registration. If we define the energy function as follow:
There are several strategies for updating the matching candidates. In one application an update strategy, referred to here as “some-looser-take-nothing” is used to update the matching candidates (step 52). First, according to the quality value of G_{I} _{ a } _{I} _{ b }(i), we sort each element of MP_{I} _{ a } _{,I} _{ b }is sorted in increasing order (step 52). Then, a predetermined percentage Q% of matches in are eliminated as the impossible matches. Thus, the remaining (100−Q%) of matches are selected as potential matches for further relaxation (step 54). In our implementation, Q is set to 25. Three iterations are generally sufficient for achieving relaxation (step 53).
On the other hand, in order to make the matching results more reliable, the method includes an “angle consistency” constraint (step _{i}} and {p_{j} q_{j}} are well matched, the angle between {right arrow over (p_{i}p_{j})} and {right arrow over (q_{i}q_{j})} must be close to zero. In this case, during this first iteration, for each element {p_{i} q_{i}} in MP_{I} _{ a } _{,I} _{ b }, a counter CA_{i }is used to record the number of matches {p_{k} q_{k}} in MP_{I} _{ a } _{,I} _{ b }where the angle between {right arrow over (p_{i}p_{k})} and {right arrow over (q_{i}q_{k})} is less than a predefined threshold θ. According to the value of CA_{i}, the elements of MP_{I} _{ a } _{,I} _{ b }are sorted in increasing order. The first Q% of matches in MP_{I} _{ a } _{,I} _{ b }are considered to be impossible matches.
After applying the relaxation process, a set of reliable matches is obtained. Referring to _{i}}_{i=1, 2 . . . Ne}, where N_{e }is the total number of elements in CP_{I} _{ a } _{,I} _{ b }. In general, the 2-D point sets {u_{i}} and {v_{i}} satisfy the following relation:
v _{i}=u_{i}+T, for i=1,2,3 . . . , N_{e},where T is the desired solution. However, in real cases, different pairs {u _{i} v_{i}} will lead to different offsets T_{i}. Therefore, a voting technique is used to measure the quality of different solutions T_{i}. Let S(i) denote a counter which records the number of solutions T_{k }consistent with T_{i}. Two solutions T_{i }and T_{k }are said to be consistent if the distance between T_{i }and T_{k }is less than a predefined threshold. Since there are N_{e }elements in CP_{I} _{ a } _{,I} _{ b }, the total number of consistency tests will be N_{e}(N_{e}−1)/2. After applying the consistency test, the offset Note that due to noise and image quality, the positions of feature points will not be precisely located and the accuracy of T is affected. In practical implementation, the y-component _{I} _{ a } _{,I} _{ b }having the highest quality value. Given point u and the offset _{h }using the correlation technique. However, if there is little texture information within the local neighborhood of the point u, the above approach will not necessarily provide a satisfactory solution. Let
In general, when stitching two adjacent images, discontinuities of intensity exist between their common areas. Therefore, step of I In particular, referring to
After this adjusting step, the intensities of I The second stage uses a ray-casting method to blend different pixel intensities together. Referring to Referring to Referring to Referring to Patent Citations
Non-Patent Citations
Referenced by
Classifications
Legal Events
Rotate |