US 7289138 B2 Abstract Provides a system for detecting an intersection between more than one panoramic video sequence and detecting the orientation of the sequences forming the intersection. Video images and corresponding location data are received. If required, the images and location data is processed to ensure the images contain location data. An intersection between two paths is then derived from the video images by deriving a rough intersection between two images, determining a neighborhood for the two images, and dividing each image in the neighborhood into strips. An identifying value is derived from each strip to create a row of strip values which are then converted to the frequency domain. A distance measure is taken between strips in the frequency domain, and the intersection is determined from the images having the smallest distance measure between them. The orientation between the two paths may also be determined in the frequency domain by using the phases of signals representing the images in the Fourier domain or performing a circular cross correlation of two vectors representing the images.
Claims(36) 1. A method for detecting an intersection as displayed in two intersecting panoramic video frame sequences, the method comprising:
receiving data representing two intersecting panoramic video frame sequences of an identical location captured at different times and from different directions, the sequences each including a plurality of frames depicting an image of the intersection between the video frame sequences;
detecting an intersection frame from each of the panoramic video frame sequences using only the data representing the two panoramic video frame sequences, the intersection comprised of the intersection frame from each panoramic video frame sequence; wherein detecting the intersection frame from each of the panoramic video frame sequences includes detecting frame images that are most similar from each of the panoramic video frame sequences; and
displaying one of the two intersecting panoramic video frame sequences up to and including the intersection frame, then displaying the other of the two intersecting panoramic video frame sequences starting from the intersection frame.
2. A method for detecting an intersection as displayed in two intersecting panoramic video frame sequences, the method comprising:
receiving data representing two intersecting panoramic video frame sequences, the sequences each including a plurality of frames depicting an image of the intersection between the video frame sequences;
detecting an intersection frame from each of the panoramic video frame sequences using only the data representing the two panoramic video frame sequences, the intersection comprised of the intersection frame from each panoramic video frame sequence; wherein detecting the intersection frame from each of the panoramic video frame sequences includes detecting frame images that are most similar from each of the panoramic video frame sequences, and wherein detecting frame images that are most similar includes:
segmenting each frame into at least one strip, wherein the frames are divided into rows and at least one strip is associated with each row;
determining an identifying value for each strip; and
determining a distance measurement between strips in corresponding rows in each frame to determine which frames from the two sequences are closest to each other, wherein taking the distance measure includes processing the identifying value of each strip; and
displaying one of the two intersecting panoramic video frame sequences up to and including the intersection frame, then displaying the other of the two intersecting panoramic video frame sequences starting from the intersection frame.
3. A method for detecting an intersection as displayed in two intersecting panoramic video frame sequences, the method comprising:
receiving data representing two intersecting panoramic video frame sequences, the sequences each including a plurality of frames depicting an image of the intersection between the video frame sequences;
detecting an intersection frame from each of the panoramic video frame sequences using only the data representing the two panoramic video frame sequences, the intersection comprised of the intersection frame from each panoramic video frame sequence; wherein detecting the intersection frame from each of the panoramic video frame sequences includes detecting frame images that are most similar from each of the panoramic video frame sequences, and wherein detecting frame images that are most similar includes:
segmenting each frame into at least one strip, wherein the frames are divided into rows and at least one strip is associated with each row;
determining an identifying value for each strip; and
determining a distance measurement between strips in corresponding rows in each frame to determine which frames from the two sequences are closest to each other, wherein taking the distance measure includes processing the identifying value of each strip; wherein the identifying value is an average intensity of the pixel values for the pixels contained in the strip; and
displaying one of the two intersecting panoramic video frame sequences up to and including the intersection frame, then displaying the other of the two intersecting panoramic video frame sequences starting from the intersection frame.
4. A method for detecting an intersection as displayed in two intersecting panoramic video frame sequences, the method comprising:
receiving data representing two intersecting panoramic video frame sequences, the sequences each including a plurality of frames depicting an image of the intersection between the video frame sequences;
detecting an intersection frame from each of the panoramic video frame sequences using only the data representing the two panoramic video frame sequences, the intersection comprised of the intersection frame from each panoramic video frame sequence; wherein detecting the intersection frame from each of the panoramic video frame sequences includes detecting frame images that are most similar from each of the panoramic video frame sequences, and wherein detecting frame images that are most similar includes:
segmenting each frame into at least one strip, wherein the frames are divided into rows and at least one strip is associated with each row;
determining an identifying value for each strip; and
determining a distance measurement between strips in corresponding rows in each frame to determine which frames from the two sequences are closest to each other, wherein taking the distance measure includes processing the identifying value of each strip; wherein taking the distance measurement includes:
receiving Fourier spectra data associated with the strips in a corresponding row of each frame; and
comparing the Fourier spectra data to derive a distance measurement between the frames of each panoramic video frame sequence; and
5. A method for detecting an intersection as displayed in two intersecting panoramic video frame sequences, the method comprising:
detecting an intersection frame from each of the panoramic video frame sequences using only the data representing the two panoramic video frame sequences, the intersection comprised of the intersection frame from each panoramic video frame sequence; wherein detecting the intersection frame from each of the panoramic video frame sequences includes detecting frame images that are most similar from each of the panoramic video frame sequences, and wherein detecting frame images that are most similar includes:
determining an identifying value for each strip; and
determining a distance measurement between strips in corresponding rows in each frame to determine which frames from the two sequences are closest to each other, wherein taking the distance measure includes processing the identifying value of each strip; wherein taking the distance measurement includes:
receiving Fourier spectra data associated with the strips in a corresponding row of each frame; and
comparing the Fourier spectra data to derive a distance measurement between the frames of each panoramic video frame sequence; wherein
determining the intersection is to be associated with the frames having the shortest distance measurement between them; and
6. A method for detecting an intersection as displayed in two intersecting panoramic video frame sequences, the method comprising:
detecting an intersection frame from each of the panoramic video frame sequences using only the data representing the two panoramic video frame sequences, the intersection comprised of the intersection frame from each panoramic video frame sequence; wherein detecting the intersection frame from each of the panoramic video frame sequences includes detecting frame images that are most similar from each of the panoramic video frame sequences, and wherein detecting frame images that are most similar includes:
determining an identifying value for each strip; and
determining a distance measurement between strips in corresponding rows in each frame to determine which frames from the two sequences are closest to each other, wherein taking the distance measure includes processing the identifying value of each strip; wherein taking the distance measurement includes:
receiving Fourier spectra data associated with the strips in a corresponding row of each frame; and
comparing the Fourier spectra data to derive a distance measurement between the frames of each panoramic video frame sequence; wherein the distance measurement is one of a Euclidean, cosine, and normalized distance; and
7. A method for detecting an intersection as displayed in two intersecting panoramic video frame sequences, the method comprising:
detecting an intersection frame from each of the panoramic video frame sequences using only the data representing the two panoramic video frame sequences, the intersection comprised of the intersection frame from each panoramic video frame sequence; wherein detecting the intersection frame from each of the panoramic video frame sequences includes detecting frame images that are most similar from each of the panoramic video frame sequences, and wherein detecting frame images that are most similar includes:
determining an identifying value for each strip; and
determining a distance measurement between strips in corresponding rows in each frame to determine which frames from the two sequences are closest to each other, wherein taking the distance measure includes processing the identifying value of each strip; wherein taking the distance measurement includes:
receiving Fourier spectra data associated with the strips in a corresponding row of each frame; and
comparing the Fourier spectra data to derive a distance measurement between the frames of each panoramic video frame sequence; wherein the distance measurement is one of a Euclidean, cosine, and normalized distance; wherein the distance measurement is weighted; and
8. A method for detecting an intersection as displayed in two intersecting panoramic video frame sequences, the method comprising:
detecting an intersection frame from each of the panoramic video frame sequences using only the data representing the two panoramic video frame sequences, the intersection comprised of the intersection frame from each panoramic video frame sequence; wherein detecting the intersection frame from each of the panoramic video frame sequences includes detecting frame images that are most similar from each of the panoramic video frame sequences, and wherein detecting frame images that are most similar includes:
determining an identifying value for each strip; and
determining a distance measurement between strips in corresponding rows in each frame to determine which frames from the two sequences are closest to each other, wherein taking the distance measure includes processing the identifying value of each strip; wherein taking the distance measurement includes:
receiving Fourier spectra data associated with the strips in a corresponding row of each frame; and
comparing the Fourier spectra data to derive a distance measurement between the frames of each panoramic video frame sequence; wherein the distance measurement is one of a Euclidean, cosine, and normalized distance; wherein the distance measurement is weighted; wherein the weighted distance measurement includes weights to reduce a high and low band of the Fourier spectra; and
9. A method for detecting an intersection as displayed in two intersecting panoramic video frame sequences, the method comprising:
receiving a first set of data representing two panoramic video frame sequences, the sequences each including a plurality of frames depicting an image of an intersection, the first set of data including pixel data and location data, the location data associated with at least one frame from each sequence;
determining missing location data for any frame not already associated with location data;
detecting the intersection using only the first set of data representing the two panoramic video frame sequences; wherein detecting the intersection includes:
determining a rough intersection between the two panoramic video frame sequences, the rough intersection determined by comparing the location data for each frame;
determining a neighborhood having a set of neighborhood frames, the neighborhood including the rough intersection, the neighborhood frames including frames from each of the panoramic video sequences located within a parameter relative to the rough intersection; and
comparing data derived from the neighborhood frames to determine the most similar frame images from each of the panoramic video frame sequences; and
10. A method for detecting an intersection as displayed in two intersecting panoramic video frame sequences, the method comprising:
receiving a first set of data representing two panoramic video frame sequences, the sequences each including a plurality of frames depicting an image of an intersection, the first set of data including pixel data and location data, the location data associated with at least one frame from each sequence;
determining missing location data for any frame not already associated with location data;
detecting the intersection using only the first set of data representing the two panoramic video frame sequences; wherein detecting the intersection includes:
determining a rough intersection between the two panoramic video frame sequences, the rough intersection determined by comparing the location data for each frame;
determining a neighborhood having a set of neighborhood frames, the neighborhood including the rough intersection, the neighborhood frames including frames from each of the panoramic video sequences located within a parameter relative to the rough intersection; and
comparing data derived from the neighborhood frames to determine the most similar frame images from each of the panoramic video frame sequences; wherein comparing data includes:
deriving strips from the panoramic video frames;
determining an identifying value for each strip; and
processing the identifying values to determine a distance measurement between images; and
11. A method for detecting an intersection as displayed in two intersecting panoramic video frame sequences, the method comprising:
receiving a first set of data representing two panoramic video frame sequences, the sequences each including a plurality of frames depicting an image of an intersection, the first set of data including pixel data and location data, the location data associated with at least one frame from each sequence;
determining missing location data for any frame not already associated with location data;
detecting the intersection using only the first set of data representing the two panoramic video frame sequences; wherein detecting the intersection includes:
determining a rough intersection between the two panoramic video frame sequences, the rough intersection determined by comparing the location data for each frame;
determining a neighborhood having a set of neighborhood frames, the neighborhood including the rough intersection, the neighborhood frames including frames from each of the panoramic video sequences located within a parameter relative to the rough intersection; and
comparing data derived from the neighborhood frames to determine the most similar frame images from each of the panoramic video frame sequences; wherein comparing data includes:
deriving strips from the panoramic video frames;
determining an identifying value for each strip; and
processing the identifying values to determine a distance measurement between images; wherein the parameter is a minimum distance from the rough intersection frames; and
12. A method for detecting an intersection as displayed in two intersecting panoramic video frame sequences, the method comprising:
determining missing location data for any frame not already associated with location data;
detecting the intersection using only the first set of data representing the two panoramic video frame sequences; wherein detecting the intersection includes:
comparing data derived from the neighborhood frames to determine the most similar frame images from each of the panoramic video frame sequences; wherein comparing data includes:
deriving strips from the panoramic video frames;
determining an identifying value for each strip; and
processing the identifying values to determine a distance measurement between images; wherein the parameter is a minimum time from the rough intersection frames; and
13. A method for detecting the orientation between two intersecting panoramic video frame sequences at an intersection as displayed in the two video sequences, the method comprising:
receiving data representing two intersecting panoramic video frame sequences, the sequences each including a plurality of frames depicting an image of the intersection between the panoramic video frame sequences;
detecting an intersection frame from each panoramic video sequence, the intersection frames containing frame images that are most similar between the two intersecting panoramic video frame sequences, the intersection frames detected using only the data representing the two panoramic video frame sequences;
detecting a relative orientation between the intersection frames of the panoramic video sequence using only the data representing the two panoramic video frame sequences; wherein detecting a relative orientation includes:
segmenting a frame into at least one strip, wherein the frames are divided into rows and at least one strip is associated with each row;
determining an identifying value for each strip; and
processing the identifying values to determine the relative orientation between images; and
14. A method for detecting the orientation between two intersecting panoramic video frame sequences at an intersection as displayed in the two video sequences, the method comprising:
receiving data representing two intersecting panoramic video frame sequences, the sequences each including a plurality of frames depicting an image of the intersection between the panoramic video frame sequences;
detecting an intersection frame from each panoramic video sequence, the intersection frames containing frame images that are most similar between the two intersecting panoramic video frame sequences, the intersection frames detected using only the data representing the two panoramic video frame sequences;
detecting a relative orientation between the intersection frames of the panoramic video sequence using only the data representing the two panoramic video frame sequences; wherein detecting a relative orientation includes:
segmenting a frame into at least one strip, wherein the frames are divided into rows and at least one strip is associated with each row;
determining an identifying value for each strip; and
processing the identifying values to determine the relative orientation between images; wherein processing the identifying values includes:
receiving Fourier spectra data associated with the identifying values of the strips; and
determining relative orientation as the slope of the phase difference between the Fourier spectra associated with the corresponding rows of the frames; and
15. A method for detecting the orientation between two intersecting panoramic video frame sequences at an intersection as displayed in the two video sequences, the method comprising:
receiving data representing two intersecting panoramic video frame sequences, the sequences each including a plurality of frames depicting an image of the intersection between the panoramic video frame sequences;
detecting an intersection frame from each panoramic video sequence, the intersection frames containing frame images between the two intersecting panoramic video frame sequences, the intersection frames detected using only the data representing the two panoramic video frame sequences;
detecting a relative orientation between the intersection frames of the panoramic video sequence using only the data representing the two panoramic video frame sequences; wherein detecting a relative orientation includes:
segmenting a frame into at least one strip, wherein the frames are divided into rows and at least one strip is associated with each row;
determining an identifying value for each strip; and
processing the identifying values to determine the relative orientation between images; wherein processing the identifying values includes:
receiving Fourier spectra data associated with the identifying values of the strips; and
determining the relative orientation by deriving the maximum of the cross correlation between the identifying data of the first and second panoramic video images, the cross correlation values derived from the Fourier spectra; and
16. A method for detecting the orientation between two intersecting panoramic video frame sequences at an intersection as displayed in the two video sequences, the method comprising:
detecting an intersection frame from each panoramic video sequence, the intersection frames containing frame images that are most similar between the two intersecting panoramic video frame sequences, the intersection frames detected using only the data representing the two panoramic video frame sequences;
detecting a relative orientation between the intersection frames of the panoramic video sequence using only the data representing the two panoramic video frame sequences; wherein detecting a relative orientation includes:
determining an identifying value for each strip; and
processing the identifying values to determine the relative orientation between images; wherein processing the identifying values includes:
receiving Fourier spectra data associated with the identifying values of the strips; and
determining the relative orientation by deriving the maximum of the cross correlation between the identifying data of the first and second panoramic video images, the cross correlation values derived from the Fourier spectra; wherein deriving the maximum cross correlation includes:
processing the Fourier spectra using complex conjugation to derive a set of data; and
taking the inverse DFT of the set of data; and
17. A method for detecting the orientation between two intersecting panoramic video frame sequences at an intersection as displayed in the two video sequences, the method comprising:
detecting a relative orientation between the intersection frames of the panoramic video sequence using only the data representing the two panoramic video frame sequences; wherein detecting a relative orientation includes:
determining an identifying value for each strip; and
processing the identifying values to determine the relative orientation between images; wherein processing the identifying values includes:
receiving Fourier spectra data associated with the identifying values of the strips; and
determining the relative orientation by deriving the maximum of the cross correlation between the identifying data of the first and second panoramic video images, the cross correlation values derived from the Fourier spectra; wherein deriving the maximum cross correlation includes:
processing the Fourier spectra using complex conjugation to derive a set of data; and
taking the inverse DET of the set of data; wherein the distance measurement is weighted; and
18. A method for detecting the orientation between two intersecting panoramic video frame sequences at an intersection as displayed in the two video sequences, the method comprising:
detecting a relative orientation between the intersection frames of the panoramic video sequence using only the data representing the two panoramic video frame sequences; wherein detecting a relative orientation includes:
determining an identifying value for each strip; and
processing the identifying values to determine the relative orientation between images; wherein processing the identifying values includes:
receiving Fourier spectra data associated with the identifying values of the strips; and
determining the relative orientation by deriving the maximum of the cross correlation between the identifying data of the first and second panoramic video images, the cross correlation values derived from the Fourier spectra; wherein deriving the maximum cross correlation includes:
processing the Fourier spectra using complex conjugation to derive a set of data; and
taking the inverse DFT of the set of data; wherein the distance measurement is weighted; wherein the weighted distance measurement includes weights to reduce a high and low band of the Fourier spectra; and
19. A method for detecting the orientation between two intersecting panoramic video frame sequences at an intersection as displayed in the two video sequences, the method comprising:
receiving pixel data representing two intersection panoramic video frames, wherein each of the two panoramic video frames is associated with a separate panoramic video frame sequence, the panoramic video frame sequences intersecting at the intersection, wherein the intersection panoramic video frames depict the most similar image between the video frame sequences;
detecting a relative orientation between the intersection panoramic video frames of the panoramic video sequence using only the pixel data representing the two panoramic video frame sequences; wherein detecting a relative orientation includes:
determining an identifying value for each strip; and
processing the identifying values to determine the relative orientation between images; and
20. A method for detecting the orientation between two intersecting panoramic video frame sequences at an intersection as displayed in the two video sequences, the method comprising:
receiving pixel data representing two intersection panoramic video frames, wherein each of the two panoramic video frames is associated with a separate panoramic video frame sequence, the panoramic video frame sequences intersecting at the intersection, wherein the intersection panoramic video frames depict the most similar image between the video frame sequences;
detecting a relative orientation between the intersection panoramic video frames of the panoramic video sequence using only the pixel data representing the two panoramic video frame sequences; wherein detecting a relative orientation includes:
determining an identifying value for each strip; and
processing the identifying values to determine the relative orientation between images; wherein processing the identifying values includes:
receiving Fourier spectra data associated with the identifying values of the strips; and
determining relative orientation as the slope of the phase difference between the Fourier spectra associated with the corresponding rows of the frames; and
21. A method for detecting the orientation between two intersecting panoramic video frame sequences at an intersection as displayed in the two video sequences, the method comprising:
receiving pixel data representing two intersection panoramic video frames, wherein each of the two panoramic video frames is associated with a separate panoramic video frame sequence, the panoramic video frame sequences intersecting at the intersection, wherein the intersection panoramic video frames depict the most similar image between the video frame sequences;
detecting a relative orientation between the intersection panoramic video frames of the panoramic video sequence using only the pixel data representing the two panoramic video frame sequences; wherein detecting a relative orientation includes:
determining an identifying value for each strip; and
processing the identifying values to determine the relative orientation between images; wherein processing the identifying values includes:
receiving Fourier spectra data associated with the identifying values of the strips; and
determining the relative orientation by deriving the maximum of the cross correlation between the identifying data of the first and second panoramic video images, the cross correlation values derived from the Fourier spectra; and
22. A method for detecting the orientation between two intersecting panoramic video frame sequences at an intersection as displayed in the two video sequences, the method comprising:
detecting a relative orientation between the intersection panoramic video frames of the panoramic video sequence using only the pixel data representing the two panoramic video frame sequences; wherein detecting a relative orientation includes:
determining an identifying value for each strip; and
processing the identifying values to determine the relative orientation between images; wherein processing the identifying values includes:
receiving Fourier spectra data associated with the identifying values of the strips; and
determining the relative orientation by deriving the maximum of the cross correlation between the identifying data of the first and second panoramic video images, the cross correlation values derived from the Fourier spectra; wherein deriving the maximum cross correlation includes:
processing the Fourier spectra using complex conjugation to derive a set of data; and
taking the inverse DFT of the set of and displaying one of the two intersecting panoramic video frame sequences up to and including the intersection frame, then displaying the other of the two intersecting panoramic video frame sequences starting from the intersection frame.
23. A method for detecting the orientation between two intersecting panoramic video frame sequences at an intersection as displayed in the two video sequences, the method comprising:
detecting a relative orientation between the intersection panoramic video frames of the panoramic video sequence using only the pixel data representing the two panoramic video frame sequences; wherein detecting a relative orientation includes:
determining an identifying value for each strip; and
processing the identifying values to determine the relative orientation between images; wherein processing the identifying values includes:
receiving Fourier spectra data associated with the identifying values of the strips; and
determining the relative orientation by deriving the maximum of the cross correlation between the identifying data of the first and second panoramic video images, the cross correlation values derived from the Fourier spectra; wherein deriving the maximum cross correlation includes:
processing the Fourier spectra using complex conjugation to derive a set of data; and
taking the inverse DFT of the set of data; wherein the distance measurement is weighted to reduce a high and low band of the Fourier spectra; and
24. A method for determining an intersection and orientation between two intersecting panoramic video frame sequences, the method comprising:
receiving a first set of data representing two intersecting panoramic video frame sequences, the first set of data including pixel data and location data, the location data associated with at least one frame from each sequence, the sequences each including a plurality of frames depicting an image of the intersection between the video frame sequences;
determining location data for any frame not already associated with location data;
determining a rough estimate of the intersection between the two panoramic video frame sequences, the rough intersection determined by comparing the location data for each frame;
determining a neighborhood having a set of neighborhood frames, the neighborhood including the rough intersection, the neighborhood frames including frames from each of the panoramic video sequences within a parameter of the rough estimate of the intersection;
segmenting the neighborhood frames into at least one strip, wherein the frames are divided into rows and at least one strip is associated with each row, each strip comprised of a plurality of pixels;
determining an identifying value for each strip;
receiving Fourier spectra data associated with the strips in a corresponding row of each frame;
comparing the Fourier spectra data to derive a distance measurement between the frames of the panoramic video sequences;
determining the relative orientation between the frames having the shortest distance measurement using the Fourier spectra data; and
25. The method of
26. The method of
27. The method of
28. A computer readable medium stored thereon a computer program, when executed by a computer, the computer program to determine an intersection and orientation between two intersecting panoramic video frame sequences, comprising:
computer code for receiving a first set of data representing two intersecting panoramic video frame sequences, the first set of data including pixel data and location data, the location data associated with at least one frame from each sequence, the sequences each including a plurality of frames depicting an image of the intersection between the video frame sequences;
computer code for determining location data for any frame not already associated with location data;
computer code for determining a rough estimate of the intersection between the two panoramic video frame sequences, the rough intersection determined by comparing the location data for each frame;
computer code for determining a neighborhood having a set of neighborhood frames, the neighborhood including the rough intersection, the neighborhood frames including frames from each of the panoramic video sequences within a parameter of the rough estimate of the intersection;
computer code for segmenting the neighborhood frames into at least one strip, wherein the frames are divided into rows and at least one strip is associated with each row, each strip comprised of a plurality of pixels;
computer code for determining an identifying value for each strip;
computer code for receiving Fourier spectra data associated with the strips in a corresponding row of each frame;
computer code for comparing the Fourier spectra data associated with the corresponding rows of the frames to derive a distance measurement between the frames of the panoramic video sequences; and
computer code for determining the relative orientation between the frames having the shortest distance measurement using the Fourier spectra data.
29. The method of
30. The method of
31. A computer readable medium stored thereon a computer program, when executed by a computer, the computer program to detect an intersection as displayed in two intersecting panoramic video frame sequences, comprising:
computer code for receiving pixel information representing two intersecting panoramic video frame sequences, the sequences each including a plurality of frames depicting an image of the intersection between the video frame sequences; and
computer code for segmenting a frame into at least one strip, wherein the frames are divided into rows and at least one strip is associated with each row, each strip containing multiple pixels;
computer code for determining an identifying value for each strip;
computer code for receiving Fourier spectra data associated with the strips in a corresponding row of each frame; and
computer code for comparing the Fourier spectra data associated with the corresponding rows of the frames to derive a distance measurement between each frame.
32. A computer readable medium stored thereon a computer program, when executed by a computer, the computer program to detect the orientation between two intersecting panoramic video frame sequences at an intersection as displayed in the two video sequences, comprising:
computer code for receiving data representing two intersecting panoramic video frame sequences, the sequences each including a plurality of frames depicting an image of the intersection between the panoramic video frame sequences;
computer code for detecting an intersection frame from each panoramic video sequence, the intersection frames containing frame images that are most similar between the intersecting panoramic video frame sequences, the intersection frames detected using only the data representing the two panoramic video frame sequences;
computer code for segmenting a frame into at least one strip, wherein the frames are divided into rows and at least one strip is associated with each row;
computer code for determining an identifying value for each strip;
computer code for receiving Fourier spectra data associated with the identifying values of the strips; and
computer code for determining the relative orientation between the intersection frames using the Fourier spectra data.
33. The computer readable medium of
34. The computer readable medium of
35. The computer readable medium of
computer code for processing the Fourier spectra data using complex conjugation to derive a second set of data; and
computer code for taking the inverse DFT of the second set of data.
36. An apparatus for determining an intersection and orientation between two intersecting panoramic video frame sequences, comprising:
a processor; and
a processor readable storage medium coupled to the processor; said processor readable storage medium containing program code for programming the apparatus to perform a method for determining an intersection and orientation between two intersecting panoramic video frame sequences, the method comprising the steps of:
receiving a first set of data representing two intersecting panoramic video frame sequences, the first set of data including pixel data and location data, the location data associated with at least one frame from each sequence, the sequences each including a plurality of frames depicting an image of the intersection between the video frame sequences; determining location data for any frame not already associated with location data;
determining a rough estimate of the intersection between the two panoramic video frame sequences, the rough intersection determined by comparing the location data for each frame;
determining a neighborhood having a set of neighborhood frames, the neighborhood including the rough intersection, the neighborhood frames including frames from each of the panoramic video sequences within a parameter of the rough estimate of the intersection;
segmenting the neighborhood frames into at least one strip, wherein the frames are divided into rows and at least one strip is associated with each row, each strip comprised of a plurality of pixels;
determining an identifying value for each strip;
receiving Fourier spectra data associated with the strips in a corresponding row of each frame;
comparing the Fourier spectra data associated with the corresponding rows of the frames to derive a distance measurement between each frame; and
determining the relative orientation between the panoramic video frames having the shortest distance measurement between them, the relative orientation determined using the Fourier spectra data.
Description The present application is related to the following United States Patents and Patent Applications, which patents/applications are assigned to the owner of the present invention, and which patents/applications are incorporated by reference herein in their entirety: U.S. Pat. No. 7,096,428, entitled “SYSTEMS AND METHODS FOR PROVIDING A SPATIALLY INDEXED PANORAMIC VIDEO,” by inventors Jonathan T. Foote and Donald G. Kimber, issued Aug. 22, 2006. A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. The current invention relates generally to video image processing, and more particularly to detecting intersection information in video sequences. Video applications have expanded in recent years for use with camcorders, televisions, and computers. As a result, use of panoramic video has expanded as well. One common use of panoramic video is to represent travel along a path. The path may be a street, a building corridor, or some path in abstract space. Representation of travel along a one way path is straightforward. A video sequence displaying travel along the path is provided to a user, thereby allowing a user to “proceed” down the path. A more difficult problem arises relates to intersections as displayed in video. The problem arises in finding an efficient basis in which to allow a user who is proceeding down a path to view travel through different paths of an intersection as displayed in video. Capturing video in every combination of **possible paths available at an intersection is impractical due to data storage, data processing, and the time and resources required to capture the video. Another possible solution is to provide a single video sequence along each direction of an intersection (thus, two video sequences for an intersection of two streets). When a change in direction is required at an intersection, the video sequence displayed for a user may change from the video sequence in which the entry to the intersection is displayed to the video sequence displaying the desired exit path from the intersection. Several problems arise with this method of providing video at an intersection. The problem may be broken into two parts. The first part is detecting when an intersection in the panoramic video appears. An intersection is an image of an identical location that occurs at different times and from different directions in the video. For example, an intersection of two streets, one oriented north-south and one oriented east-west, may be captured in panoramic video from an east-west direction as well as a north-south direction. A panoramic video sequence traveling through the intersection along each direction will capture the same location at different times and from different directions. One method for detecting an intersection is to go through all the video frames of a sequence and label the intersections manually. From a practical point of view, this process would become extremely tedious and time consuming for large scale applications, such as labeling the intersections on a city street map. Selecting the intersections by hand would also be less accurate then automatic methods, as this method would rely on the human eye for choosing between images that may have been taken only 1/30th of a second apart for video frames displayed at 30 Hz. Further, adding additional information to the video frames to be analyzed by a processor would require more processing power. The second part of the problem is to determine how to provide a smooth transition between the separate panoramic video sequences. A smooth transition is desirable to make the change from one video sequence to another video sequence appear as a single video sequence. One solution would be to label each video sequence with orientation data at the corresponding intersection frames to indicate the angle the video sequence is captured relative to the intersection. As with adding intersection data, the addition of intersection orientation data would become extremely tedious and time consuming for large scale applications, such as labeling the intersections on a city street map. Further, adding additional information to the video frames to be analyzed by a processor would require more processing power. What is needed is a system for detecting the intersection of an intersection between two separate panoramic video sequences captured while traveling through the intersection from different directions. The system should determine where the intersection is and the orientation of the images depicting the intersection. In one embodiment, the present invention provides a system for detecting an intersection in more than one panoramic video sequence. Video images and corresponding location data are received. The images and location data may be processed to ensure all the images contain location data. An intersection between two paths is derived from the video images. The orientation between the two paths may also be determined from the video images. In one embodiment, detecting an intersection between two paths includes deriving a rough intersection between two images using location data associated with each frame image. A neighborhood is then determined for the two images. Next, each image in the neighborhood is divided into strips. An identifying vector is derived from each strip to create a row of strip values. The strip values are converted to the frequency domain, a distance measure is taken between strips in the frequency domain, and the intersection is determined from the images having the smallest distance measure between them. In one embodiment, detecting the relative orientation of two panoramic images includes using the phases of signals representing the images in the Fourier domain. The slope of the phase difference between the Fourier representation of the two images may be considered to be relative orientation of the two candidate images. In another embodiment, detecting the relative orientation of two panoramic images includes performing a circular cross correlation of two vectors representing the images. In one embodiment, the present invention provides a system for detecting an intersection in more than one panoramic video sequence where the panoramic sequences capture the intersection. Video images and corresponding location data are received. The images and location data may be processed to ensure all the images contain location data. An intersection between two paths is derived from the video images. In one embodiment, the intersection is determined to include a frame from each panoramic video frame sequence, where the frames depict the most similar image of the intersection. The relative orientation between the two paths may also be determined from the video images. In one embodiment, performing the intersection detection and orientation detection include processing image information in the frequency domain. Among the areas invention may be applicable include panoramic based virtual reality systems as well as any application displaying video images where a common point or image displayed in at least two video sequences is to be recognized or processed. The present invention may be used to detect an intersection in two or more panoramic video sequences where an intersection is captured or displayed from a similar vertical position and spatial position but from any angle. A panoramic camera may provide a cylindrical panoramic image that repeats periodically every 360 degrees. Two panoramic images taken at the same spatial location with the same cylindrical axis differ in relative orientation. In this disclosure, camera rotation corresponds to the column direction, and height corresponds to row direction of panoramic images. A method In addition to video image sequences, spatial location data is received in step Once the video images and spatial location data are received in step After additional spatial location data is determined in step In the case where the intersection of two paths is being determined, the data points from the two sets of candidate panoramic images may be expressed as:
In one embodiment of the present invention, the distance between two panoramic frames may be calculated using a Fourier transform. A Fourier transform works well for matching purposes because the magnitude of the Fourier transform is invariant to phase. In one embodiment, to apply a Fourier transform, each image may be divided into rows. A set of values may then be derived from each row. In one embodiment, a Fourier transform could be taken on a specified row of pixel values of each panoramic image. The resultant Fourier spectra derived from different frames within the neighborhood could then be compared to determine the closest images. However, practical limitations may prohibit direct application of such spectra matching. Limitations such as different axis of projection of a camera and camera position displacement can cause changes in a panoramic image and the rows within the images. In another embodiment of the present invention, a Fourier transform is taken on only a portion of the panoramic image. In this embodiment, a panoramic image having H rows and W columns of pixels is segmented into R strips and C columns, wherein R is less than H and C is less than W. Thus, each strip is comprised of a number of pixel rows equaling H/R and a number of pixel columns equaling W/C. The number of strips R and columns C an image should be segmented into depends upon several factors, including the detail within the image, the original image resolution, effects such as lighting variation and camera displacement. In one embodiment, one or more strips may overlap each other. This step is illustrated as step After the strips are configured for a neighborhood of images, an identifying value is assigned to each strip as shown in step
In Once the strips and average rows are determined from the average pixel values, the average row values are converted to the frequency domain as illustrated in step A two dimensional texture could be used instead of a one dimensional texture if the panoramic images had a spatial constraint in the row direction. After the FFT is taken of the average row values and/or the Fourier spectra data associated with the strips in a row is received, the distance between two panoramic images is computed in step In the embodiment of the invention discussed above, the texture coefficients are truncated to middle frequencies. There are several advantages to this method. Low frequency and DC components of an image spectrum are often very sensitive to changes in illumination. Thus, to minimize the effect of different lighting on the same location in two different images, the low frequency components may be eliminated. Removing the high frequency spectra helps to reduce the noise and reduce the effect of different objects at the same location (such as a car present at the intersection in one image but not another). The distance measure between two panoramic images can then be computed as the sum of the distances between the one dimensional texture vectors for two images at corresponding row positions: In one embodiment, once the two closest frames are detected, the panoramic orientation of each image may be determined. The panoramic orientation may be used to determine the relative angle between the paths and the amount that one image may need to be rotated to match another image. One method of determining a rough estimate of the orientation of each path is by taking the derivative of the spatial location data of the frames. This derivative will give a spatial direction for each path. However, more precise methods can be used that examine the panoramic image data rather than the location data. One method using panoramic image data uses the phase of the signals once the averaged rows are converted to the Fourier domain. As discussed above, two panoramic images are divided into strips of rows and columns. An identifying numerical value is taken from each strip, such as the intensity of the pixels within the strip. Thus, a first image A may be represented by a signal a[n] with a period of W, wherein the strip has a width of W. Similarly, a second image B may be represented by a signal b[n] with a period W, wherein the strip has the same width W. The strips from A and B are taken at similar locations in the panoramic image but with the camera rotated an amount θ radians about a cylindrical axis, so that b[n]=a[(n−d)mod W]. Here, d=Wθ/2π. Next, a DFT is performed on each strip. The Fourier spectra A[k] and B[k] of a[n] and b[n], respectively, will be related by B[k]=A[K] exp(−jdk2π/W). The phase difference between A[k] and B[k] can be written as:
The function unwrap (D[k],θ) adds the appropriate integer multiple of 2π to each value of D[k] to minimize the excursion from the linear estimate based on the current value of θ. G[k] is an arbitrary weighting function that may be used to emphasize the contribution of some frequencies over others. For example, frequencies with noise may be de-emphasized with low pass weighting or a weighting that is derived from the image data itself that provides noise level information at each frequency component (such as weighting each frequency by the power at that frequency). In another embodiment of the present invention, the iterative search method of equation (7) is avoided by performing the linear fit to D[k] incrementally starting from the lowest frequency bin and an initial slope value estimate of zero. At the next highest frequency bin, the phase difference is unwrapped based upon the current estimate of θ. The estimate of the slope is then updated with the newly unwrapped value. The rotation is estimated in this manner for each strip of the image. The mean or median of the estimated rotations is used to determine the final estimate of the image rotation. In yet another embodiment, the error in equation (7) may be minimized over all the image strips simultaneously by summing the phase fit error for each strip and finding the rotation that minimizes the sum of the phase fit errors according to: In another embodiment of the present invention, the phase difference θ may be determined by finding the shift d that corresponds to the maximum of the circular cross correlation c[s] of a[n] and b[n], where:
Accordingly, C[k]=DFT {c[s]} satisfies C[k]=A[k]B*[k], where * indicates complex conjugation is to be performed, and c[s] may be computed by taking the inverse DFT of C[k]. The value of s for which c[s] is maximized is taken as the estimate of the rotation d. In one embodiment, the computation for d is performed for each band of the corresponding panoramic images. The mean or median value of the d values is used for the overall estimate of rotation between the panoramic images. In another embodiment of the present invention, the accuracy of the correlation process may be improved by implementing a weighting function into the cross correlation. Weighting the cross correlation in the frequency domain with an arbitrary function G[k] may increase or decrease the contribution of certain frequencies as desired (such as decreasing the contribution of a frequency having high noise). With a weighting function, the generalized cross correlation can be expressed as:
The weighting function may be derived from a-priori knowledge or dynamically from the statistics of the two images being aligned. An example of an a-priori knowledge-based weighting function may include using a low pass filter to filter out high frequencies known to have more noise then low frequencies. An example of a dynamic weighting function may make G[k] equal to the reciprocal of an estimate of the variance or noise at each frequency. Yet another example of a weighting function could implement G[k] as the magnitude-squared coherence function estimated from the image data at each frequency, expressed as:
In one embodiment, the present invention may be used to generate composite panoramic video frame sequences about an intersection. In this embodiment, orientation and intersection information derived from the images themselves as discussed above maybe used to combine two intersecting video frame sequences. For purposes of the example discussed herein, in an intersection having a north-south direction and an east-west direction, it shall be assumed that a first panoramic video sequence contains frames captured while traveling through the intersection in the north-south direction and contains a first intersection frame. It shall also be assumed that a second panoramic sequence having frames captured while traveling through the intersection in the east-west direction contains a second intersection frame, where the first and second intersection frames are determined to be the two frames with the most similar image between the sequences and thereby associated with the intersection of the sequences. When the two intersection frames are known, a composite frame sequence maybe generated that contains frames from different video frame sequences with a common intersection. In one embodiment, the composite video sequence may include frames from the first panoramic video frame sequence up until the intersection frame from the first sequence. Once the intersection frame from the first video sequence is reached, the next frame in the composite panoramic video frame sequence may be the intersection frame from the second video sequence. The frames in the composite video sequence may then consist of frames located after the intersection frame of the second video sequence. Thus, the composite video sequence may consist of two frame sequences, wherein the first sequence is taken from the first panoramic video frame sequence and consists of consecutive frames in the first sequence up until the first intersection frame, and the second sequence is taken from the second panoramic video frame sequence and consists of consecutive frames starting from the second intersection frame. Using the detection information derived using the images themselves as discussed above, a composite panoramic video sequence can be created with frames depicting travel through the intersection in any direction as indicated by a user. In another embodiment, the transition between the two intersection frames in a composite panoramic video frame sequence may be smoothed out by utilizing relative orientation information associated with the video frame sequences. Instead of providing a transition between two intersection frames where the view of the image is suddenly shifted by the amount of the relative orientation between them, the transition maybe smoothed out by rotating the view in the images. The smoothing effect maybe derived by using the orientation information to provide a “turning” effect from one frame sequence to the other. Once the orientations of the video frame sequences have been aligned, the two video streams can by slowly cross faded to reduce the artifacts of an abrubt transition at the intersection. This may require the relative orientation to be detected for a time window before and after the intersection so that the cross-fading is done between matching sequences such as the matching panoramas. In one embodiment, though both video frame sequences include captured image data representing a panoramic image, the actual image displayed to a user may be less than the entire panoramic data. Thus, instead of providing a 360 degree view of an image of a street to a user as the frames are shown, the image presented to a user may only provide a 90 degree view facing in the direction the consecutive frames were captured (ie, from north to south or from east to west). In this embodiment, a first and second panoramic video frame sequence may display a reduced view of the middle one fourth of the frame image. In another embodiment of the present invention, the reduced view may include video that is not taken from panoramic video. The reduced view could be taken from 4:3 aspect ratio video of some other format of video. For purposes of example only, the following discussion will assume the first and second video sequence have a panoramic format. When the first panoramic video frame sequence displaying the reduced view reaches a first intersection frame, the displayed view of the first intersection frame may by adjusted or rotated to until the view of first intersection frame is similar to the displayed view (the middle one fourth of the frame image) of the second intersection frame in the second panoramic video frame sequence. In order to provide a smooth rotation over time, the composite video frame sequence may include multiple frames of the first intersection frame, wherein a different view within the first or second intersection frame is provided in each frame, in order to provide for a shift in the view of the frame. The amount of the view rotation would correspond to the relative orientation that was derived from the frame image data as discussed above. Alternatively, both views of the intersection frames could be rotated towards each other to meet at a point of view somewhere in the middle of the relative orientation between them. In another embodiment of the present invention for generating a composite panoramic video frame sequence from two intersecting panoramic video frame sequences, frames from a first panoramic video frame sequence are displayed. As the displayed frames in the first sequence approach the first intersection frame, the view of the currently displayed frame may be shifted in a direction towards the relative orientation between the intersection frames. The intersection frames from each panoramic video frame sequence are then adjusted in view slightly to compensate for a portion of their relative orientation, followed by adjusting the views of several frames immediately after the second intersection frame in the second panoramic video frame sequence until the orientation between the sequences is completely compensated for. Thus, instead of adjusting the view from only the intersection frames to compensate for the relative orientation, the relative orientation is gradually compensated for by adjusting the views of several frames located closest to the intersection frames as well as the intersection frames themselves. This provides for a video frame sequence having a smoother, more gradual change in view. This embodiment provides a view having more detail in the direction the user with will eventually be heading, the direction ultimately captured by the second sequence of frames, as the user approaches the turn that will lead in that direction. This type of gradual view change is similar to that of someone driving a car who glances in the direction of where she or he will be turning before actually making the turn. The amount of the view change or compensation maybe determined mathematically or manually such that the generated composite video frame sequence provides a smooth transition between the frame sequences. Further, image averaging and smoothing techniques may be used to compensate for any discrepancies between the images of the two intersection frames. In one embodiment, the present invention provides a system for detecting an intersection in more than one video sequence. Video images and location data are received. The images and location data may be processed to ensure all the images contain location data. An intersection between two paths is derived from the video images. The orientation between the two paths may also be determined from the video images. In one embodiment, performing the intersection detection and orientation detection include processing image information in the frequency domain. In one embodiment, the images are panoramic video images. In addition to an embodiment consisting of specifically designed integrated circuits or other electronics, the present invention maybe conveniently implemented using a conventional general purpose or a specialized digital computer or microprocessor programmed according to the teachings of the present disclosure, as will be apparent to those skilled in the computer art. Appropriate software coding can readily be prepared by skilled programmers based on the teachings of the present disclosure, as will be apparent to those skilled in the software art. The invention may also be implemented by the preparation of application specific integrated circuits or by interconnecting an appropriate network of conventional component circuits, as will be readily apparent to those skilled in the art. The present invention includes a computer program product which is a storage medium (media) having instructions stored thereon/in which can be used to program a computer to perform any of the processes of the present invention. The storage medium can include, but is not limited to, any type of disk including floppy disks, optical discs, DVD, CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of media or device suitable for storing instructions and/or data. Stored on any one of the computer readable medium (media), the present invention includes software for controlling both the hardware of the general purpose/specialized computer or microprocessor, and for enabling the computer or microprocessor to interact with a human user or other mechanism utilizing the results of the present invention. Such software may include, but is not limited to, device drivers, operating systems, and user applications. Ultimately, such computer readable media further includes software for performing at least one of additive model representation and reconstruction. Included in the programming (software) of the general/specialized computer or microprocessor are software modules for implementing the teachings of the present invention, including, but not limited to, separating planes of a source image, averaging at least one of foreground and background colors, replacing colors, and compensating for error introduced by color replacement in one plane by feeding error into a second plane, storage, communication of results, and reconstructing an image according to the processes of the present invention. Other features, aspects and objects of the invention can be obtained from a review of the figures and the claims. It is to be understood that other embodiments of the invention can be developed and fall within the spirit and scope of the invention and claims. The foregoing description of preferred embodiments of the present invention has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Obviously, many modifications and variations will be apparent to the practitioner skilled in the art. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to understand the invention for various embodiments and with various modifications that are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the following claims and their equivalence. Patent Citations
Non-Patent Citations
Referenced by
Classifications
Legal Events
Rotate |