|Publication number||US7123777 B2|
|Application number||US 10/256,743|
|Publication date||Oct 17, 2006|
|Filing date||Sep 26, 2002|
|Priority date||Sep 27, 2001|
|Also published as||US20030068098, WO2003027766A2, WO2003027766A3|
|Publication number||10256743, 256743, US 7123777 B2, US 7123777B2, US-B2-7123777, US7123777 B2, US7123777B2|
|Inventors||Michael Rondinelli, Herman Herman, Sanjiv Singh|
|Original Assignee||Eyesee360, Inc.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (64), Non-Patent Citations (3), Referenced by (64), Classifications (12), Legal Events (6)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This application is a continuation-in-part of U.S. patent application Ser. No. 10/080,834 filed Feb. 22, 2002 now U.S. Pat. No. 6,856,472; U.S. patent application Ser. No. 10/081,433 filed Feb. 22, 2002 now abandoned; U.S. patent application Ser. No. 10/081,545 filed Feb. 22, 2002 now abandoned; and U.S. patent application Ser. No. 10/227,136 filed Aug. 23, 2002, which are all incorporated herein by reference. This application also claims the benefit of U.S. Provisional Application Ser. No. 60/326,013 filed Sep. 27, 2001 and U.S. Provisional Application Ser. No. 60/346,717 filed Jan. 7, 2002.
The present invention relates to panoramic imaging, and more particularly relates to a system for processing panoramic photographic images.
Recent work has shown the benefits of panoramic imaging, which is able to capture a large azimuth view with a significant elevation angle. If instead of providing a small conic section of a view, a camera could capture an entire half-sphere or more at once, several advantages could be realized. Specifically, if the entire environment is visible at the same time, it is not necessary to move the camera to fixate on an object of interest or to perform exploratory camera movements. Additionally, this means that it is not necessary to stitch multiple, individual images together to form a panoramic image. This also means that the same panoramic image or panoramic video can be supplied to multiple viewers, and each viewer can view a different portion of the image or video, independent from the other viewers.
One method for capturing a large field of view in a single image is to use an ultra-wide angle lens. A drawback to this is the fact that a typical 180-degree lens can cause substantial amounts of optical distortion in the resulting image.
A video or still camera placed below a convex reflective surface can provide a large field of view provided an appropriate mirror shape is used. Such a configuration is suited to miniaturization and can be produced relatively inexpensively. Spherical mirrors have been used in such panoramic imaging systems. Spherical mirrors have constant curvatures and are easy to manufacture, but do not provide optimal imaging or resolution.
Hyperboloidal mirrors have been proposed for use in panoramic imaging systems. The rays of light which are reflected off of the hyperboloidal surface, no matter where the point of origin, all converge at a single point, enabling perspective viewing. A major drawback to this system lies in the fact that the rays of light that make up the reflected image converge at the focal point of the reflector. As a result, positioning of the sensor relative to the reflecting surface is critical, and even a slight disturbance of the mirror will impair the quality of the image. Another disadvantage is that the use of a perspective-projections model inherently requires that, as the distance between the sensor and the mirror increases, the cross-section of the mirror must increase. Therefore, in order to keep the mirror at a reasonable size, the mirror must be placed close to the sensor. This causes complications to arise with respect to the design of the image sensor optics.
Another proposed panoramic imaging system uses a parabolic mirror and an orthographic lens for producing perspective images. A disadvantage of this system is that many of the light rays are not orthographically reflected by the parabolic mirror. Therefore, the system requires an orthographic lens to be used with the parabolic mirror.
The use of equi-angular mirrors has been proposed for panoramic imaging systems. Equi-angular mirrors are designed so that each pixel spans an equal angle irrespective of its distance from the center of the image. An equi-angular mirror such as this can provide a resolution superior to the systems discussed above. However, when this system is combined with a camera lens, the combination of the lens and the equi-angular mirror is no longer a projective device, and each pixel does not span exactly the same angle. Therefore, the resolution of the equi-angular mirror is reduced when the mirror is combined with a camera lens.
Ollis, Hernan, and Singh, “Analysis and Design of Panoramic Stereo Vision Using Equi-Angular Pixel Cameras”, CMU-RI-TR-99-04, Technical Report, Robotics Institute, Carnegie Mellon University, January 1999, disclose an improved equi-angular mirror that is specifically shaped to account for the perspective effect a camera lens adds when it is combined with such a mirror. This improved equi-angular mirror mounted in front of a camera lens provides a simple system for producing panoramic images that have a very high resolution. However, this system does not take into account the fact that there may be certain areas of the resulting panoramic image that a viewer may have no desire to see. Therefore, some of the superior image resolution resources of the mirror are wasted on non-usable portions of the image.
Panoramic imaging systems also typically require large amounts of computing resources in order to produce viewable panoramic images, especially when displaying the images at an appropriate frequency for video. A single panoramic image may be composed of more than a million pixels. Due to the non-linear mappings of many mirrors and lenses used in existing panoramic imaging systems, and the characteristics of the hardware, software, and/or other computing resources used in conjunction with these mirrors, many of these systems require large amounts of processor resources, processing times, and expert operators in order to produce viewable panoramic images. These problems are particularly apparent when multiple panoramic images are captured and shown sequentially at a frequency rate suitable for video.
The present invention has been developed in view of the foregoing and to address other deficiencies of the prior art.
The present invention provides a system for processing panoramic photographic images.
An aspect of the present invention is to provide a system for processing images including a mirror for reflecting an image of a scene, a mounting assembly for mounting the mirror on an axis, wherein the mirror includes a convex reflective surface defined by rotating around the axis: an equi-angular shape or a compensated equi-angular shape, a camera for capturing the image reflected by the mirror, a digital converter device for producing pixel data representative of the captured image, and means for radially linearly mapping the pixel data into a viewable image.
Another aspect of the present invention is to provide a system for processing images including a mirror for reflecting an image of a scene, means for mounting the mirror on an axis, wherein the mirror includes a convex reflective surface defined by rotating around the axis: an equi-angular shape or a compensated equi-angular shape, means for capturing the image reflected by the mirror, means for producing pixel data representative of the captured image, and means for radially linearly mapping the pixel data into a viewable image.
A further aspect of the present invention is to provide a method of processing images including the steps of providing a mirror for reflecting an image of a scene, mounting the mirror on an axis, wherein the mirror includes a convex reflective surface defined by rotating around the axis: an equi-angular shape or a compensated equi-angular shape, capturing the image reflected by the mirror, producing pixel data representative of the captured image, and radially linearly mapping the pixel data into a viewable image.
Another aspect of the present invention is to provide a method of processing images including the steps of retrieving a source image including pixel data, creating a first texture map memory buffer, transferring the pixel data from the source image to the first texture map memory buffer, producing a plurality of vertices for a first model of a viewable image, wherein the vertices are representative of one or more points corresponding to one or more space vectors of the source image, computing one or more texture map coordinates for each of the vertices, wherein the one or more texture map coordinates are representative of one or more pieces of pixel data in the first texture map memory buffer corresponding to one or more pieces of pixel data in the source image, transferring the first model, including the vertices and the one or more texture map coordinates, to a graphics hardware device, and instructing the graphics hardware device to use the pixel data to complete the first model and display the completed model as a viewable panoramic image.
A further aspect of the present invention is to provide an apparatus for processing images including means for receiving a source image including pixel data, a processor for creating a texture map memory buffer, for transferring the pixel data from the source image to the texture map memory buffer, for producing a plurality of vertices for a model of a viewable image, wherein the vertices are representative of one or more points corresponding to one or more space vectors of the source image, and for computing one or more texture map coordinates for each of the vertices, wherein the one or more texture map coordinates are representative of one or more pieces of pixel data in the texture map memory buffer corresponding to one or more pieces of pixel data in the source image, and a graphics hardware device for receiving the model, including the vertices and the one or more texture map coordinates, for utilizing the pixel data to complete the model, and for displaying the completed model as a viewable image.
These and other aspects of the present invention will be more apparent from the following description.
The present invention provides a system for processing panoramic photographic images. Referring to the drawings,
As used herein, the term “panoramic images” means wide-angle images taken from a field of view of from about 60° to 360°, typically from about 90° to 360°. Preferably, the panoramic visual images comprise a field of view from about 180° to 360°. In a particular embodiment, the field of view is up to 360° in a principal axis, which is often oriented to provide a 360° horizontal field of view. In this embodiment, a secondary axis may be defined, e.g., a vertical field of view. The vertical field of view may be defined with respect to the optical axis of a camera lens, with the optical axis representing 0°. Such a vertical field of view may range from 0.1° to 180°, for example, from 1° to 160°. In one embodiment, the vertical field of view may be controlled in order to maximize the resolution of the portion of the panoramic image that the viewer is most interested in seeing. In order to maximize the resolution of the portion of the panoramic image that the viewer desires to see, the vertical field of view may be controlled in an attempt to eliminate unwanted portions of the panoramic image from the resulting viewable panoramic image. However, the particular controlled vertical field of view chosen may not fully eliminate unwanted portions of the panoramic image from the viewable panoramic image. For example, in order to provide a panoramic image with improved resolution and minimal unwanted portions of the panoramic image, the controlled vertical field of view may range from about 2° to about 160°, preferably from about 5° to about 150°. A particularly preferred controlled vertical field of view that provides panoramic images with improved resolution and minimal unwanted portions of the panoramic image ranges from about 10° to about 140°.
As used herein, the terms “high-resolution” and/or “improved resolution” mean panoramic images having a viewable resolution of at least 0.3 M pixel, preferably having a viewable resolution of at least at least 0.75 M pixel. In a particular embodiment, the terms “high-resolution” and/or “improved resolution” mean panoramic images having a viewable resolution of at least 1 M pixel.
A panoramic image is typically captured with a system, such as the system 10 of
One common application of such a system is to capture a raw 360° image with the convex reflective surface, and unwarp the raw 360° image into a viewable panoramic image.
The use of such imagery has distinct advantages. It is a passive sensor, so power requirements are minimal. It has the potential to be extremely robust, since the sensor is purely solid state and has no moving parts. Furthermore, curved mirrors can be made free of optical distortion that is typically seen in lenses. In addition, the large field of view available offers substantial advantages for panoramic photography, target tracking, obstacle detection, localization, and tele-navigation of machinery.
In the system 10 of
For example, in the system 10 of
An embodiment of the present invention provides a high-resolution panoramic mirror designed with a controlled vertical field of view. As used herein, the term “controlled vertical field of view” refers to a vertical field of view that is adjusted in order to minimize unwanted images from being captured by the panoramic mirror and thereby appearing in the viewable panoramic image, and to maximize the resolution of the portion of the viewable panoramic image that the user desires to see. The controlled vertical field of view may range from about 2° to about 170°, preferably from about 5° to about 150°. A particularly preferred controlled vertical field of view that provides panoramic images with improved resolution and minimal unwanted portions of the panoramic image ranges from about 10° to about 140°. In this embodiment, the high-resolution qualities of the mirror provide resulting high-resolution panoramic images, while the controlled vertical field of view further increases the resolution of the resulting viewable panoramic image.
In a preferred embodiment, a mirror shape may be used that is truly equi-angular when combined with camera optics. In such an equi-angular mirror/camera system, each pixel in the image spans an equal angle irrespective of its distance from the center of the image, and the shape of the mirror is modified in order to compensate for the perspective effect a camera lens adds when combined with the mirror, thereby providing improved high-resolution panoramic images.
For different values of α, mirrors can be produced with a high degree of curvature or a low degree of curvature, while still maintaining their equi-angular properties. In one embodiment, α ranges from about 3 to about 15, preferably from about 5 to about 12. In a particular embodiment, α is chosen to be 11.
It has been determined that the addition of a camera with a lens introduces an effect such that each pixel does not span the same angle. This is because the combination of the mirror and the camera is no longer a projective device. Hence, to be exactly equi-angular, the mirror may be shaped to account for the perspective effect of the lens and the algorithms must be modified. Such a modified equi-angular mirror shape is defined herein as a “compensated equi-angular mirror.”
It is possible to make a small angle approximation by assuming that each pixel spans an equal angle. The following equation (2) can be used to derive the mirror shape:
Since the camera is still a projective device this typically only works for small fields of view. Surfaces of mirrors in which each pixel truly corresponds to an equal angle are shapes that satisfy the polar coordinate equation (3) below:
The advantage of using equation (2) is that the surfaces produced have a closed-form solution, whereas equation (3) must be solved numerically. However, the result of solving equation (3) numerically is that it produces a profile of the mirror that produces a truly equi-angular relation where each pixel in the image has the same vertical field of view.
A typical convex mirror will typically have a continuous surface across any diameter. Because of this constraint, a significant portion of the imaged surface area of the mirror is likely to reflect portions of a panoramic image that the viewer is least interested in seeing. The pixels in the resulting photograph that reflect such unwanted portions of the panoramic image end up not being efficiently utilized. It is desirable to minimize these unwanted portions of the panoramic image. This is especially important when resolution is at a premium, as is the case with panoramic mirrors.
In one embodiment, a panoramic mirror is fabricated with a controlled vertical field of view. By fabricating a mirror with such a controlled vertical field of view, less desired portions of the panoramic image can be substantially reduced or eliminated from the resulting panoramic image. A compensated equi-angular mirror is most suited to be used in this embodiment. This is because the uniform distribution of resolution along any radius of the mirror provides the most effective elimination of less desired portions of the panoramic image, in addition to producing high-resolution panoramic images.
In one embodiment, in order to select the lower limit of the controlled vertical field of view, a convex shaped panoramic mirror, such as a compensated equi-angular panoramic mirror, can be fabricated into a point at the center of the mirror. As an illustration, a two-dimensional profile of such a mirror can be depicted by removing a conical portion from the center of the two-dimensional mirror profile and constricting the resulting two-dimensional mirror profile at the center to form a point. This constricted shape is illustrated in the sectional views shown in
As another illustration, shown in
Once a two-dimensional mirror profile is developed, as shown in
In one embodiment, Angle C ranges from about 0.5° to about 20°, preferably from about 1° to about 10°, more preferably from about 2° to about 8°. In a particular embodiment, angle C is about 5°.
For a compensated equi-angular panoramic mirror manufactured with a total cone angle of 2A removed from the center of the mirror, the relationship that describes the resulting mirror profile can now be written in equation (5) as:
As is the case with equation (3), equation (5) must also be solved numerically based on various values substituted for θ. θ is the angle that a light ray makes with the central axis as it reflects off of a point on the surface of the mirror and into the camera lens.
In another embodiment, the upper limit of the controlled vertical field of view can be denoted by angle B, shown in
Once a two-dimensional mirror profile is developed with an angle B chosen, as shown in
Angle D ranges from about 50° to about 100°, preferably from about 65° to about 90°, more preferably from about 70° to about 85°. In a particular embodiment, angle D is about 76°.
In practice, a panoramic mirror with a controlled vertical field of view may be formed by generating a two-dimensional profile of such a mirror with the selected angle A, as depicted in
In an embodiment of the invention, A is chosen to be 10°, B is chosen to be 140°, and α is chosen to be 11. Substituting these values in equation (5), and solving the equation numerically, a unique mirror shape is produced with an angle C of about 5° and an angle D of about 76°. This unique mirror shape reflects panoramic images with a resolution unparalleled in the prior art. This superior resolution is obtained from a combination of the compensated equi-angular properties of the panoramic mirror, and the fact that the resolution has been further optimized by controlling the appropriate vertical field of view for the mirror. In this embodiment, the primary concern is providing a high-resolution viewable panoramic image, not eliminating central obscurations from the viewable panoramic image.
In one embodiment, in order to provide the portion of the viewable panoramic image that the user is most interested in seeing at the best resolution possible, all of the unwanted portions of the viewable panoramic image may not be fully eliminated. These unwanted portions may include, for example, the camera, the camera mount, the camera lens, the mount holding the mirror in front of the camera and other unwanted foreground images. For example, the vertical field of view of the viewable panoramic image that the viewer wishes to see may be 40° to 140°, while the controlled vertical field of view of the viewable panoramic image may be 10° to 140°. As used herein the term “desired vertical field of view” means the vertical field of view corresponding to the portion of the viewable panoramic image that the viewer is interested in viewing. The desired vertical field of view may be equal to or less than the controlled vertical field of view. The desired vertical field of view may range from about 2° to about 170°, preferably from about 15° to about 150°. A particularly preferred desired vertical field of view that a viewer would typically be interested in viewing ranges from about 40° to about 140°.
In one embodiment, a compensated equi-angular mirror with a controlled vertical field of view may be manufactured with a hole centered at the axial center 28 of the mirror in order to accommodate various mounting devices. The mounting hole may range in diameter from about 0.05 cm to about 15 cm, preferably from about 0.1 cm to about 5 cm. In a particular embodiment the mounting hole is 0.64 cm in diameter.
In one embodiment, as shown schematically in
E=(atan(r R /r camera)+α·atan(r R /r camera)+A)/2 (7)
In equation (7), rR is the radius of the rod. Angle E ranges from about 5° to about 30°, preferably from about 10° to about 20°, more preferably from about 12° to about 16°. In a particular embodiment, angle E is about 14°.
In another embodiment, a compensated equi-angular mirror with a controlled vertical field of view can be mounted in front of a camera with a mounting assembly as schematically illustrated in
In another embodiment, as shown schematically in
In another embodiment, as shown schematically in
In a preferred embodiment, a compensated equi-angular mirror with a desired vertical field of view having a lower limit A′ of about 40° and an upper limit B′ of about 140° is designed with a controlled vertical field of view having an angle A equal to about 10° and an angle B equal to about 140°, an α equal to about 11, and a diameter DM of about 8 cm. The mirror may be placed at a distance rcamera from the camera of about 12 cm, and may placed on a mounting device with a diameter dmount of about 4.25 cm. The mirror is typically placed at a distance rmount from the widest portion of the mirror mount of about 4.7 cm. In this embodiment, the mirror may mounted in front of a camera sold under the designation NIKON 990 by NIKON, or a camera sold under the designation NIKON 995 by NIKON. The mirror may mounted on a rod that is about 0.64 cm thick. In this embodiment, a unique mirror shape is produced with an angle E of about 14° and an angle D of about 76°. In this embodiment, the primary concern is providing a high-resolution viewable panoramic image, not eliminating central obscurations from the viewable panoramic image.
A unique aspect of the present invention is that any video or still camera that will focus on the mirror surface may be used. Since the mirror shape can be designed to account for different distances that the mirror may be placed from a lens of a camera, virtually any video or still camera will work with the system of the present invention.
Once a camera has captured an image of a scene reflected from an attached mirror, this raw image must be converted or “unwarped” into a viewable panoramic image.
In one embodiment, a method and apparatus for processing raw images of a scene reflected by a mirror and captured with a camera may be used with the system of the present invention as disclosed in copending commonly owned U.S. patent application Ser. No. 10/081,545 filed Feb. 22, 2002, which is hereby incorporated by reference. In this embodiment, image processing may be performed using a software application, hereinafter called PhotoWarp, that can be used on various types of computers, such as Mac OS 9, Mac OS X, and Windows platforms. The software can process images captured with a panoramic imaging device, such as the device 12 of
The PhotoWarp Job 120 contains a source list 122 that identifies one or more source image groups, for example 124 and 126. The source image groups can contain multiple input files as shown in blocks 128 and 130. The PhotoWarp Job 120 also contains a destination list 132 that identifies one or more destination groups 134 and 136. The destination groups can contain multiple output files as shown in blocks 138 and 140. A Job item list 142 identifies the image transformation operations that are to be performed, as illustrated by blocks 144 and 146. The PhotoWarp Job can be converted to XML or alternatively created in XML as shown by block 148.
The preferred embodiment of the software includes a PhotoWarp Core that serves as a cross-platform “engine” which drives the functionality of PhotoWarp. The PhotoWarp Core handles all the processing tasks of PhotoWarp, including the reprojection or “unwarping” process that is central to the application's function.
PhotoWarp preferably uses a layered structure that maximizes code reuse, cross-platform functionality and expandability. The preferred embodiment of the software is written in the C and C++ languages, and uses many object-oriented methodologies. The main layers of the application are the interface, jobs, a remapping engine, and output tasks.
The PhotoWarp Core refers to the combination of the Remapping Engine, Output Tasks, and the Job Processor that together do the work of the application. The interface allows users to access this functionality.
The Remapping Engine, or simply the “Engine” is an object-oriented construct designed to perform arbitrary transformations between well-defined geometric projections. The Engine was designed to be platform independent, conforming to the ANSI C++ specification and using only C and C++ standard library functions. The Engine's basic construct is an image object, represented as an object of the CImage class. An image is simply a two-dimensional array of pixels. Pixels are considered to be an abstract data type to allow for the large variety of color models, encodings and bit depths. In one example, a Pixel is a 32-bit value consisting of four 8-bit channels: alpha, red, green and blue.
Using the described process, the warped source image can be converted into an image with a more traditional projection using an unwarping process. For example, it may be desirable to unwarp an equi-angular source image into an equi-rectangular projection image, where pixels in the horizontal direction are directly proportional to the pan (longitudinal) angles (in degrees) of the panorama, and pixels in the vertical direction are directly proportional to the tilt (latitudinal) angles (also in degrees) of the panorama.
The algorithm for the unwarping process determines the one-to-one mapping between pixels in the unwarped image and those in the warped image, then uses this mapping to extract pixels from the warped image and to place those pixels in the unwarped image, possibly using an interpolation algorithm for smoothness. Since the mapping between the unwarped and warped images may not always translate into integer coordinates in the source image space, it may be necessary to determine a value for pixels in between other pixels. Bi-directional interpolation algorithms (such as bilinear, bicubic, spline, or sinc functions) can be used to determine such values.
The unique shape and properties of the compensated equi-angular mirror combined with the functionality of the PhotoWarp software may substantially reduces a processing time associated with processing the pixel data into the viewable image. Specifically, since each pixel reflected by the mirror and captured by the camera corresponds to an equal angle, simple first order equations can be processed with the PhotoWarp software and used to quickly determine the angle for the output pixel and the corresponding source pixel for the angle, and the proper source pixel value can then be mapped to the output pixel of the viewable panoramic image. These pixels reflected by such a compensated equi-angular mirror may be referred to as equi-angular pixels, and such a mapping scheme may be referred to as a radially linear mapping scheme. This simple radially linear pixel mapping substantially reduces the processing time and the complexity of the software code needed to produce a viewable panoramic image by as much as 20 to 40 percent when compared to panoramic imaging systems that do not utilize a mirror that provides radially linear mapping between the source pixels and the output pixels. This improvement in processing time is achieved by the reduced number of calculations that must be performed on every pixel in the image.
As an example, radially linearly mapping an equi-angular source image to an equi-rectangular destination image can be quickly achieved by pre-calculating sine and cosine values for a particular pan angle in the output image, then proceeding linearly along the radius of the source image to produce columns of destination pixels. Only two multiply-add computations would be needed for each pixel in the output image, and the system memory would typically not need to be accessed to perform these calculations. A non-radially linear source mapping would require either more calculations for each pixel, or would need to generate a lookup table for radial pixels, which on modern processors can incur a performance penalty for accessing system memory.
In another embodiment, image processing may be performed using a software application, hereinafter called VideoWarp, that can also be used on various types of computers, such as Mac OS 9, Mac OS X, and Windows platforms. This software may be combined with a graphics hardware device, such as a 3-D graphics card commonly known in the art, to process images captured with a panoramic imaging device, such as the device 12 of
Typically, video is made up of a plurality of still images displayed in sequence. The images are usually displayed at a high rate speed, sufficient to make the changing events in the individual images appear fluid and connected. A minimum image display rate is often approximately 30 images per second, although other display rates may be sufficient depending on the characteristics of the equipment used for processing the images. While software alone may be sufficient for processing the often one million or more pixels needed for a viewable panoramic image and displaying the viewable panoramic image, software alone is typically not capable of calculating and displaying the one million or more pixels of a viewable panoramic image 30 or more times a second in order to produce a real time video feed. Therefore, in one embodiment the VideoWarp software may be used in conjunction with a graphics hardware device to process panoramic video that can be viewed and manipulated in real time, or recorded for later use, such as on a video disc (e.g. as a QuickTime movie) for storage and distribution.
VideoWarp preferably uses a layered structure that maximizes code reuse, cross-platform functionality and expandability. The preferred embodiment of the software is written in the C and C++ languages, and uses many object-oriented methodologies. The main components of the application are the user interface, source, model, projection and renderer.
The VideoWarp Core refers to the combination of the source, model, projection and renderer classes that together do the work of the application. The interface allows users to access this functionality.
The Source component manages and retrieves frames of video data from a video source. Source is an abstract class which allows the rendering of panoramic video to be independent of the particular source chosen for display. The source can be switched at any time during the execution of VideoWarp. The source is responsible for communicating with any video source devices (when applicable), retrieving frames of video, and transferring each frame of video into a memory buffer called a texture map. The texture map may represent image data in memory in several ways. In one embodiment, each pixel may be represented by a single Red, Green and Blue channel (RGB) value. In another embodiment, pixel data may be represented by luminance values for each pixel and chroma values for a group of one or more pixels, which is commonly referred to in the art as YUV format. The source may use the most efficient means possible to represent image data on the host computer system to achieve maximum performance and quality. For example, the source will attempt to use the YUV format if the graphics hardware device appears to support the YUV format. More than one source may be utilized at any given time by the renderer to obtain a more complete field-of-view.
A source may retrieve its video data from a video camera attached to the host computer, either through an analog to digital converter device to digitize analog video signals from a video camera, or through a direct digital interface with a digital camera (such as a DV or IIDC camera connected through an IEEE-1394 bus), or a digital camera connected through a camera link interface. Additionally, the source may retrieve video data from a tape deck or external storage device made to reproduce the signals of a video camera from a recording. The source may also retrieve video data from a prerecorded video file on a computer disk, computer memory device, CD-ROM, DVD-ROM, computer network or other suitable digital storage device. The source may retrieve video data from a recorded Digital Video Disc (DVD). The source may retrieve video data from a streaming video server over a network or Internet. Additionally, the source may retrieve video data from a television broadcast.
The model component is responsible for producing vertices for a virtual three-dimensional model.
Model is an abstract class which allows the rendering of panoramic video to be independent of the particular model chosen for display. The model can be switched at any time during the execution of VideoWarp. If the model is switched, the vertices will need to be calculated again. The model may represent a cube or hexahedron, a sphere or ellipsoid, a cylinder having closed ends, an icosahedron, or any arbitrary three-dimensional model. The model preferably will encompass a 360 degree horizontal field of view from a viewpoint in the interior, and a vertical field of view between 90 degrees and 180 degrees. The model may encompass a lesser area should the coverage of the source video be less than that of the model, or to the boundary of the area to visible to the user. Models can be varied over time to provide transitions or animations to the user display. Transitions may be used between models to smooth or “morph” between displays that represent different views of the panoramic video to the user.
The projection component is used by the model to compute texture map coordinates for each vertex in the model. Texture map coordinates refer to a particular point or location within a source texture map, which can be represented by s and t. The projection defines the relationship between each pixel in the source texture map and a direction (θ, φ) of the panoramic source image for that pixel. The direction (θ, φ) also corresponds to a particular vertex of the virtual model, as described above. Projection provides a function which converts the (θ, φ) coordinates provided for a vertex of the model to the corresponding s and t texture map coordinate. When the viewable image is displayed, the point (s, t) of the texture map will be pinned to the corresponding vertex, producing a “skin” over the skeleton of the model which will be used to eventually reproduce substantially the entire original appearance of the captured scene to the user. This is also illustrated in
Projection is an abstract class which allows the rendering of panoramic video to be independent of the particular projection chosen to represent the source image. The parameters of the projection may be changed over time as the source video dictates. The projection itself may be changed at any time during the execution of VideoWarp. If the projection is changed, the texture map coordinates will need to be calculated again. The projection may represent an equi-angular mirror, an unrolled cylinder, an equi-rectangular map projection, the faces of a cube or other polyhedron, or any other projection which provides a 1-to-1 mapping between directional vectors (θ, φ) and texture map coordinates (s,t).
In one embodiment, the projection may utilize an encoding method and apparatus to provide an encoded projection, as disclosed in copending commonly owned U.S. patent application Ser. No. 10/227,136 filed Aug. 23, 2002, which is hereby incorporated by reference. Such an encoding method and apparatus may be utilized for making the pixel data of the images more suitable for transmitting over a computer network and/or for storing on a computer system. In this embodiment, in order to make the pixel data more suitable for transmitting over a computer network, such as compressing the pixel data, the projection may be a partial equi-rectangular projection, which can be defined as a rectangular array of pixels representing a portion of an equi-rectangular projection of the panoramic image. More specifically, pixel data may be divided into equi-rectangular blocks of pixels having a width substantially equal to a width of a macro-block of pixel data and a length substantially equal to an integer multiple of a length of a macro-block of pixel data. As used herein, the term “macro-block” refers to a group or block of pixels, wherein the macro-block has a width w that may be measured in pixels and a length 1 that may be measured in pixels. In this way, it will be ensured that a compression artifact source, such as a sharp line, will only occur on a perimeter of a macro-block of pixel data. As used herein, the term “compression artifact source” refers to a line, boundary, or other portion of an uncompressed image that crosses over at least one individual pixel of the image and may cause visible distortions in the image when the image is compressed and then later uncompressed for viewing. Such a partial equi-rectangular projection 218 is shown in
In order to make the pixel data more suitable for storing on a computer system, the partial equi-rectangular projection 218 of
In one embodiment of the invention, a data table may be created containing pixel attribute data corresponding to the pixel data of the partial equi-rectangular projection 226 shown in
Other modified partial equi-rectangular projections containing patterns of arranged equi-rectangular blocks may be used. In one embodiment, a linear increasing phi major pattern may be used. As used herein, the term “phi major pattern” refers to the ordering of macro-blocks first by their minimum phi angle as stored in the data table representation.
The equi-rectangular blocks of a partial equi-rectangular projection may also be arranged in a modified partial equi-rectangular projection having a purely random pattern. This may be used as a form of encryption, as the receiver of the random patterned projection would need to have the corresponding data table so that the equi-rectangular blocks may be placed back in a partial equi-rectangular projection in the proper order.
Although the use of such an encoding scheme is particularly suitable for video applications, because of the large amounts of bandwidth and computer processing resources often required to process such video, this encoding scheme may be used in conjunction with the PhotoWarp software for producing encoded panoramic images, and such a use is within the present scope of the invention.
The renderer component manages the interactions of all the other components in VideoWarp. Renderer is an abstract class which allows the rendering of panoramic video to be independent of the particular host operating system, 3D graphics framework, and 3D graphics architecture. A particular renderer is chosen which is compatible with the host computer and will achieve the maximum performance. The Renderer is in use for the lifetime of the application.
At the start of the application, the renderer uses the facilities of the host operating system to initialize the graphics hardware device, often using a framework such as OpenGL or Direct3D. The renderer may then determine the initial source, model and projection to use for the session and initializes their status. Once initialized, the renderer begins a loop to display panoramic video:
The renderer may execute some of the above processes simultaneously by using a preemptive threading architecture on the host platform. This is used to improve performance and update at a smooth, consistent rate. For example, the renderer may spawn a preemptive thread that is responsible for continually retrieving new source video frames and updating the source texture map. It may also spawn a preemptive thread responsible for issuing redraw requests to the graphics hardware device at the maximum rate possible by the hardware. Additionally, the renderer may make use of the features of a host system to execute direct memory access between the source texture map and the graphics hardware device. This typically eliminates the interaction of the computer CPU from transferring the large amounts of image data, which frees the CPU to perform other duties and may greatly improve the performance of the system. The renderer may also pass along important information about the host system to the source, model and projection components to improve performance or quality. For example, the renderer may inform the source that the graphics hardware device is compatible with YUV encoded pixel data. For many forms of digital video, YUV is the native encoding of pixel data and is more space-efficient than the standard RGB pixel format. The source can then work natively with YUV pixels, avoiding a computationally expensive conversion to RGB, saving memory and bandwidth. This will often result in considerable performance and quality improvements.
The Interface layer is the part of the VideoWarp application visible to the user. It shelters the user from the complexity of the underlying core, while providing an easy to use, attractive front end for their utility. VideoWarp can provide a simple one-window interface suitable for displaying panoramic video captured with a reflective mirror optic. Specifically, VideoWarp enables the following capabilities:
The implementation of the interface layer varies by host platform and operating system. The appearance of the interface is similar on all platforms to allow easy switching between platforms for users.
In some instances, the resolution of a captured source image may be so great that a single texture map may not be able to accommodate all of the pixel data from the captured image. In many instances the graphics hardware device may only allow the texture map to be a maximum size, such as 2048 by 2048 pixels, or 4096 by 4096 pixels. If an image is captured having a resolution of 8192 by 8192 pixels, the single texture map would not be able to accommodate it. In one embodiment, multiple texture maps may be created, and the texture map coordinates may be computed for the multiple texture maps. When the texture map coordinates are computed, the multiple texture maps may be considered as a “single” texture map, so that stitching effects commonly associated with multiple texture maps will not appear in the resulting viewable image or images.
The speed realized from the combination of the VideoWarp software and a graphics hardware device can be utilized to display interlaced video. The term interlaced video refers to video having video frames consisting of two fields displayed in two passes. Each field contains every other horizontal line of the video frame. An interlaced video system displays the first field as a frame of alternating lines over the entire screen, and then displays the second field to fill in the alternating gaps left by the first field. One field can consist of the “even” lines of the video frame and can be referred to as an even frame of video, and the other field can consist of the “odd” lines of the video frame and can be referred to as an odd frame of video. Many video cameras on the market exclusively capture video in an interlaced fashion. Interlaced is preferred, often in NTSC or PAL television broadcasts, due to its ability to provide persistence of vision at lower bandwidths, since only half of the data required to fill an entire frame of video is transmitted at one time. However, a drawback of using an interlaced video scheme is that each “half” frame of video must typically be displayed at an interlaced video rate, such as 1/60th of a second intervals, in order to achieve an overall video frame rate of 1/30th of a second. VideoWarp combined with a graphics hardware device provides an appropriate speed for displaying interlaced video.
In one embodiment, two texture map memory buffers may be created, one for storing the pixel data of the even lines of an interlaced video frame, and one for storing the odd lines of an interlaced video frame. These buffers may be half the size of a buffer needed to store a full frame of video. The VideoWarp software and graphics hardware device can then process the incoming pixel data in the same manner as already described herein. When the graphics hardware device utilizes the pixel data from the texture map memory buffers to complete the virtual model, the texture map coordinates can be scaled by one half in the vertical direction, which will effectively “stretch” the odd or even lines of video back to a full frame size, and then an interpolation scheme can be used to complete the frame. By utilizing such an interpolation scheme, the quality of the resulting video can be improved and the interlacing effect will not be visible. The graphics hardware device can then show the even and odd frames of video in an alternating fashion, at a rate of approximately 60 frames per second. In this embodiment, a viewer may notice a slight discontinuity or “jitter” in the video stream as the even and odd frames are displayed. To eliminate the discontinuity, the texture map coordinates of the even frames of video may be shifted in the vertical direction by one half of the distance spanned by a pixel. Although this procedure typically eliminates the discontinuity in the displayed video, the texture map coordinates will now change with every complete frame of video displayed and will have to be re-calculated each time. This may be remedied by instructing the software and hardware to construct two virtual models, one to be used for even frames of video, and one to be used for odd frames of video. Two sets of texture coordinates could be calculated initially, and then utilized for rendering the entire video stream, provided the camera and mirror in relation to the camera are not moved, and/or if the source projection is changed. Alternatively, a technique known in the art and referred to as multi-texturing may used if the graphics hardware device supports this technique. Only one virtual model would be typically be needed if multi-texturing is used.
In another embodiment, two full frame size texture map memory buffers may be created, one for storing the pixel data of the even lines of an interlaced video frame, and one for storing the odd lines of an interlaced video frame. Viewing the frames alone, the odd lines of the even video frame would appear as a solid color, and the even lines of the odd video frame would appear as a solid color. An interlaced filter, which is well known in the art, could be used to interpolate the even lines of the even video frame across the odd lines, and to interpolate the odd lines of the odd video frame across the even lines. The frames can then be displayed in an alternating sequence as described above.
The speed realized from the combination of the VideoWarp software and a graphics hardware device can also be utilized to interactively eliminate a skew effect from a viewable panoramic image in real time, i.e., to eliminate any undesired horizontal or vertical offset of an image that may cause it to appear “crooked”. Specifically, a view within a particular panoramic image can be represented by a particular set of coordinates (p,y,r,f) for the pitch (tilt), yaw (pan), roll (rotation) and field-of-view. For a panoramic image taken with a camera leveled relative to the ground, these coordinates will typically be correct and the viewable image will have the proper alignment within the viewing frame. However, if the camera was not level when the image was captured, the view may appear crooked, i.e., the pitch (tilt), yaw (pan), roll (rotation) and field-of-view coordinates may not have the proper values needed to present an aligned image. The amount of deviation from the normal for the camera in such a case can be represented with three coordinates (∂p, ∂y, ∂r). The “crookedness” apparent in the view can be compensated by adding offsets to the view which negate the deviation in the original image. For an image taken that was deviated from the norm by (∂p,∂y,∂r), the corrected viewing coordinates for a desired view (p,y,r,f) may be represented by (p-∂, p), (y-∂, y), (r-∂, r) and f. By using the VideoWarp software combined with a graphics hardware device, a user could quickly be presented with a real-time preview of what the captured panoramic image would look like. If it appears to the user that the captured image is skewed, the user could utilize the software interface to automatically adjust the pitch, roll, yaw and/or field of view of the image until the skew effect is eliminated. As the user manipulates the pitch, roll, yaw and/or field of view through the software interface, the graphics hardware could continuously calculate updated values for (p-∂, p), (y-∂, y), (r-∂, r) and f and update the image in real time, essentially presenting a sequence of still viewable panoramic images as a “mini” video, with-each still image having a slightly less skewed effect. When the user is presented with a viewable image that has the desired pitch, yaw, roll, and/or field of view, that particular image could be saved via the software interface as the final corrected viewable panoramic image. The skew may also be corrected automatically, by utilizing a device that can measure the pitch, yaw, and roll of the mirror. Software and/or hardware could then utilize the measurements provided by the device to compensate and correct the potentially skewed image.
The VideoWarp software combined with a graphics hardware device may also be able to eliminate “jitter” effects that can often be noticed in video, due to the camera capturing the video not being held perfectly steady. Portions of the video may be tracked from frame to frame, and the software and/or hardware may analyze the portions as they change, determining if the tracked portions represent changes that would be indicative of the camera being slightly rotated or shaken. The software and/or hardware may then compensate for the difference in the tracked portions, thus stabilizing the video.
The user interface component of both the PhotoWarp and VideoWarp software allows a viewer to change the viewing perspective of the resulting viewable panoramic image. In the VideoWarp context, the speed with which frames of video may be produced provides a substantial real-time update of the resulting video as the user changes the viewing perspective, without noticeable lag or latency. The viewing perspective may be altered by allowing the user to “look” up and concentrate on the top portion of the resulting viewable panoramic images, to “look” down and concentrate more on the bottom portion of the resulting viewable panoramic images, to pan around the entire 360° horizontal field of view of the resulting viewable panoramic images, as if from a stationary reference point in the captured scene, and/or to “zoom” in or out on portions of the resulting viewable panoramic images. In the VideoWarp context, the viewing perspective may be rendered by placing a “virtual” camera in the center of the model, which typically simulates a user's head and the view they would see if they were standing in the middle of the model. A user requesting a change in the viewing direction can be likened to the user altering the roll, pitch, and/or yaw of his or her head. As the roll, pitch, and/or yaw changes, the orientation of the virtual camera can be altered accordingly, thus changing the viewing perspective of the resulting viewable image or images. The user or viewer may use a mouse, a keyboard, a track ball or any other haptic device to facilitate altering the viewing perspective of the viewable panoramic images. In another embodiment, the viewer may use a head tracker coupled with a head mounted device to facilitate altering the viewing perspective of the viewable panoramic images. In this embodiment, the viewer is given the sense that he or she is standing in the center of the scene that was captured with the panoramic camera.
In one embodiment of the present invention, a target apparatus, such as a fixed target, may be provided that attaches to the base of a mirror, such as the mirror 14 of the system 10. The plane of the target apparatus may be placed substantially perpendicular to the optical axis of the camera, and may be placed behind the mirror at such a distance as to not obscure useful panoramic image data.
The target apparatus may include as target elements a barcode or other indicia containing parameters describing the shape of the panoramic mirror; a series of marks for determining the center and the radius of the mirror, such as perpendicular marks drawn on radial lines outwards from the center of the mirror, marks drawn tangent to the edge of the mirror, or marks comprising a combination of perpendicular marks drawn on radial lines outwards from the center of the mirror and marks drawn tangent to the edge of the mirror such as the marks 278 shown in
In one embodiment, the image pixel data of a captured scene may be transferred to a server computer for processing in a client-server computer network, as disclosed in copending commonly owned U.S. patent application Ser. No. 10/081,433 filed Feb. 22, 2002, which is hereby incorporated by reference. Such processing may include, for example, converting the raw 2-dimensional array of pixels captured with the panoramic imaging device into an image suitable for viewing.
Once the raw photographic image is resident on the client computer 288, the image is transmitted, as illustrated at 290, to a server computer 292. The images may be transmitted from the client computer 288 to the server computer 292 using, for example, an Internet connection therebetween, a wireless connection, a phone line, or other suitable networking medium. Furthermore, the images may be transmitted using various network protocols, including e-mail, File Transfer Protocol (FTP), Hypertext Transfer Protocol (HTTP), or other suitable networking protocols.
Once the raw images have been transmitted to the server computer 292 and are resident thereon, the server computer 292 may process the raw photographic image to obtain a viewable panoramic image. Such processing may be accomplished with the PhotoWarp software and/or the VideoWarp software in combination with a graphics hardware device, as previously described herein. The processing on the server computer may also include evaluating information obtained from a target apparatus and adjusting the raw image accordingly, as previously described herein.
Once the raw image has been processed to obtain a corresponding viewable panoramic image, the panoramic image may then be transmitted, as illustrated at 294, back to the client computer 288. The panoramic image may be transmitted from the server computer 292 to the client computer 288 in a similar manner as described herein for transmitting the raw images from the client computer 288 to the server computer 292. Once the panoramic images have been transmitted back to the client computer 288 and are resident thereon, a user may then display, view and/or use the processed panoramic images as desired. The client computer 288 may have installed thereon, software capable of viewing the panoramic images, such as Quicktime VR software available from Apple Computer, Inc.
Such a client server embodiment may include several variations. For example, a processed viewable panoramic image may be transmitted to an additional viewing computer or web server, rather than being transmitted back to the client computer 288. Alternatively, rather than transmitting the captured image from the camera to a user or client computer as illustrated in
In another embodiment of the invention, the ability to generate still panoramic images and/or panoramic video having multiple perspective views for different users at the same time is made available. This may be accomplished by rendering images with different viewing directions. Utilizing a client-server situation as described above, multiple users can elect to view different portions of the captured surrounding scene. Each user may independently alter the viewing perspective of the portion of the panoramic image they are viewing. The speed realized with the combination of the VideoWarp software and the graphics hardware device can provide panoramic video streams having multiple views and being requested by multiple users with almost no loss of performance and very little latency. In this embodiment, the video could be processed on the client side and then transferred to the server for viewing.
Although the present invention has been primarily described utilizing a compensated equi-angular mirror, it is to be understood that a parabolic shaped mirror, a hyperbolic shaped mirror, a spherical shaped mirror, or any other convex shaped mirror may be used, and these mirrors may or may not be combined with lenses of various types. Additionally, multiple mirrors may be combined in particular configurations, which may increase the resolution and/or available field of view of the resulting image or images. Such uses are within the scope of the present invention.
Although the panoramic imaging system of the present invention has been primarily described as using a computer system combined with software to process and produce images suitable for viewing, it is to be understood that a dedicated hardware system or other embedded computing device may also be used, and is within the scope of the present invention.
Whereas particular embodiments of this invention have been described above for purposes of illustration, it will be evident to those skilled in the art that numerous variations of the details of the present invention may be made without departing from the invention as defined in the appended claims.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US994935||Jun 18, 1909||Jun 13, 1911||Louis Henry Kleinschmidt||Apparatus for producing topographic views.|
|US3846809||Oct 18, 1973||Nov 5, 1974||Pinzone J||Reflectors and mounts for panoramic optical systems|
|US4429957||Jul 30, 1981||Feb 7, 1984||King-Bell Optics, Inc.||Panoramic zoom lens assembly|
|US4549208||Dec 21, 1983||Oct 22, 1985||Hitachi, Ltd.||Picture processing apparatus|
|US4734690||Apr 6, 1987||Mar 29, 1988||Tektronix, Inc.||Method and apparatus for spherical panning|
|US4757383||Apr 20, 1987||Jul 12, 1988||Fuji Photo Film Co., Ltd.||Method of image signal encoding by orthogonal transformation|
|US4797944||Jan 27, 1987||Jan 10, 1989||Fuji Photo Film Co., Ltd.||Image signal encoding method by orthogonal transformation|
|US4805030||Jan 27, 1987||Feb 14, 1989||Fuji Photo Film Co., Ltd.||Method of image signal encoding by orthogonal transformation|
|US4807042||Jan 27, 1987||Feb 21, 1989||Fuji Photo Film Co., Ltd.||Method of image signal encoding by orthogonal transformation|
|US4965753||Feb 23, 1990||Oct 23, 1990||Cae-Link Corporation, Link Flight||System for constructing images in 3-dimension from digital data to display a changing scene in real time in computer image generators|
|US5067019||Mar 31, 1989||Nov 19, 1991||The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration||Programmable remapper for image processing|
|US5115266||Apr 11, 1991||May 19, 1992||Troje Gerald J||Optical system for recording or projecting a panoramic image|
|US5175808||Sep 12, 1989||Dec 29, 1992||Pixar||Method and apparatus for non-affine image warping|
|US5185667||May 13, 1991||Feb 9, 1993||Telerobotics International, Inc.||Omniview motionless camera orientation system|
|US5359363||Feb 8, 1993||Oct 25, 1994||Telerobotics International, Inc.||Omniview motionless camera surveillance system|
|US5396583||Oct 13, 1992||Mar 7, 1995||Apple Computer, Inc.||Cylindrical to planar image mapping using scanline coherence|
|US5444478||Dec 28, 1993||Aug 22, 1995||U.S. Philips Corporation||Image processing method and device for constructing an image from adjacent images|
|US5452413||Dec 18, 1992||Sep 19, 1995||International Business Machines Corporation||Method and system for manipulating wide-angle images|
|US5574836||Jan 22, 1996||Nov 12, 1996||Broemmelsiek; Raymond M.||Interactive display apparatus and method with viewer position compensation|
|US5586231||Dec 20, 1994||Dec 17, 1996||U.S. Philips Corporation||Method and device for processing an image in order to construct from a source image a target image with charge of perspective|
|US5594845||Dec 20, 1994||Jan 14, 1997||U.S. Philips Corporation||Method and device for processing an image in order to construct a target image from a plurality of contiguous source images|
|US5640496||Feb 4, 1991||Jun 17, 1997||Medical Instrumentation And Diagnostics Corp. (Midco)||Method and apparatus for management of image data by linked lists of pixel values|
|US5657073||Jun 1, 1995||Aug 12, 1997||Panoramic Viewing Systems, Inc.||Seamless multi-camera panoramic imaging with distortion correction and selectable field of view|
|US5760826||May 10, 1996||Jun 2, 1998||The Trustees Of Columbia University||Omnidirectional imaging apparatus|
|US5790181||Aug 25, 1994||Aug 4, 1998||Australian National University||Panoramic surveillance system|
|US5796426||May 27, 1994||Aug 18, 1998||Warp, Ltd.||Wide-angle image dewarping method and apparatus|
|US5920376||Aug 30, 1996||Jul 6, 1999||Lucent Technologies, Inc.||Method and system for panoramic viewing with curved surface mirrors|
|US5963213||May 7, 1997||Oct 5, 1999||Olivr Corporation Ltd.||Method and system for accelerating warping|
|US5990941||Aug 18, 1995||Nov 23, 1999||Interactive Pictures Corporation||Method and apparatus for the interactive display of any portion of a spherical image|
|US6005611||Aug 4, 1998||Dec 21, 1999||Be Here Corporation||Wide-angle image dewarping method and apparatus|
|US6043837||May 8, 1997||Mar 28, 2000||Be Here Corporation||Method and apparatus for electronically distributing images from a panoptic camera system|
|US6118474||Dec 5, 1997||Sep 12, 2000||The Trustees Of Columbia University In The City Of New York||Omnidirectional imaging apparatus|
|US6157018||Jun 29, 1998||Dec 5, 2000||Ishiguro; Hiroshi||Omni directional vision photograph device|
|US6157385||May 17, 1999||Dec 5, 2000||Oxaal; Ford||Method of and apparatus for performing perspective transformation of visible stimuli|
|US6175454||Jan 13, 1999||Jan 16, 2001||Behere Corporation||Panoramic imaging arrangement|
|US6204855||Jun 19, 1998||Mar 20, 2001||Intel Corporation||Computer system for interpolating a value for a pixel|
|US6211864||Feb 16, 2000||Apr 3, 2001||Oak Technology, Inc.||Method and apparatus for optimizing storage of compressed images in memory|
|US6219089||Jun 24, 1999||Apr 17, 2001||Be Here Corporation||Method and apparatus for electronically distributing images from a panoptic camera system|
|US6222683||Jul 31, 2000||Apr 24, 2001||Be Here Corporation||Panoramic imaging arrangement|
|US6233004||Oct 15, 1997||May 15, 2001||Canon Kabushiki Kaisha||Image processing method and apparatus|
|US6246413||Aug 17, 1998||Jun 12, 2001||Mgi Software Corporation||Method and system for creating panoramas|
|US6256061||Jun 9, 1998||Jul 3, 2001||Interactive Pictures Corporation||Method and apparatus for providing perceived video viewing experiences using still images|
|US6271855||Jun 18, 1998||Aug 7, 2001||Microsoft Corporation||Interactive construction of 3D models from panoramic images employing hard and soft constraint characterization and decomposing techniques|
|US6304285||Jun 16, 1998||Oct 16, 2001||Zheng Jason Geng||Method and apparatus for omnidirectional imaging|
|US6313865||Jan 10, 2000||Nov 6, 2001||Be Here Corporation||Method and apparatus for implementing a panoptic camera system|
|US6320584||Dec 13, 1999||Nov 20, 2001||Imove Inc.||Method and apparatus for simulating movement in multidimensional space with polygonal projections from subhemispherical imagery|
|US6323862||Jul 8, 1999||Nov 27, 2001||Ford Oxaal||Apparatus for generating and interactively viewing spherical image data and memory thereof|
|US6331869||Aug 7, 1998||Dec 18, 2001||Be Here Corporation||Method and apparatus for electronically distributing motion panoramic images|
|US6333826||Mar 7, 1998||Dec 25, 2001||Jeffrey R. Charles||Omniramic optical system having central coverage means which is associated with a camera, projector, or similar article|
|US6337708||Apr 21, 2000||Jan 8, 2002||Be Here Corporation||Method and apparatus for electronically distributing motion panoramic images|
|US6369818||Nov 25, 1998||Apr 9, 2002||Be Here Corporation||Method, apparatus and computer program product for generating perspective corrected data from warped information|
|US6412961 *||Mar 9, 2001||Jul 2, 2002||Robert Andrew Hicks||Rectifying mirror|
|US6754614 *||Feb 23, 2000||Jun 22, 2004||Interscience, Inc.||Linearized static panoramic optical mirror|
|US20010010555||Jun 11, 1997||Aug 2, 2001||Edward Driscoll Jr||Panoramic camera|
|US20010015751||Mar 29, 2001||Aug 23, 2001||Genex Technologies, Inc.||Method and apparatus for omnidirectional imaging|
|US20020006000||Jul 13, 2001||Jan 17, 2002||Kiyoshi Kumata||Omnidirectional vision sensor|
|US20020041326||Apr 3, 2001||Apr 11, 2002||Edward Driscoll||Method and apparatus for electronically distributing images from a panoptic camera system|
|US20020064306||Aug 22, 2001||May 30, 2002||Ulrich Pilz||Image encoding method and image encoder|
|USD312263||Aug 3, 1987||Nov 20, 1990||Wide angle reflector attachment for a camera or similar article|
|USRE36207||Jul 12, 1996||May 4, 1999||Omniview, Inc.||Omniview motionless camera orientation system|
|DE10000673A1||Jan 11, 2000||Jul 12, 2001||Brains 3 Gmbh & Co Kg||Optical arrangement has automation software, converts photographic image to image strip by mirrored sphere with distortion by factor of 3.6, converts Cartesian data into linear data|
|DE19542355A1||Nov 14, 1995||May 15, 1997||Hans Thilo Richter||Verfahren und Vorrichtung zur Erzeugung eines Weitwinkelbildes|
|FR1234341A||Title not available|
|WO2000011512A1||Sep 25, 1998||Mar 2, 2000||Be Here Corporation||A panoramic imaging arrangement|
|1||J. S. Chahl et al., "Reflective Surfaces for Panoramic Imaging," Applied Optics, vol. 36, No. 31, Nov. 1, 1997, pp. 8275-8285.|
|2||M. Ollis et al., "Analysis and Design of Panoramic Stereo Vision Using Equi-Angular Pixel Cameras," The Robotics Institute, Carnegie Mellon University, CMU-RI-TR-99-04, Jan. 1999, pp. 1-43.|
|3||R. A. Hicks et al., "Catadioptric Sensors That Approximate Wide-Angle Perspective Projections," Drexel University and National Science Foundation, pp. 1-7.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7609910 *||Apr 6, 2005||Oct 27, 2009||Siemens Medical Solutions Usa, Inc.||System and method for creating a panoramic view of a volumetric image|
|US7620220 *||Mar 21, 2003||Nov 17, 2009||Boston Scientific Scimed, Inc.||Scan conversion of medical imaging data from polar format to cartesian format|
|US7978931 *||Mar 9, 2007||Jul 12, 2011||Sony Corporation||Optimized video stitching method|
|US8044953 *||Mar 28, 2006||Oct 25, 2011||Autodesk, Inc.||System for interactive 3D navigation for proximal object inspection|
|US8077906 *||May 2, 2007||Dec 13, 2011||Electronics And Telecommunications Research Institute||Apparatus for extracting camera motion, system and method for supporting augmented reality in ocean scene using the same|
|US8106936||Mar 14, 2008||Jan 31, 2012||Kollmorgen Corporation||Panoramic video imaging and display system|
|US8203596||Dec 20, 2007||Jun 19, 2012||Lockheed Martin Corporation||Panoramic imaging system with dual imagers|
|US8478074 *||Jul 7, 2006||Jul 2, 2013||Microsoft Corporation||Providing multiple and native representations of an image|
|US8730127 *||Oct 1, 2009||May 20, 2014||Rafael Advanced Defense Systems Ltd.||Method and system for displaying a panoramic view to an operator|
|US8767037||Mar 17, 2010||Jul 1, 2014||The Trustees Of Columbia University In The City Of New York||Systems and methods for panoramic imaging|
|US8817066||Apr 22, 2011||Aug 26, 2014||The Trustees Of Columbia University In The City Of New York||Systems and methods for panoramic imaging|
|US8836783||Apr 4, 2014||Sep 16, 2014||Eyesee360, Inc.||Telepresence using panoramic imaging and directional sound and motion|
|US8942513||Mar 18, 2013||Jan 27, 2015||Vivotek Inc.||De-warping processing method for digital images|
|US9165338 *||Aug 1, 2014||Oct 20, 2015||Tencent Technology (Shenzhen) Company Limited||Method, device and computer-readable storage medium for panoramic image completion|
|US9204042||Sep 12, 2014||Dec 1, 2015||Eyesee360, Inc.||Method of recording and synchronizing panoramic images and motion|
|US9256056||Jun 24, 2014||Feb 9, 2016||Duke University||Monocentric lens-based multi-scale optical systems and methods of use|
|US9286673||Oct 3, 2013||Mar 15, 2016||Volcano Corporation||Systems for correcting distortions in a medical image and methods of use thereof|
|US9292918||Mar 13, 2014||Mar 22, 2016||Volcano Corporation||Methods and systems for transforming luminal images|
|US9301687||Mar 12, 2014||Apr 5, 2016||Volcano Corporation||System and method for OCT depth calibration|
|US9307926||Oct 2, 2013||Apr 12, 2016||Volcano Corporation||Automatic stent detection|
|US9324141||Oct 2, 2013||Apr 26, 2016||Volcano Corporation||Removal of A-scan streaking artifact|
|US9360630||Aug 31, 2012||Jun 7, 2016||Volcano Corporation||Optical-electrical rotary joint and methods of use|
|US9367965||Sep 25, 2013||Jun 14, 2016||Volcano Corporation||Systems and methods for generating images of tissue|
|US9383263||Dec 17, 2013||Jul 5, 2016||Volcano Corporation||Systems and methods for narrowing a wavelength emission of light|
|US9395617 *||Jan 11, 2013||Jul 19, 2016||Applied Quantum Technologies, Inc.||Panoramic multi-scale imager and method therefor|
|US9432591||May 7, 2013||Aug 30, 2016||Duke University||Multiscale optical system having dynamic camera settings|
|US9438796||Oct 30, 2015||Sep 6, 2016||Eyesee360, Inc.||Apparatus and method for capturing panoramic images and motion|
|US9478940||Oct 4, 2013||Oct 25, 2016||Volcano Corporation||Systems and methods for amplifying light|
|US9486143||Dec 20, 2013||Nov 8, 2016||Volcano Corporation||Intravascular forward imaging device|
|US9494771||Feb 20, 2014||Nov 15, 2016||Duke University||Quasi-monocentric-lens-based multi-scale optical system|
|US9596993||Feb 22, 2012||Mar 21, 2017||Volcano Corporation||Automatic calibration systems and methods of use|
|US9612105||Dec 16, 2013||Apr 4, 2017||Volcano Corporation||Polarization sensitive optical coherence tomography system|
|US9622706||Jul 14, 2008||Apr 18, 2017||Volcano Corporation||Catheter for in vivo imaging|
|US9635253||Oct 31, 2013||Apr 25, 2017||Duke University||Multiscale telescopic imaging system|
|US9709379||Dec 16, 2013||Jul 18, 2017||Volcano Corporation||Optical coherence tomography system that is reconfigurable between different imaging modes|
|US9730613||Dec 20, 2013||Aug 15, 2017||Volcano Corporation||Locating intravascular images|
|US9762813||Dec 22, 2015||Sep 12, 2017||Duke University||Monocentric lens-based multi-scale optical systems and methods of use|
|US9770172||Mar 7, 2014||Sep 26, 2017||Volcano Corporation||Multimodal segmentation in intravascular images|
|US20030220971 *||Aug 16, 2002||Nov 27, 2003||International Business Machines Corporation||Method and apparatus for video conferencing with audio redirection within a 360 degree view|
|US20040186369 *||Mar 21, 2003||Sep 23, 2004||Scimed Life Systems, Inc.||Scan conversion of medical imaging data|
|US20040196282 *||Feb 17, 2004||Oct 7, 2004||Oh Byong Mok||Modeling and editing image panoramas|
|US20050099500 *||Nov 10, 2004||May 12, 2005||Canon Kabushiki Kaisha||Image processing apparatus, network camera system, image processing method and program|
|US20050199645 *||Jun 25, 2003||Sep 15, 2005||Ronald Sivertsen||Device for recognising containers|
|US20050226483 *||Apr 6, 2005||Oct 13, 2005||Bernhard Geiger||System and method for creating a panoramic view of a volumetric image|
|US20050226491 *||Jun 25, 2003||Oct 13, 2005||Andreas Nordbryhn||Device for recognising containers|
|US20060227134 *||Mar 28, 2006||Oct 12, 2006||Autodesk Inc.||System for interactive 3D navigation for proximal object inspection|
|US20070117082 *||Nov 21, 2006||May 24, 2007||Winneg Douglas M||Systems, methods and apparatus for monitoring exams|
|US20070117083 *||Nov 21, 2006||May 24, 2007||Winneg Douglas M||Systems, methods and apparatus for monitoring exams|
|US20070211934 *||Mar 9, 2007||Sep 13, 2007||Sony Taiwan Limited||Optimized video stitching method|
|US20080008392 *||Jul 7, 2006||Jan 10, 2008||Microsoft Corporation||Providing multiple and native representations of an image|
|US20080075358 *||May 2, 2007||Mar 27, 2008||Electronics And Telecommunications Research Institute||Apparatus for extracting camera motion, system and method for supporting augmented reality in ocean scene using the same|
|US20090058988 *||Mar 14, 2008||Mar 5, 2009||Kollmorgen Corporation||System for Panoramic Image Processing|
|US20090059018 *||Sep 5, 2007||Mar 5, 2009||Micron Technology, Inc.||Navigation assisted mosaic photography|
|US20110018964 *||Mar 17, 2010||Jan 27, 2011||The Trustees Of Columbia University In The City Of New York||Systems and methods for panoramic imaging|
|US20110175796 *||Oct 1, 2009||Jul 21, 2011||Rafael Advanced Defense Systems Ltd.||Method and system for displaying a panoramic view to an operator|
|US20110211039 *||Apr 22, 2011||Sep 1, 2011||Gurunandan Krishnan||Systems And Methods For Panoramic Imaging|
|US20120105601 *||Jun 1, 2011||May 3, 2012||Samsung Electronics Co., Ltd.||Apparatus and method for creating three-dimensional panoramic image by using single camera|
|US20120206566 *||Oct 11, 2011||Aug 16, 2012||Teachscape, Inc.||Methods and systems for relating to the capture of multimedia content of observed persons performing a task for evaluation|
|US20120229596 *||Dec 28, 2011||Sep 13, 2012||Michael Kenneth Rose||Panoramic Imaging and Display System With Intelligent Driver's Viewer|
|US20140009571 *||Jun 24, 2013||Jan 9, 2014||Zheng Jason Geng||Wide Field of View Reflector and Method of Designing and Making Same|
|US20140139623 *||Jan 11, 2013||May 22, 2014||Duke University||Panoramic multi-scale imager and method therefor|
|US20150170332 *||Aug 1, 2014||Jun 18, 2015||Tencent Technology (Shenzhen) Company Limited||Method, Device and Computer-Readable Storage Medium for Panoramic Image Completion|
|WO2008115416A1 *||Mar 14, 2008||Sep 25, 2008||Kollmorgen Corporation||System for panoramic image processing|
|WO2012145317A1||Apr 17, 2012||Oct 26, 2012||Eyesee360, Inc.||Apparatus and method for panoramic video imaging with mobile computing devices|
|U.S. Classification||382/284, 382/154, 359/725, 382/276|
|International Classification||G06K9/36, G06T3/00, G02B13/06, G03B37/00|
|Cooperative Classification||G02B13/06, G06T3/0062, G03B37/06|
|Dec 2, 2002||AS||Assignment|
Owner name: EYESEE360, INC., PENNSYLVANIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:RONDINELLI, MICHAEL;HERMAN, HERMAN;SINGH, SANJIV;REEL/FRAME:013539/0361
Effective date: 20021120
|Jul 3, 2007||CC||Certificate of correction|
|Apr 16, 2010||FPAY||Fee payment|
Year of fee payment: 4
|Apr 16, 2014||FPAY||Fee payment|
Year of fee payment: 8
|Mar 29, 2016||AS||Assignment|
Owner name: 360FLY, INC., PENNSYLVANIA
Free format text: CHANGE OF NAME;ASSIGNOR:EYESEE360, INC.;REEL/FRAME:038278/0584
Effective date: 20140723
|Jun 16, 2017||AS||Assignment|
Owner name: HYDRA VENTURES B.V., AS COLLATERAL AGENT, NETHERLA
Free format text: SECURITY INTEREST;ASSIGNOR:360FLY, INC.;REEL/FRAME:042859/0928
Effective date: 20170614