CA2232757C - Real-time image rendering with layered depth images - Google Patents
Real-time image rendering with layered depth images Download PDFInfo
- Publication number
- CA2232757C CA2232757C CA002232757A CA2232757A CA2232757C CA 2232757 C CA2232757 C CA 2232757C CA 002232757 A CA002232757 A CA 002232757A CA 2232757 A CA2232757 A CA 2232757A CA 2232757 C CA2232757 C CA 2232757C
- Authority
- CA
- Canada
- Prior art keywords
- depth image
- layered depth
- image
- layered
- viewpoint
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Lifetime
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
- G06T15/205—Image-based rendering
Abstract
A method and computer product for rendering real-time three-dimensional images on a display based on view manipulation of prestored depth images in a global coordinate space. First, a layered depth image is generated from multiple depth images based on a predetermined display viewpoint. If the determined viewpoint is within a predetermined threshold of the layered depth image, the generated layered depth image is warped based on the determined display viewpoint, pixels from the layered depth image are splatted onto the warped image, and an output image is generated and displayed based on the splat pixels. If the determined viewpoint is outside the predetermined threshold of the previously generated layered depth image, a next closest layered depth image is generated. If the next closest layered depth image is not fully generated, the previously generated layered depth image is used to generate an output image. If the next closest layered depth image is fully generated, the process is repeated for the generated next closest layered depth image.
Description
REAL-TIME IMAGE RENDERING WTTH LAYERED DEPTH IlVIAGES
Field of the Inyen-tion This invention relates to imaging rendering and, more particularly, to an improved method, apparatus and computer product for space transformation in an iunage based rendered scene.
Background of the Invention Image based rendering (IBR) techniques are efficient ways of rendering real and synthetic objects in a three-dimensional scene. With traditional rendering techniques, the time required to render an image becomes unbounded as the geometric complexity of the scene increases. The rendering time also increases as the shading computations become more complex.
In the simplest IBR technique, one synthesizes a new image from a single input depth image (DI). A pI is an image with z-buffer information stored with each pixel. FTidden surfaces are not included in the input image, and thus the image has an effective depth complexity of one_ Shading computations that have been computed for the input image can be reused by subsequent images. Finally, if a depth image is obtained from a real world scene using real images, new views can be created vwith IBR methods without first creating a traditional geometric representation of the scene.
Because the pixels of an image form a regular grid, IBR computations are largely incremental and inexpensive. Moreover, McMillan, in Leonard McMillan, "A
list-priority rendering algorithm for redisplaying projected surfaces", L71VC
Technical Report, 95-005, University of North Carolina, 1995, presents an ordering algorithm that ensures that pixels in the synthesized image are drawn back to front, and thus no
Field of the Inyen-tion This invention relates to imaging rendering and, more particularly, to an improved method, apparatus and computer product for space transformation in an iunage based rendered scene.
Background of the Invention Image based rendering (IBR) techniques are efficient ways of rendering real and synthetic objects in a three-dimensional scene. With traditional rendering techniques, the time required to render an image becomes unbounded as the geometric complexity of the scene increases. The rendering time also increases as the shading computations become more complex.
In the simplest IBR technique, one synthesizes a new image from a single input depth image (DI). A pI is an image with z-buffer information stored with each pixel. FTidden surfaces are not included in the input image, and thus the image has an effective depth complexity of one_ Shading computations that have been computed for the input image can be reused by subsequent images. Finally, if a depth image is obtained from a real world scene using real images, new views can be created vwith IBR methods without first creating a traditional geometric representation of the scene.
Because the pixels of an image form a regular grid, IBR computations are largely incremental and inexpensive. Moreover, McMillan, in Leonard McMillan, "A
list-priority rendering algorithm for redisplaying projected surfaces", L71VC
Technical Report, 95-005, University of North Carolina, 1995, presents an ordering algorithm that ensures that pixels in the synthesized image are drawn back to front, and thus no
-2-depth comparisons are required. This also permits proper alpha compositing or blending of pixels without depth sorting.
Despite these advantages, there still exist marty problems with current IBR
methods. For example, if the viewer moves slightly and thereby uncovers a surface, no relevant information is available for this newly unoccluded surface_ This occurs because a single DI has no information about hidden surfaces. A simple solution to this problem is the use of more than one input DI. If n input images are used, the size of the scene description is multiplied by n, and the rendering cost increases accordingly. Moreover, with more than one input DX, hidden surface removal must be performed.
Another difficulty arises because the input DI has a di$erent sampling pattern and density than the output image. When mapping the discrete pixels forward from the input DI, many pixels might squeeze together in an output pixel. These pixels must be properly blended for anti-alia.sing. Also, forward mapping of the image spreads the pixels apart, creating gaps in the output image. One solution includes performing a backwards rnapping from the output image location to the input DI.
This is an expensive operation that requires some amount of searching in the input DI.
Another solution is to think of the input DI as a mesh of micro-polygons, and to scan-convert these polygons in the output image. This is also expensive, because it requires a polygon scan-convert setup for each input pixel.
The simplest solution to fill gaps in the output irrtage is to predict the projected size of an input pixel in the new projected view, and to "splat" the input pixel into the output image using a precomputed footprint. For the splats to combine smoothiy in the output image, the outer regions of the splat shoald have fractional alpha values and be composed into the new image using an ordering algoxithm.
This requires the output pixels to be drawn in depth order. But, McMillan's ordering algorithm cannot be applied when more than one input DI is used, and so a depth sort is required.
Nelson Max, in "Rendering trees from precomputed z-buffer views, Sixth Eurographics Workshop on Rendering", Eurographics, June 1995, discusses using layered depth images (LDI) for the purpose of high quality anti-aliasing_ LDIs are images that include information of objects that are hidden by foreground objects. In other words, an LDI includes multiple depth layers. Max warps from n input LDIs with different camera information to an output LDI, thereby rendering at about five
Despite these advantages, there still exist marty problems with current IBR
methods. For example, if the viewer moves slightly and thereby uncovers a surface, no relevant information is available for this newly unoccluded surface_ This occurs because a single DI has no information about hidden surfaces. A simple solution to this problem is the use of more than one input DI. If n input images are used, the size of the scene description is multiplied by n, and the rendering cost increases accordingly. Moreover, with more than one input DX, hidden surface removal must be performed.
Another difficulty arises because the input DI has a di$erent sampling pattern and density than the output image. When mapping the discrete pixels forward from the input DI, many pixels might squeeze together in an output pixel. These pixels must be properly blended for anti-alia.sing. Also, forward mapping of the image spreads the pixels apart, creating gaps in the output image. One solution includes performing a backwards rnapping from the output image location to the input DI.
This is an expensive operation that requires some amount of searching in the input DI.
Another solution is to think of the input DI as a mesh of micro-polygons, and to scan-convert these polygons in the output image. This is also expensive, because it requires a polygon scan-convert setup for each input pixel.
The simplest solution to fill gaps in the output irrtage is to predict the projected size of an input pixel in the new projected view, and to "splat" the input pixel into the output image using a precomputed footprint. For the splats to combine smoothiy in the output image, the outer regions of the splat shoald have fractional alpha values and be composed into the new image using an ordering algoxithm.
This requires the output pixels to be drawn in depth order. But, McMillan's ordering algorithm cannot be applied when more than one input DI is used, and so a depth sort is required.
Nelson Max, in "Rendering trees from precomputed z-buffer views, Sixth Eurographics Workshop on Rendering", Eurographics, June 1995, discusses using layered depth images (LDI) for the purpose of high quality anti-aliasing_ LDIs are images that include information of objects that are hidden by foreground objects. In other words, an LDI includes multiple depth layers. Max warps from n input LDIs with different camera information to an output LDI, thereby rendering at about five
-3-minutes per frame. This technique generates high quality anti-aliasing of the output picture, but is expensive and cannot run in real time.
The present invention is directed to overcoming the foregoing and other disadvantages. More specifically, the present invention is directed to providing a method, apparatus, and computer product suitable for real-time IBR of objects in a transversable three-dimensional space.
Summaa of the Inyention In accordance with this invention, a method and computer product for rendering real-time three-dimensional images on a display based on viewpoint manipulation of prestored depth images in a global coordinate space is provided.
The method formed in accordance with this invention includes warping pixels from one or more depth images to a layered depth image. The warping of pixels from one or more depth images to a layered depth image is performed by retrieving depth image pixels that correspond to a ray traced location within the layered depth image, comparing the z-value of the retrieved pixels to the z-values of the pixels previouisly stored at z-value layers within the layered depth image at the ray traced locations with,in the layered depth image that correspond to the retrieved pixels from the depth images, and saving the retrieved pixels in the layered depth image based on the ray traced location and the retrieved pixel's z-value, if no previously stored pixels have a compared z-value that is less than a preset value from the retrieved pixels' z-values.
The retrieved pixels are averaged with previously stored pixels in the layered depth image that have compared z-values that are less than a preset value from the retrieved pixel's z-value. The averaged result is saved based on the the ray traced locations and the retiieved pixels' z-values.
In accordance with other aspects of this inventioN a method for rendering real-time three-dimensional images from prestored depth images in a global coordinate space on a display based on a viewpoint manipulation is provided. A
layered depth image is generated from the prestored depth images based on a
The present invention is directed to overcoming the foregoing and other disadvantages. More specifically, the present invention is directed to providing a method, apparatus, and computer product suitable for real-time IBR of objects in a transversable three-dimensional space.
Summaa of the Inyention In accordance with this invention, a method and computer product for rendering real-time three-dimensional images on a display based on viewpoint manipulation of prestored depth images in a global coordinate space is provided.
The method formed in accordance with this invention includes warping pixels from one or more depth images to a layered depth image. The warping of pixels from one or more depth images to a layered depth image is performed by retrieving depth image pixels that correspond to a ray traced location within the layered depth image, comparing the z-value of the retrieved pixels to the z-values of the pixels previouisly stored at z-value layers within the layered depth image at the ray traced locations with,in the layered depth image that correspond to the retrieved pixels from the depth images, and saving the retrieved pixels in the layered depth image based on the ray traced location and the retrieved pixel's z-value, if no previously stored pixels have a compared z-value that is less than a preset value from the retrieved pixels' z-values.
The retrieved pixels are averaged with previously stored pixels in the layered depth image that have compared z-values that are less than a preset value from the retrieved pixel's z-value. The averaged result is saved based on the the ray traced locations and the retiieved pixels' z-values.
In accordance with other aspects of this inventioN a method for rendering real-time three-dimensional images from prestored depth images in a global coordinate space on a display based on a viewpoint manipulation is provided. A
layered depth image is generated from the prestored depth images based on a
-4-predeterrnined display viewpoint. Then, the generated layered depth image is assigned as the current layered depth image. An output image is generated from the current layered depth image based on the predetermined display viewpoint.
Next, it is determined if the display viewpoint of the current layered depth image has been manipulated within the global coordinate space. An output image is generated from the current layered depth image based on the deternuned display viewpoint, if the display viewpoint has been manipulated and the manipulation is within a predetennined threshold distance from the current layered depth image's viewpoint.
A next closest layered depth image is generated, if the display viewpoint has been manipulated and the manipulation is outside the predetermined threshold distance of the current layered depth image's viewpoint. An output image is generating from the next closest layered depth image based on the determined display viewpoint and the next closest layered depth image is assigned as the current layered depth image, if the next closest layered depth image has been fully generated.
In accordance with still other aspects of this invention, the generating of an output image from the current and next closest layered depth image includes warping the generated layered depth image based on the manipulated display viewpoint, splatting the pixels from the layered depth image onto the warped image, and generating and displaying an output image based on the splat pixels.
In accordatnce with further other aspects of this invention, the splatting includes selecting one of two or more splat sizes and splatting pixels based on the splat size selected.
In accordance with still further other aspects of this invention, each splat size comprises a predefined splat mask.
In accordance with still yet further other aspects of this invention, real-time three-dimensional stereo images are rendered from prestored depth images based on the techniques described above.
Next, it is determined if the display viewpoint of the current layered depth image has been manipulated within the global coordinate space. An output image is generated from the current layered depth image based on the deternuned display viewpoint, if the display viewpoint has been manipulated and the manipulation is within a predetennined threshold distance from the current layered depth image's viewpoint.
A next closest layered depth image is generated, if the display viewpoint has been manipulated and the manipulation is outside the predetermined threshold distance of the current layered depth image's viewpoint. An output image is generating from the next closest layered depth image based on the determined display viewpoint and the next closest layered depth image is assigned as the current layered depth image, if the next closest layered depth image has been fully generated.
In accordance with still other aspects of this invention, the generating of an output image from the current and next closest layered depth image includes warping the generated layered depth image based on the manipulated display viewpoint, splatting the pixels from the layered depth image onto the warped image, and generating and displaying an output image based on the splat pixels.
In accordatnce with further other aspects of this invention, the splatting includes selecting one of two or more splat sizes and splatting pixels based on the splat size selected.
In accordance with still further other aspects of this invention, each splat size comprises a predefined splat mask.
In accordance with still yet further other aspects of this invention, real-time three-dimensional stereo images are rendered from prestored depth images based on the techniques described above.
-5-As vvill be readily appreciated ;from the foregoing summary, the invention provides a new and improved method, apparatus, and computer product for rendering real-time three-dimensional images on a display based on viewpoint manipulation of prestored depth images in a global coordinate space. Because the apparatus does not warp from multiple input DIs, for each frame generated, the disadvantages due to such warping is avoided.
Brief Description of the Drawings The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same becomes better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:
FIGURE I is an illustration of a camera view of objects in a three-dimensional space;
FIGURE 2 is a top view of the camera view of FIGURE 1;
FIGURES 3 and 4 are diagrams of systems operable with the present invention;
FIGYTRES 5A-C are flow diagrams of the present invention;
FIGURE 6 is a perspective view of camera views and their geometric relatioilship as determined based on the present invention;
FIGITRL 7 is a top view of multiple camera views of an object in space; and FIGURE 8 is an illustration of geometric relationships of an object viewed from a single camera based on the present invention.
Detailed Descriation of the Preferred Emb diment As will be better understood from the following description, the present invention is directed to an image based rendering system that renders multiple frames per second of an unage of objects located in a transversable three-dimensional space.
Each image to be rendered is generated from a layered depth image (LDI) that comprises information from multiple precreated depth images (DI).
A DI is an image with single depth or z information in addition to the color (RGB), and optionally a normal stored for each pixel. As will be described in more detail below with FIGURE 5, the construction of the LDX is decoupled from the final rendering of images from desired viewpoints. Thus, the LDI construction does not need to run at multiple frames per second to allow for interactive camera motion. The LDI is then processed in real-time, multiple frames per second, into an output image based on a desired camera view.
Brief Description of the Drawings The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same becomes better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:
FIGURE I is an illustration of a camera view of objects in a three-dimensional space;
FIGURE 2 is a top view of the camera view of FIGURE 1;
FIGURES 3 and 4 are diagrams of systems operable with the present invention;
FIGYTRES 5A-C are flow diagrams of the present invention;
FIGURE 6 is a perspective view of camera views and their geometric relatioilship as determined based on the present invention;
FIGITRL 7 is a top view of multiple camera views of an object in space; and FIGURE 8 is an illustration of geometric relationships of an object viewed from a single camera based on the present invention.
Detailed Descriation of the Preferred Emb diment As will be better understood from the following description, the present invention is directed to an image based rendering system that renders multiple frames per second of an unage of objects located in a transversable three-dimensional space.
Each image to be rendered is generated from a layered depth image (LDI) that comprises information from multiple precreated depth images (DI).
A DI is an image with single depth or z information in addition to the color (RGB), and optionally a normal stored for each pixel. As will be described in more detail below with FIGURE 5, the construction of the LDX is decoupled from the final rendering of images from desired viewpoints. Thus, the LDI construction does not need to run at multiple frames per second to allow for interactive camera motion. The LDI is then processed in real-time, multiple frames per second, into an output image based on a desired camera view.
-6-FIGURES 1 and 2 illustrate the geornetric relationships based on the present invention. The location of camera view 10 is identified by x, y, and z coordinates within a global coordinate system. A projection plane or display area 12 is identified by X,= by Yres. The camera view 10 is at some fixed predefined focal length from the display area 12. The display area 12 includes a predefined number of pixels 14.
fiach LDI contains camera viewpoint information relating to various features of the camera view and an array of layered depth pixels. The array size is Xres by Y,,.. Each layered depth pixel includes pixel image data and an integer indicating the number of valid layers contained in that pixel. For pixel 14 in the ezample shown, two valid layers are present: one from object 16, and one from object 18. The pixel image data relates to the objects in the three-dimensional space along a particular line-of-sight or ray from the camera position. In FIGURE 1, the closest object for pixel 14 is object 16. The pixel image data includes the color, the depth of the object, and an index into a table. The index is formed from a combination of the normal of the object seen 19 and the distance from the camera 20 that originally captured the pixel. The index is used to compute splat size and is described in more detail below with FIGUIZE 6.
FIGURE 3 is an illustrative embodiment of a computer system 19 capable of implementing the method of the present invention, described in more detail below with respect to FIGL7RE 5. The computer system 19 includes a depth image memory 20 for storing pregenerated DIs, an image processor 22 connected to the first memory 20 and a layered depth image memory 26 connected to the image processor 22 for storing LDIs. While shown as separate memories for ease of illustration, the depth image memory 20 and the layered depth image memory 26 could be located in a single memory, such as removable memory or on a hard disc drive. The computer system 19 also includes a fast warping based renderer 28 connected to the layered depth image memory 26, at least one user interface ('M
device 30 connected to the image processor 22 and renderer 28, and a display connected to the renderer 28. The UI devices 30 allow a user to request different displayable viewpoints of the image. As will be readily appreciated by those of ordinary skill in the art of user interfaces, the UI devices may be a mouse, keyboard, a voice connmand generator, or other type interactive device capable of directing movement through the image by generating viewpoint signals.
fiach LDI contains camera viewpoint information relating to various features of the camera view and an array of layered depth pixels. The array size is Xres by Y,,.. Each layered depth pixel includes pixel image data and an integer indicating the number of valid layers contained in that pixel. For pixel 14 in the ezample shown, two valid layers are present: one from object 16, and one from object 18. The pixel image data relates to the objects in the three-dimensional space along a particular line-of-sight or ray from the camera position. In FIGURE 1, the closest object for pixel 14 is object 16. The pixel image data includes the color, the depth of the object, and an index into a table. The index is formed from a combination of the normal of the object seen 19 and the distance from the camera 20 that originally captured the pixel. The index is used to compute splat size and is described in more detail below with FIGUIZE 6.
FIGURE 3 is an illustrative embodiment of a computer system 19 capable of implementing the method of the present invention, described in more detail below with respect to FIGL7RE 5. The computer system 19 includes a depth image memory 20 for storing pregenerated DIs, an image processor 22 connected to the first memory 20 and a layered depth image memory 26 connected to the image processor 22 for storing LDIs. While shown as separate memories for ease of illustration, the depth image memory 20 and the layered depth image memory 26 could be located in a single memory, such as removable memory or on a hard disc drive. The computer system 19 also includes a fast warping based renderer 28 connected to the layered depth image memory 26, at least one user interface ('M
device 30 connected to the image processor 22 and renderer 28, and a display connected to the renderer 28. The UI devices 30 allow a user to request different displayable viewpoints of the image. As will be readily appreciated by those of ordinary skill in the art of user interfaces, the UI devices may be a mouse, keyboard, a voice connmand generator, or other type interactive device capable of directing movement through the image by generating viewpoint signals.
-7-The image processor 22 generates LDIs from DIs stored in the depth image memory 20 and the renderer 28 outputs images to the display 32 processing the LDI
based on the generated viewpoint signals. The processing of the LDI by the renderer 28 is performed in real-time and is described in more detail below with respect to FIGU'RES 5A and C. LDI generation is described in more detail below with respect to FIGURE 5B. Real-time based on the present invention relates to producing images at multiple frames per second. A viewpoint signal as desired by a user through operation of the UI device is sent to the image processor 22 and the renderer 28. The renderer 28 is preferably a high speed processor for generating multiple frames per second. The image processor 22 does not require to be as fast as renderer 28. As will readily appreciated by those of ordinary skill in the art of data processing, processors and renderers may be components of a central processing unit or separate components.
FIGURE 4 is an illustrative embodiment of an alternate computer system 33 formed in accordance with the present invention. The FIGURE 4 computer system 33 includes an image processor or server 22a and a renderer 28a that are remotely located froim each other. The computer system 33 is impleinented over a network 34. The network 34 is a public data networiS such as the Internet, or a private data network. Time intensive image processing operations are performed at the server 22a which is connected over the network 34 to a user's system 36.
The setver 22a includes a database 20a for storing prede~ined DIs. In this configuration the server 22a is exclusively used for generating LDIs from DIs. The LDIs are downloaded to a user's system 36 across the network 34. The user's computer system 36 is typically a desktop computer connected to the network 34 through one of its data ports, but may be any other functionally comparable components.
When the user's system 36 receives an LDI across the network 34 from the server 22a, it is displayed on display 32a thereby allowing interaction. If the user desires a new viewpoint, a desired viewpoint signal is sent back through the network 34 to the server 22a, thereby requesting that another LDI be generated, if the desired viewpoint is greater than a predefined amount. Therefore, the present invention is a useful tool for traversing through complex three-dimensional environments without requiring, at the local level, the intense processing required for generating LDIs from DIs.
An example of a type of usage over a network would be a car company wanting to let customers view and travel through, at their leisure, pregenerated images of a new automobile they plan on selling. This example is used for illustrative purposes only.
As vvill be readily appreciated by those of ordinary skill in the art, various computer system, configurations may be applied to the present invention provided they include components for performirg the method of the present invention. In other words, image processing and rendering may be performed within a conventional central processing unit or specialty processors or depth image memory may be permanent memory at various locations.
FIGURES SA-C illustrate the method briefly described above and performed by the computer systems 19 and 33 shown in FIGURBS 3 and 4. First, at block 100, an LDI is generated from a predetermined number of prestored DIs based on a predetermined or default viewpoint. LDI generation is described in more detail below with respect to FIGURE 5B. Then, at block 104, the LDI is saved as the current L)DI. The current LDI is a memory location identifier. At block 108, the process determines what viewpoint is desired based on a user desired viewpoint that is generated by interaction with the displayed image using the UI device, (i_e., viewpoint manipulation). This interaction may be the simple act of activating the arrow buttons on a keyboard or controlling the cursor by a mouse. Next, the process determines if the viewpoint desired is outside a predetermined threshold distance from the current LDI, at decision block 112. If the determixted viewpoint is not outside the predetermined threshold distance, at block 114, an output image is generated from the current I.DI. Generation of an output iraage from an LDI is described in more detail below with respect to FIGURE 5C. If the determined viewpoint is outside the predeteitnined threshold distance, the process determines if the determined viewpoint is within the predetermined threshold distance from an LDI stored as a last LDI. See decision block 118. Storage of an LDI as the last LDI is described below. I.f no LDI
is stored as a last LDI or if the determined viewpoint is not within the predetermined threshold distance from the last LDI, at block 122, a next closest LDI from prestored depth image is based on the determined viewpoint. Neact closest LDI generation is described in more detail below with respect to FIGITRE 5B. Since LDI
generation is slow as coznpared to the generation of output images from LDIs, the process determines if the next closest LDI has been fully generated. See decision block 126.
If the next closest LDI has not been fully generated, the output image is generated from the current LDI and returns to the decision perfortned in decision block 126, as shown in block 130. Once the next closest LDI has been fully generated an output image is generated from the next closest LDI. Again, output image generation from an LDI is described in more de2ail below with respect to FIGURE SC. See block 134.
-~-Next, at block 138, the current LDI is saved as the last LDI and the generated next closest LDI is saved as the current LDI. Essentially, what is being performed by the step at block 13 8 is a swapping of the current and last LDIs in meinory. The process then returns to the viewpoint determination at block 108.
If, at decision block 118, the viewpoint was determined to be within the predetermined threshold distance from the last LDI, an output image is generated from the last LDI. See block 142. Again, output image generation from an LDI
is described in more detail below with respect to FIGURE 5C. Then, at block 146, the current LDI is saved as the last LDI and the previous last LDI is saved as the current LDI. Essentially, this step is petforming a storage location swap of the last and current LDIs. Once the saving is complete, the process returns to the viewpoint deterntination step shown at block 108. As will be readily appreciated by those of ordinary skill in the art of three-dimensional display generation, the viewpoint deterinination process performed at block 108 is continually performed throughout the process shown in FIGYJRE 5A. Specifically, the output image generations shown in blocks 130 and 134 must have up-to-date viewpoint information in order to generate accurate output images.
FIGURE 5B illustrates LDI generation from prestored depth images from block 100 in FIGURE SA. First, at block 200, the pixel from the DI that corresponds or warps to the first pixel location in the LDI is selected_ Then, at decision block 204, the process determines if the LDI pixel location that corcesponds to the selected DI
pixel includes one or more pixels at distinct layers. If there are no pixels stored at distinct layers within the LDI at the pixel location that corresponds to the selected DI
pixel, the selected DI pixel is stored at the LDI pixel location based on the DI pixel's z-value. See block 208. If at decision block 212 the LDI pixel location that corresponds to the selected DI pixel is the last pixel location in the LDI, the process selects from the DI the pixel that warps or corresponds to the next pixel location in the LDI, at block 216, and returns to decision block 204. If at decision block 204 the process deterrrrines that there are pixels located at the pixel location that corresponds to the selected DI pixel, the process compares the z-value for the selected DI
pixel to the z-value(s) of the located one or more pixels. See block 220_ Then, at decision block 224, the process determines if any of the z-values of the one or more pixels at distinct layers within the LDI differ by less than a preset value from the selected pixel z-value. If the one or more pixels do not differ by less than the preset value, the selected DI pixel is added as a new layer to the LDI pixel. See block 228. If, however, the z-values of the one or more pixels of the LDX differ by less than the preset value, the process takes the average of the one or more pixels with z-values that differ by less than the preset value to the selected DI pixel. See block 232. Once the steps in blocks 228 and 232 are cortplete the process continues onto decision block 212, thereby continuing until the DI has been fully mapped to the LDI.
Essentially what is being performed in the process described above for FIGURE
5 is the incorporation of the pixels of prestored depth images into a single layered depth image. The process is determining if pixels have already been mapped to a common location in the layered depth image (from previous depth images) when a mapping is occurrixtg from a pixel in a present depth image. If pixel information at the LDI
already exists the rnapped pixel from the depth image is stored at a z-value location if its z-value is greater by a preset value than all the other pixels stored at that I.DI pixel location. If there already exists a pixel at the same z-value or a z-value that is within the preset value then the mapped pixel is averaged with those pixels. As willl be readily appreciated by those of ordinary skill in the art of image warping, the terminology applied to the above description is intended to describe the warping of an image to a different viewpoint.
FIGURE 5C illustrates the output image generation from an LDI. The process illustrated in FIGURE 5C is that performed in blocks 114, 130, 134 and of FIGCTRE 5A- First, at block 252 the process begins by selecting the first pixel in the LDI to be used to generate the output image. At block 254, the selected pixel is warped to the location required by the desired viewpoint for the output image.
Then, the pixel is splatted into the warp location, at block 256, thereby generating an output image for display. If, at decision block 258, more unwarped pixels remain from the LDI, the next pixel is selected and the process returns to block 254. Once no more unwarped pixels remain, the output image is fully generated and displayed.
The process illustrated in FIGURE 5C is capable of generating a stereographic output image with a few minor changes (not shown). Since a stereographic image includes two images from two dif'ferent viewpoints, the distance between the two views is a predetermined value. The process of generating an output image, as shown in FIGURE 5C, generates two output images based on the predetermined distance value for stereographic image generation. 'fhen, the two generated output images are displayed on a stereographic output device.
The LDI warping process, performed in an iIlustrative embodiment of the present invention, begins with a four-by-four matrix C that represents the camera view for each LDI. The matrix C, includes information relating to the position of the camera or viewpoint in global 3-D space, the dimensions of the camera viewing angle, the resolution level for the camera's projection plane or display area, and the distance of camera to the display area.
Cl transforms a point from the global coordinate system that represents the three-dimensional environment into the image coordinate system of a first LDI
camera view. The image coordinates (xt, Y1) index a pixel address and are obtained after multiplying each point's global coordinates by C1 and dividing out w(homogenous coordinate). The zt coordinate is used for depth comparisons in a z buffer.
A transfer matrix is defined as Ti2 Z = C2 - Cl 1. The transfer matrix computes the image coordinates as seen in the output camera view, based on the image coordinates seen in the LDI camera. In the folloving description, the output camera view refers to the user-desired view requested through the user interface device.
Equation 1 determines the coordinates (x2,, y2, s2) of the LDI warped to the desired viewpoint.
xi x2=w yl yZ ' = resultYec (1) T1, Z z1 z2 = w 1 w The coordinates (xz, y2) obtained after dividing out w, index a pixel address in the output camera's image.
As can be readily appreciated by those of ordinary skill in the art, this matrix multiply can be factored to allow reuse of much of the computation as one iterates through the layers of a layered depth pixel, as shown in Equations 2 and 3.
xX xl 0 T1,2 = il T1,2 - 0 +zi = T1,2 = ~ = startYec+zl -depthYec (2) To compute the warped position of the next layered depth pixel along a scanline, the new start'V'ec is simply incremented, as shown in Equation 3.
x1+1 xl 1 T.2= l =T,2= Ol +T,2 0= startYec + incrYec (3) CC2Ci 1 J xs = x2s (4) C2-output camera (desired viewpoint) The warping algorithm, Equation 4, is similar to McMillan's ordering algorithm discussed in "A list-priority rendering algorithm for redisplaying projected surfaces", UNC Technical Report 95-005, University of North Carolina, 1995. As shown in FIGURE 6, first the depth order of layered depth pixels is computed by first finding the projection of the output camera's location in the LDI camera's image plane 278. This is the intersection of the line joining the two camera locations with the LDI
camera's image plane 278. The line joining the two camera locations is called the epipolar line 280 and the intersection with the image plane is called an epipolar point 282. The LDI
image is then split horizontally and vertically at the epipolar point 282 creating four image regions. Two or one regions may exist, if the epipolar point 282 lies off the image plane 278. For each region, the LDI is traversed in (possibly reverse) scan line order. At the beginning of each scan line, startVec is computed. The sign of incrVec is determined by the direction of processing in the region. Each layered depth pixel in the scan line is then warped to the output image. This procedure visits each of the layers of a pixel in back to front order and computes resultVec to determine its location in the output image.
As in texture mapping, a divide is required per pixel. Finally, the depth pixel's color is splatted at this location in the output image.
To splat the LDI into the output image, the projected area or size of the warped pixel is approximated. The proper size is computed as shown in Equation 5.
size =(dl ) 2 cos (02 ) res2tan (. 5 fovi ) (d2)2 cos (81) res1tan ( 5fov2) (5) As shown in FIGURE 7, dl is the distance from the sampled surface point to the LDI
camera 290, fovl is the field of view of the LDI camera 290, resl is the pixel resolution of the LDI camera 290, and 01 is the angle between the surface normal and the line of sight to the LDI camera 290. 02, d2, and res2 refer to similar values with respect to the output camera view 292.
It is more efficient to compute an approximation of the square root of size, as shown in Equation 6.
size d1 cos (92) resZtan (. 5 fovl ) d2 cos (91) resttan (. 5 fov2 ) (6) 1 dl cos (02)res2tan (. 5 fov1 ) (7) M cos (O1) resitan (.5 fov2 ) ~_ z2=lookup[nx,ny,dl] (8) As shown in Equation 7, 0 is approximated as the angles 0 between the surface normal vector and the z axes of the camera's coordinate systems. Also, d2 is approximated by using ze2, which is the z coordinate of the sampled point in the output camera's unprojected eye coordinate system.
Since this embodiment provides three splat sizes, the lengthy computations of Equations 6 and 7 are unnecessary. Therefore, as shown in Equation 8, an approximation of the size computation is implemented using a lookup table. For each pixel in the LDI, two bits represent dl, and four bits encode the normal (two bits for nx, and two for ny), as shown in FIGURE 8. This produces a six-bit lookup table with 64 possible indexes.
Before rendering each new image, the new output camera information precomputes values for the 64 possible Iookup table indexes. During rendering a projection matrix P2 included within C2 is chosen such that z2 = 1/ze2. At each pixel, size is obtained by multiplying the computed z2 by the value found in the lookup table.
The three splat sizes are a one pixel, a three by three pixel, and a five by five pixel footprint, as shown below. Each pixel in a footprint has an alpha value of one, one-half, or one-fourth, therefore the alpha blending is performed with integer shifts and adds. The following splat masks are used:
1 and 25 2 4 2 and 25 2 4 4 4 Z
As will be appreciated by those of ordinary skill in the art of pixel splatting, various splat patterns and sizes may be chosen depending upon system capabilities, and desired processing speed and image quality.
While the present invention has been illustra.ted with reference to exeannplaty eznbodiments, those slcilled in the art will appreciate that various changes in form and detaal may be made without departing from the intended scope of the present invention as defined in the appended claims_ Because of the variations that can be applied to the illustrated and described embodiments of the invention, the invention should be defined solely with reference to the appended claims.
based on the generated viewpoint signals. The processing of the LDI by the renderer 28 is performed in real-time and is described in more detail below with respect to FIGU'RES 5A and C. LDI generation is described in more detail below with respect to FIGURE 5B. Real-time based on the present invention relates to producing images at multiple frames per second. A viewpoint signal as desired by a user through operation of the UI device is sent to the image processor 22 and the renderer 28. The renderer 28 is preferably a high speed processor for generating multiple frames per second. The image processor 22 does not require to be as fast as renderer 28. As will readily appreciated by those of ordinary skill in the art of data processing, processors and renderers may be components of a central processing unit or separate components.
FIGURE 4 is an illustrative embodiment of an alternate computer system 33 formed in accordance with the present invention. The FIGURE 4 computer system 33 includes an image processor or server 22a and a renderer 28a that are remotely located froim each other. The computer system 33 is impleinented over a network 34. The network 34 is a public data networiS such as the Internet, or a private data network. Time intensive image processing operations are performed at the server 22a which is connected over the network 34 to a user's system 36.
The setver 22a includes a database 20a for storing prede~ined DIs. In this configuration the server 22a is exclusively used for generating LDIs from DIs. The LDIs are downloaded to a user's system 36 across the network 34. The user's computer system 36 is typically a desktop computer connected to the network 34 through one of its data ports, but may be any other functionally comparable components.
When the user's system 36 receives an LDI across the network 34 from the server 22a, it is displayed on display 32a thereby allowing interaction. If the user desires a new viewpoint, a desired viewpoint signal is sent back through the network 34 to the server 22a, thereby requesting that another LDI be generated, if the desired viewpoint is greater than a predefined amount. Therefore, the present invention is a useful tool for traversing through complex three-dimensional environments without requiring, at the local level, the intense processing required for generating LDIs from DIs.
An example of a type of usage over a network would be a car company wanting to let customers view and travel through, at their leisure, pregenerated images of a new automobile they plan on selling. This example is used for illustrative purposes only.
As vvill be readily appreciated by those of ordinary skill in the art, various computer system, configurations may be applied to the present invention provided they include components for performirg the method of the present invention. In other words, image processing and rendering may be performed within a conventional central processing unit or specialty processors or depth image memory may be permanent memory at various locations.
FIGURES SA-C illustrate the method briefly described above and performed by the computer systems 19 and 33 shown in FIGURBS 3 and 4. First, at block 100, an LDI is generated from a predetermined number of prestored DIs based on a predetermined or default viewpoint. LDI generation is described in more detail below with respect to FIGURE 5B. Then, at block 104, the LDI is saved as the current L)DI. The current LDI is a memory location identifier. At block 108, the process determines what viewpoint is desired based on a user desired viewpoint that is generated by interaction with the displayed image using the UI device, (i_e., viewpoint manipulation). This interaction may be the simple act of activating the arrow buttons on a keyboard or controlling the cursor by a mouse. Next, the process determines if the viewpoint desired is outside a predetermined threshold distance from the current LDI, at decision block 112. If the determixted viewpoint is not outside the predetermined threshold distance, at block 114, an output image is generated from the current I.DI. Generation of an output iraage from an LDI is described in more detail below with respect to FIGURE 5C. If the determined viewpoint is outside the predeteitnined threshold distance, the process determines if the determined viewpoint is within the predetermined threshold distance from an LDI stored as a last LDI. See decision block 118. Storage of an LDI as the last LDI is described below. I.f no LDI
is stored as a last LDI or if the determined viewpoint is not within the predetermined threshold distance from the last LDI, at block 122, a next closest LDI from prestored depth image is based on the determined viewpoint. Neact closest LDI generation is described in more detail below with respect to FIGITRE 5B. Since LDI
generation is slow as coznpared to the generation of output images from LDIs, the process determines if the next closest LDI has been fully generated. See decision block 126.
If the next closest LDI has not been fully generated, the output image is generated from the current LDI and returns to the decision perfortned in decision block 126, as shown in block 130. Once the next closest LDI has been fully generated an output image is generated from the next closest LDI. Again, output image generation from an LDI is described in more de2ail below with respect to FIGURE SC. See block 134.
-~-Next, at block 138, the current LDI is saved as the last LDI and the generated next closest LDI is saved as the current LDI. Essentially, what is being performed by the step at block 13 8 is a swapping of the current and last LDIs in meinory. The process then returns to the viewpoint determination at block 108.
If, at decision block 118, the viewpoint was determined to be within the predetermined threshold distance from the last LDI, an output image is generated from the last LDI. See block 142. Again, output image generation from an LDI
is described in more detail below with respect to FIGURE 5C. Then, at block 146, the current LDI is saved as the last LDI and the previous last LDI is saved as the current LDI. Essentially, this step is petforming a storage location swap of the last and current LDIs. Once the saving is complete, the process returns to the viewpoint deterntination step shown at block 108. As will be readily appreciated by those of ordinary skill in the art of three-dimensional display generation, the viewpoint deterinination process performed at block 108 is continually performed throughout the process shown in FIGYJRE 5A. Specifically, the output image generations shown in blocks 130 and 134 must have up-to-date viewpoint information in order to generate accurate output images.
FIGURE 5B illustrates LDI generation from prestored depth images from block 100 in FIGURE SA. First, at block 200, the pixel from the DI that corresponds or warps to the first pixel location in the LDI is selected_ Then, at decision block 204, the process determines if the LDI pixel location that corcesponds to the selected DI
pixel includes one or more pixels at distinct layers. If there are no pixels stored at distinct layers within the LDI at the pixel location that corresponds to the selected DI
pixel, the selected DI pixel is stored at the LDI pixel location based on the DI pixel's z-value. See block 208. If at decision block 212 the LDI pixel location that corresponds to the selected DI pixel is the last pixel location in the LDI, the process selects from the DI the pixel that warps or corresponds to the next pixel location in the LDI, at block 216, and returns to decision block 204. If at decision block 204 the process deterrrrines that there are pixels located at the pixel location that corresponds to the selected DI pixel, the process compares the z-value for the selected DI
pixel to the z-value(s) of the located one or more pixels. See block 220_ Then, at decision block 224, the process determines if any of the z-values of the one or more pixels at distinct layers within the LDI differ by less than a preset value from the selected pixel z-value. If the one or more pixels do not differ by less than the preset value, the selected DI pixel is added as a new layer to the LDI pixel. See block 228. If, however, the z-values of the one or more pixels of the LDX differ by less than the preset value, the process takes the average of the one or more pixels with z-values that differ by less than the preset value to the selected DI pixel. See block 232. Once the steps in blocks 228 and 232 are cortplete the process continues onto decision block 212, thereby continuing until the DI has been fully mapped to the LDI.
Essentially what is being performed in the process described above for FIGURE
5 is the incorporation of the pixels of prestored depth images into a single layered depth image. The process is determining if pixels have already been mapped to a common location in the layered depth image (from previous depth images) when a mapping is occurrixtg from a pixel in a present depth image. If pixel information at the LDI
already exists the rnapped pixel from the depth image is stored at a z-value location if its z-value is greater by a preset value than all the other pixels stored at that I.DI pixel location. If there already exists a pixel at the same z-value or a z-value that is within the preset value then the mapped pixel is averaged with those pixels. As willl be readily appreciated by those of ordinary skill in the art of image warping, the terminology applied to the above description is intended to describe the warping of an image to a different viewpoint.
FIGURE 5C illustrates the output image generation from an LDI. The process illustrated in FIGURE 5C is that performed in blocks 114, 130, 134 and of FIGCTRE 5A- First, at block 252 the process begins by selecting the first pixel in the LDI to be used to generate the output image. At block 254, the selected pixel is warped to the location required by the desired viewpoint for the output image.
Then, the pixel is splatted into the warp location, at block 256, thereby generating an output image for display. If, at decision block 258, more unwarped pixels remain from the LDI, the next pixel is selected and the process returns to block 254. Once no more unwarped pixels remain, the output image is fully generated and displayed.
The process illustrated in FIGURE 5C is capable of generating a stereographic output image with a few minor changes (not shown). Since a stereographic image includes two images from two dif'ferent viewpoints, the distance between the two views is a predetermined value. The process of generating an output image, as shown in FIGURE 5C, generates two output images based on the predetermined distance value for stereographic image generation. 'fhen, the two generated output images are displayed on a stereographic output device.
The LDI warping process, performed in an iIlustrative embodiment of the present invention, begins with a four-by-four matrix C that represents the camera view for each LDI. The matrix C, includes information relating to the position of the camera or viewpoint in global 3-D space, the dimensions of the camera viewing angle, the resolution level for the camera's projection plane or display area, and the distance of camera to the display area.
Cl transforms a point from the global coordinate system that represents the three-dimensional environment into the image coordinate system of a first LDI
camera view. The image coordinates (xt, Y1) index a pixel address and are obtained after multiplying each point's global coordinates by C1 and dividing out w(homogenous coordinate). The zt coordinate is used for depth comparisons in a z buffer.
A transfer matrix is defined as Ti2 Z = C2 - Cl 1. The transfer matrix computes the image coordinates as seen in the output camera view, based on the image coordinates seen in the LDI camera. In the folloving description, the output camera view refers to the user-desired view requested through the user interface device.
Equation 1 determines the coordinates (x2,, y2, s2) of the LDI warped to the desired viewpoint.
xi x2=w yl yZ ' = resultYec (1) T1, Z z1 z2 = w 1 w The coordinates (xz, y2) obtained after dividing out w, index a pixel address in the output camera's image.
As can be readily appreciated by those of ordinary skill in the art, this matrix multiply can be factored to allow reuse of much of the computation as one iterates through the layers of a layered depth pixel, as shown in Equations 2 and 3.
xX xl 0 T1,2 = il T1,2 - 0 +zi = T1,2 = ~ = startYec+zl -depthYec (2) To compute the warped position of the next layered depth pixel along a scanline, the new start'V'ec is simply incremented, as shown in Equation 3.
x1+1 xl 1 T.2= l =T,2= Ol +T,2 0= startYec + incrYec (3) CC2Ci 1 J xs = x2s (4) C2-output camera (desired viewpoint) The warping algorithm, Equation 4, is similar to McMillan's ordering algorithm discussed in "A list-priority rendering algorithm for redisplaying projected surfaces", UNC Technical Report 95-005, University of North Carolina, 1995. As shown in FIGURE 6, first the depth order of layered depth pixels is computed by first finding the projection of the output camera's location in the LDI camera's image plane 278. This is the intersection of the line joining the two camera locations with the LDI
camera's image plane 278. The line joining the two camera locations is called the epipolar line 280 and the intersection with the image plane is called an epipolar point 282. The LDI
image is then split horizontally and vertically at the epipolar point 282 creating four image regions. Two or one regions may exist, if the epipolar point 282 lies off the image plane 278. For each region, the LDI is traversed in (possibly reverse) scan line order. At the beginning of each scan line, startVec is computed. The sign of incrVec is determined by the direction of processing in the region. Each layered depth pixel in the scan line is then warped to the output image. This procedure visits each of the layers of a pixel in back to front order and computes resultVec to determine its location in the output image.
As in texture mapping, a divide is required per pixel. Finally, the depth pixel's color is splatted at this location in the output image.
To splat the LDI into the output image, the projected area or size of the warped pixel is approximated. The proper size is computed as shown in Equation 5.
size =(dl ) 2 cos (02 ) res2tan (. 5 fovi ) (d2)2 cos (81) res1tan ( 5fov2) (5) As shown in FIGURE 7, dl is the distance from the sampled surface point to the LDI
camera 290, fovl is the field of view of the LDI camera 290, resl is the pixel resolution of the LDI camera 290, and 01 is the angle between the surface normal and the line of sight to the LDI camera 290. 02, d2, and res2 refer to similar values with respect to the output camera view 292.
It is more efficient to compute an approximation of the square root of size, as shown in Equation 6.
size d1 cos (92) resZtan (. 5 fovl ) d2 cos (91) resttan (. 5 fov2 ) (6) 1 dl cos (02)res2tan (. 5 fov1 ) (7) M cos (O1) resitan (.5 fov2 ) ~_ z2=lookup[nx,ny,dl] (8) As shown in Equation 7, 0 is approximated as the angles 0 between the surface normal vector and the z axes of the camera's coordinate systems. Also, d2 is approximated by using ze2, which is the z coordinate of the sampled point in the output camera's unprojected eye coordinate system.
Since this embodiment provides three splat sizes, the lengthy computations of Equations 6 and 7 are unnecessary. Therefore, as shown in Equation 8, an approximation of the size computation is implemented using a lookup table. For each pixel in the LDI, two bits represent dl, and four bits encode the normal (two bits for nx, and two for ny), as shown in FIGURE 8. This produces a six-bit lookup table with 64 possible indexes.
Before rendering each new image, the new output camera information precomputes values for the 64 possible Iookup table indexes. During rendering a projection matrix P2 included within C2 is chosen such that z2 = 1/ze2. At each pixel, size is obtained by multiplying the computed z2 by the value found in the lookup table.
The three splat sizes are a one pixel, a three by three pixel, and a five by five pixel footprint, as shown below. Each pixel in a footprint has an alpha value of one, one-half, or one-fourth, therefore the alpha blending is performed with integer shifts and adds. The following splat masks are used:
1 and 25 2 4 2 and 25 2 4 4 4 Z
As will be appreciated by those of ordinary skill in the art of pixel splatting, various splat patterns and sizes may be chosen depending upon system capabilities, and desired processing speed and image quality.
While the present invention has been illustra.ted with reference to exeannplaty eznbodiments, those slcilled in the art will appreciate that various changes in form and detaal may be made without departing from the intended scope of the present invention as defined in the appended claims_ Because of the variations that can be applied to the illustrated and described embodiments of the invention, the invention should be defined solely with reference to the appended claims.
Claims (14)
1. A method for rendering real-time three-dimensional images from prestored depth images in a global coordinate space on a display based on a viewpoint manipulation, said method comprising:
(a) generating a layered depth image from the prestored depth images based on a predetermined perspective display viewpoint;
(b) assigning the generated layered depth image as the current layered depth image;
(c) rendering an output image only from the current layered depth image based on the predetermined perspective display viewpoint;
(d) determining if the perspective display viewpoint of the current layered depth image has been manipulated within the global coordinate space;
(e) rendering an output image only from the current layered depth image based on the determined perspective display viewpoint, if the perspective display viewpoint has been manipulated and the manipulation is within a predetermined threshold distance from the current layered depth image's perspective viewpoint;
(f) generating a next closest layered depth image, if the perspective display viewpoint has been manipulated and the manipulation is outside the predetermined threshold distance of the current layered depth image's perspective viewpoint;
and (g) rendering an output image only from the next closest layered depth image based on the determined perspective display viewpoint and assigning the next closest layered depth image as the current layered depth image and repeating (d)-(g), if the next closest layered depth image has been fully generated.
(a) generating a layered depth image from the prestored depth images based on a predetermined perspective display viewpoint;
(b) assigning the generated layered depth image as the current layered depth image;
(c) rendering an output image only from the current layered depth image based on the predetermined perspective display viewpoint;
(d) determining if the perspective display viewpoint of the current layered depth image has been manipulated within the global coordinate space;
(e) rendering an output image only from the current layered depth image based on the determined perspective display viewpoint, if the perspective display viewpoint has been manipulated and the manipulation is within a predetermined threshold distance from the current layered depth image's perspective viewpoint;
(f) generating a next closest layered depth image, if the perspective display viewpoint has been manipulated and the manipulation is outside the predetermined threshold distance of the current layered depth image's perspective viewpoint;
and (g) rendering an output image only from the next closest layered depth image based on the determined perspective display viewpoint and assigning the next closest layered depth image as the current layered depth image and repeating (d)-(g), if the next closest layered depth image has been fully generated.
2. The method of Claim 1, wherein the step of generating a layered depth image from the prestored de-pth images is performed separate from the step of rendering an output image.
3. The method of Claim 1, wherein rendering an output image directly from the current and next closest layered depth image further comprises:
i) warping the generated layered depth image based on the manipulated perspective display viewpoint;
ii) splatting pixels from the layered depth image onto the warped image;
and iii) generating and displaying an output image based on the splatted pixels.
i) warping the generated layered depth image based on the manipulated perspective display viewpoint;
ii) splatting pixels from the layered depth image onto the warped image;
and iii) generating and displaying an output image based on the splatted pixels.
4. The method of Claim 3, wherein the splatting comprises selecting one of two or more splat sizes and splatting pixels based on the splat size selected.
5. The method of Claim 4, wherein each splat size comprises a predefined splat mask.
6. The method of Claim 1, wherein generating a layered depth image from the prestored depth images based on a predetermined perspective display viewpoint further comprises:
i) retrieving a pixel from a depth image that corresponds to a ray-traced location within the layered depth image;
ii) comparing the z-value of the retrieved pixel to the z-values of pixels previously stored at z-value layers within the layered depth image at the ray-traced location within the layered depth image that corresponds to the retrieved pixel from the depth image;
iii) saving the retrieved pixel in the layered depth image based on the ray-traced location and the retrieved pixel's z-value, if no previously stored pixels have a compared z-value that is less than a preset value from the retrieved pixel's z-value;
iv) averaging the retrieved pixel in the layered depth image to previously stored pixels that have compared z-values that are less than the preset value from the retrieved pixel's z-value to obtain an averaged result; and v) saving the averaged result based on the ray-traced location and the retrieved pixel's z-value.
i) retrieving a pixel from a depth image that corresponds to a ray-traced location within the layered depth image;
ii) comparing the z-value of the retrieved pixel to the z-values of pixels previously stored at z-value layers within the layered depth image at the ray-traced location within the layered depth image that corresponds to the retrieved pixel from the depth image;
iii) saving the retrieved pixel in the layered depth image based on the ray-traced location and the retrieved pixel's z-value, if no previously stored pixels have a compared z-value that is less than a preset value from the retrieved pixel's z-value;
iv) averaging the retrieved pixel in the layered depth image to previously stored pixels that have compared z-values that are less than the preset value from the retrieved pixel's z-value to obtain an averaged result; and v) saving the averaged result based on the ray-traced location and the retrieved pixel's z-value.
7. The method of Claim 6, further comprising:
vi) repeating i)-v) until all ray traced locations of the layered depth image have been analyzed with respect to each of the one or more depth images.
vi) repeating i)-v) until all ray traced locations of the layered depth image have been analyzed with respect to each of the one or more depth images.
8. A computer program product comprising a memory having stored thereon computer-executable instructions that when executed by a computer carry out the method of Claim 1.
9. A computer program product comprising a memory having stored therein computer-executable instructions that when executed by a computer carry out the method of Claim 3.
10. A method for rendering real-time three-dimensional stereo images from prestored depth images in a global coordinate space on a display based on a viewpoint manipulation, said method comprising:
(a) generating a layered depth image from the prestored depth images based on a predetermined perspective display viewpoint;
(b) assigning the generated layered depth image as the current layered depth image;
(c) rendering stereo output images only from the current layered depth image based on the predetermined perspective display viewpoint and predetermined stereo image requirement ;
(d) determining if the perspective display viewpoint of the current layered depth image has been manipulated within the global coordinate space;
(e) rendering stereo output images only from the current layered depth image based on the determined perspective display viewpoint and the predetermined stereo image requirement, if the perspective display viewpoint has been manipulated and the manipulation is within a predetermined threshold distance from the current layered depth image's perspective viewpoint;
(f) generating a next closest layered depth image, if the perspective display viewpoint has been manipulated and the manipulation is outside the predetermined threshold distance of the current layered depth image's perspective viewpoint;
and (g) rendering stereo output images only from the next closest layered depth image based on the determined perspective display viewpoint and the predetermined stereo image requirement., and assigning the next closest layered depth image as the current layered depth image and repeating (d)-(g), if the next closest layered depth image has been fully generated.
(a) generating a layered depth image from the prestored depth images based on a predetermined perspective display viewpoint;
(b) assigning the generated layered depth image as the current layered depth image;
(c) rendering stereo output images only from the current layered depth image based on the predetermined perspective display viewpoint and predetermined stereo image requirement ;
(d) determining if the perspective display viewpoint of the current layered depth image has been manipulated within the global coordinate space;
(e) rendering stereo output images only from the current layered depth image based on the determined perspective display viewpoint and the predetermined stereo image requirement, if the perspective display viewpoint has been manipulated and the manipulation is within a predetermined threshold distance from the current layered depth image's perspective viewpoint;
(f) generating a next closest layered depth image, if the perspective display viewpoint has been manipulated and the manipulation is outside the predetermined threshold distance of the current layered depth image's perspective viewpoint;
and (g) rendering stereo output images only from the next closest layered depth image based on the determined perspective display viewpoint and the predetermined stereo image requirement., and assigning the next closest layered depth image as the current layered depth image and repeating (d)-(g), if the next closest layered depth image has been fully generated.
11. The method of Claim 10, wherein rendering stereo output images directly from the current and next closest layered depth image further comprises:
i) warping the generated layered depth image based on the manipulated perspective display viewpoint and the predetermined stereo image requirement;
ii) splatting pixels from the layered depth image onto the warped image;
and iii) generating and displaying output images based on the splatted pixels.
i) warping the generated layered depth image based on the manipulated perspective display viewpoint and the predetermined stereo image requirement;
ii) splatting pixels from the layered depth image onto the warped image;
and iii) generating and displaying output images based on the splatted pixels.
12. The method of Claim 10, wherein said generating a layered depth image from the prestored depth images comprises warping pixels from one or more depth images into each layered depth image.
13. The method of Claim 12, wherein said warping pixels from one or more depth images into each layered depth image further comprises:
i) retrieving a pixel from the depth image that corresponds to a ray traced location within the layered depth image;
ii) comparing the z-value of the retrieved pixel to the z-values of pixels previously stored at z-value layers within the layered depth image at the ray traced location within the layered depth image that corresponds to the retrieved pixel from the depth image;
iii) saving the retrieved pixel in the layered depth image based on the ray traced location and the retrieved pixel's z-value, if no previously stored pixels have a compared z-value that is less than a preset value from the retrieved pixel's z-value;
iv) averaging the retrieved pixel in the layered depth image to previously stored pixels that have compared z-values that are less than the preset value from the retrieved pixel's z-value to obtain an averaged result; and v) saving the averaged result based on the ray traced location and the retrieved pixel's z-value.
i) retrieving a pixel from the depth image that corresponds to a ray traced location within the layered depth image;
ii) comparing the z-value of the retrieved pixel to the z-values of pixels previously stored at z-value layers within the layered depth image at the ray traced location within the layered depth image that corresponds to the retrieved pixel from the depth image;
iii) saving the retrieved pixel in the layered depth image based on the ray traced location and the retrieved pixel's z-value, if no previously stored pixels have a compared z-value that is less than a preset value from the retrieved pixel's z-value;
iv) averaging the retrieved pixel in the layered depth image to previously stored pixels that have compared z-values that are less than the preset value from the retrieved pixel's z-value to obtain an averaged result; and v) saving the averaged result based on the ray traced location and the retrieved pixel's z-value.
14. The method of Claim 13, further comprising:
vi) repeating i)-v) until all ray traced locations of the layered depth image have been analyzed with respect to each of the one or more depth images.
vi) repeating i)-v) until all ray traced locations of the layered depth image have been analyzed with respect to each of the one or more depth images.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US09/040,941 US6466207B1 (en) | 1998-03-18 | 1998-03-18 | Real-time image rendering with layered depth images |
| US09/040,941 | 1998-03-18 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CA2232757A1 CA2232757A1 (en) | 1999-09-18 |
| CA2232757C true CA2232757C (en) | 2007-05-22 |
Family
ID=21913835
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002232757A Expired - Lifetime CA2232757C (en) | 1998-03-18 | 1998-03-19 | Real-time image rendering with layered depth images |
Country Status (2)
| Country | Link |
|---|---|
| US (1) | US6466207B1 (en) |
| CA (1) | CA2232757C (en) |
Families Citing this family (74)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2354416B (en) * | 1999-09-17 | 2004-04-21 | Technologies Limit Imagination | Depth based blending for 3D graphics systems |
| US6639597B1 (en) * | 2000-02-28 | 2003-10-28 | Mitsubishi Electric Research Laboratories Inc | Visibility splatting and image reconstruction for surface elements |
| US6738533B1 (en) * | 2000-03-29 | 2004-05-18 | Microsoft Corporation | Minimum sampling rate and minimum sampling curve for image-based rendering |
| US6677942B1 (en) * | 2000-08-01 | 2004-01-13 | International Business Machines Corporation | Method and apparatus for acquiring normals maps consistent with low spatial resolution meshes |
| US6580424B1 (en) * | 2000-10-21 | 2003-06-17 | Microsoft Corporation | System and method for interactive room modeling from camera images |
| US20020188669A1 (en) * | 2001-06-11 | 2002-12-12 | Levine Marc Jay | Integrated method for disseminating large spatial data sets in a distributed form via the internet |
| KR100561837B1 (en) * | 2001-07-09 | 2006-03-16 | 삼성전자주식회사 | Method for representing image-based rendering information in a 3D scene |
| JP4148671B2 (en) * | 2001-11-06 | 2008-09-10 | ソニー株式会社 | Display image control processing apparatus, moving image information transmission / reception system, display image control processing method, moving image information transmission / reception method, and computer program |
| GB2392072B (en) * | 2002-08-14 | 2005-10-19 | Autodesk Canada Inc | Generating Image Data |
| JP3855053B2 (en) * | 2003-01-30 | 2006-12-06 | 国立大学法人 東京大学 | Image processing apparatus, image processing method, and image processing program |
| US7538774B2 (en) * | 2003-06-20 | 2009-05-26 | Nippon Telegraph And Telephone Corporation | Virtual visual point image generating method and 3-d image display method and device |
| GB2410663A (en) * | 2004-01-29 | 2005-08-03 | Univ London | 3d computer graphics processing system |
| KR100519779B1 (en) * | 2004-02-10 | 2005-10-07 | 삼성전자주식회사 | Method and apparatus for high speed visualization of depth image-based 3D graphic data |
| US8698844B1 (en) * | 2005-04-16 | 2014-04-15 | Apple Inc. | Processing cursor movements in a graphical user interface of a multimedia application |
| US7298370B1 (en) * | 2005-04-16 | 2007-11-20 | Apple Inc. | Depth ordering of planes and displaying interconnects having an appearance indicating data characteristics |
| WO2006136978A1 (en) * | 2005-06-22 | 2006-12-28 | Koninklijke Philips Electronics N.V. | Method and system for rendering image data on a 3d display |
| KR100714672B1 (en) * | 2005-11-09 | 2007-05-07 | 삼성전자주식회사 | Method for depth based rendering by using splats and system of enabling the method |
| US8559705B2 (en) | 2006-12-01 | 2013-10-15 | Lytro, Inc. | Interactive refocusing of electronic images |
| US10298834B2 (en) * | 2006-12-01 | 2019-05-21 | Google Llc | Video refocusing |
| KR100858086B1 (en) * | 2007-01-30 | 2008-09-10 | 삼성전자주식회사 | Rendering apparatus and method |
| US8384718B2 (en) * | 2008-01-10 | 2013-02-26 | Sony Corporation | System and method for navigating a 3D graphical user interface |
| US9098647B2 (en) * | 2008-03-10 | 2015-08-04 | Apple Inc. | Dynamic viewing of a three dimensional space |
| US8089479B2 (en) | 2008-04-11 | 2012-01-03 | Apple Inc. | Directing camera behavior in 3-D imaging system |
| CN101271583B (en) * | 2008-04-28 | 2010-04-21 | 清华大学 | Fast image drafting method based on depth drawing |
| US8106924B2 (en) * | 2008-07-31 | 2012-01-31 | Stmicroelectronics S.R.L. | Method and system for video rendering, computer program product therefor |
| US9619917B2 (en) * | 2008-10-03 | 2017-04-11 | Apple Inc. | Depth of field for a camera in a media-editing application |
| EP2180449A1 (en) * | 2008-10-21 | 2010-04-28 | Koninklijke Philips Electronics N.V. | Method and device for providing a layered depth model of a scene |
| TWI542190B (en) * | 2008-11-04 | 2016-07-11 | 皇家飛利浦電子股份有限公司 | Method and system for encoding a 3d image signal, encoded 3d image signal, method and system for decoding a 3d image signal |
| CN101695139B (en) * | 2009-10-14 | 2011-08-17 | 宁波大学 | Gradable block-based virtual viewpoint image drawing method |
| KR101760323B1 (en) * | 2010-01-13 | 2017-07-21 | 삼성전자주식회사 | Method and system for rendering three dimensional views of a scene |
| JP5450330B2 (en) * | 2010-09-16 | 2014-03-26 | 株式会社ジャパンディスプレイ | Image processing apparatus and method, and stereoscopic image display apparatus |
| US9122053B2 (en) | 2010-10-15 | 2015-09-01 | Microsoft Technology Licensing, Llc | Realistic occlusion for a head mounted augmented reality display |
| US8884984B2 (en) | 2010-10-15 | 2014-11-11 | Microsoft Corporation | Fusing virtual content into real content |
| US9213405B2 (en) | 2010-12-16 | 2015-12-15 | Microsoft Technology Licensing, Llc | Comprehension and intent-based content for augmented reality displays |
| EP2472880A1 (en) * | 2010-12-28 | 2012-07-04 | ST-Ericsson SA | Method and device for generating an image view for 3D display |
| US9401046B2 (en) * | 2011-02-07 | 2016-07-26 | Intel Corporation | Micropolygon splatting |
| TWI419078B (en) * | 2011-03-25 | 2013-12-11 | Univ Chung Hua | Apparatus for generating a real-time stereoscopic image and method thereof |
| CN102186095B (en) * | 2011-05-03 | 2012-12-12 | 四川虹微技术有限公司 | Matching error correction method applicable for depth-image-based rendering |
| US9153195B2 (en) | 2011-08-17 | 2015-10-06 | Microsoft Technology Licensing, Llc | Providing contextual personal information by a mixed reality device |
| WO2013028908A1 (en) | 2011-08-24 | 2013-02-28 | Microsoft Corporation | Touch and social cues as inputs into a computer |
| US9323325B2 (en) | 2011-08-30 | 2016-04-26 | Microsoft Technology Licensing, Llc | Enhancing an object of interest in a see-through, mixed reality display device |
| WO2013169671A1 (en) | 2012-05-09 | 2013-11-14 | Lytro, Inc. | Optimization of optical systems for improved light field capture and manipulation |
| US9607424B2 (en) | 2012-06-26 | 2017-03-28 | Lytro, Inc. | Depth-assigned content for depth-enhanced pictures |
| US10129524B2 (en) | 2012-06-26 | 2018-11-13 | Google Llc | Depth-assigned content for depth-enhanced virtual reality images |
| US9858649B2 (en) | 2015-09-30 | 2018-01-02 | Lytro, Inc. | Depth-based image blurring |
| EP2962290B1 (en) * | 2013-02-21 | 2019-07-17 | HERE Global B.V. | Relaying 3d information by depth simulation using 2d pixel displacement |
| US10334151B2 (en) | 2013-04-22 | 2019-06-25 | Google Llc | Phase detection autofocus using subaperture images |
| DE112014005866B4 (en) | 2013-12-24 | 2018-08-02 | Lytro, Inc. | Improvement of plenoptic camera resolution |
| WO2015178217A1 (en) * | 2014-05-21 | 2015-11-26 | ソニー株式会社 | Image processing apparatus and method |
| US8988317B1 (en) | 2014-06-12 | 2015-03-24 | Lytro, Inc. | Depth determination for light field images |
| US9444991B2 (en) | 2014-11-13 | 2016-09-13 | Lytro, Inc. | Robust layered light-field rendering |
| US9852539B2 (en) * | 2015-02-26 | 2017-12-26 | Qualcomm Incorporated | Single pass surface splatting |
| US10540818B2 (en) | 2015-04-15 | 2020-01-21 | Google Llc | Stereo image generation and interactive playback |
| US10419737B2 (en) | 2015-04-15 | 2019-09-17 | Google Llc | Data structures and delivery methods for expediting virtual reality playback |
| US10440407B2 (en) | 2017-05-09 | 2019-10-08 | Google Llc | Adaptive control for immersive experience delivery |
| US10275898B1 (en) | 2015-04-15 | 2019-04-30 | Google Llc | Wedge-based light-field video capture |
| US10567464B2 (en) | 2015-04-15 | 2020-02-18 | Google Llc | Video compression with adaptive view-dependent lighting removal |
| US10444931B2 (en) | 2017-05-09 | 2019-10-15 | Google Llc | Vantage generation and interactive playback |
| US10469873B2 (en) | 2015-04-15 | 2019-11-05 | Google Llc | Encoding and decoding virtual reality video |
| US10412373B2 (en) | 2015-04-15 | 2019-09-10 | Google Llc | Image capture for virtual reality displays |
| US10565734B2 (en) | 2015-04-15 | 2020-02-18 | Google Llc | Video capture, processing, calibration, computational fiber artifact removal, and light-field pipeline |
| US11328446B2 (en) | 2015-04-15 | 2022-05-10 | Google Llc | Combining light-field data with active depth data for depth map generation |
| US10546424B2 (en) | 2015-04-15 | 2020-01-28 | Google Llc | Layered content delivery for virtual and augmented reality experiences |
| US10341632B2 (en) | 2015-04-15 | 2019-07-02 | Google Llc. | Spatial random access enabled video system with a three-dimensional viewing volume |
| US9979909B2 (en) | 2015-07-24 | 2018-05-22 | Lytro, Inc. | Automatic lens flare detection and correction for light-field images |
| US10275892B2 (en) | 2016-06-09 | 2019-04-30 | Google Llc | Multi-view scene segmentation and propagation |
| US10679361B2 (en) | 2016-12-05 | 2020-06-09 | Google Llc | Multi-view rotoscope contour propagation |
| US10594945B2 (en) | 2017-04-03 | 2020-03-17 | Google Llc | Generating dolly zoom effect using light field image data |
| US10474227B2 (en) | 2017-05-09 | 2019-11-12 | Google Llc | Generation of virtual reality with 6 degrees of freedom from limited viewer data |
| US10354399B2 (en) | 2017-05-25 | 2019-07-16 | Google Llc | Multi-view back-projection to a light-field |
| US10403032B2 (en) * | 2017-08-22 | 2019-09-03 | Qualcomm Incorporated | Rendering an image from computer graphics using two rendering computing devices |
| US10545215B2 (en) | 2017-09-13 | 2020-01-28 | Google Llc | 4D camera tracking and optical stabilization |
| US10089796B1 (en) * | 2017-11-01 | 2018-10-02 | Google Llc | High quality layered depth image texture rasterization |
| US10965862B2 (en) | 2018-01-18 | 2021-03-30 | Google Llc | Multi-camera navigation interface |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4807158A (en) * | 1986-09-30 | 1989-02-21 | Daleco/Ivex Partners, Ltd. | Method and apparatus for sampling images to simulate movement within a multidimensional space |
| IL108668A (en) * | 1993-02-25 | 1998-09-24 | Hughes Training Inc | Method and system for generating a plurality of images of a three-dimensional scene |
-
1998
- 1998-03-18 US US09/040,941 patent/US6466207B1/en not_active Expired - Lifetime
- 1998-03-19 CA CA002232757A patent/CA2232757C/en not_active Expired - Lifetime
Also Published As
| Publication number | Publication date |
|---|---|
| CA2232757A1 (en) | 1999-09-18 |
| US6466207B1 (en) | 2002-10-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2232757C (en) | Real-time image rendering with layered depth images | |
| US5613048A (en) | Three-dimensional image synthesis using view interpolation | |
| US5805782A (en) | Method and apparatus for projective texture mapping rendered from arbitrarily positioned and oriented light source | |
| Oliveira et al. | Relief texture mapping | |
| US5377313A (en) | Computer graphics display method and system with shadow generation | |
| Pulli et al. | View-based rendering: Visualizing real objects from scanned range and color data | |
| US6525731B1 (en) | Dynamic view-dependent texture mapping | |
| El-Hakim et al. | A multi-sensor approach to creating accurate virtual environments | |
| Raskar et al. | Image precision silhouette edges | |
| Matusik et al. | Polyhedral visual hulls for real-time rendering | |
| US5704024A (en) | Method and an apparatus for generating reflection vectors which can be unnormalized and for using these reflection vectors to index locations on an environment map | |
| US6081273A (en) | Method and system for building three-dimensional object models | |
| US7256791B2 (en) | Rasterization of three dimensional images | |
| EP1550984A2 (en) | Integrating particle rendering and three-dimensional geometry rendering | |
| US5193145A (en) | Method and apparatus for producing a visually improved image in a computer system | |
| JPH0778267A (en) | Method for display of shadow and computer-controlled display system | |
| JPH10508396A (en) | Method of generating composite image | |
| US7528831B2 (en) | Generation of texture maps for use in 3D computer graphics | |
| US20040179262A1 (en) | Open GL | |
| Gortler et al. | Rendering layered depth images | |
| US6975319B1 (en) | System, method and article of manufacture for calculating a level of detail (LOD) during computer graphics processing | |
| US5793372A (en) | Methods and apparatus for rapidly rendering photo-realistic surfaces on 3-dimensional wire frames automatically using user defined points | |
| KR100381817B1 (en) | Generating method of stereographic image using Z-buffer | |
| US6690369B1 (en) | Hardware-accelerated photoreal rendering | |
| JPH07225854A (en) | System and method for generating two-dimensional display of three-dimensional object |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| MKEX | Expiry |
Effective date: 20180319 |
|
| MKEX | Expiry |
Effective date: 20180319 |