CA2242150C - Display techniques for three dimensional virtual reality - Google Patents
Display techniques for three dimensional virtual reality Download PDFInfo
- Publication number
- CA2242150C CA2242150C CA002242150A CA2242150A CA2242150C CA 2242150 C CA2242150 C CA 2242150C CA 002242150 A CA002242150 A CA 002242150A CA 2242150 A CA2242150 A CA 2242150A CA 2242150 C CA2242150 C CA 2242150C
- Authority
- CA
- Canada
- Prior art keywords
- video
- world
- video screen
- screen
- computer graphics
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
Abstract
A limitation of a three-dimensional world in which objects in the distance may be represented in only two dimensions as a video on a screen is that when an object within the field represented by the video undergoes a trajectory that takes it to a location in the world that is not represented by the video but instead is a location in the foreground which is represented by computer graphics, namely, any portion of the object that is no longer on the video screen disappears. To overcome this limitation, when an object within the field represented by the video undergoes a trajectory that takes it to a location in the world that is not represented by the video but instead is a location in the foreground which is represented by computer graphics, such an object, or portion thereof, is made to continue to be visible to the user by representing it at the foreground location to which its trajectory was carrying it using computer graphic techniques, rather than video. Thus, the video object "pops" out of the video and becomes visible, e.g., in front of, or to the side of, the video screen, rather than becoming invisible because it is no longer on the video screen.
Description
DISPLAY TECHNIQUES FOR THREE DIMENSIONAL VIRTUAL REALITY
Technical Field This invention relates to the integration of computer graphics and video to provide a realistic three dimensional virtual reality experience.
Background of the Invention The display of a three dimensional world to a viewer requires considerable computation power, and it is typically costly to develop the necessary highly detailed models required for doing so. In order to simplify the problem, a portion of the world that is in the distance may be represented in only two dimensions as a video displayed on a surface, e.g., a screen. By video it is meant the common usage of the term, such as the placing or projecting of predefined images on the surface, e.g., the electronic version of filmed moving pictures. Thus, such a world is essentially truncated in length to the screen on which the video is displayed.
A great reduction in computation power and cost can be achieved by such an arrangement.
Summary of the Invention We have recognized that a limitation of such a world occurs when an object within the field represented by the video undergoes a trajectory that takes it to a location in the world that is not represented by the video but instead is a location in the foreground which is represented by computer graphics, namely, any portion of the object that is :no longer on the video screen, disappears. Therefore, in accordance with the principles of the invention, when an object within the field represented by the video undergoes a trajectory that takes it, or a portion thereof, to a location in the world that is not represented by the video but insi:ead is a location in the foreground which is represented by computer graphics, such an object or portion is made to continue to be visible to the user and is represented at the foreground location using computer graphic techniques, rather than video. Thus, the video obj ect "pops"
out of the video and becomes visible, e.g., in front of, or to the side of, the video screen, rather than becoming invisible because it is no longer on the video screen.
In accordance with one aspect of the present invention there is provided an apparatus for use in processing a view of a three dimensional world in which a first portion of said world is modeled as computer graphics and a second portion of said world is represented by a video, 3 0 wherein said video is made up of a sequence of images displayed on a video screen which exists in said first portion of said world, said video displaying a plurality of objects, said apparatus comprising: means for determining that an object which is represented as part of said video on said video screen has moved, from the point of view of a viewer of said world, to a location that is off of said video screen, so that said object can no longer be displayed on said video screen and will disappear from said world; and means for changing the representation of said object from a video representation to one modeled as computer graphics whereby said object remains visible in said world, albeit not on said video screen, and said video screen continues to exist in said world.
In accordance with another aspect of the present invention there is provided an apparatus for use in processing a view on a three dimensional world in which a first portion of said world is modeled as computer graphics and a second portion of said world is represented by a video, comprising: means for determining that an object being represented by said video has moved to a location that is being modeled as computer graphics; and means for revealing a representation of said object as computer graphics.
In accordance with yet another aspect of the present invention there is provided a system for displaying an apparently three dimensional world to a user, in which a first part of said three dimensional world is displayed as computer graphics and a second part is displayed as video on a video screen, wherein said video is made up of a sequence of images displayed on a video screen which exists in said first portion of said world, said video displaying a plurality of objects, the method, said system comprising: a processor for determining that an element displayed as video on said video screen transited, from the point of view of a viewer of said world, from said second part to said first part so that said object can no longer be displayed on said video screen and will disappear from said world; and a rendering engine for displaying said world with said element displayed in said first part rather than said second part whereby said object remains visible in said world, albeit not on said video screen, and said video screen continues to exist in said world.
In accordance with still yet another aspect of the present invention there is provided a method for use in processing a view of a three dimensional world in which a first portion of said world is modeled as computer graphics and a second portion of said world is represented by a video, wherein said video is made up of a sequence of images displayed on a video screen which exists in said first portion of said world, said video displaying a plurality -la-of objects, the method comprising the steps of: determining that an object which is represented as part of said video on said video screen has moved, from the point of view of a viewer of said world, to a location that is off of said video screen, so that said object can no longer be displayed on said video screen and will disappear from said world;
and changing the representation of said object from a video representation to one modeled as computer graphics whereby said object remains visible in said world, albeit not on said video screen, and said video screen continues to exist in said world.
Brief Description of the Drawings In the drawings:
FIGs. 1-3 show an example of a video object "popping" out of a video and so becoming visible in front of, or to the side of, the video screen, in accordance with the principles of the invention;
FIG. 4 shows an exemplary process by which objects within the field represented by the video surface that, due to their projected motion are to be in the three dimensional world, are so displayed using computer graphics techniques, e.g., the object "pops out" of the video surface, in accordance with the principles of the invention;
FIG. 5 shows a flow chart of a process of another embodiment of the invention;
FIG. 6 is a representation of a three dimensional world and an associated view processing system according to an embodiment oI'the present invention; and FIG. 7 is a representation of a three dimensional world and an associated three dimensional display system according to an embodiment of the present invention.
Detailed Description To better illustrate the invention, FIGS. 1-3 show an example of that which a user sees according to the invention when an object, that is displayed as video on a video screen of a world that has a portion of the world distant from the point of view of the user represented in only two dimensions as a video on the video screen, undergoes a trajectory that takes at least a portion of it to a location in the world that is not represented by the video but instead is a location in the world that is represented by computer graphics, and such an object or portion is made to yet be visible to the user by representing the object at its location off of the video screen using computer graphic techniques, rather than video. In other words, FIGS. 1-3 show an example of a video object "popping" out of the video and becoming visible in front of, to the side of, above, or below the video screen, rather than simply becoming invisible because it is no longer on the video screen. For simplification of terminology purposes, a portion of an object may simply be referred to as an object, since any portion of an object may be considered an object in its own right.
FIG. 1 shows world 101, which is the Champs Elysees in Paris, France, as one approaches La Place de 1'Etoile in the center of which is the Arc de Triomphe, built by Napoleon Bonaparte. World 101 is divided into two portions, video screen 103, on which is shown the current frame of a video and the remainder of the world 1 O5, which is represented using computer graphics techniques, and is thus referred to herein as computer graphics part (CG Part) 105. The current frame of video being displayed on video screen 103 includes police van 113 and Arc de Triomphe 115. Within CG Part i05 there are various elements, such as bicyclist 107, representing the user, road 109, and sky 111.
Note that the viewpoint of the user is actually behind the representation of the user in the form of bicyclist 107. Also note that police van 113, which is part of the current -2a-video frame being shown on video screen 103, is moving slower than bicyclist 107, so that police van 113 will eventually be passed by bicyclist 107 as he continues to ride toward Arc de Triumph 115.
FIG. 2 shows world 101 of FIG. 1, but at a later time. At the time of FIG. 2, the frame of video being shown on screen 103 is from a view closer to Arc de Triumph 115.
Such a frame may have resulted, for example, from moving the camera that captured the video closer to Arc de Triumph 115. As a result of the camera location when the frame of video on screen 103 that is shown in FIG. 2 was taken, only a portion of police van 113 was captured video frame. The rest of the police van 113 was out of view of the 1 o camera, and hence is not visible within the current frame of video on screen 103 that is shown in FIG. 2. However, from the viewpoint of the user, looking at world 101 as it appears in FIG. 2, the remaining portion of police van 113 should be visible, notwithstanding that it is no longer within the boundaries of video screen 103. Therefore, in accordance with the principles of the invention, the remaining portion of police van 113 is displayed using computer graphic techniques within world 101 as element 201, which is a part of CG Part 105.
FIG. 3 shows world 101 of FIG. 1, but at a time even later than that of FIG.
2.
Thus, at the time of FIG. 3, the frame of video being shown on screen 103 is from a view still closer to Arc de Triumph 115 than that of FIG. . As a result of the camera location 2o when the frame of video on screen 103 that is shown in FIG. 3 was taken, none of police van 113 is visible within the current frame of video on screen 103 that is shown in FIG. 3.
However, from the viewpoint of the user, looking at world 101 as it appears in FIG. 3, police van 113 should be visible to the user's left, notwithstanding that it is no longer within the boundaries of video screen 103. Therefore, in accordance with the principles of the invention, the entirety of police van 113 is displayed using computer graphic techniques within world 101 as element 201, which is a part of CG Part 105. Ai least a portion of element 201 will continue to be displayed as part of CG Part 105 until element 201 passes completely from the user's viewpoint.
FIG. 4 shows an exemplary process by which an objects within the field 3o represented by the video surface that, due to its projected motion is to be displayed in the three dimensional world, e.g., in front of, to the side of, above, or below, the video surface, is so displayed using computer graphics techniques, so that the object "pops out"
of the video surface. In typical embodiments of the invention, the objects to be displayed using computer graphics are predefined, and their location in the video and the time associated with its position within the video is known, so that the time and extent at which the object needs to be popped out of the video is known. However, the techniques of the invention may be employed with computers and software which are sufficiently sophisticated to track recognizable objects within the video surface and to develop computer graphics models from such video representations in order to create the popped out computer graphics.
The process is entered in step 401 when the user selects a video to view and the first frame of the video is to be displayed to a user. In step 403, the video frame to be displayed is retrieved. This may be achieved by retrieving data representing the frame which is pointed to by a pointer. Such a frame pointer is set initially, e.g., in step 401, to point to the first frame of the video. Next, in step 405, it is determined if there are any objects with the current video frame which is to be displayed on the video screen for which popping out needs to be initiated. This may be achieved by employing additional information associated with the frame that describes any objects that need to be popped out. Such additional information may also include data necessary to render the object as well as motion data which indicates the path of the object. Alternatively, this step may be achieved by analyzing the content of the video frame, perhaps in conjunction with prior frames and future frames, to determine data necessary to render the object as well as motion data which indicates the path of the object. In addition, the viewpoint given to the user by the overall display may be incorporated in the determination. Such a viewpoint is determined by camera controls, which may be set by the user. Of course, a combination of the foregoing techniques may also be employed. In step 407, a computer graphics display engine routine is initiated for each of the objects determined in step 405. Each of these computer graphics routines are supplied with the information necessary to generate their respective objects as time passes with each frame for display on the video surface. -For example, the information supplied for each object may include the aforementioned motion data which indicates the path of the object.
Next, steps 409 and 411 are performed substantially in parallel with step 413.
The degree of parallelism to be achieved should enable the popped out objects to be displayed substantially while the frame is displayed so that the user perceives them as being displayed together at the same time.
3o In step 409, a location on the viewscreen is determined for each of objects for which a computer graphics display engine routine is executing. In step 411, each of the objects for which a computer graphics display engine routine is executing that are still within the view of the viewer are rendered. Optionally, the computer graphics display engine routine for any object that has moved so that it is no longer within the view of the viewer may be deleted. However, such computer graphics display engine routines should only be deleted provided that it is known that the motion path of the object or the viewpoint of the viewer will not again bring that object into view.
The video frame is displayed in step 413. Control then passes from both steps 411 and 413 to step 415. In step 41 S the next video frame is advanced to, provided that there is one. This may be achieved by incrementing or otherwise updating the value of the frame pointer. Thereafter, control passes back to step 403 and the process continues as described above.
In another implementation of the basic process of FIG. 4, the video screen is moving within the overall three-dimensional world being displayed to the user.
Instead of determining which objects require the initiating of popping out based on information about objects in the frame being displayed on the video surface, as described above, the initiation may be based on the current location of the video screen within the overall three-dimensional world being displayed to the user. For example, if the video screen is displaying a video which gives the user the feeling of traveling down the Champs Elysees each successive video frame is actually a representation of the view from a location further and further down the street. As the frames are displayed and the user's viewpoint moves further down the street, it may be necessary for objects, e.g., a bus, a pedestrian, or a parked car, to pop off the video screen and become represented by computer graphics.
The computer graphics display engine routine for each of the objects may be initiated as a 2o function of the distance traveled by the video screen down the street, which, for example, may be either indicated in additional information associated with the frame, or, for simple situations, determined as a predetermined function of the time that the video has been running.
In another embodiment of the invention, a three-dimensional world ready for rendering, including all the elements that must pop out of the video surface, is modeled in memory. The three-dimensional world includes information such as the spatial location, size and orientation for each of the elements that must pop out. Motion information for the pop out elements may also be included. In this embodiment, the video screen is moving within the overall three-dimensional world that is being displayed to the user, as 3o described above. Each frame of the video has spatial information which describes the location of the video screen within the three-dimensional world stored in memory. Such information may be implied, e.g., if the video screen is moving at a constant rate of speed, or it may be in separate information that is associated with the video frame. Using the information about each video pop out element as well as the information about the location of the video screen allows the spatial location of each video pop out element, as well as the portion of each video pop out element that is not blocked by the video screen and is within the viewpoint of the user, to be determined for each frame.
Therefore, when the video screen is at a particular location, the portion of any video pop out element that is visible, e.g., in front of or to the side of, above, or below, the video screen may be rendered on the display. Thus, a match may be achieved between what is seen on the video screen and the computer graphic pop out elements.
A process for implementing this embodiment is shown, in flow chart form, in FIG. S. Note that prior to entering the process shown in FIG. 5 a three-dimensional world including all the elements that must pop out of the video surface must be pregenerated and stored in memory. Also, spatial information which describes the location of the video screen within the three-dimensional world must be associated with frames of the video and stored in memory.
The process is entered in step 501 when the user selects a video to view and the first frame of the video is to be displayed to a user. In step 503, the video frame to be displayed is retrieved. This may be achieved by retrieving data representing the frame which is pointed to by a pointer. Such a frame pointer is set initially, e.g., in step 501, to point to the first frame of the video. In step 505, the location of the video screen within the three-dimensional world is determined. As noted, such information may be implied, e.g., if the video screen is moving at a constant rate of speed, or it may be in separate information that is associated with the video frame. Next, in step 507, the visible portion of any pop out elements is determined as a function of the current video surface location and the user viewpoint.
Thereafter, steps 507 and 509 are performed substantially in parallel. The degree of parallelism to be achieved should enable the popped out objects to be displayed substantially while the frame is displayed, so that the user perceives them as being displayed together at the same time. In step 507, the determined visible portion of each of the pop out elements is rendered. The video frame is displayed in step 509.
Control then passes from both steps 507 and 509 to step 511. In step 511 the next video frame is advanced to, provided that there is one. This may be achieved by incrementing or otherwise updating the value of the frame pointer. Thereafter, control passes back to step 503 and the process continues as described above.
The foregoing assumes that the video frame to be displayed is set, which determines the location of the video screen within the three-dimensional world.
However, the converse may also be implemented, i.e., the location of the video screen within the three-dimensional world is set, and this determines which video frame is to be displayed. To do so, in step 501 br 511, in which the video frame to be displayed next is determined, the desired location of the video screen within the three-dimensional world is ascertained. This location may be specified as a function of user input. For example, if the scene being displayed corresponds to the view of a user on a bicycle, the location of the video screen may be determined by sensors mounted to a stationary bicycle which detects riding of the bicycle by the user. A search of the information identifying the location of the frames is conducted to determine which frame corresponds to the current location of the screen, and the frame pointer is set to point to that frame.
The various methods of determining which frame is displayed may be used separately, or their usage may be intermixed, depending on the need of the application being implemented.
Note that frames for which it is known that no pop out elements will be displayed need not be associated with spatial information which describes the location of the video screen within the three-dimensional world stored in memory. The video is simply displayed on the video screen and no pop out elements are rendered.
Any prestored information, such as frames, and information associated therewith, as well as models of the three-dimensional world, may be stored in different segments. Such segments may be located at a single physical location, e.g., within a single computer, or at different physical locations, such as in various computers that are connected by a network, and made available for use with embodiments of the invention. Moreover, the elements which process the information to generate the display may, but need not be, contained within a single computer.
FIG. 6 shows a view processing system 40 for processing a view of a three dimensional world 30 to be displayed by a display screen 33 according to an embodiment of the present invention. The three dimensional world 30 represented on the display screen 33 comprises a first portion 31 modeled as computer graphics and a second portion represented by a video.
With reference to FIG. 6, a video object locating mechanism 42 determines that an object (e.g. see police van 113 in FIG. 1 - FIG. 3) which is represented as part of the video on the second portion 32 (video) of the display screen 33 has moved, from the point of view of a viewer of the world 30, to a location that is off of the second portion 32 (video) of the display screen 33, so that the object cannot be displayed and will disappear from the world 30. The video object locating mechanism 42 relays information about the determined objects to a video-to-graphics conversion mechanism 41. The video-to-graphics conversion mechanism 41 changes the representation of the object from a video representation to one modeled as computer graphics whereby the object remains visible in the world 30, albeit not on the second portion 32 (video) of the display screen 33, however the second portion 32 continues to exist in the world 30.
FIG. 7 shows a three dimensional world display system 50 for displaying an apparently three dimensional world 30 to a user on a display screen 33. The three dimensional world 30 represented on the display screen 33 comprises a first part 61 displayed as computer graphics and a second part 62 displayed as video. The video is made up of a sequence of images displayed on the second part 62 (video) of the display screen 33 which exists in the second part 62 of the world 30. The video displays a plurality of objects.
The arrangement of first and second parts 6l, 62 is for pedagogic illustrative purposes only and is not representative. The first and second parts 61, 62 can also be arranged vertically or within each other (i.e. second part 62 within first part 61 ), for example.
With reference to FIG. 7, a processor 52 determines that an element (e.g. see police van 113 in FIG. 1- FIG. 3) displayed as video on the second part 62 (video) of the display screen 33, transited, from the point of view of the viewer of the world 30, from the second part 62 to the first part 61 so that the element can no longer be displayed on the second part (video) of the display screen 33 and will disappear from the world 30. A
rendering engine S 1 displays the world 30 with the element displayed in the first part 61 of the world 30. In this way, the element remains visible in the world 30, albeit not on the second part 62 (video) of the display screen 33, and the video screen 32 continues to exist in the world 30.
The foregoing merely illustrates the principles of the inventions. It will thus be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope.
-7a-
Technical Field This invention relates to the integration of computer graphics and video to provide a realistic three dimensional virtual reality experience.
Background of the Invention The display of a three dimensional world to a viewer requires considerable computation power, and it is typically costly to develop the necessary highly detailed models required for doing so. In order to simplify the problem, a portion of the world that is in the distance may be represented in only two dimensions as a video displayed on a surface, e.g., a screen. By video it is meant the common usage of the term, such as the placing or projecting of predefined images on the surface, e.g., the electronic version of filmed moving pictures. Thus, such a world is essentially truncated in length to the screen on which the video is displayed.
A great reduction in computation power and cost can be achieved by such an arrangement.
Summary of the Invention We have recognized that a limitation of such a world occurs when an object within the field represented by the video undergoes a trajectory that takes it to a location in the world that is not represented by the video but instead is a location in the foreground which is represented by computer graphics, namely, any portion of the object that is :no longer on the video screen, disappears. Therefore, in accordance with the principles of the invention, when an object within the field represented by the video undergoes a trajectory that takes it, or a portion thereof, to a location in the world that is not represented by the video but insi:ead is a location in the foreground which is represented by computer graphics, such an object or portion is made to continue to be visible to the user and is represented at the foreground location using computer graphic techniques, rather than video. Thus, the video obj ect "pops"
out of the video and becomes visible, e.g., in front of, or to the side of, the video screen, rather than becoming invisible because it is no longer on the video screen.
In accordance with one aspect of the present invention there is provided an apparatus for use in processing a view of a three dimensional world in which a first portion of said world is modeled as computer graphics and a second portion of said world is represented by a video, 3 0 wherein said video is made up of a sequence of images displayed on a video screen which exists in said first portion of said world, said video displaying a plurality of objects, said apparatus comprising: means for determining that an object which is represented as part of said video on said video screen has moved, from the point of view of a viewer of said world, to a location that is off of said video screen, so that said object can no longer be displayed on said video screen and will disappear from said world; and means for changing the representation of said object from a video representation to one modeled as computer graphics whereby said object remains visible in said world, albeit not on said video screen, and said video screen continues to exist in said world.
In accordance with another aspect of the present invention there is provided an apparatus for use in processing a view on a three dimensional world in which a first portion of said world is modeled as computer graphics and a second portion of said world is represented by a video, comprising: means for determining that an object being represented by said video has moved to a location that is being modeled as computer graphics; and means for revealing a representation of said object as computer graphics.
In accordance with yet another aspect of the present invention there is provided a system for displaying an apparently three dimensional world to a user, in which a first part of said three dimensional world is displayed as computer graphics and a second part is displayed as video on a video screen, wherein said video is made up of a sequence of images displayed on a video screen which exists in said first portion of said world, said video displaying a plurality of objects, the method, said system comprising: a processor for determining that an element displayed as video on said video screen transited, from the point of view of a viewer of said world, from said second part to said first part so that said object can no longer be displayed on said video screen and will disappear from said world; and a rendering engine for displaying said world with said element displayed in said first part rather than said second part whereby said object remains visible in said world, albeit not on said video screen, and said video screen continues to exist in said world.
In accordance with still yet another aspect of the present invention there is provided a method for use in processing a view of a three dimensional world in which a first portion of said world is modeled as computer graphics and a second portion of said world is represented by a video, wherein said video is made up of a sequence of images displayed on a video screen which exists in said first portion of said world, said video displaying a plurality -la-of objects, the method comprising the steps of: determining that an object which is represented as part of said video on said video screen has moved, from the point of view of a viewer of said world, to a location that is off of said video screen, so that said object can no longer be displayed on said video screen and will disappear from said world;
and changing the representation of said object from a video representation to one modeled as computer graphics whereby said object remains visible in said world, albeit not on said video screen, and said video screen continues to exist in said world.
Brief Description of the Drawings In the drawings:
FIGs. 1-3 show an example of a video object "popping" out of a video and so becoming visible in front of, or to the side of, the video screen, in accordance with the principles of the invention;
FIG. 4 shows an exemplary process by which objects within the field represented by the video surface that, due to their projected motion are to be in the three dimensional world, are so displayed using computer graphics techniques, e.g., the object "pops out" of the video surface, in accordance with the principles of the invention;
FIG. 5 shows a flow chart of a process of another embodiment of the invention;
FIG. 6 is a representation of a three dimensional world and an associated view processing system according to an embodiment oI'the present invention; and FIG. 7 is a representation of a three dimensional world and an associated three dimensional display system according to an embodiment of the present invention.
Detailed Description To better illustrate the invention, FIGS. 1-3 show an example of that which a user sees according to the invention when an object, that is displayed as video on a video screen of a world that has a portion of the world distant from the point of view of the user represented in only two dimensions as a video on the video screen, undergoes a trajectory that takes at least a portion of it to a location in the world that is not represented by the video but instead is a location in the world that is represented by computer graphics, and such an object or portion is made to yet be visible to the user by representing the object at its location off of the video screen using computer graphic techniques, rather than video. In other words, FIGS. 1-3 show an example of a video object "popping" out of the video and becoming visible in front of, to the side of, above, or below the video screen, rather than simply becoming invisible because it is no longer on the video screen. For simplification of terminology purposes, a portion of an object may simply be referred to as an object, since any portion of an object may be considered an object in its own right.
FIG. 1 shows world 101, which is the Champs Elysees in Paris, France, as one approaches La Place de 1'Etoile in the center of which is the Arc de Triomphe, built by Napoleon Bonaparte. World 101 is divided into two portions, video screen 103, on which is shown the current frame of a video and the remainder of the world 1 O5, which is represented using computer graphics techniques, and is thus referred to herein as computer graphics part (CG Part) 105. The current frame of video being displayed on video screen 103 includes police van 113 and Arc de Triomphe 115. Within CG Part i05 there are various elements, such as bicyclist 107, representing the user, road 109, and sky 111.
Note that the viewpoint of the user is actually behind the representation of the user in the form of bicyclist 107. Also note that police van 113, which is part of the current -2a-video frame being shown on video screen 103, is moving slower than bicyclist 107, so that police van 113 will eventually be passed by bicyclist 107 as he continues to ride toward Arc de Triumph 115.
FIG. 2 shows world 101 of FIG. 1, but at a later time. At the time of FIG. 2, the frame of video being shown on screen 103 is from a view closer to Arc de Triumph 115.
Such a frame may have resulted, for example, from moving the camera that captured the video closer to Arc de Triumph 115. As a result of the camera location when the frame of video on screen 103 that is shown in FIG. 2 was taken, only a portion of police van 113 was captured video frame. The rest of the police van 113 was out of view of the 1 o camera, and hence is not visible within the current frame of video on screen 103 that is shown in FIG. 2. However, from the viewpoint of the user, looking at world 101 as it appears in FIG. 2, the remaining portion of police van 113 should be visible, notwithstanding that it is no longer within the boundaries of video screen 103. Therefore, in accordance with the principles of the invention, the remaining portion of police van 113 is displayed using computer graphic techniques within world 101 as element 201, which is a part of CG Part 105.
FIG. 3 shows world 101 of FIG. 1, but at a time even later than that of FIG.
2.
Thus, at the time of FIG. 3, the frame of video being shown on screen 103 is from a view still closer to Arc de Triumph 115 than that of FIG. . As a result of the camera location 2o when the frame of video on screen 103 that is shown in FIG. 3 was taken, none of police van 113 is visible within the current frame of video on screen 103 that is shown in FIG. 3.
However, from the viewpoint of the user, looking at world 101 as it appears in FIG. 3, police van 113 should be visible to the user's left, notwithstanding that it is no longer within the boundaries of video screen 103. Therefore, in accordance with the principles of the invention, the entirety of police van 113 is displayed using computer graphic techniques within world 101 as element 201, which is a part of CG Part 105. Ai least a portion of element 201 will continue to be displayed as part of CG Part 105 until element 201 passes completely from the user's viewpoint.
FIG. 4 shows an exemplary process by which an objects within the field 3o represented by the video surface that, due to its projected motion is to be displayed in the three dimensional world, e.g., in front of, to the side of, above, or below, the video surface, is so displayed using computer graphics techniques, so that the object "pops out"
of the video surface. In typical embodiments of the invention, the objects to be displayed using computer graphics are predefined, and their location in the video and the time associated with its position within the video is known, so that the time and extent at which the object needs to be popped out of the video is known. However, the techniques of the invention may be employed with computers and software which are sufficiently sophisticated to track recognizable objects within the video surface and to develop computer graphics models from such video representations in order to create the popped out computer graphics.
The process is entered in step 401 when the user selects a video to view and the first frame of the video is to be displayed to a user. In step 403, the video frame to be displayed is retrieved. This may be achieved by retrieving data representing the frame which is pointed to by a pointer. Such a frame pointer is set initially, e.g., in step 401, to point to the first frame of the video. Next, in step 405, it is determined if there are any objects with the current video frame which is to be displayed on the video screen for which popping out needs to be initiated. This may be achieved by employing additional information associated with the frame that describes any objects that need to be popped out. Such additional information may also include data necessary to render the object as well as motion data which indicates the path of the object. Alternatively, this step may be achieved by analyzing the content of the video frame, perhaps in conjunction with prior frames and future frames, to determine data necessary to render the object as well as motion data which indicates the path of the object. In addition, the viewpoint given to the user by the overall display may be incorporated in the determination. Such a viewpoint is determined by camera controls, which may be set by the user. Of course, a combination of the foregoing techniques may also be employed. In step 407, a computer graphics display engine routine is initiated for each of the objects determined in step 405. Each of these computer graphics routines are supplied with the information necessary to generate their respective objects as time passes with each frame for display on the video surface. -For example, the information supplied for each object may include the aforementioned motion data which indicates the path of the object.
Next, steps 409 and 411 are performed substantially in parallel with step 413.
The degree of parallelism to be achieved should enable the popped out objects to be displayed substantially while the frame is displayed so that the user perceives them as being displayed together at the same time.
3o In step 409, a location on the viewscreen is determined for each of objects for which a computer graphics display engine routine is executing. In step 411, each of the objects for which a computer graphics display engine routine is executing that are still within the view of the viewer are rendered. Optionally, the computer graphics display engine routine for any object that has moved so that it is no longer within the view of the viewer may be deleted. However, such computer graphics display engine routines should only be deleted provided that it is known that the motion path of the object or the viewpoint of the viewer will not again bring that object into view.
The video frame is displayed in step 413. Control then passes from both steps 411 and 413 to step 415. In step 41 S the next video frame is advanced to, provided that there is one. This may be achieved by incrementing or otherwise updating the value of the frame pointer. Thereafter, control passes back to step 403 and the process continues as described above.
In another implementation of the basic process of FIG. 4, the video screen is moving within the overall three-dimensional world being displayed to the user.
Instead of determining which objects require the initiating of popping out based on information about objects in the frame being displayed on the video surface, as described above, the initiation may be based on the current location of the video screen within the overall three-dimensional world being displayed to the user. For example, if the video screen is displaying a video which gives the user the feeling of traveling down the Champs Elysees each successive video frame is actually a representation of the view from a location further and further down the street. As the frames are displayed and the user's viewpoint moves further down the street, it may be necessary for objects, e.g., a bus, a pedestrian, or a parked car, to pop off the video screen and become represented by computer graphics.
The computer graphics display engine routine for each of the objects may be initiated as a 2o function of the distance traveled by the video screen down the street, which, for example, may be either indicated in additional information associated with the frame, or, for simple situations, determined as a predetermined function of the time that the video has been running.
In another embodiment of the invention, a three-dimensional world ready for rendering, including all the elements that must pop out of the video surface, is modeled in memory. The three-dimensional world includes information such as the spatial location, size and orientation for each of the elements that must pop out. Motion information for the pop out elements may also be included. In this embodiment, the video screen is moving within the overall three-dimensional world that is being displayed to the user, as 3o described above. Each frame of the video has spatial information which describes the location of the video screen within the three-dimensional world stored in memory. Such information may be implied, e.g., if the video screen is moving at a constant rate of speed, or it may be in separate information that is associated with the video frame. Using the information about each video pop out element as well as the information about the location of the video screen allows the spatial location of each video pop out element, as well as the portion of each video pop out element that is not blocked by the video screen and is within the viewpoint of the user, to be determined for each frame.
Therefore, when the video screen is at a particular location, the portion of any video pop out element that is visible, e.g., in front of or to the side of, above, or below, the video screen may be rendered on the display. Thus, a match may be achieved between what is seen on the video screen and the computer graphic pop out elements.
A process for implementing this embodiment is shown, in flow chart form, in FIG. S. Note that prior to entering the process shown in FIG. 5 a three-dimensional world including all the elements that must pop out of the video surface must be pregenerated and stored in memory. Also, spatial information which describes the location of the video screen within the three-dimensional world must be associated with frames of the video and stored in memory.
The process is entered in step 501 when the user selects a video to view and the first frame of the video is to be displayed to a user. In step 503, the video frame to be displayed is retrieved. This may be achieved by retrieving data representing the frame which is pointed to by a pointer. Such a frame pointer is set initially, e.g., in step 501, to point to the first frame of the video. In step 505, the location of the video screen within the three-dimensional world is determined. As noted, such information may be implied, e.g., if the video screen is moving at a constant rate of speed, or it may be in separate information that is associated with the video frame. Next, in step 507, the visible portion of any pop out elements is determined as a function of the current video surface location and the user viewpoint.
Thereafter, steps 507 and 509 are performed substantially in parallel. The degree of parallelism to be achieved should enable the popped out objects to be displayed substantially while the frame is displayed, so that the user perceives them as being displayed together at the same time. In step 507, the determined visible portion of each of the pop out elements is rendered. The video frame is displayed in step 509.
Control then passes from both steps 507 and 509 to step 511. In step 511 the next video frame is advanced to, provided that there is one. This may be achieved by incrementing or otherwise updating the value of the frame pointer. Thereafter, control passes back to step 503 and the process continues as described above.
The foregoing assumes that the video frame to be displayed is set, which determines the location of the video screen within the three-dimensional world.
However, the converse may also be implemented, i.e., the location of the video screen within the three-dimensional world is set, and this determines which video frame is to be displayed. To do so, in step 501 br 511, in which the video frame to be displayed next is determined, the desired location of the video screen within the three-dimensional world is ascertained. This location may be specified as a function of user input. For example, if the scene being displayed corresponds to the view of a user on a bicycle, the location of the video screen may be determined by sensors mounted to a stationary bicycle which detects riding of the bicycle by the user. A search of the information identifying the location of the frames is conducted to determine which frame corresponds to the current location of the screen, and the frame pointer is set to point to that frame.
The various methods of determining which frame is displayed may be used separately, or their usage may be intermixed, depending on the need of the application being implemented.
Note that frames for which it is known that no pop out elements will be displayed need not be associated with spatial information which describes the location of the video screen within the three-dimensional world stored in memory. The video is simply displayed on the video screen and no pop out elements are rendered.
Any prestored information, such as frames, and information associated therewith, as well as models of the three-dimensional world, may be stored in different segments. Such segments may be located at a single physical location, e.g., within a single computer, or at different physical locations, such as in various computers that are connected by a network, and made available for use with embodiments of the invention. Moreover, the elements which process the information to generate the display may, but need not be, contained within a single computer.
FIG. 6 shows a view processing system 40 for processing a view of a three dimensional world 30 to be displayed by a display screen 33 according to an embodiment of the present invention. The three dimensional world 30 represented on the display screen 33 comprises a first portion 31 modeled as computer graphics and a second portion represented by a video.
With reference to FIG. 6, a video object locating mechanism 42 determines that an object (e.g. see police van 113 in FIG. 1 - FIG. 3) which is represented as part of the video on the second portion 32 (video) of the display screen 33 has moved, from the point of view of a viewer of the world 30, to a location that is off of the second portion 32 (video) of the display screen 33, so that the object cannot be displayed and will disappear from the world 30. The video object locating mechanism 42 relays information about the determined objects to a video-to-graphics conversion mechanism 41. The video-to-graphics conversion mechanism 41 changes the representation of the object from a video representation to one modeled as computer graphics whereby the object remains visible in the world 30, albeit not on the second portion 32 (video) of the display screen 33, however the second portion 32 continues to exist in the world 30.
FIG. 7 shows a three dimensional world display system 50 for displaying an apparently three dimensional world 30 to a user on a display screen 33. The three dimensional world 30 represented on the display screen 33 comprises a first part 61 displayed as computer graphics and a second part 62 displayed as video. The video is made up of a sequence of images displayed on the second part 62 (video) of the display screen 33 which exists in the second part 62 of the world 30. The video displays a plurality of objects.
The arrangement of first and second parts 6l, 62 is for pedagogic illustrative purposes only and is not representative. The first and second parts 61, 62 can also be arranged vertically or within each other (i.e. second part 62 within first part 61 ), for example.
With reference to FIG. 7, a processor 52 determines that an element (e.g. see police van 113 in FIG. 1- FIG. 3) displayed as video on the second part 62 (video) of the display screen 33, transited, from the point of view of the viewer of the world 30, from the second part 62 to the first part 61 so that the element can no longer be displayed on the second part (video) of the display screen 33 and will disappear from the world 30. A
rendering engine S 1 displays the world 30 with the element displayed in the first part 61 of the world 30. In this way, the element remains visible in the world 30, albeit not on the second part 62 (video) of the display screen 33, and the video screen 32 continues to exist in the world 30.
The foregoing merely illustrates the principles of the inventions. It will thus be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope.
-7a-
Claims (21)
1. A method for use in processing a view of a three dimensional world in which a first portion of said world is modeled as computer graphics and a second portion of said world is represented by a video, wherein said video is made up of a sequence of images displayed on a video screen which exists in said first portion of said world, said video displaying a plurality of objects, the method comprising the steps of:
determining that an object which is represented as part of said video on said video screen has moved, from the point of view of a viewer of said world, to a location that is off of said video screen, so that said object can no longer be displayed on said video screen and will disappear from said world; and changing the representation of said object from a video representation to one modeled as computer graphics whereby said object remains visible in said world, albeit not on said video screen, and said video screen continues to exist in said world.
determining that an object which is represented as part of said video on said video screen has moved, from the point of view of a viewer of said world, to a location that is off of said video screen, so that said object can no longer be displayed on said video screen and will disappear from said world; and changing the representation of said object from a video representation to one modeled as computer graphics whereby said object remains visible in said world, albeit not on said video screen, and said video screen continues to exist in said world.
2. The method as defined in claim 1 wherein said object has moved to a location in said three dimensional world that is at a location which is one or more of in front of, to the side of, above, or below, a screen on which said video is being rendered.
3. The method as defined in claim 1 wherein said determining step includes the step of analyzing said video for changes in identifiable objects therein.
4. The method as defined in claim 1 wherein said determining step includes the step of obtaining additional information about said object which is necessary to render said object as computer graphics.
5. The method as defined in claim 1 wherein said determining step includes the step of obtaining additional information about said object which is necessary to render said object as computer graphics and wherein said additional information is associated with a frame of said video.
6. The method as defined in claim 1 wherein said determining step includes the step of obtaining additional information identifying said object.
7. The method as defined in claim 1 wherein said determining step includes the step of obtaining additional information and wherein said additional information includes information about said object in at least one category from the set of categories including spatial location, size, orientation, and motion information.
8. The method as defined in claim 1 wherein said video is displayed on a video screen that is moving within said first portion of said world which is modeled as computer graphics.
9. A method for use in processing a view of a three dimensional world in which a first portion of said world is modeled as computer graphics and a second portion of said world is represented by a video, wherein said video is made up of a sequence of images displayed on a video screen which exists in said first portion of said world, said video displaying a plurality of objects, the method comprising the steps of:
determining that an object which is represented as part of said video on said video screen has moved, from the point of view of a viewer of said world to a location that is off of said video screen, so that said object can no longer be displayed on said video screen and will disappear from said world; and revealing a representation of said object as computer graphics whereby said object remains visible in said world, albeit not on said video screen and said video screen continues to exist in said world.
determining that an object which is represented as part of said video on said video screen has moved, from the point of view of a viewer of said world to a location that is off of said video screen, so that said object can no longer be displayed on said video screen and will disappear from said world; and revealing a representation of said object as computer graphics whereby said object remains visible in said world, albeit not on said video screen and said video screen continues to exist in said world.
10. The method as defined in claim 9 further including the step of prestoring said representation of said object as computer graphics.
11. The method as defined in claim 9 wherein said representation of said object as computer graphics is prestored, said representation including information for at least one category from the set of categories including spatial location, size, orientation, and motion information.
12. The method as defined in claim 9 wherein said video screen is moveable in said world and wherein said step of revealing is initiated as a function of a position of said video screen in said world.
13. The method as defined in claim 9 wherein said video screen is moveable in said world and wherein said step of revealing, is initiated by said video screen moving past said representation of said object as computer graphics.
14. The method as defined in claim 9 wherein said step of revealing is performed by a computer graphics display engine, and wherein said step of revealing is initiated as a function of a position of said video screen in said world.
15. Apparatus for use in processing a view of a three dimensional world to a user, in which a first portion of said world is modeled as computer graphics and a second portion is represented by a video, wherein said video is made up of a sequence of images displayed on a video screen which exists in said first portion of said world, said video displaying a plurality of objects, said apparatus comprising:
means for determining that an object which is represented as part of said video on said video screen has moved, from the point of view of a viewer of said world, to a location off of said video screen, so that said object can no longer be displayed on said video screen and will disappear from said world; and means for changing the representation of said object from a video representation to one modeled as computer graphics whereby said object remains visible in said world, albeit not on said video screen, and said video screen continues to exist in said world.
means for determining that an object which is represented as part of said video on said video screen has moved, from the point of view of a viewer of said world, to a location off of said video screen, so that said object can no longer be displayed on said video screen and will disappear from said world; and means for changing the representation of said object from a video representation to one modeled as computer graphics whereby said object remains visible in said world, albeit not on said video screen, and said video screen continues to exist in said world.
16. The apparatus as defined in claim 15 wherein said object has moved to a location in said three dimensional world that is at a location which is one or more of in front of, to the side of, above, or below, a screen on which said video is being rendered.
17. Apparatus for use in processing a view on a three dimensional world in which a first portion of said world is modeled as computer graphics and a second portion of said world is represented by a video, comprising:
means for determining that an object being represented by said video has moved to a location that is being modeled as computer graphics; and means for revealing a representation of said object as computer graphics.
means for determining that an object being represented by said video has moved to a location that is being modeled as computer graphics; and means for revealing a representation of said object as computer graphics.
18. The apparatus as defined in claim 17 further said object has moved to a location in said three dimensional world that is at a location which is one or more of in front of, to the side of, above, or below, a screen on which said video is being rendered.
19. A system for displaying an apparently three dimensional world to a user, in which a first part of said three dimensional world is displayed as computer graphics and a second part is displayed as video on a video screen, wherein said video is made up of a sequence of images displayed on the video screen which exists in said second part of said world, said video displaying a plurality of objects, said system comprising:
a processor for determining that an element displayed as video on said video screen transited, from the point of view of a viewer of said world, from said second part to said first part so that said element can no longer be displayed on said video screen and will disappear from said world; and a rendering engine for displaying said world with said element displayed in said first part rather than said second part whereby said element remains visible in said world, albeit not on said video screen, and said video screen continues to exist in said world.
a processor for determining that an element displayed as video on said video screen transited, from the point of view of a viewer of said world, from said second part to said first part so that said element can no longer be displayed on said video screen and will disappear from said world; and a rendering engine for displaying said world with said element displayed in said first part rather than said second part whereby said element remains visible in said world, albeit not on said video screen, and said video screen continues to exist in said world.
20. The system as defined in claim 19 wherein rendering engine includes said processor.
21. The system as defined in claim 20 wherein said video screen is within said first part.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US08/906,214 US6320589B1 (en) | 1997-08-04 | 1997-08-04 | Display techniques for three dimensional virtual reality |
| US08/906,214 | 1997-08-04 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CA2242150A1 CA2242150A1 (en) | 1999-02-04 |
| CA2242150C true CA2242150C (en) | 2005-09-06 |
Family
ID=25422099
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002242150A Expired - Fee Related CA2242150C (en) | 1997-08-04 | 1998-06-30 | Display techniques for three dimensional virtual reality |
Country Status (4)
| Country | Link |
|---|---|
| US (1) | US6320589B1 (en) |
| AU (1) | AU742529B2 (en) |
| CA (1) | CA2242150C (en) |
| GB (1) | GB2328102B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101416235B1 (en) * | 2008-02-12 | 2014-07-07 | 삼성전자주식회사 | Method and apparatus for 3D location input |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3024145B2 (en) * | 1989-07-12 | 2000-03-21 | 株式会社日立製作所 | Texture mapping method |
| US5261041A (en) * | 1990-12-28 | 1993-11-09 | Apple Computer, Inc. | Computer controlled animation system based on definitional animated objects and methods of manipulating same |
| DE19636028C1 (en) * | 1996-09-05 | 1997-11-20 | Daimler Benz Ag | Stereo image object detection, esp. for road vehicles |
| US6226009B1 (en) * | 1998-06-29 | 2001-05-01 | Lucent Technologies Inc. | Display techniques for three dimensional virtual reality |
-
1997
- 1997-08-04 US US08/906,214 patent/US6320589B1/en not_active Expired - Lifetime
-
1998
- 1998-06-30 CA CA002242150A patent/CA2242150C/en not_active Expired - Fee Related
- 1998-07-28 GB GB9816368A patent/GB2328102B/en not_active Expired - Fee Related
- 1998-08-03 AU AU78696/98A patent/AU742529B2/en not_active Ceased
Also Published As
| Publication number | Publication date |
|---|---|
| CA2242150A1 (en) | 1999-02-04 |
| US6320589B1 (en) | 2001-11-20 |
| GB2328102B (en) | 1999-10-13 |
| AU742529B2 (en) | 2002-01-03 |
| GB9816368D0 (en) | 1998-09-23 |
| AU7869698A (en) | 1999-02-11 |
| GB2328102A (en) | 1999-02-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6256043B1 (en) | Three dimensional virtual reality enhancement techniques | |
| EP0930584B1 (en) | Method and apparatus for displaying panoramas with video data | |
| US6690393B2 (en) | 3D environment labelling | |
| US7667700B1 (en) | System and method for navigating operating in a virtual environment | |
| KR101773018B1 (en) | Augmented reality based on imaged object characteristics | |
| EP0899691B1 (en) | Three dimensional virtual reality enhancement techniques | |
| US6226009B1 (en) | Display techniques for three dimensional virtual reality | |
| JP2022089817A5 (en) | Program, object drawing method and information processing system | |
| CA2242150C (en) | Display techniques for three dimensional virtual reality | |
| US20210247200A1 (en) | Virtual-space-image providing device and program for providing virtual space image | |
| Hirose | CABIN-A multiscreen display for computer experiments | |
| CA2242166C (en) | Display techniques for three dimensional virtual reality | |
| EP1111546A2 (en) | Display techniques for three-dimensional virtual reality | |
| US6256044B1 (en) | Display techniques for three-dimensional virtual reality | |
| EP0969416A2 (en) | Display techniques for object visibility in a three-dimensional virtual world | |
| CN114935977B (en) | Spatial anchor point processing method and device, electronic equipment and storage medium | |
| WO2021111752A1 (en) | Information processing apparatus, information processing method, and program | |
| US11483486B1 (en) | Virtual experience imaging products | |
| CN115861950A (en) | Vehicle-mounted display method, vehicle-mounted display system, intelligent vehicle window and vehicle | |
| CN117622182A (en) | Display system and display method | |
| Olofinboba | Unique high-fidelity wraparound driving simulator for human factors research applications | |
| JPH10123929A (en) | Experiencing device | |
| CN116415991A (en) | Time-limited shopping method and device | |
| Burns | Simulating the visual environment in real-time via software | |
| You et al. | Creation of Augmented Virtual Environments by Dynamic Fusion of Imagery and 3D Models |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| MKLA | Lapsed | ||
| MKLA | Lapsed |
Effective date: 20090630 |