|Publication number||US5999187 A|
|Application number||US 08/883,781|
|Publication date||Dec 7, 1999|
|Filing date||Jun 27, 1997|
|Priority date||Jun 28, 1996|
|Also published as||WO1998000811A1|
|Publication number||08883781, 883781, US 5999187 A, US 5999187A, US-A-5999187, US5999187 A, US5999187A|
|Inventors||James Dehmlow, John Gass, Lynne M. Evans, Craig Daw|
|Original Assignee||Resolution Technologies, Inc.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (24), Non-Patent Citations (10), Referenced by (205), Classifications (14), Legal Events (4)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This application claims priority based on U.S. provisional application Ser. No. 06/020,524 filed Jun. 28, 1996 which is incorporated herein by reference.
The present invention is directed to a computer-aided design system, including a system implemented by computer software, and in particular to a fly-through, three-dimensional design system with multiple levels of details.
A computer-aided design system typically includes software for displaying on a two-dimensional computer display screen or other display device, an image of one or more predefined objects. In a typical application, the objects are parts or components of a product such as a machine or other device such as a vehicle, aircraft, ship, and the like. In a typical situation, a designer will provide input which defines the size and shape of the various components. The computer-aided design system provides a display indicating how the various components will mesh or fit together in three dimensions (or, in some cases, will indicate that the components do not properly fit together, thus alerting the designer that there is an error which needs to be corrected). In many computer-aided design systems, a user may select the point of view of the object or collection of objects, for example, the user may wish to see a front view, a side view, a top view, a perspective view, etc. In some systems, the definition of point of view is essentially static, i.e., the user inputs (e.g., via a keyboard or otherwise) a single point of view, and the software displays an image in accordance with the defined point of view.
In some systems, however, it is desired to simulate a real-time change in the point of view. For purposes of discussion, it is useful to consider the point of view in terms of a "virtual camera," i.e., what the collection of objects would look like if the display simulates the output of a camera as it moved with respect to the simulated objects. In theory, the virtual camera can rotate in any of three rotational directions, and can translate parallel any of three mutually perpendicular translational axes or any combination thereof. In a system in which all six types of motions are possible, the camera is said to possess six degrees of freedom. In such a system the virtual camera can translate or rotate in any direction with respect to the simulated collection of objects.
A particularly desirable design system provides a "fly-through" ability. In a fly-through system, a user may indicate a desired direction of movement and/or rotation of the virtual camera (e.g., using a mouse, joystick, trackball, head tracker, eye tracker, voice actuated command, and the like). In response, the computer display screen, under control of the design system software, will depict a series of images indicating the various views that would be taken by the virtual camera as it moves in the indicated direction.
A fly-through system is particularly useful in the design of complex structures such as aircraft or automobiles which may have thousands or millions of components. A fly-through system provides the user with the ability to inspect any desired portion of the device by quickly and easily moving to the region-of-interest and/or moving the virtual camera around the region-of-interest to inspect it from different points of view. In order to achieve a fly-through system which is useful for location and inspection purposes, and which can be used for relatively long periods of time without unduly tiring the viewer, the series of images presented on the screen during a fly-through operation must have a frame frequency sufficiently high that an illusion of motion, similar to a motion picture, is provided. A frame rate which is too low can make it difficult to locate desired volumes and can cause eye strain or other undesirable visual effect.
Although, for reasons noted above, a high frame-frequency fly-through design system is desirable, it has been difficult to implement, particularly at a reasonable cost, since rendering of images, particularly images which depict a three-dimensional object or scene, is one of the most computationally-intensive and resource-intensive (especially memory-intensive) tasks that can be performed on a computer. In particular, it has been difficult to implement a useful fly-through three-dimensional design system on a workstation-level system, at reasonable expense.
Accordingly, it would be useful to provide a design system which provides for fly-through capability at a high-frame rate, capable of use for designs having millions of components but which can be implemented on a system at reasonable cost, preferably with little or no special hardware requirements beyond those found on a workstation-level computer.
According to the present invention, certain techniques are employed to simplify the image during a fly-through procedure so that a high frame rate can be retained during fly-through. A number of simplification techniques are described which avoid interfering with the navigational and/or inspection functions of a fly-through procedure.
A volume-based simplification technique is described. The simplification involves rendering faces of volumetric regions or cells (such as cubes) or collections of cells. A cell is rendered if a surface (not volume) of at least one part intersects the cell. Preferably data is stored indicating which cells are marked "on" (i.e., are intersected by at least one part). A plurality of cell granularities preferably organized as an octree, provide multiple levels of detail.
Preferably, parts are grouped based on their spatial proximity. The user can define a boundary so that the processing occurs in a spatially limited volume or "world." Because each defined "world" is finite, the computational procedures are bounded, which assists in providing the desired performance level such as a high frame rate (e.g., about 10 hertz or more).
The data defining the parts is pre-processed (i.e., processed prior to loading into the viewer for display) in a fashion which organizes the data to achieve very rapid image processing. The simplified image can be presented in multiple levels of details, and preferably the user may control the tradeoff between image quality (i.e., the level of detail), and the interactive nature (e.g., the frame rate of the system). Several techniques provide the effect of using a finer level of detail for those parts that are closer to the virtual camera, and coarser levels of detail for parts that are further away.
The computational load is also kept within bounds by a particularly fast view frustum culling procedure. Additionally, the system can, relatively rapidly, determine which faces of various image components of cells will ultimately need to be drawn, and which are occluded.
In one embodiment, images are rendered by slices in front-to-back order (i.e., away from the point of view or virtual camera). For each slice, a prediction of rendering time is used to determine if the target level of detail can be used without impinging on the overall frame rate. If not, a coarser level of detail is automatically is substituted. The prediction of rendering time can be adjusted based on the history of actual rendering times of various slices.
Although various options or techniques for achieving a high frame rate and/or simplifying an image can be used, the techniques disclosed herein have been found to be particularly effective for implementation at a reasonable cost and with reasonable hardware and/or memory requirements. Although a number of the techniques disclosed herein provide useful advantages in themselves, several techniques also interact with one another in an advantageous manner to produce results which exceed those that would be expected from the sum of the advantages of the individual techniques.
FIGS. 1A through 1C depict a coordinate system having increasing granularity;
FIG. 2A depicts an example of a simple object of a type that may be displayed by the techniques of the present invention;
FIG. 2B depicts the object of FIG. 2A in a coordinate system with a first granularity;
FIG. 2C depicts a simplified rendering of the object of FIG. 2A in a coordinate system having the granularity of FIG. 2B;
FIG. 2D depicts the object of FIG. 2A in a coordinate system having a second granularity;
FIG. 2E depicts a simplified image of the object of FIG. 2A provided in a system having the granularity of FIG. 2D, according to an embodiment of the present invention;
FIG. 3 is a flowchart depicting a process according to an embodiment of the present invention;
FIG. 4 is a flowchart depicting a process according to an embodiment of the present invention;
FIG. 5 is a flowchart depicting a process according to an embodiment of the present invention;
FIGS. 6A and 6B depict file structures for storing or transmitting data according to embodiments of the present invention;
FIG. 7A is a schematic perspective view depicting a view frustum according to an embodiment of the present invention;
FIG. 7B is a top plan view of the region of FIG. 7A;
FIG. 8 is a flowchart depicting a process according to an embodiment of the present invention;
FIG. 9 is a flowchart depicting a process according to an embodiment of the present invention;
FIG. 10 is a flowchart depicting a process according to an embodiment of the present invention;
FIG. 11 is a flowchart depicting a process according to an embodiment of the present invention;
FIG. 12 is a flowchart depicting a process according to an embodiment of the present invention; and
FIG. 13 is a flowchart depicting a process according to an embodiment of the present invention.
Before describing the details of the present invention, it is believed useful to provide a discussion of certain basic concepts in connection with image rendering.
Among the various approaches that can be taken when considering a design system are world approaches versus parametric approaches. In a parametric system, locations in space are defined or described in terms of variables or parameters which are, theoretically, of unlimited range. For example, geometric objects can be defined in a Cartesian coordinate system using well-known analytic geometry techniques. This system of description is theoretically unbounded and values or parameters can take on any size or magnitude. In such a system it is impossible to know in advance size of a given computational problem since the system is unbounded.
In contrast, the present invention uses a world system in which a bounded coordinate system defines where the object or objects exist in space. In the world system, a coordinate system is used to define where the product sits in space and a boundary is established in this coordinate system relative to the product. The boundary can be defined in various fashions, e.g. to encompass the entire product or collection of objects, a portion of the product that is of interest, the product and its surroundings (e.g., associated tooling or use environment), and the like. In any case, however, the system is bounded. The bounded world system of the present invention is useful in a number of fashion, as will be apparent from the discussion below. The world system of the present invention provides the opportunity for pre-processing data for spatial organization and rendering performance. The user may define different worlds to suit his or her purpose, such as a product-only world and a product-environment world. A default viewing position (or "home" position) can be defined for each world, which is particularly useful in complex images having millions of parts in which a user can easily become "lost." Different color, surface or texture mapping can be defined for each world, e.g., to give meaning to, or indicate function of the parts displayed in the world.
Boundaries for a given world can be defined in a number of fashions. FIGS. 1A through 1C depict a relatively simple cubic-bounded coordinate system populated with a single cubic object. A given bounded volume can be subdivided in a number of fashions to provide different granularities. In FIG. 1A, the cubic object 112 fits exactly in a single cell of the coordinate system. FIG. 1B depicts the same object 112 in which the granularity has increased (i.e., in a system having a finer granularity). The conversion of the system of FIG. 1A to the system of FIG. 1B involves doubling the number of density of coordinate planes. Accordingly, as can be seen from FIG. 1B, the object 112 now occupies eight cells at the increased granularity. The granularity increase achieved in going from the system of FIG. 1A to the system of FIG. 1B can be applied in a recursive fashion, e.g., to subdivide each of the cells of FIG. 1B into eight equal cubes, a depicted in FIG. 1C. The octree data representation is used in several stages, including: data preprocessing 316, volume-based geometry simplification 346 view frustum culling 348 and cell occlusion (all discussed below in greater detail). The octree representation may be thought of as a hierarchy of nested cells or cubes. Each cell has 8 internal child cells whose sides are half the length of the corresponding side of the parent cells. There is no theoretical limit to the number of levels in the hierarchy but a practical limit is imposed by memory constraints.
The process can be repeated as many times a desired to achieve a "tree" of hierarchy of coordinate systems of ever-increasing granularity. In this example, because each recursion involves subdividing a unit cube into eight equal cubes, the series of increasing-granularity coordinate systems is referred to as an octree.
A series of increasing-granularity coordinate systems can be used to define different levels of detail, and an octree system is particularly useful in connection with the present invention, as will be clear from the description below. To illustrate the relation of granularity to multiple levels of detail, a simple object, depicted in FIG. 2A, will be provided as an example. The depicted object is a parallelepiped 212 surmounted by a pyramid 214. The three-dimensional shape and placement of the object can be better understood by noting the shape of the projections 216, 218, of the object 210, on the vertical bounding planes 220, 222. FIG. 2A also shows a horizontal bounding plane 219.
FIG. 2B depicts the object 210, positioned in a coordinate system having the granularity of FIG. 1A. As an example (and as will be described in greater detail below), a simplified rendering of the object 210 is depicted in FIG. 2C, in which the simplification is achieved by creating an object consisting of the faces of a plurality of the cells of the coordinate system, namely those cells which are intersected by a surface of the object 210.
FIG. 2D is similar to the depiction of FIG. 2B, but depicts the object 210 in a coordinate system having the granularity of FIG. 1B, i.e., the next-finer granularity in an octree system. When the same simplification technique used to create object 224 is applied in the finer-granularity system of FIG. 2D, the resultant simplified object, while still highly-simplified, bears a somewhat greater resemblance to the original object 210 than the coarser level of detail object 224. Thus, in each octree level or each increasing level of granularity, a finer level of detail of an object can be achieved.
FIG. 3 is a flow chart showing a general overview of a process according to one embodiment of the present invention. The process of FIG. 3 generally includes a first portion 312 containing steps that are performed prior to display of any images, and a second portion 314 which contains the steps used during image display. The first portion 312 includes pre-processing procedures 316 and data organization procedures 318. The goal of the pre-processing phase 316 is to convert the input data into a form that is relatively easy to load and manipulate at run time 314. Selecting or identifying the world 320 which the user desires to display establishes a finite volumetric boundary which will form the basis for subsequent spatial analysis. As will be described below, a given set of input data (a given collection of parts to be displayed) can be associated with two or more world views. Such as a first world view which shows the parts in a particular environment, a second world view which shows the parts in isolation, and the like. The world which is selected provides a central focus in the display environment, both from user's perspective and from the viewpoint of organizing the underlying data structures. The world is defined to establish boundaries that are useful to the user for a particular purpose. The world definition is used in managing the loading and unloading of part data, forms the basis for orienting the parts for viewing (e.g., rotating all parts with a common transformation) specifies the granularity of the level of detail (LOD) geometric simplifications, and specified color availability and precedence.
Color precedence data provides one way of selecting the color (or texture or "wallpaper") with which to color a given cell face, when a cell is occupied by two or more parts which are to be rendered in different colors. In one embodiment, a color precedence table is implemented as an array of color indices where the precedence is implicit in the array index (i.e., the color index associated with the first element in the array has the highest precedent, the second element has the second highest precedence, and so on). When two parts with different colors occupy a given cell, the cell faces will be colored with that color that has the highest precedence according to the color precedence table.
As noted above, the various parts which are to be depicted i the design system have shapes and locations that are mathematically (geometrically) defined. The data may be input in any a number of formats, including stereolithography (STL) format, an output format used by many computer-aided design (CAD) systems. The STL format defines the vertices of independent triangles that form (or approximately) the surface of the part. Normals to the triangles are defined on a facet basis (one normal for the three vertices that define the triangle face). Examples of formats that may be used for an input file include virtual reality markup language (VRML), and the unigraphics (UGU) solid pattern facet (FAC) file. The facet file format also defines independent triangles, with each triangle defined by three vertices and three normals.
Regardless of the format in which the part data is stored, in order to provide a common data form for pre-processing, a translator procedure extracts a list of triangles for each part from the input files 322. Although it is theoretically possible to use polygons other than triangles, triangles are preferred because of the simplicity and speed of processing. If the input data includes color information, the translator, in addition to extracting the list of polygons, can use the world's color table to convert the input color index into an RGB color specification. If no color input is provided, a default color index for the part can be generated. It may happen that some of the parts defined in the input files are not positioned anywhere within the boundaries defined by the world. Although cell files 326 will be defined for these parts, the list of intersecting cells will always be null and thus such parts will never be displayed for that particular world view.
Once the list of triangles for each part is obtained, certain groups of triangles may be combined or grouped into surface descriptions which can be optimized for rendering. This building of surface descriptions is referred to as "meshing" 324. According to one embodiment, meshing is performed by sorting the triangles and combining adjacent triangle edges. Preferably the meshing procedure is configured to accommodate double sided surfaces.
After meshing, if triangle normals are not already appropriately included in the input data (as they typically are, e.g. in for so-called "smooth" normals), triangle normals may be recalculated. If desired, the normals may be averaged for a given volume or mesh (i.e., in order to support smooth, shaded rendering).
To assist in efficiently displaying simplified views, (as described more fully below) for each part or component, it is determined, for at least one level of detail, which cells the surfaces of that part will intersect when the part is positioned in its defined location in the particular world view. For a pre-defined (preferably the finest) level of granularity of that world view 326. By defining intersections at the finest level of granularity, the intersection results can be propagated up through the octree hierarchy in an efficient manner, as described more fully below.
FIG. 4 depicts one manner of determining whether a given triangle intersects a cell. The triangle vertices are used to define a bounding box 412 of initial candidate cells at some level of granularity n where n is a coarser level of granularity than level m (the finest level of granularity available in the system), i.e. n<m. A recursive intersection test is performed until a list of candidate cells 414 at level m is obtained. Any candidate cells which are already marked "on" for that part 416 (i.e., cells that have already been determined to contain the part's surface as a result of analysis of another triangle of the part's surface) are ignored 418. Otherwise, the translation and scale factor that would be needed to transform each candidate cell (at whatever level) to the unit cube centered on the origin is determined 420. This transformation is then applied 422 to the triangle to test the transform triangle against a unit cube centered at the origin 424. Those cells that are determined to intersect are marked 426.
When the lists for each part are built, the data is output for data organization purposes. In one embodiment, cell intersections are represented in a 3-D run length encoded (3DRLE) format. The 3DRLE data is expressed as a set of minimum and maximum i, j, k, coordinates at a specified cell level in the octree data representation. The cell at the origin of the world (at each level) has i=0, j=0, j=0 coordinates. For example, 3DRLE data with the values (222333) at cell level 2 represents a cube that is composed of 8 level 2 cells. In this case, the same volume could equivalently be represented as (111111) at cell level 1, or (444777) at cell level 3. In general, it is always possible to represent a volume with smaller cells at another level. The converse representation only exists if cell boundary alignment conditions are satisfied.
FIG. 5 depicts a procedure for constructing 3DRLE data. The intersection results from the procedure of FIG. 4 are stored in a temporary structure such as a data array (so-called 3-D array) containing the marked state of each cell 512. The temporary data structure is traversed, e.g., in the left-to-right, back-to-front, top-to-bottom manner, until an "on" cell is found 514. Once an on cell is found, the data is examined to find the longest contiguous run of cells in each direction (i, j, k) that results in a volume with rectangular sides 516. The corners of this volume are stored as a proposed RLE value. The cells identified in the previous step 516 are then marked "off" 518 and traversal of the temporary data structure continues until all cells have been examined 519.
In one embodiment, the data is output in a single data file. In another embodiment, the date is output in two separate files: a geometry file which stores the surface representation of the part (i.e., the meshes and their associated vertices and normals) and a cell file which contains the result of the spatial analysis (i.e., the list of cells that are intersected by the surfacer (not the volume) of the part. The geometry file is independent of the selection of the world. However, the cell file depends on both the world and the part geometry. Thus, if either the world or the part changes, the cell file needs to be updated.
Various formats can be used for the output file. FIGS. 6A and 6B depict formats for geometry and cell output files respectively, according to one embodiment. The geometry file includes a part name 612 the number of triangles 614 bounding box 616, the number of meshes 618, a color index 620, vertex data 622, mesh data 624, normal data 626, and end markers 628. The vertices and normals of the meshes are unique but may be referenced multiple times by meshes, thus avoiding data duplications. Multiple parts may be provided in a given file by repeating portion 612 through 626.
The cell files contain a version number 632, a world name or signature 634, an indication of the cell level for the data 636, the name of the part 638, the RLE data 642, and end markers 634. It is noted that the vertices and normals for each part are stored in separate arrays. Each mesh 624 is an array of alternating normal and vertex indices that sequentially define the points on the surface of the mesh. Multiple parts per file can be supported by repeating portion 632-644. The world signature 634 preferably includes the minimum and maximum coordinates of the world bounding box, the maximum cell level for comparison with the cell file 3DRLE cell level at load time viewing rotations about the XYZ axis.
The depicted organization is particularly useful when using the geometry files for rendering the part in full detail 342. For full detail rendering, each mesh is processed, the normals and vertices for each point on the mesh are dereferenced (via the indices) and the XYZ components of both the normals and vertices 626, 622 are sent to the graphics system for rendering.
After the data output files are constructed, certain data organization is performed to assist in later processing. Preferably a cell-to-part mapping is performed 328, i.e., for each cell, a list of those parts that reside or intersect that cell is formed. In one embodiment this mapping is performed for only a single octree hierarchy level, (e.g., level 5 in a 7-level system). Although there is no theoretical reason why the mapping could not be performed for multiple levels, a single level typically suffices for the purpose of full detail rendering and avoids using an undue amount of memory. The cell-to-part mapping 328 is used in the full detail rendering 342. As noted below, view frustum culling 348 is used to define a set of cells that will be within the "cone" of visibility for a given virtual camera position and other cells are "culled out." This cell culling can be used whether simplified rendering 346 or full detail 342 is being performed. If full detail is being performed, it will be necessary to know, for those cells which are within the view area, which parts are within those cells, and this is the purpose of the cell-to-part mapping 328. If a part intersects more than one cell it will be referenced in the cell-to-part mapping for each cell. If view frustum culling identifies cells at a different level from that of the cell-to-part mapping, the octree data representation is traversed (as discussed below) to collect the list of parts from the cell-to-part mapping.
In the depicted embodiment, each cell in the octree has an associated color index 332. The value of this color index is determined by the color indices of the parts contained in the cell 620 and the world color precedence table, discussed above. In one embodiment, a byte is associated with each cell to contain both the color mapping 332 and an indication of whether that cell is "on" or "off" 334. Since a byte can contain 256 different values, the byte can be used to indicate up to 255 colors, with the last "color" being used to mark an off state (color is only meaningful for those cells that are marked on).
Once cell data indicating color and on/off status for each cell at the finest level of detail (finest granularity) has been stored, this data can be used for rendering simplified representations at any higher level by a relatively simple and rapid propagation method. Starting with the lowest level (finest granularity) octree, the next-coarser granularity octree can be populated by marking a parent cell "on" if any one of the child cells is marked on. If two different child cells have two different colors, the parent cell color will be determined by the color priority table.
Additional information may be stored for each cell for various purposes. A one-byte "octcode" data type may be used to define a particular child of a parent cell in the octree data representation (each bit in the byte corresponds to a specific one of the 8 child cells). As discussed more thoroughly below, bit shift operations can be used in conjunctions with the octcodes to traverse to child cells.
Another byte ("octlist") can be used to define a state for each child cell of a parent. For example, this byte can be used to encode which children cells of a given parent cell are marked on or off.
In a number of operations (described below) it is useful to traverse the octree hierarchy, i.e., to be able to start with a given cell at a given octree hierarchy level and rapidly determine those cells that are parent or child cells in the next-higher or next-lower octree hierarchy level. Preferably both traversal upward (traversal to an ancestor) and downward (traversal to descendants) can be achieved. In order to determine the destination cell index, bit shift operations can be used on each of the i, j, k, components of the current cell index. A cell index of an immediate ancestor (parent) is calculated by applying a right shift of one bit to the current cell index. In general, for an ancestor N level above, a right shift of N bits is used. To calculate the cell index of an immediate descendant (child) the current cell index is left shifted one bit and the octcode of the specified child is used to determine if the vacated right bit should be toggled (from 0 to 1).
In one embodiment a flag may be associated with each cell to indicate if all descendants are marked on. In certain embodiments, described below, cell rendering can be more rapid by rendering a single parent cell instead of 8 child cells, if all of the child cells are "on."
In some embodiments a density map 336 is provided for storing the number of descendent cells at a particular level that are marked on. A density map can be used in a fashion similar to the cell attribute flag discussed above i.e., it may be useful in some cases to render a parent cell even if not all child cells are "on" provided at least a certain minimum number of the child cells are on.
After the first portion 312 has been completed, the system is prepared to begin displaying in a fly-through interactive three-dimensional fashion 314. In one embodiment, the user presses a mouse button and points the cursor in the direction desired, in order to initiate fly-through. To start the display 338, preferably the user is permitted to set certain parameters for the display such as the level of detail and/or the frame rate. Because of the interaction between the level of detail and the frame rate (the finer level of detail, the slower the maximum frame rate that can be achieved), in some embodiments, the user may be permitted to select one parameter which will control value of the outer (or to select a range or certain discrete values). In one embodiment, the user may select a desired level of detail and the system will thereupon render some or all portions of the view at a level of detail which is equal to or finer than the specified level and will adjust the frame rate as needed to maintain this level of detail. In another embodiment, the user may specify that the frame rate should never fall below a specified value and system will automatically render in those levels of detail necessary to maintain the minimum frame rate.
In some embodiments it is desired to maintain some representation of all parts that are visible in the scene. In other embodiments, it may be desirable to discontinue certain visible cell representations (e.g., those in the distant background) in order to achieve certain rendering performance gains.
There are some situations in which the time required to draw a simplified version of a part might be longer than the time required to draw the part in full detail (e.g., a part with a large surface area defined by large triangles). In some embodiments, (particularly when the number of parts is relatively small) these situations are identified on a part-by-part basis and a more rapid rendering method is selected. In another embodiment, no account is taken of this possibility and simplified representations are presumed to take less time than a full-detail representation of a given part.
In one embodiment, a part, especially a large or complex part, may be "split" (i.e. treated as two or more parts) in order to improve rendering performance. Preferably any such splitting will be accommodated during the pre-processing 316 or data organization 318 phases, i.e. before image generation occurs. The criteria for splitting the part can be based on the spatial location of individual polygons relative to this axis-aligned cell boundaries that intersect the part. All of the sub-parts that result from this processing are present in the cell-to-part mappings contained in the cells. In general, this will increase the memory requirement for the cell-to-part mappings since there are now more parts (i.e. one part is split into multiple sub-parts) that span the same volume of space. However, rendering performance is improved in cases when the view frustum only intersects a portion of the original part because only the sub-parts within the view frustum need be rendered as opposed to the entire part.
Preferably simplified representation of parts are drawn in the background portion of a scene and at least some detailed rendering is maintained for those parts relatively close to the virtual camera, when possible.
In a typical situation, simplified rendering is necessary only when the camera is moving, i.e., during a "fly-through" procedure. Thus, if the camera is not moving 340, the scene will be rendered in full detail 342. If the camera is in motion, (i.e., if the user is providing input via a mouse, joystick, etc., indicating the user's desire to move the view point or "virtual camera" the system will then determine whether it is time for a new frame 343. At any given time, a frame rate is in effect. For example, in one embodiment a frame rate of 10 frames per second or more is preferably maintained.
If it is determined that anew frame is to be rendered, the system will calculate the point of view for that frame. The point of view will depend upon, for example, the input being provided by the user (where the user is placing the mouse or cursor). In one embodiment, the user can set a desired flight speed. Preferably a default speed is defined and only certain speed increments are permitted. Preferably the default speed and speed increments are established based on the extent of the current world. In some cases, however, it is desired to control the apparent acceleration and/or velocity of the virtual camera 344. In one embodiment if the viewpoint was previously static and the user has requested movement to a new location, it is preferred, in some embodiments, to limit the apparent acceleration of the camera so that there is no appearance of "infinite" acceleration from a static position, instantaneously, to a moving status, since such instantaneous acceleration may make it difficult for a user to properly navigate through the scenes (e.g., may result in over-shooting or under-shooting). Thus in one embodiment, within the constraints of limits on acceleration, the new point of view is calculated based on the user input 344. Preferably virtual camera positions are determined as a function of acceleration and velocity. Acceleration is particularly important for fine-grained motion control, when close to parts of interest. Preferably a velocity maximum is pre-defined and thus acceleration is only important until the maximum velocity is obtained. Deceleration, in one embodiment, is allowed to be instantaneous, i.e., is a step function, unlike acceleration which is a ramp function. The distance the virtual camera travels between frames is a function of the current velocity and the elapsed time. If the target frame rate is achieved, then the current velocity and the actual elapsed time are used to calculate the camera position. If not, then an ideal elapsed time (based on the target frame rate) is used instead of the actual elapsed time. This prevents "jumps" in the camera position when the frame rate decreases.
As noted above, one of the techniques used to reduce computational requirements is view frustum culling. The idea behind view frustum culling is schematically illustrated in FIG. 7. If it is assumed that the virtual camera 712 is positioned as shown in FIG. 7, the view which would be seen by such virtual camera defines a volume which generally has the shape of a truncated pyramid 714 bounded by four view frustum planes 715a, 715b, 715c, 715d and near and far bounding planes 715e, 715f. Accordingly, in the situation depicted in FIG. 7, the scene can be adequately rendered by rendering only those parts (either in full detail or in simplified fashion) which lie within the frustum 714 and "culling" volumes outside that cone 714. According to one embodiment, view frustum culling is achieved in a fashion which is hierarchical, tightly integrated with the octree data representation, incremental (supports collecting cells one "slice" at a time), designed to produce a tight cell fit at the boundaries of the cone frustum 714, and is based on optimized plane or cell intersection tests.
FIG. 8 depicts a process for a view frustum culling procedure according to an embodiment of the present invention. As depicted in FIG. 8, after defining the planes that bound the view frustum 812, cells are identified which are at least partially inside the view frustum and are marked "on" 814. This analysis is, in one embodiment, performed at a single pre-defined octree hierarchy level, such as level 5. Those identified cells which are frustum boundary cells are recorded 816, and it is determined which frustum boundary plane intersects each boundary cell 818. In one embodiment, in order to determine if a cell is within the frustum, a plane/cell corner test is used. Preferably, the cell-in-volume test can be performed with no more than 12 inequalities. The test reveals if a point (the cell corner) is in front of or behind a plane. This test can be performed using a simple dot product and thus is relatively rapid and easy to execute. According to this test, if the corner of the cell closest to the plane (assuming the cell is in front of the plane) is selected for the test, the results can be extrapolated to determine if a cell is completely or possibly partially in front of a plane. To determine if a cell intersects a plane, these results are combined with a second test where the orientation is reversed. In this second test, the normal on the other side of the plane is used to establish the forward direction and a new cell corner is used. If either test reveals that a cell is in front of the plane, then the cell does not intersect the plane. If both tests indicate the cell is partially in front of the plane, then the cell intersects the plane. For cells close to the position of the virtual camera, an axis-aligned plane (roughly coplanar with the far clipping plane) is used to exclude cells directly behind the virtual camera position.
The cells are sorted in order of distance from the virtual camera 820. As described below, scene processing is performed slice-wise. The slice planes can be defined in a number of fashions. In one embodiment slice planes are defined as parallel to the far clipping plane (i.e., the bounding plane of the view frustum). Sort distance can be calculated as the dot product over the norm. The position of the current slice plane is updated. The distance from the virtual camera to this plane is calculated 822. The slice planes are used to incrementally partition the scene so it can be processed one-slice-at-a-time in a front-to-back order. Next, those slice cells which are in front of the slice plane but have not been completely processed are located 824. This is done by comparing the distances from the virtual camera to the cell and the slice plane. The cell level which is required is determined by the level of detail that will be used for rendering the current slice. It is common for the cells collected at the pre-defined level to be traversed in order to find descendent (smaller) slice cells. When traversing the hierarchy to find descendant slice cells, the cell type (interior or boundary) and plane mask (indicating which boundary planes intersect the cell) are used extensively. Specifically, if the cell type is "interior," then there is no need to test the descendent cell against the boundary planes of the new frustum. If the cell type is "boundary," then the plane mask is used to optimize the boundary testing. The cell processing state is updated at the conclusion of this step. "Processing state" is an indication, for a given cell, of whether processing has not started, has been partially completed, or is complete. For example, if all of the cell descendants have been allocated to slices, then the cell is updated to indicate processing is complete. At this point the slice is rendered 826 and the position of the slice plane is updated 822 in order to continue processing. The last slice plane does not need to be defined, since collecting all of the non-completed cells is preferably supported.
A number of aspects of the frustum culling can be used to improve or maintain performance. By making an assumption that the near clipping plane (the view plane immediately in front of the camera) is close to the position of the virtual camera, the view frustum boundary can be defined by five points 718a, 718b, 718c, 718d, 718e (i.e., a pyramid) (rather than eight points that would be required to define a truncated pyramid).
The five-point view frustum boundary can be used to define the perspective transformation for the underlying graphics system (e.g. OpenGL). In one embodiment the far clipping plane 715f' is coincident with the five-point view frustum plane that is perpendicular to the virtual camera look direction 722 (i.e. the base of the pyramid). The near clipping plane 715e is positioned slightly in front of the virtual camera at a distance 724 that is proportional to the size of the five-point view frustum.
In another embodiment the near and far clipping planes are positioned dynamically based on the part(s) (i.e. the cells that have non-null cell-to-art mappings) that are present within the five-point view frustum boundary 714. The near and far clipping planes are positioned as close to the part(s) as possible without actually clipping any of the part(s) from the resulting image that is generated. This clip plane "clamping" reduces the distance between the clipping planes and improves the z-buffer resolution of the graphics system. This is turn improves the accuracy of the graphics selection mechanism. Furthermore, surfaces that are close together are displayed more precisely (with respect to how the surfaces occlude each other). The actual position of the clipping planes is determined after the cells have been sorted in order of distance from the virtual camera and before any primitives are sent to the graphics system for rendering. The closest point on the closest cell (with a non-null cell-to-part mapping) to the virtual camera is used to position the near clipping plane. In a similar fashion, the farthest point on the farthest cell (again with a non-null cell-to-part mapping) is used to position the far clipping plane. Since the cells are already sorted for the subsequent scene processing, this procedure can be used at interactive frame rates.
Once view frustum culling is performed 348 and it is known which cells need to be rendered, scene processing is used 352 to provide procedures intended to deliver the "best" (substantially most useful or pleasing) image possible with the available resources. In general, the scene is rendered 354 in slices from front to back. This spatial partitioning divides the work to be done, allows slice primitives to be sent to the graphics pipeline early in the processing sequence (i.e., drawing starts before the entire scene is processed), sets implicit priorities (i.e., the slices in front may be rendered in greater detail than those behind), and allows rendering progress to be monitored. A rendering slice is a set of cells that is processed and drawn together. Typically a slice contains all of the cells that are within some distance from the virtual camera. In one embodiment, a single level of detail is used for any given slice. As used hereinafter, a "region" is the largest spatially contiguous set of slices in a scene that share a an LOD.
Monitoring the time which is spent rendering is useful in controlling the LOD transitions and provides a natural checkpoint for testing if the user has requested an interrupt (e.g., in a static scene). As noted above, a motion scene is drawn while the camera is in motion. The goal is to provide enough visual detail to permit navigation while still maintaining high frame rates. Typically a motion scene is composed of multiple regions, each with a different level of detail. The region with the finest level of detail (in some cases full detail) is closest to the virtual camera and the coarsest LOD is the farthest away. Static scenes on the other hand are drawn when the camera stops moving. The goal is to render the entire scene in full detail. Preferably, interrupts are supported to allow a user to pause briefly and resume moving the virtual camera, without waiting for the static scene to be drawn completely.
In one embodiment, the time available for drawing a scene (the reciprocal of the frame rate) is allocated according to a precalculated region-time-distribution. This distribution essentially determines what percentage of the available time will be spend in each region, in order to produce the most useful image. The available rendering time is bounded by the requested frame rate. The distributions are based partly on the number of LOD settings. For example, if the user has specified that a motion scene can use three different LOD settings, the time may be distributed such that 70% of the time is used in the first region employing the finest level of detail, 20% in the second region and 10% in the third region at the lowest level of detail.
Multiple region-time-distributions are provided for each possible level of detail count in order to address the variation of drawing complexity and machine capability. In one embodiment, these are ordered in tables so entries for low drawing complexity/high machine capability combinations are first, and high drawing complexity/low machine capability combinations are last. In one embodiment, an indication of the drawing complexity is divided by an indication of the machine capability and the result is used as an index into this table. A number of items can be used to indicate drawing complexity. In one embodiment, the approximate number of polygons that were drawn in the previous frame is used. This approach is based on the existence of a certain amount of frame-to-frame coherence and provides the advantage of avoiding spending time determining a polygon count for the current frame. The polygons for each part are known from the data pre-processing phase 316, and the polygons used to draw cells can be readily collected. One manner of indicating the machine capability is to base a metric on the number of polygons the machine can render per unit time. The machine capability factor may be calculated (or automatically updated) based on statistics gathered during use or may be manually set (effectively allowing the user to "push" the full detail boundary farther back into the scene).
The processing for both motion and static scenes is similar except that when a motion scene starts to run out of time, compromises are made in image quality (i.e., the remaining regions are assigned coarser LOD). When a static scene runs out of time, drawing is interrupted to determine if the user has requested an interrupt (the static scene code is re-entrant).
FIG. 9 depicts a scene processing algorithm according to in one embodiment of the present invention. At the beginning of the procedure a timer is started 912. The virtual camera position is updated 914 and the view frustum is defined 916. The view frustum culling is performed 918 (e.g., a depicted in FIG. 8) and the region-time-distribution table is consulted to set the current level of detail 920. A spatial extent of the parts in the scene is determined 922. This can be done in a number of fashions. In one embodiment, two boundaries are defined by planes parallel to the far clipping plane. In other embodiments, it may be desirable to calculate density or complexity maps for this purpose. The spacing between slice planes is determined 924. Proper spacing is useful to avoid excessive processing overhead. Preferably the spacing is based on the current LOD (920). For example, if octree hierarchy level 5 is currently being rendered, then the slice plane spacing is set based on the size of a level 5 cell. The current slice plane location is updated (based on a previous plane location and current spacing 926). The cells in front of the current slice plane that have not yet been rendered are identified 928. As noted above, the view frustum culling algorithm provides this list of cells. At this point, a prediction is performed to determine if the cells can be drawn at the current LOD within the time remaining for the current region 932. If not, the LOD is updated to a coarser representation 934. The prediction is based on the region-time-distribution table and the value of the timer 912. If the cells can be drawn at the current LOD, the cells are rendered 936. Rendering progress is checked and the current LOD will be updated if necessary 942.
In one embodiment, the user can specify a "focal length" which defines the distance from the virtual camera to the boundary between a first LOD region and a second region at another (e.g. coarser) LOD. If desired, this focal length can controlled exclusively by the user and can be configured so that it is not influenced by the parts present in the current scene. The scene processing algorithm proceeds normally, but with the additional constraint that the boundary between the first two regions is dictated by the focal length, not the time remaining for rendering a particular region. If all of the parts are within the focal length distance from the virtual camera they are rendered in the first region at the finer LOD. If all of the parts are beyond the focal length distance, none of them are rendered at the first LOD specified by the user. Once the current slice plane is beyond the focal length distance, the region-time-distribution tables are used to guide the scene processing algorithm and the focal length constraint is no longer in effect. This focal length control gives users greater control over the final image quality.
The rendering of the scene (i.e., attending to display of an image on the display screen) is performed differently depending on whether the simplified 346 or full detail 342 scene is being rendered. Certain features are in common, however. A number of communication channels can be used for rendering, such as the X-Window system or, preferably, the Open GL pipeline. Open GL is software which is available from Silicon Graphics, Inc. Preferably, double buffering is used so the current frame can be displayed while the next frame is being composed. This ensures that only completed frames are seen and aids in performance. In both instances, z-buffering techniques known to those of skill in the art are available to support hidden surface removal. The scene can be rendered to simulate various directions of lighting, including ambient light and/or a positional light. In one embodiment, the positional light moves with the virtual camera and is offset to an elevated value (i.e., along the y axis) to provide more realistic lighting and avoid glare. In one embodiment, flat shading is used. In another embodiment, smooth shading is used. The graphics window is preferably implemented by a GlwM drawing area widget, available from Silicon Graphics, Inc. It provides a window with the appropriate visual and color maps needed for Open GL based on supplied parameters.
Full detail rendering 342 uses data which is organized such that each part has one or more meshes (defined by referencing normal and vertex indices), a normal array and a vertex array that the meshes reference and a color. Preferably a default set of material properties is applied to all parts. A frame number counter is initialized at the start of each session and is incremented each time a frame is drawn. In one embodiment the current frame number is used as a flag in data structures to indicate that work for the frame is complete. This technique is used to identify the parts that have been drawn in a particular frame. This alleviates the need to maintain a list of drawn parts and ensures that part rendering will not be duplicated in a single frame.
FIG. 10 depicts a full detail rendering process according to an embodiment of present invention. After setting the color and material properties for the part 1012, the procedure loops through each mesh in the part 1014. The normal and vertex indices for each mesh point are dereferenced 1016 and the mesh points are packaged in the fashion supported by the particular communication channel for rendering this is being used. If open GL is being used, the mesh points are packaged into a triangle strip primitive 1018. After each mesh in the part has been completed 1020, the part is tagged with the current frame number 1022 to indicate that the rendering of the part has been completed and the system loops 1024 to select the next part 1010.
The simplified cell rendering procedure 354 is preferably configured so as to draw the minimum number of cell faces to present a facade that is visually identical to the image that would result from drawing all six faces of each cell. At most, only three faces of a cell are visible from any orientation (assuming no stereoscopic viewing). As depicted in FIG. 11, one of the first steps in cell rendering is to determine which three faces of a cell are potentially visible 1112. This is a function of a position of the virtual camera relative to the cells. To further reduce the number of cell faces that are rendered, faces that have an adjacent neighbor 1121 (a cell that contains one or more parts) do not need to be drawn in most cases 1116. This is due to the fact that if the neighboring cell is drawn, it is guaranteed to occlude the cell face. To assure that a cell neighbor will occlude, a number of steps should be taken. Neighbor cells within the range of the near clipping plane (near the apex of the view frustum for the current camera position) are invalidated as potentially occluding cells. Otherwise, gaps in the cell faces may be seen where the near plane clips drawn cell faces (of the neighbors) from the resulting image. To simplify the calculation, a range of cell coordinates at each level is determined that effectively acts as a bounding box for the near clipping plane. When the cell neighbor check is performed, the cell coordinates of the neighbor are preferably compared against this range (instead of, e.g. performing a time-consuming distance calculation). If the neighbor cell is too close to the virtual camera, the candidate cell face is drawn (i.e., there is no assumption of cell neighbor occlusion).
When rendering multiple levels of detail, it becomes important to know how a neighbor cell will be rendered. For example, if the neighbor cell is rendered in full detail (i.e., the parts in the cell are to be drawn, not the cell itself), then the cell cannot be used as a neighbor cell in the cell neighbor occlusion algorithms since there is no guarantee that the candidate cell face is occluded 1118. To handle this case, the data structures preferably store the LOD used to render cells. Unless a neighbor cell is invalidated by steps 1116 or 1118, cell faces with an adjacent neighbor are not drawn 1120.
In order to improve rendering, the cell faces are collected in a buffer 1122 until an optimal number are present, whereupon the buffer is flushed. This is accomplished, in one embodiment, by sending quad primitives for each cell face to the graphics pipeline. When Open GL systems are used, all the primitives are enclosed in one Open GL begin/end pair to reduce the overhead of the graphics calls. The cell face buffer contains, for each cell face, vertex coordinates, an index for the associated normal, and the color index. Preferably, to avoid redundant state changes, color changes are not only sent to the graphics pipeline each time the buffer is flushed (at the beginning) and when cell faces change color. This process can also be repeated for cell material properties such as specular properties and shininess properties. In one embodiment, specular and shininess properties are held constant. The size of the cell face buffer can be determined empirically, if desired. Preferably, the size should be sufficient to provide a optimal or reasonable number of quad primitives per begin/end package on the particular target hardware. In one embodiment, the buffer is 500 (quad primatives). Preferably, if all of the child cells of a particular parent cell are marked "on," 1114 the parent cell is rendered rather than the individual child cells 1124. The total size of the rendered cells is the same in either case, but when the parent cell is drawn the number of graphics primitives and code traversal is reduced. The potential disadvantage is that the parent cell will be drawing with a single color (determined from a color precedence table) whereas they may be different colors for the various child cells.
In one embodiment, cell face masks are used to maintain the list of faces that need to be drawn for a cell. In one embodiment, this mask may be implemented as a byte data type. Preferably, a mapping convention defines the bits of that byte which are associated with each of the six cell faces. The cell face mask is initiated with the three visible faces of the cell and these are subsequently pruned (if possible) as the algorithm 1114-1124 proceeds.
As noted above, the frame rate is important to how the rendered scene will appear during fly-through. Preferably, frame rate is controlled 356 to maintain a constant frame rate, or to at least bound the rate of change. In one embodiment, both predictive and reactive techniques are used to achieve this goal. (356)
Predictive techniques for supporting relatively high virtual camera velocities and rotations, particularly in worlds that have a wide range of graphical complexities (i.e., many parts in some areas and few in others). This approach assumes that the information gleaned from rendering the last frame is not sufficient for rendering the best image for the current frame. According to one embodiment, the time required to render a slice at the current LOD is predicted 1212 (FIG. 12) and this prediction is used as basis for deciding if the current LOD should be updated before drawing (i.e., if the drawing will take too long) 1214. Both the polygon draw rate and the pixel fill rate factor into these slice predictions. As noted above, the polygon draw rate is calculated by dividing the number of polygons in the slice by the tendering capability of the targeting machine. According to one embodiment, the pixel fill rate is based on the projected area of the parts and images space relative to the size of the view port. The target machine pixel fill rate is used to determine the time required to pixel-fill the entire view port. Therefore, the time needed to pixel-fill the part is some percentage of the time required to pixel-fill the entire view port. The face of the part-bounding box (roughly perpendicular and closest to the virtual camera) is used to approximate the projected area of a part. Although this is a rough approximation, it is relatively rapid to obtain. The pixel fill rate is particular important for low end machines with software (not hardware) graphic support.
Reactive techniques are used, e.g., to automatically adjust parameters (used in the predictive estimates) e.g., to fine tune for machine load. The reactive techniques essentially form a dynamic feedback loop and it may be useful to provide damping, e.g., in order to prevent oscillations (e.g., visual flashing) in the LOD boundaries within the scene. Damping, in one embodiment, can be achieved by accumulating results over a number of frames before dynamically altering the parameters (as opposed to updating immediately after each frame).
In light of the above description, a number of advantages of the present invention can be seen. The present invention is particularly useful for facilitating communication in a design context, preferably early in the manufacturing process, by integrating parts throughout the design phase to ensure that the entire product fits together seamlessly and meeds the needs of the customer. When complicated components are integrated together in a virtual environment, engineers, product designers, manufacturing personnel and customers can view the results, identify problems and find solutions early in the manufacturing life cycle, reducing costs and leading to higher quality products. The present invention provides a design system which is based on bounded volumes or worlds which assists in organizing data, optimizing rendering and focusing activity in the design environment.
It is contemplated that a major use of the present invention will be in connection with for Computer-Aided Design (CAD). However, the invention is believed to be, in general powerful and extensible and it is also contemplated that the invention may be useable in software or other systems (e.g., in application products) for animation systems, game development, simulation systems, and internet/intranet viewers. It is anticipated that some or all aspects of the invention may be incorporated in a general purpose software development toolkit that would enable application product developers to incorporate the technology into their products as they see fit. The present invention can also be used in digital studios for entertainment media creation purposes, operating systems, such as a 3D viewer of the local and available file-systems and applications, medical imaging, weather forecasting, computation fluid dynamics, engineering analysis, computer-aided manufacturing, architectural engineering and design, electronic design, operations analysis, and the like.
Preferably, the system can handle large number of parts, such as one million or more, at high (interactive) frame rates (at least about 5, preferably about 10 hertz or more). According to the invention, parts or groups of parts are automatically simplified for rendering, e.g., for a fly-through procedure, the user is not required to define simplified representations on a part-by-part basis. Preprocessing organizes the data such that at load time, the volume occupied by the parts is known and may be merged with the volumes occupied by other parts already loaded in the system. Since the data is already spatially organized, processing time required to update simplified geometric representations is reduced. The simplified geometric representation of the present invention not only helps to achieve the desired performance, but is simplified in a visually apparent obvious fashion which helps to avoid potential confusion in a virtual manufacturing environment. For example, the user may need to know if two parts intersect. If the parts do not intersect but the simplified representations do, the user may be confused if the distinction between actual and simplified representations is not apparent.
A number of variations and modifications of the present invention can be used. Although the present invention has been described in terms of a computer-aided design application, the techniques disclosed herein can also be used for other image display purposes, such as computer games, high-definition television, virtual reality displays, computer animation and the like. In one embodiment, the cell faces are provided with textures according to various texture maps. Preferably the required texture maps are created during a pre-processing step. In some embodiments this approach precludes dynamic loading and unloading of parts during a viewer session (which is otherwise possible).
The system can be provided with the capability for capturing the cells and the outline of the view frustum associated with a particular frame. This makes it possible to navigate through the captured scene and analyze the output of various procedures. Preferably, the size of the frustum is increased in this mode to allow viewing from positions that provide a global perspective (i.e., back up far enough to see the entire outline of the view frustum without interference from a clipping plane). In one embodiment, it is possible to freeze the graphics primitives associated with the frame and rotate the image to view the cell facade. The cells that were not drawn become apparent as the scene is rotated. This optimization was not visible before rotation (i.e., the goal of the cell rendering is to present this illusion).
It is possible to provide for capturing the outline of the view frustum and the cells for a particular frame. By alternating the colors used to draw the cell outlines, the cells associated with each slice can be discerned.
In some embodiments it is possible to provide for additional efficiency by avoiding rendering of parts that are subject to geometric occlusion. Geometric occlusion involves determining parts of a scene that are occluded geometrically and do not need to be drawn. In one approach, as a scene is proceeded from front to back, for each slice, the world coordinates of coordinates of significant parts are transformed into image space, i.e., parallel to the far clipping plane. Shadows or projections on the image space plane are accumulated as the image based plane moves back through the scene during processing and the shadows were used to preclude drawing parts that are "in the dark." Such an approach may be used in conjunction with pre-processing to avoid making assumptions that result in improper display (e.g., assuring a part is solid when, in fact, the surface has holes). Pre-processing in this context may also be useful for optimization, e.g., to establish thresholds for part shadow casting that take into account the proximity of the part to the virtual camera and relative orientation of the virtual camera and the part.
In one embodiment the center of the screen is used as a reference point for calculating virtual camera adjustments. Preferably there is a "dead" spot near the center of the screen such that the angle of adjustment is not affected when the cursor is in that area. This helps the user avoid overly-sensitive input adjustments.
In one embodiment, in addition to a fly mode, the user is permitted to initiate an "inspect" mode which provides a direct manipulation interface based on quaternion rotation vectors. An example of a procedure of this general type is described in Bell, Gavin "Software for a Virtual Trackball Implementation" Computer Graphics, Siggraph, August 1988, pp. 121-129, incorporated herein by reference. Preferably the user is permitted to select individual parts among those which are displayed and the system may provide detailed information on the parts such as by printing statistical information (name and color index) and the like.
In one embodiment, flight paths (a series of virtual camera positions and orientations) may be recorded and played back. When recorded, the type of scene (motion or static) and the virtual camera location is written to a file for each frame. During such playback mode, the user preferably is permitted to manually pause and resume animation. In one embodiment, playback may be paused automatically at user-specified frame count intervals. This made may be useful for manually capturing images, e.g., for animation.
If desired, certain procedures can be used to assist in the transition from the motion mode to a static scene mode, particularly since, once motion stops, it may take a considerable amount of time for the static (full detail) scene to be rendered. During this transition time, in some embodiments, the user is allowed to interrupt the static scene generation and resume moving. Preferably the user is presented with information indicating the progress being made on drawing the static scene. In one embodiment, the static scene is produced by producing successively-refined intermediate visual representations of the static scene, and, at intervals, checking to see if user has requested an interrupt. Preferably, the successive refinement provides the user with good resolution on closer parts first, without waiting for the remainder of the static scene to be drawn. In one embodiment this is achieved by moving the full-detail boundary farther back into the scene with each intermediate visual representation, until the entire static scene is rendered in full detail.
A stencil buffer may be used to keep track of what is drawn in full detail and what is drawn as an LOD representation. FIG. 13 depicts a procedure which can be used for this purpose. First the stencil buffer is set to 0 312. The current slice is drawn in full detail and the portions of the stencil buffer corresponding to portions of the scene that are drawn in full detail is marked with a 1. 1314. The next slice is drawn in a simplified (with simplified rendering for this slice) manner, only for those areas where the stencil buffer is not equal to 1 and those areas are then marked with a value equal to two 1315. All the remaining slices are rendered for those areas where the stencil buffer is not equal to 1 but no marking of the stencil buffer is done for those areas 1316. Next, all areas of the frame buffer pixels associated with stencil value buffer equal to 2 are cleared 1318 (this clears the LOD representation that will be replaced with full detail on the next pass) and those areas are set to 0. The procedure returns 1314 to process the scene beginning with the next slice to be done in full detail.
Although as described above, view frustum culling can be performed using cell/plane intersection tests, it is also possible to use an approach that employs a table to look up discrete 3D representations of lines, namely the lines that form boundary edges of the view frustum. In this approach, the procedure walks the edges of the view frustum and looks up the cells along the edge in the table. The procedures interpolates between these cells to find the boundary cells for view frustum slices.
Although the invention has been described in terms of worlds having uniform cells, it is possible to provide the worlds that are not necessarily uniform. In one embodiment, a cell structure can be defined that allows an arbitrary number of uniform cells to be configured at the top. In this case, all of the traversal operations would be required to consult with the layer of cells before continuing down each of the separate hierarchies.
By providing additional data reference structure and cell color distribution, it may be possible to support dynamic geometry changes. In one embodiment, it may be possible to dynamically or quickly toggle the visibility of a particular part or groups of parts, e.g., by optimizing the manner in which a part's cell information is propagated.
Although color precedence was described as being established with respect to a particular world (and thus held static over the course of the session), the system could be configured to allow the color precedence table to updated dynamically. This would support altering the coloring of cells based on the types of parts they contain. For example, if hydraulic parts were blue, electric parts were green, and these parts were common to many cells, the color precedence table could be changed to emphasize either type of part. This might assist in navigational efforts if the user needed to focus on one of the part types.
Although the region-time-distribution was described as being based on a pre-calculation scheme, in another embodiment the time distribution tables may be dynamically updated.
In situations where large geometries are involved, the task may be split in order to optimize rendering. In some embodiments it may be useful to render a particular cell at a coarser LOD if cells in front contain parts that are drawn in a high LOD or in full detail (such that cells behind are at least partially obscured and therefore less important).
Although the present invention can be used to avoid the need for "hand building" LOD representations on a part-by-part basis, in some embodiments it may be useful to provide at least some such hand-built LOD representations, e.g., depending on the complexity of the scene. In some situations, thresholds for using different techniques can be established such as rendering all parts in full detail for low complexity scenes, building LOD representations on a part-by-part basis for medium complexity scenes, grouping parts based on spatial coherence and building LOD representations on a group-by-group basis for high complexity scenes and using the above-described volume-based LOD approach for very high complexity scenes.
In some embodiments it is desired to support hierarchical (i.e. nested) "worlds" and provide the user with the ability to navigate through these hierarchies. Each world in the hierarchy can be defined to encompass a particular part or group of parts that are of interest to the user. Tailoring the size of each world optimizes the geometric simplifications used in that world (i.e. the cells at a particular level in the octree are scaled appropriately). The user can navigate between the worlds by selecting world from a list or by moving the virtual camera with the understanding that this is used as input to another algorithm that selects the most appropriate world based on the current position of the virtual camera and other user-supplied parameters.
Although the above description was generally described based on cubic geometric simplification, volume based simplification can also be provided based on other shapes such as convex polyhedral geometries. This approach may be further useful in avoiding problems with axis-aligned normals.
Although the invention has been defined by way of a preferred embodiment and certain variations and modifications, other variations and modifications can also be used in the invention being defined by the following claims:
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4694404 *||Jan 12, 1984||Sep 15, 1987||Key Bank N.A.||High-speed image generation of complex solid objects using octree encoding|
|US4710876 *||Jun 5, 1985||Dec 1, 1987||General Electric Company||System and method for the display of surface structures contained within the interior region of a solid body|
|US4719585 *||Aug 28, 1985||Jan 12, 1988||General Electric Company||Dividing cubes system and method for the display of surface structures contained within the interior region of a solid body|
|US4729098 *||Jun 5, 1985||Mar 1, 1988||General Electric Company||System and method employing nonlinear interpolation for the display of surface structures contained within the interior region of a solid body|
|US4865423 *||Jul 5, 1988||Sep 12, 1989||International Business Machines Corporation||Method for generating images|
|US4882679 *||Nov 27, 1987||Nov 21, 1989||Picker International, Inc.||System to reformat images for three-dimensional display|
|US4961154 *||Jun 2, 1987||Oct 2, 1990||Scitex Corporation Ltd.||Three dimensional modelling apparatus|
|US5095521 *||Feb 27, 1990||Mar 10, 1992||General Electric Cgr S.A.||Method for the computing and imaging of views of an object|
|US5123084 *||Dec 21, 1990||Jun 16, 1992||General Electric Cgr S.A.||Method for the 3d display of octree-encoded objects and device for the application of this method|
|US5170347 *||Apr 30, 1990||Dec 8, 1992||Picker International, Inc.||System to reformat images for three-dimensional display using unique spatial encoding and non-planar bisectioning|
|US5218671 *||May 31, 1989||Jun 8, 1993||Computer Design, Inc.||Image color correction system and method|
|US5345490 *||Jun 28, 1991||Sep 6, 1994||General Electric Company||Method and apparatus for converting computed tomography (CT) data into finite element models|
|US5381518 *||Dec 18, 1992||Jan 10, 1995||Pixar||Method and apparatus for imaging volume data using voxel values|
|US5412763 *||May 24, 1991||May 2, 1995||General Electric Cgr S.A.||Method for displaying an image of a portion of the inside of a three-dimensional physical structure|
|US5414801 *||Jun 11, 1991||May 9, 1995||Virtus Corporation||Computerized method and apparatus using containment relationships to represent objects in a three-dimensional space, and for moving therethrough|
|US5475422 *||Jun 20, 1994||Dec 12, 1995||Nippon Telegraph And Telephone Corporation||Method and apparatus for reconstructing three-dimensional objects|
|US5517602 *||Dec 3, 1992||May 14, 1996||Hewlett-Packard Company||Method and apparatus for generating a topologically consistent visual representation of a three dimensional surface|
|US5579455 *||Jul 30, 1993||Nov 26, 1996||Apple Computer, Inc.||Rendering of 3D scenes on a display using hierarchical z-buffer visibility|
|US5596690 *||Jul 21, 1993||Jan 21, 1997||Xerox Corporation||Method and apparatus for operating on an object-based model data structure to produce a second image in the spatial context of a first image|
|US5613049 *||Oct 26, 1994||Mar 18, 1997||The Boeing Company||Method for creating spatially balanced bounding volume hierarchies for use in a computer generated display of a complex structure|
|US5615321 *||Aug 12, 1994||Mar 25, 1997||Dassault Systemes Of America Corp.||Automatic identification of geometric relationships between elements of a computer-generated drawing|
|US5638190 *||Mar 29, 1994||Jun 10, 1997||Clemson University||Context sensitive color quantization system and method|
|US5729254 *||Dec 27, 1995||Mar 17, 1998||Mitsubishi Electric Information Technology Center America, Inc.||Method and apparatus for differential object modeling using automatically generated constructive solid geometry (CSG) through an evolutionary process|
|US5825998 *||Jul 9, 1996||Oct 20, 1998||The Boeing Company||Method for controlling the level of detail displayed in a computer generated screen display of a complex structure|
|1||Chamberlain, B., T. DeRose, D. Lischinski, D. Salesin and J. Snyder, "Fast Rendering of Complex Environments Using a Spatial Hierarchy", Graphics Interface, May 1996.|
|2||*||Chamberlain, B., T. DeRose, D. Lischinski, D. Salesin and J. Snyder, Fast Rendering of Complex Environments Using a Spatial Hierarchy , Graphics Interface, May 1996.|
|3||Funkhouser, T.A. and C.H. Sequin, Aug. 1993, "Adaptive Display Algorithm for Interactive Frame Rates During Visualization of Complex Virtual Environments", In Computer Graphics Proceedings, Annaul Conference Series, ACM SIGGRAPH, 247-254.|
|4||*||Funkhouser, T.A. and C.H. Sequin, Aug. 1993, Adaptive Display Algorithm for Interactive Frame Rates During Visualization of Complex Virtual Environments , In Computer Graphics Proceedings, Annaul Conference Series, ACM SIGGRAPH, 247 254.|
|5||Greene, N., M. Kass and G.iller, Aug. 1993, "Hierarchical 2-Buffer Visitality", In Computer Graphics Proceedings, Annual Conference Series, ACM SIGGRAPHIC, 231-238.|
|6||*||Greene, N., M. Kass and G.iller, Aug. 1993, Hierarchical 2 Buffer Visitality , In Computer Graphics Proceedings, Annual Conference Series, ACM SIGGRAPHIC, 231 238.|
|7||Lanr, D. And P. Hanrahan, Jul. 1991, "Hierarchical Splatting: A Progressive Refinement Algorrithm for Volume Rendering", Computer Graphics (SIGGRAPHIC '91 Proceedings) 25(4), 285-288.|
|8||*||Lanr, D. And P. Hanrahan, Jul. 1991, Hierarchical Splatting: A Progressive Refinement Algorrithm for Volume Rendering , Computer Graphics (SIGGRAPHIC 91 Proceedings) 25(4), 285 288.|
|9||*||Rohif, J. And J. Heiman, Jul. 1994, Iris Performer: A High Performance Multiprocessing Toolkit for Real Time 3D Graphics, In Computer Graphics Proceedings, Annual Conference Series, ACM SIGGRAPH, 381 394.|
|10||Rohif, J. And J. Heiman, Jul. 1994, Iris Performer: A High Performance Multiprocessing Toolkit for Real-Time 3D Graphics, In Computer Graphics Proceedings, Annual Conference Series, ACM SIGGRAPH, 381-394.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US6222555 *||Jun 18, 1998||Apr 24, 2001||Christofferson Enterprises, Llc||Method for automatically smoothing object level of detail transitions for regular objects in a computer graphics display system|
|US6304680 *||Oct 22, 1998||Oct 16, 2001||Assembly Guidance Systems, Inc.||High resolution, high accuracy process monitoring system|
|US6313838 *||Feb 16, 1999||Nov 6, 2001||Sun Microsystems, Inc.||Estimating graphics system performance for polygons|
|US6320596 *||Feb 24, 1999||Nov 20, 2001||Intel Corporation||Processing polygon strips|
|US6348921 *||May 1, 1998||Feb 19, 2002||Ze Hong Zhao||System and method for displaying different portions of an object in different levels of detail|
|US6384822 *||Oct 13, 1999||May 7, 2002||Creative Technology Ltd.||Method for rendering shadows using a shadow volume and a stencil buffer|
|US6424343 *||Feb 17, 1999||Jul 23, 2002||Sun Microsystems, Inc.||Graphics system with programmable real-time sample filtering|
|US6459428 *||Oct 3, 2001||Oct 1, 2002||Sun Microsystems, Inc.||Programmable sample filtering for image rendering|
|US6483504 *||Oct 6, 1999||Nov 19, 2002||Sun Microsystems, Inc.||Graphics system having a super sampled-sample buffer with efficient storage of sample position information|
|US6489956 *||Oct 6, 1999||Dec 3, 2002||Sun Microsystems, Inc.||Graphics system having a super-sampled sample buffer with generation of output pixels using selective adjustment of filtering for implementation of display effects|
|US6496186 *||Oct 6, 1999||Dec 17, 2002||Sun Microsystems, Inc.||Graphics system having a super-sampled sample buffer with generation of output pixels using selective adjustment of filtering for reduced artifacts|
|US6496187 *||Dec 27, 1999||Dec 17, 2002||Sun Microsystems, Inc.||Graphics system configured to perform parallel sample to pixel calculation|
|US6525723 *||Oct 6, 1999||Feb 25, 2003||Sun Microsystems, Inc.||Graphics system which renders samples into a sample buffer and generates pixels in response to stored samples at different rates|
|US6573896 *||Jul 8, 1999||Jun 3, 2003||Dassault Systemes||Three-dimensional arrow|
|US6574360 *||Jul 23, 1999||Jun 3, 2003||International Business Machines Corp.||Accelerated occlusion culling using directional discretized occluders and system therefore|
|US6664957 *||Nov 5, 1999||Dec 16, 2003||Fujitsu Limited||Apparatus and method for three-dimensional graphics drawing through occlusion culling|
|US6778171||Oct 31, 2000||Aug 17, 2004||Eagle New Media Investments, Llc||Real world/virtual world correlation system using 3D graphics pipeline|
|US6788302 *||Aug 3, 2000||Sep 7, 2004||International Business Machines Corporation||Partitioning and load balancing graphical shape data for parallel applications|
|US6828974||May 9, 2003||Dec 7, 2004||Dassault Systemes||Three-dimensional arrow|
|US6836274||Sep 19, 2000||Dec 28, 2004||Eagle New Media Investments, Llc||Three dimensional light electronic programming guide|
|US6941255||Jun 29, 2001||Sep 6, 2005||Object Reservoir, Inc.||Feature modeling in a finite element model|
|US6963431 *||Jan 28, 2002||Nov 8, 2005||Zebra Imaging, Inc.||Rendering methods for full parallax autostereoscopic displays|
|US7002590||Jan 28, 2002||Feb 21, 2006||Yakov Kamen||Modifying object geometry to accommodate information attributes|
|US7006951 *||Jun 29, 2001||Feb 28, 2006||Object Reservoir, Inc.||Method for solving finite element models using time slabbing|
|US7027964||Jun 29, 2001||Apr 11, 2006||Object Reservoir, Inc.||Method and system for solving finite element models using multi-phase physics|
|US7042455 *||May 26, 2004||May 9, 2006||Sand Codex Llc||System and method for multiple node display|
|US7043413||Jun 29, 2001||May 9, 2006||Object Reservoir, Inc.||Method for modeling an arbitrary well path in a hydrocarbon reservoir using adaptive meshing|
|US7057670||Apr 27, 2001||Jun 6, 2006||Dan Kikinis||Cursor control system|
|US7069202||Jan 11, 2002||Jun 27, 2006||Ford Global Technologies, Llc||System and method for virtual interactive design and evaluation and manipulation of vehicle mechanisms|
|US7079996 *||May 30, 2001||Jul 18, 2006||Ford Global Technologies, Llc||System and method for design of experiments using direct surface manipulation of a mesh model|
|US7103658 *||Dec 31, 2001||Sep 5, 2006||Fujitsu Limited||Rendering calculation processing status monitoring program, and storage medium, apparatus, and method therefor|
|US7106338||Oct 18, 2000||Sep 12, 2006||Eagle New Media Investments, Llc||Method and system for optimal usage of memory for storing scheduling and guiding data in 3D-enabled EPG|
|US7149671||Jun 29, 2001||Dec 12, 2006||Object Reservoir, Inc.||Method and system for coordinate transformation to model radial flow near a singularity|
|US7155673||Feb 1, 2001||Dec 26, 2006||Ford Global Technologies, Llc||System and method of interactive evaluation of a geometric model|
|US7174280||Apr 23, 2002||Feb 6, 2007||Ford Global Technologies, Llc||System and method for replacing parametrically described surface features with independent surface patches|
|US7213256||Dec 29, 2000||May 1, 2007||Dan Kikinis||Method and apparatus for finding the same of similar shows|
|US7224361||Apr 20, 2005||May 29, 2007||Microsoft Corporation||System and method for multiple node display|
|US7254271||Mar 17, 2005||Aug 7, 2007||Seadragon Software, Inc.||Method for encoding and serving geospatial or other vector data as images|
|US7260508||Jun 29, 2001||Aug 21, 2007||Object Reservoir, Inc.||Method and system for high-resolution modeling of a well bore in a hydrocarbon reservoir|
|US7263236 *||Dec 5, 2003||Aug 28, 2007||Samsung Electronics Co., Ltd.||Method and apparatus for encoding and decoding three-dimensional object data|
|US7286708||Dec 29, 2006||Oct 23, 2007||Microsoft Corporation||Method for encoding and serving geospatial or other vector data as images|
|US7312796||Sep 19, 2000||Dec 25, 2007||Jlb Ventures Llc||Perpendicular view three dimensional electronic programming guide|
|US7322009||Dec 1, 2004||Jan 22, 2008||Jlb Ventures Llc||Three dimensional light electronic programming guide|
|US7375727||Mar 3, 2005||May 20, 2008||Nvidia Corporation||System, method and computer program product for geometrically transforming geometric objects|
|US7375732||Sep 29, 2006||May 20, 2008||Microsoft Corporation||Methods and apparatus for navigating an image|
|US7386875||Mar 1, 2002||Jun 10, 2008||Jlb Ventures, Llc||Freezing live video frames with interactive links|
|US7425952 *||Apr 25, 2005||Sep 16, 2008||Metavr, Inc.||Three-dimensional visualization architecture|
|US7450120 *||Dec 19, 2003||Nov 11, 2008||Nvidia Corporation||Apparatus, system, and method for Z-culling|
|US7483827 *||Dec 30, 2004||Jan 27, 2009||Autodesk, Inc.||Translating objects between software applications which employ different data formats|
|US7519449 *||Dec 31, 2002||Apr 14, 2009||Samsung Electronics Co., Ltd.||Rendering system and method and recording medium therefor|
|US7525544 *||Apr 16, 2004||Apr 28, 2009||Research In Motion Limited||System and method of converting edge record based graphics to polygon based graphics|
|US7546419||Jun 1, 2005||Jun 9, 2009||Aguera Y Arcas Blaise||Efficient data cache|
|US7554543||Jun 30, 2009||Microsoft Corporation||System and method for exact rendering in a zooming user interface|
|US7609910 *||Apr 6, 2005||Oct 27, 2009||Siemens Medical Solutions Usa, Inc.||System and method for creating a panoramic view of a volumetric image|
|US7610552||Jan 8, 2008||Oct 27, 2009||Jlb Ventures Llc||Three dimensional light electronic programming guide|
|US7685534||Mar 23, 2010||Jlb Ventures Llc||Method and apparatus for a three-dimensional web-navigator|
|US7689935 *||Mar 11, 2003||Mar 30, 2010||Gould Eric J||Method, apparatus and article of manufacture for displaying content in a multi-dimensional topic space|
|US7698727||Apr 13, 2010||Jlb Ventures Llc||Method and apparatus for notifying users of interactive functions using a remote device|
|US7701468||Jan 28, 2002||Apr 20, 2010||Jlb Ventures Llc||Mechanism and apparatus for construction of modular-based interactive programming guides and TV portals|
|US7711527||Jan 7, 2005||May 4, 2010||Autodesk, Inc||Translating objects between software applications which employ different data formats|
|US7724965||Sep 19, 2007||May 25, 2010||Microsoft Corporation||Method for encoding and serving geospatial or other vector data as images|
|US7733341 *||May 30, 2008||Jun 8, 2010||Nokia Corporation||Three dimensional image processing|
|US7755624||Jul 13, 2010||Nvidia Corporation||Apparatus, system, and method for Z-culling|
|US7761269||Jul 20, 2010||Ford Global Technologies, Llc||System and method of subjective evaluation of a vehicle design within a virtual environment using a virtual reality|
|US7796127||Nov 29, 2005||Sep 14, 2010||Samsung Electronics Co., Ltd.||Adaptive rendering apparatus and method using hierarchical structure in 3D image and computer-readable recording medium storing computer program for executing the adaptive rendering method|
|US7800609||Jan 2, 2008||Sep 21, 2010||Sensable Technologies, Inc.||Method and apparatus for generating and interfacing with a haptic virtual reality environment|
|US7864173||Jul 27, 2004||Jan 4, 2011||Sensable Technologies, Inc.||Systems and methods for creating virtual objects in a sketch mode in a haptic virtual reality environment|
|US7880737||Feb 1, 2011||Vijayvardhan Elchuri||Graphical method and system for making drawings directly in three-dimensions on a computer monitor or other display device|
|US7886330||Feb 8, 2011||Jlb Ventures Llc||Enhanced home entertainment system with removable long-term storage|
|US7889195||Feb 15, 2011||Sensable Technologies, Inc.||Systems and methods for sculpting virtual objects in a haptic virtual reality environment|
|US7912299||Mar 22, 2011||Microsoft Corporation||System and method for efficiently encoding data|
|US7930434||Oct 17, 2005||Apr 19, 2011||Microsoft Corporation||System and method for managing communication and/or storage of image data|
|US7952579||Nov 1, 2007||May 31, 2011||Nvidia Corporation||System, method and computer program product for geometrically transforming geometric objects|
|US7952591||May 31, 2011||Jlb Ventures Llc||Navigating programming content via multi-dimensional surfaces|
|US7956865||Jun 7, 2011||Jlb Ventures Llc||Method and system for modification of EPG object attributes|
|US7969436||Jun 28, 2011||Nvidia Corporation||System, method and computer program product for transforming a polynomial equation from a coordinate frame of one tile to a coordinate frame of another tile at a finer level of a hierachy|
|US7990374 *||Aug 2, 2011||Sensable Technologies, Inc.||Apparatus and methods for haptic rendering using data in a graphics pipeline|
|US7999805||Aug 16, 2011||Research In Motion Limited||System and method of converting edge record based graphics to polygon based graphics|
|US8098243 *||Jan 17, 2012||Samsung Electronics Co., Ltd.||Method and apparatus for efficiently handling query for 3D display|
|US8130233||Apr 27, 2011||Mar 6, 2012||Jlb Ventures Llc||Modifying screen objects|
|US8132207||May 11, 2001||Mar 6, 2012||Jlb Ventures Llc||Universal programming system and method for EPG with localized interactive content|
|US8176439||Feb 4, 2010||May 8, 2012||Jlb Ventures Llc||Navigating content|
|US8194070||Jul 11, 2011||Jun 5, 2012||Research In Motion Limited||System and method of converting edge record based graphics to polygon based graphics|
|US8259122||Nov 19, 2007||Sep 4, 2012||Nvidia Corporation||System, method and article of manufacture for a programmable processing model with instruction set|
|US8264492||Sep 11, 2012||Nvidia Corporation||System, method and article of manufacture for a programmable processing model with instruction set|
|US8269768 *||Sep 8, 2003||Sep 18, 2012||Nvidia Corporation||System, method and computer program product for updating a far clipping plane in association with a hierarchical depth buffer|
|US8314810||Nov 20, 2012||Jlb Ventures Llc||Modifying screen objects|
|US8336069||Dec 18, 2012||Jlb Ventures Llc||Method and system for adaptive electronic programming guide|
|US8341676||Nov 28, 2001||Dec 25, 2012||JLB Ventures LLC.||Method and system for performing an alphabetic search using a single application of keys|
|US8400451 *||Sep 12, 2011||Mar 19, 2013||Perry Peterson||Close-packed, uniformly adjacent multiresolutional, overlapping spatial data ordering|
|US8438594||Feb 13, 2001||May 7, 2013||Jlb Ventures Llc||Method and system for 3-D enhanced advertising for TV broadcast of 2-D video|
|US8522266||Sep 13, 2000||Aug 27, 2013||Jlb Ventures Llc||System and method for insertion of recorded media into a broadcast|
|US8531488 *||Dec 14, 2012||Sep 10, 2013||Sony Corporation||Image processing apparatus and method, content creating support apparatus and method and data structure of image file with speed map to determine speed of viewpoint change|
|US8576222||Jan 3, 2011||Nov 5, 2013||3D Systems, Inc.||Systems and methods for interfacing with a virtual object in a haptic virtual environment|
|US8595764||Feb 5, 2002||Nov 26, 2013||Jlb Ventures, Llc||Image-oriented electronic programming guide|
|US8601511||Jan 18, 2012||Dec 3, 2013||Jlb Ventures Llc||Providing an interactive electronic programmimg guide with a layout that is defined based on objects|
|US8756635||Jun 5, 2001||Jun 17, 2014||JLB Ventures, LLC.||Remote control of program scheduling|
|US8789114||Jun 5, 2001||Jul 22, 2014||Jlb Ventures Llc||Remote control of program scheduling|
|US8970583 *||Oct 1, 2012||Mar 3, 2015||Google Inc.||Image space stylization of level of detail artifacts in a real-time rendering engine|
|US9030411||May 13, 2014||May 12, 2015||3D Systems, Inc.||Apparatus and methods for haptic rendering using a haptic camera view|
|US9153011 *||Dec 7, 2012||Oct 6, 2015||Google Inc.||Movement based level of detail adjustments|
|US9171396 *||Apr 12, 2012||Oct 27, 2015||Primal Space Systems Inc.||System and method of procedural visibility for interactive and broadcast streaming of entertainment, advertising, and tactical 3D graphical information using a visibility event codec|
|US9200919||Aug 24, 2012||Dec 1, 2015||Apple Inc.||Method, system and apparatus for selectively obtaining map image data according to virtual camera velocity|
|US9384548 *||Jan 23, 2015||Jul 5, 2016||Kabushiki Kaisha Toshiba||Image processing method and apparatus|
|US9472019||Jun 29, 2011||Oct 18, 2016||Primal Space Systems, Inc.||System and method of from-region visibility determination and delta-PVS based content streaming using conservative linearized umbral event surfaces|
|US20010015154 *||Jan 5, 2001||Aug 23, 2001||Roman Lenz||Ternary pigment compositions|
|US20020007488 *||May 15, 2001||Jan 17, 2002||Dan Kikinis||Transparent object management for removable media recorders|
|US20020010925 *||Jun 5, 2001||Jan 24, 2002||Dan Kikinis||Remote control of program scheduling|
|US20020019732 *||Jun 5, 2001||Feb 14, 2002||Dan Kikinis||Interactivity using voice commands|
|US20020019980 *||Jun 5, 2001||Feb 14, 2002||Dan Kikinis||Remote control of program scheduling|
|US20020019986 *||Jun 5, 2001||Feb 14, 2002||Dan Kikinis||Enhanced home entertainment system with removable long-term storage|
|US20020032494 *||Jun 29, 2001||Mar 14, 2002||Kennon Stephen R.||Feature modeling in a finite element model|
|US20020032550 *||Jun 29, 2001||Mar 14, 2002||Ward Steven B.||Method for modeling an arbitrary well path in a hydrocarbon reservoir using adaptive meshing|
|US20020035453 *||Jun 29, 2001||Mar 21, 2002||Pond Stuart W.||Method for solving finite element models using time slabbing|
|US20020046014 *||Jun 29, 2001||Apr 18, 2002||Kennon Stephen R.||Method and system for solving finite element models using multi-phase physics|
|US20020059597 *||Jun 5, 2001||May 16, 2002||Dan Kikinis||Method and apparatus for notifying users of interactive functions|
|US20020059605 *||May 11, 2001||May 16, 2002||Dan Kikinis||Universal programming system and method for EPG with localized interactive content|
|US20020059606 *||May 11, 2001||May 16, 2002||Dan Kikinis||Universal programming system and method for EPG with added offline archive|
|US20020067373 *||Jun 29, 2001||Jun 6, 2002||Eric Roe||System and method for defining and displaying a reservoir model|
|US20020067521 *||Jan 28, 2002||Jun 6, 2002||Holzbach Mark E.||Rendering methods for full parallax autostereoscopic displays|
|US20020072883 *||Jun 29, 2001||Jun 13, 2002||Kok-Thye Lim||Method and system for high-resolution modeling of a well bore in a hydrocarbon reservoir|
|US20020073428 *||Dec 8, 2000||Jun 13, 2002||Leonid Gurevich||Downloading and transfer of audio or video data from video broadcasts|
|US20020082813 *||Jun 29, 2001||Jun 27, 2002||Kok-Thye Lim||Method and system for coordinate transformation to model radial flow near a singularity|
|US20020101400 *||Feb 1, 2001||Aug 1, 2002||Stewart Paul Joseph||System and method of interactive evaluation of a geometric model|
|US20020101417 *||Oct 3, 2001||Aug 1, 2002||Burk Wayne Eric||Programmable sample filtering for image rendering|
|US20020105551 *||Feb 15, 2001||Aug 8, 2002||Yakov Kamen||Method and apparatus for a three-dimensional web-navigator|
|US20020113798 *||Jan 28, 2002||Aug 22, 2002||Yakov Kamen||Modifying object geometry to accommodate information attributes|
|US20020122079 *||Mar 1, 2002||Sep 5, 2002||Yakov Kamen||Multimedia interactive device function selection based upon button depression duration|
|US20020129369 *||Jan 28, 2002||Sep 12, 2002||Yakov Kamen||Mechanism and apparatus for construction of modular-based interactive programming guides and TV portals|
|US20020166122 *||Feb 5, 2002||Nov 7, 2002||Dan Kikinis||Image-oriented electronic programming guide|
|US20020183986 *||May 30, 2001||Dec 5, 2002||Stewart Paul Joseph||System and method for design of experiments using direct surface manipulation of a mesh model|
|US20030055904 *||Dec 31, 2001||Mar 20, 2003||Fujitsu Limited||Rendering calculation processing status monitoring program, and storage medium, apparatus, and method therefor|
|US20030132968 *||Jan 11, 2002||Jul 17, 2003||Stewart Paul Joseph||System and method for virtual interactive design and evaluation and manipulation of vehicle mechanisms|
|US20030172366 *||Dec 31, 2002||Sep 11, 2003||Samsung Electronics Co., Ltd.||Rendering system and method and recording medium therefor|
|US20030200066 *||Apr 23, 2002||Oct 23, 2003||Stewart Paul Joseph||System and method for replacing parametrically described surface features with independent surface patches|
|US20030231213 *||Mar 11, 2003||Dec 18, 2003||Gould Eric J.||Method, apparatus and article of manufacture for displaying content in a multi-dimensional topic space|
|US20040150639 *||Dec 5, 2003||Aug 5, 2004||Samsung Electronics Co., Ltd.||Method and apparatus for encoding and decoding three-dimensional object data|
|US20040196939 *||Apr 1, 2003||Oct 7, 2004||Co Ramon S.||All-Digital Phase Modulator/Demodulator Using Multi-Phase Clocks and Digital PLL|
|US20040208370 *||Apr 16, 2004||Oct 21, 2004||Ken Whatmough||System and method of converting edge record based graphics to polygon based graphics|
|US20050001849 *||May 26, 2004||Jan 6, 2005||Arcas Blaise Aguera Y||System and method for multiple node display|
|US20050033559 *||Nov 21, 2002||Feb 10, 2005||Stefan Wander||Construction method and cad system|
|US20050097603 *||Dec 1, 2004||May 5, 2005||Eagle New Media Investments Llc||Three dimensional light electronic programming guide|
|US20050131938 *||Jan 7, 2005||Jun 16, 2005||Autodesk||Translating objects between software applications which employ different data formats|
|US20050144598 *||Dec 30, 2004||Jun 30, 2005||Stewart Sabadell||Translating objects between software applications which employ different data formats|
|US20050184989 *||Apr 20, 2005||Aug 25, 2005||Arcas Blaise A.Y.||System and method for multiple node display|
|US20050268044 *||Jun 1, 2005||Dec 1, 2005||Arcas Blaise A Y||Efficient data cache|
|US20050270288 *||Mar 17, 2005||Dec 8, 2005||Arcas Blaise A Y||Method for encoding and serving geospatial or other vector data as images|
|US20060061575 *||Sep 22, 2005||Mar 23, 2006||Junichi Kamai||Image processing device and image processing method|
|US20060109266 *||Jun 28, 2005||May 25, 2006||Sensable Technologies, Inc.||Apparatus and methods for haptic rendering using data in a graphics pipeline|
|US20060109267 *||Apr 25, 2005||May 25, 2006||Metavr||Three-dimensional visualization architecture|
|US20060132484 *||Nov 29, 2005||Jun 22, 2006||Samsung Electronics Co., Ltd.||Adaptive rendering apparatus and method using hierarchical structure in 3D image and computer-readable recording medium storing computer program for executing the adaptive rendering method|
|US20060176295 *||Jul 28, 2003||Aug 10, 2006||Lattice Technology, Inc.||3-Dimensional graphics data display device|
|US20060176305 *||Oct 17, 2005||Aug 10, 2006||Arcas Blaise A Y||System and method for managing communication and/or storage of image data|
|US20060227130 *||Mar 21, 2006||Oct 12, 2006||Vijayvardhan Elchuri||Graphical method and system for making drawings directly in three-dimensions on a computer monitor or other display device|
|US20060235941 *||Mar 29, 2006||Oct 19, 2006||Microsoft Corporation||System and method for transferring web page data|
|US20060267982 *||May 22, 2006||Nov 30, 2006||Seadragon Software, Inc.||System and method for exact rendering in a zooming user interface|
|US20060268297 *||May 25, 2005||Nov 30, 2006||Lexmark International, Inc.||Method for constructing a lookup table for converting data from a first color space to a second color space|
|US20060284834 *||Jun 28, 2005||Dec 21, 2006||Sensable Technologies, Inc.||Apparatus and methods for haptic rendering using a haptic camera view|
|US20070047101 *||Sep 29, 2006||Mar 1, 2007||Seadragon Software, Inc.||Methods and apparatus for navigating an image|
|US20070047102 *||Sep 29, 2006||Mar 1, 2007||Seadragon Software, Inc.||Methods and apparatus for navigating an image|
|US20070083499 *||Sep 25, 2006||Apr 12, 2007||Samsung Electronics Co., Ltd.||Method and apparatus for efficiently handling query for 3D display|
|US20070104378 *||Dec 29, 2006||May 10, 2007||Seadragon Software, Inc.||Method for encoding and serving geospatial or other vector data as images|
|US20070182743 *||Apr 18, 2007||Aug 9, 2007||Microsoft Corporation||Displaying visual content using multiple nodes|
|US20070198695 *||Jul 22, 2004||Aug 23, 2007||At&T Corp.||Method for three-dimensional inventory link|
|US20080031527 *||Oct 11, 2005||Feb 7, 2008||Arcas Blaise Aguera Y||System and method for efficiently encoding data|
|US20080050024 *||Sep 19, 2007||Feb 28, 2008||Seadragon Software, Inc.||Method for encoding and serving geospatial or other vector data as images|
|US20080088620 *||Aug 17, 2007||Apr 17, 2008||Sensable Technologies, Inc.||Systems and methods for sculpting virtual objects in a haptic virtual reality environment|
|US20080163300 *||Jan 8, 2008||Jul 3, 2008||Jlb Ventures Llc||Three Dimensional Light Electronic Programming Guide|
|US20080198158 *||Dec 27, 2007||Aug 21, 2008||Hitachi, Ltd.||3D map display system, 3D map display method and display program|
|US20080201737 *||Apr 25, 2008||Aug 21, 2008||Jlb Ventures Llc||Freezing live video frames with interactive links|
|US20080218514 *||Jan 2, 2008||Sep 11, 2008||Sensable Technologies, Inc.||Method and apparatus for generating and interfacing with a haptic virtual reality environment|
|US20080222719 *||Mar 26, 2008||Sep 11, 2008||International Business Machines Corporation||Fine-Grained Authorization by Traversing Generational Relationships|
|US20080263607 *||Apr 28, 2008||Oct 23, 2008||Jlb Ventures, Llc||Downloading and Transfer of Audio or Video Data from Video Broadcasts|
|US20080303816 *||May 30, 2008||Dec 11, 2008||Nokia Corporation||Three dimensional image processing|
|US20090167765 *||Mar 13, 2009||Jul 2, 2009||Research In Motion Limited||System and method of converting edge record based graphics to polygon based graphics|
|US20100020075 *||Jul 6, 2009||Jan 28, 2010||Xydne, Inc.||Apparatus and method for creating a virtual three-dimensional environment, and method of generating revenue therefrom|
|US20100053170 *||Mar 4, 2010||Airbus Espana, S.L.||Methods and systems for adapting a multi-block structured mesh topology of an object to a modification in its geometry|
|US20100077434 *||Mar 25, 2010||Jlb Ventures Llc||Three Dimensional Light Electronic Programming Guide|
|US20100088429 *||Sep 3, 2008||Apr 8, 2010||Chih-Hsing Chu||Method for constructing a decomposition data structure of multiple levels of detail design feature of 3d cad model and streaming thereof|
|US20100138792 *||Feb 4, 2010||Jun 3, 2010||Jlb Ventures Llc||Navigating content|
|US20100138793 *||Dec 2, 2008||Jun 3, 2010||Microsoft Corporation||Discrete objects for building virtual environments|
|US20100162309 *||Mar 3, 2010||Jun 24, 2010||Jlb Ventures Llc||Navigating content|
|US20100175087 *||Mar 17, 2010||Jul 8, 2010||Jlb Ventures Llc||Notifying Users of Interactive Functions|
|US20110158611 *||Dec 28, 2010||Jun 30, 2011||Jlb Ventures Llc||Systems and methods for content storage|
|US20110199384 *||Aug 18, 2011||Jlb Ventures Llc||Modifying screen objects|
|US20110202959 *||Aug 18, 2011||Jlb Ventures Llc||Navigating programming content via multi-dimensional surfaces|
|US20120246082 *||Sep 27, 2012||Extraction Zones LLC||Passenger Extraction Program and a Method of Extraction|
|US20120256915 *||Oct 11, 2012||Jenkins Barry L||System and method of procedural visibility for interactive and broadcast streaming of entertainment, advertising, and tactical 3d graphical information using a visibility event codec|
|US20130100171 *||Apr 25, 2013||Sony Computer Entertainment Inc.||Image Processing Apparatus, Content Creating Support Apparatus, Image Processing Method, Content Creating Support Method, And Data Structure of Image File|
|US20130201339 *||Feb 8, 2012||Aug 8, 2013||Honeywell International Inc.||System and method of optimal video camera placement and configuration|
|US20130342537 *||Dec 7, 2012||Dec 26, 2013||Google Inc.||Movement Based Level of Detail Adjustments|
|US20140025203 *||Jun 19, 2013||Jan 23, 2014||Seiko Epson Corporation||Collision detection system, collision detection data generator, and robot|
|US20140330419 *||Jul 18, 2014||Nov 6, 2014||Stratasys, Inc.||Solid identification grid engine for calculating support material volumes, and methods of use|
|US20150193476 *||Jan 6, 2014||Jul 9, 2015||International Business Machines Corporation||Limiting the rendering of instances of recursive elements in view output|
|US20150193479 *||Nov 4, 2014||Jul 9, 2015||International Business Machines Corporation||Limiting the rendering of instances of recursive elements in view output|
|US20150235409 *||Feb 13, 2015||Aug 20, 2015||Autodesk, Inc||Techniques for cut-away stereo content in a stereoscopic display|
|DE10300527A1 *||Jan 9, 2003||Jul 22, 2004||Realtime Technology Ag||Display system for virtual three-dimensional scenes, adjusts spatial angles of camera model when trackball is rotated, so that view of virtual three-dimensional scene varies accordingly|
|EP1128328A2 *||Jan 31, 2001||Aug 29, 2001||Mitsubishi Denki Kabushiki Kaisha||Rendering pipeline for surface elements|
|EP1501036A1 *||Jul 22, 2004||Jan 26, 2005||AT&T Corp.||Method for Three-Dimensional Inventory Link|
|EP1672590A2 *||Dec 14, 2005||Jun 21, 2006||Samsung Electronics Co.,Ltd.||Adaptive rendering apparatus and method using a hierarchical structure in a 3D image|
|WO2001078014A1 *||Apr 3, 2001||Oct 18, 2001||Isurftv||Real world/virtual world correlation system using 3d graphics pipeline|
|WO2004109446A2 *||May 28, 2004||Dec 16, 2004||Sand Codex, Llc||A system and method for multiple node display|
|WO2004109446A3 *||May 28, 2004||Mar 10, 2005||Sand Codex Llc||A system and method for multiple node display|
|WO2012012161A2 *||Jun 29, 2011||Jan 26, 2012||Barry Lynn Jenkins||System and method of from-region visibility determination and delta-pvs based content streaming using conservative linearized umbral event surfaces|
|WO2012012161A3 *||Jun 29, 2011||Mar 15, 2012||Barry Lynn Jenkins||System and method of from-region visibility determination and delta-pvs based content streaming using conservative linearized umbral event surfaces|
|U.S. Classification||345/420, 345/428, 345/424, 382/199|
|International Classification||G06T15/08, G06T15/20, G06T19/00|
|Cooperative Classification||G06T19/003, G06T17/005, G06T2210/36, G06T2200/16|
|European Classification||G06T15/20, G06T19/00N, G06T19/00|
|Jun 27, 1997||AS||Assignment|
Owner name: RESOLUTION TECHNOLOGIES, INC., WASHINGTON
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:DEHMLOW, JAMES;GASS, JOHN;EVANS, LYNNE M.;AND OTHERS;REEL/FRAME:008681/0814;SIGNING DATES FROM 19970624 TO 19970625
|Jun 26, 2003||REMI||Maintenance fee reminder mailed|
|Dec 8, 2003||LAPS||Lapse for failure to pay maintenance fees|
|Feb 3, 2004||FP||Expired due to failure to pay maintenance fee|
Effective date: 20031207