|Publication number||US20020130889 A1|
|Application number||US 10/145,110|
|Publication date||Sep 19, 2002|
|Filing date||May 15, 2002|
|Priority date||Jul 18, 2000|
|Also published as||US7405734, US20020015055, WO2002007092A2, WO2002007092A3, WO2002007092A8|
|Publication number||10145110, 145110, US 2002/0130889 A1, US 2002/130889 A1, US 20020130889 A1, US 20020130889A1, US 2002130889 A1, US 2002130889A1, US-A1-20020130889, US-A1-2002130889, US2002/0130889A1, US2002/130889A1, US20020130889 A1, US20020130889A1, US2002130889 A1, US2002130889A1|
|Inventors||David Blythe, James Foran|
|Original Assignee||David Blythe, Foran James L.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (39), Referenced by (10), Classifications (5), Legal Events (4)|
|External Links: USPTO, USPTO Assignment, Espacenet|
 This application is a continuation-in-part of U.S. patent application Ser. No. 09/888,438, filed Jun. 26, 2001, which claims priority to U.S. Provisional Application No. 60/219,006, filed Jul. 18, 2000. U.S. patent application Ser. Nos. 09/888,438 and 60/219,006 are both incorporated herein by reference in their entireties.
 Not applicable.
 Not applicable.
 1. Field of the Invention
 The invention described herein relates to computer graphics, and more particularly to compositing of images.
 2. Background Art
 A common problem in computer graphics is the efficient compositing of two or more rendered images to produce a single image. The compositing process typically involves the combination of the images, pixel by pixel, and takes into account the respective color coordinates of each pixel. The process can also take into account opacity, intensity, and the relative distances of the images from a viewer.
 One way in which compositing has been accomplished in the past is through software-based readback from frame buffers. In such arrangements, two or more graphics processors each send frames of rendered graphics data to their respective frame buffers. The contents of each frame buffer are then read back, into a compositor module. The compositor module can be a compositing frame buffer or a graphics host. At the compositor module, software-based compositing is performed on corresponding pixels, one pixel from each frame buffer. A pixel from one frame buffer is composited with a pixel from the second frame buffer. This continues until all appropriate pixels in each frame buffer have been composited. A final output, comprising the resultant composited pixels, is then available from the compositor module.
 In some applications such a system can be adequate. In applications requiring faster compositing, however, such a system may not be fast enough. In video applications, for example, where the final output must be produced in real time, such compositing entails unacceptable delay.
 Hence there is a need for a system and method for fast compositing, where the compositing can take place at near real time rates.
 The invention described herein is a system, method, and computer program product for compositing rendered image data in real time or near real time. The input pixel streams that constitute the rendered image data can be video streams, for example. Each input pixel stream can originate from its own graphics processing unit. Each pixel includes a set of color coordinates, such as red, green, and blue (RGB) coordinates, and can also include an alpha value. Compositing is performed by an image combiner that can be implemented in either hardware or software. The image combiner accepts two or more input pixel streams, and performs compositing on corresponding pixels from each input pixel stream. The compositing process takes into account the color coordinates of each corresponding pixel as well as the alpha values or intensity values. The compositing process also uses depth information that defines whether a given pixel is in the foreground or in the background relative to another corresponding pixel. At the pixel level, the result of the compositing process is a resultant pixel based on the corresponding pixels of each input pixel stream.
 The foregoing and other features and advantages of the invention will be apparent from the following, more particular description of a preferred embodiment of the invention, as illustrated in the accompanying drawings.
FIG. 1 is a block diagram illustrating a system for real time or near real time combination of graphics images, according to an embodiment of the invention.
FIG. 2 is a block diagram illustrating an image combiner, according to an embodiment of the invention.
FIG. 3 is a flowchart illustrating a method for real time or near real time compositing of graphics images, according to an embodiment of the invention.
FIG. 4 is a flowchart illustrating the compositing process, according to an embodiment of the invention.
FIG. 5 is a flowchart illustrating a method for real time or near real time compositing of graphics images wherein inputs represent adjacent volumes of a three-dimensional scene, according to an embodiment of the invention.
 The invention described herein is a system, method, and computer program product for the real time compositing of two or more input pixel streams. The input pixel streams can be video streams, for example. Each input pixel stream can originate from its own graphics processing unit. Each pixel includes a set of color coordinates, such as red, green, and blue (RGB) coordinates, plus an alpha value that defines the opacity of the pixel. Compositing is performed by an image combiner that can be implemented in either hardware or software. The image combiner accepts the input pixel streams, and performs compositing on corresponding pixels from each input pixel stream. The compositing process takes into account the color coordinates of each corresponding pixel as well as the alpha values or the intensity values. The compositing process also uses depth information that defines whether a given pixel stream is in the foreground or in the background relative to another corresponding pixel stream. At the pixel level, the result of the compositing process is a resultant pixel stream based on the corresponding pixels of each input pixel stream.
 An exemplary context for the invention is illustrated in FIG. 1. Two graphics processors are shown, processors 105 and 110. Graphics processor 105 produces a frame of image data; likewise, graphics processor 110 produces a frame of image data. The image data of graphics processor 105 is then sent, as input 121, to image combiner 130. Similarly, the rendered image data produced by graphics processor 110 is sent as input 122 to combiner 130. Combiner 130 also receives depth information 132. Depth information 132 indicates the depth order of the inputs 121 and 122. The inputs can be understood as images to be combined; depth information 132 indicates which is in the foreground and which is in the background, relative to a viewer. Depth information 132 generally does not change in the context of a given frame.
 Combiner 130 performs compositing on corresponding pixels of inputs 121 and 122 based on the color coordinates of the corresponding pixels, the alpha values of the corresponding pixels, and the depth information 132. The compositing process is described in greater detail in section III below. A resultant pixel stream 135 is then produced by combiner 130. In an embodiment of the invention, the pixels of pixel stream 135 that constitute a frame are stored in a frame buffer 140. In this embodiment, the output of system 100 is a frame of image data, output 145.
 Note that while the embodiment of FIG. 1 shows two graphics processors and two associated inputs, other embodiments of the invention could feature more than two inputs. In such a case, the compositor performs compositing on all the inputs, taking into account the relative depth information of all inputs. Also, inputs 121 and 122 of FIG. 1 are received by combiner 130 from respective graphics processors 105 and 110. In an alternative context, however, inputs to combiner 130 can include the outputs of other image combiners.
 Image combiner 130 is illustrated in greater detail in FIG. 2. Combiner 130 includes a depth determination module 205. Depth determination module 205 receives depth information 132. As described above, depth information 132 indicates the depth order of the images represented by the input pixel streams. Depth determination module 205 coverts this information to a format usable for blending purposes. Output 210 of depth determination module 205 therefore conveys which input pixel stream is “over” another.
 Output 210 is sent to one or more blending modules, shown in FIG. 2 as blending modules 215 through 225. In an embodiment of the invention, each blending module is associated with a specific color coordinate. In the embodiment of FIG. 3, blending modules 215, 220, and 225 are associated with red, green, and blue coordinates, respectively. Each blending module performs blending of a color from corresponding pixels from respective input pixel streams. Hence, blending module 215 blends red coordinates R1 and R2 from corresponding pixels. The alpha values of the corresponding pixels, α1, and α2, are also input to blending module 215, along with output 210 of depth determination module 205. Inputs to blending modules 220 and 225 are analogous.
 The invention can implement any of several well known blending operations. In a preferred embodiment, blending is performed in depth order, taking into account the opacity of input pixels. Here, colors are blended linearly according to alpha values. In one embodiment, a blended red coordinate, for example, has the value α1R1+(1−α1)α2R2, assuming that input 1 is over input 2. The other color coordinates are blended analogously. Alternatively, if compositing is based on maximum intensity, the value of the resultant red coordinate is the maximum of the red coordinates of the corresponding pixels. The other color coordinates would be blended analogously. Compositing is described more fully in Computer Graphics, Principle and Practice, Foley et al., Addison-Wesley, 1990, pp.835-843 (included herein by reference in its entirety).
 The composited color coordinates from blending modules 215 through 225 are then sent to an output module 230 for formatting as resultant pixel 235.
 While the embodiment of FIG. 2 shows the blending of color coordinates in parallel, alternative embodiments can perform blending of color coordinates in serial using a single blending module.
 The method of the invention is illustrated in FIG. 3. The method begins at step 305. In steps 310 and 315, two inputs, shown here as inputs 1 and 2 are received at a compositor. In step 320, depth information is received by the compositor. In step 325, the compositing of the two inputs is performed in depth order, so as to take into account the depth information received in step 320. The process concludes at step 335.
 Compositing step 325 is illustrated in greater detail in FIG. 4, according to an embodiment of the invention. The compositing process begins at step 405. In step 410, the depth order of the input pixel streams is determined for a frame. This determination is made based on the received depth information. In step 415, the first color coordinates from corresponding pixels are blended, based on the depth order and the pixels' alpha values. Similarly, in steps 420 and 425, the second and third color coordinates are blended. In step 430, a resultant pixel is output. In step 435, a determination is made as to whether additional pixels are to be blended for the current frame. If so, the process repeats from step 415 with a new set of corresponding pixels. Otherwise, the process ends at step 440.
 As mentioned above, the process of compositing pixels based on transparency is known to persons of ordinary skill in the art, and is documented in Computer Graphics, Principle and Practice, Foley et al., supra. Moreover, while the embodiment of FIG. 4 shows the blending of color coordinates in parallel, alternative embodiments can perform blending of color coordinates in serial.
 A particular embodiment of the method of the invention is shown in FIG. 5. The method begins with step 505. In steps 510 and 515, inputs 1 and 2 are received respectively. In this embodiment however, each input represents a sub-volume of three dimensional space from a scene that has been rendered. The sub-volumes represented by inputs 1 and 2 can therefore be thought of as a back “slab” and a front slab, respectively, where the terms front and back refer to the relative positions of each volume from the current frame perspective of the viewer. In step 520, the depth information is received, as before. In step 525, composition is performed on the slabs in depth order, and the resultant pixel stream is output. The process concludes at step 535.
 Note that while the embodiments of FIGS. 3 through 5 shows two graphics processors and two associated inputs, other embodiments of the invention could feature more than two inputs. In such a case, the compositor performs compositing on all the inputs, taking into account depth information relating to the inputs.
 Referring to FIG. 2, compositor 130 may be implemented using hardware, software or a combination thereof. In particular, compositor 130 may be implemented using a computer program, and execute on a computer system or other processing system. An example of such a computer system 600 is shown in FIG. 6. The computer system 600 includes one or more processors, such as processor 604. The processor 604 is connected to a communication infrastructure 606 (e.g., a bus or network). Various software embodiments can be described in terms of this exemplary computer system. After reading this description, it will become apparent to a person skilled in the relevant art how to implement the invention using other computer systems and/or computer architectures.
 Computer system 600 also includes a main memory 608, preferably random access memory (RAM), and may also include a secondary memory 610. The secondary memory 610 may include, for example, a hard disk drive 612 and/or a removable storage drive 614, representing a magnetic medium drive, an optical disk drive, etc. The removable storage drive 614 reads from and/or writes to a removable storage unit 618 in a well known manner. Removable storage unit 618 represents a magnetic medium, optical disk, etc. As will be appreciated, the removable storage unit 618 includes a computer usable storage medium having stored therein computer software and/or data.
 Secondary memory 610 can also include other similar means for allowing computer programs or input data to be loaded into computer system 600. Such means may include, for example, a removable storage unit 622 and an interface 620. Examples of such means also include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM, or PROM) and associated socket, and other removable storage units 622 and interfaces 620 which allow software and data to be transferred from the removable storage unit 622 to computer system 600.
 Computer system 600 may also include a communications interface 624. Communications interface 624 allows software and data to be transferred between computer system 600 and external devices. Examples of communications interface 624 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, etc. Software and data transferred via communications interface 624 are in the form of signals 628 which may be electronic, electromagnetic, optical or other signals capable of being received by communications interface 624. These signals 628 are provided to communications interface 624 via a communications path (i.e., channel) 626. This channel 626 carries signals 628 into and out of computer system 600, and may be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an RF link or other communications channels. In an embodiment of the invention, signals 628 can comprise image data (such as inputs 221 and 222) and depth information (such as depth information 232).
 In this document, the terms “computer program medium” and “computer usable medium” are used to generally refer to media such as removable storage drive 614, a hard disk installed in hard disk drive 612, and signals 628. These computer program products are means for providing software to computer system 600. The invention is directed in part to such computer program products.
 Computer programs (also called computer control logic) are stored in main memory 608 and/or secondary memory 610. Computer programs may also be received via communications interface 624. Such computer programs, when executed, enable the computer system 600 to perform the features of the present invention as discussed herein. In particular, the computer programs, when executed, enable the processor 604 to perform the features of the present invention. Accordingly, such computer programs represent controllers of the computer system 600.
 While various embodiments of the present invention have been described above, it should be understood that they have been presented by way of example, and not limitation. It will be apparent to persons skilled in the relevant art that various changes in detail can be made therein without departing from the spirit and scope of the invention. Thus the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4949280 *||May 10, 1988||Aug 14, 1990||Battelle Memorial Institute||Parallel processor-based raster graphics system architecture|
|US5101475 *||Apr 17, 1989||Mar 31, 1992||The Research Foundation Of State University Of New York||Method and apparatus for generating arbitrary projections of three-dimensional voxel-based data|
|US5187660 *||Dec 1, 1989||Feb 16, 1993||At&T Bell Laboratories||Arrangement for displaying on a display volumetric data|
|US5363475 *||Dec 5, 1989||Nov 8, 1994||Rediffusion Simulation Limited||Image generator for generating perspective views from data defining a model having opaque and translucent features|
|US5434968 *||Jan 27, 1994||Jul 18, 1995||Kubota Corporation||Image data processing device with multi-processor|
|US5459823 *||Jul 3, 1991||Oct 17, 1995||Canon Kabushiki Kaisha||Graphics engine for true colour 2D graphics|
|US5511154 *||Sep 27, 1994||Apr 23, 1996||International Business Machines Corporation||Method and apparatus for managing concurrent access to multiple memories|
|US5544283 *||Jul 26, 1993||Aug 6, 1996||The Research Foundation Of State University Of New York||Method and apparatus for real-time volume rendering from an arbitrary viewing direction|
|US5546530 *||Oct 25, 1994||Aug 13, 1996||Vpl Research, Inc.||Method and apparatus for rendering graphical images using parallel processing|
|US5557711 *||Aug 9, 1995||Sep 17, 1996||Hewlett-Packard Company||Apparatus and method for volume rendering|
|US5640496 *||Feb 4, 1991||Jun 17, 1997||Medical Instrumentation And Diagnostics Corp. (Midco)||Method and apparatus for management of image data by linked lists of pixel values|
|US5734808 *||Sep 28, 1994||Mar 31, 1998||Namco Ltd.||Pipeline processing device, clipping processing device, three-dimensional simulator device and pipeline processing method|
|US5757385 *||Jan 30, 1996||May 26, 1998||International Business Machines Corporation||Method and apparatus for managing multiprocessor graphical workload distribution|
|US5760781 *||Sep 30, 1997||Jun 2, 1998||The Research Foundation Of State University Of New York||Apparatus and method for real-time volume visualization|
|US5764228 *||May 1, 1996||Jun 9, 1998||3Dlabs Inc., Ltd.||Graphics pre-processing and rendering system|
|US5774133 *||Jul 7, 1993||Jun 30, 1998||3Dlabs Ltd.||Computer system with improved pixel processing capabilities|
|US5794016 *||Dec 11, 1995||Aug 11, 1998||Dynamic Pictures, Inc.||Parallel-processor graphics architecture|
|US5841444 *||Mar 21, 1997||Nov 24, 1998||Samsung Electronics Co., Ltd.||Multiprocessor graphics system|
|US5847711 *||Aug 1, 1997||Dec 8, 1998||The Research Foundation Of State University Of New York||Apparatus and method for parallel and perspective real-time volume visualization|
|US5963212 *||Jun 11, 1997||Oct 5, 1999||Bakalash; Reuven||Parallel computing system for modeling and data processing|
|US6008813 *||Aug 1, 1997||Dec 28, 1999||Mitsubishi Electric Information Technology Center America, Inc. (Ita)||Real-time PC based volume rendering system|
|US6016150 *||Jul 30, 1997||Jan 18, 2000||Microsoft Corporation||Sprite compositor and method for performing lighting and shading operations using a compositor to combine factored image layers|
|US6052129 *||Oct 1, 1997||Apr 18, 2000||International Business Machines Corporation||Method and apparatus for deferred clipping of polygons|
|US6064393 *||Jul 30, 1997||May 16, 2000||Microsoft Corporation||Method for measuring the fidelity of warped image layer approximations in a real-time graphics rendering pipeline|
|US6100899 *||Oct 2, 1997||Aug 8, 2000||Silicon Graphics, Inc.||System and method for performing high-precision, multi-channel blending using multiple blending passes|
|US6243098 *||May 25, 1999||Jun 5, 2001||Terarecon, Inc.||Volume rendering pipelines|
|US6246421 *||Dec 6, 1999||Jun 12, 2001||Sony Corporation||Apparatus and method for parallel rendering of image pixels|
|US6339432 *||Sep 24, 1999||Jan 15, 2002||Microsoft Corporation||Using alpha values to control pixel blending|
|US6532017 *||May 20, 1999||Mar 11, 2003||Terarecon, Inc.||Volume rendering pipeline|
|US6559843 *||Feb 26, 1996||May 6, 2003||Compaq Computer Corporation||Segmented ray casting data parallel volume rendering|
|US6570579 *||Nov 9, 1999||May 27, 2003||Broadcom Corporation||Graphics display system|
|US6577317 *||Aug 20, 1999||Jun 10, 2003||Apple Computer, Inc.||Apparatus and method for geometry operations in a 3D-graphics pipeline|
|US6597363 *||Aug 20, 1999||Jul 22, 2003||Apple Computer, Inc.||Graphics processor with deferred shading|
|US6870539 *||Nov 17, 2000||Mar 22, 2005||Hewlett-Packard Development Company, L.P.||Systems for compositing graphical data|
|US6903753 *||Oct 31, 2000||Jun 7, 2005||Microsoft Corporation||Compositing images from multiple sources|
|US20010036356 *||Apr 6, 2001||Nov 1, 2001||Autodesk, Inc.||Non-linear video editing system|
|US20020015055 *||Jun 26, 2001||Feb 7, 2002||Silicon Graphics, Inc.||Method and system for presenting three-dimensional computer graphics images using multiple graphics processing units|
|US20030025699 *||Aug 23, 2002||Feb 6, 2003||Wei Tien En||Method and apparatus for a video graphics circuit having parallel pixel processing|
|US20040021659 *||Jul 31, 2002||Feb 5, 2004||Silicon Graphics Inc.||System and method for decoupling the user interface and application window in a graphics application|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7405734||Jun 26, 2001||Jul 29, 2008||Silicon Graphics, Inc.||Method and system for presenting three-dimensional computer graphics images using multiple graphics processing units|
|US7528890||Apr 30, 2004||May 5, 2009||Yoostar Entertainment Group, Inc.||Interactive system and method for video compositing|
|US7646434||Jun 11, 2009||Jan 12, 2010||Yoostar Entertainment Group, Inc.||Video compositing systems for providing interactive entertainment|
|US7649571||Jun 11, 2009||Jan 19, 2010||Yoostar Entertainment Group, Inc.||Methods for interactive video compositing|
|US8824861||Jun 30, 2009||Sep 2, 2014||Yoostar Entertainment Group, Inc.||Interactive systems and methods for video compositing|
|US8893016 *||Jun 10, 2005||Nov 18, 2014||Nvidia Corporation||Using a graphics system to enable a multi-user computer system|
|US9143721||Apr 24, 2013||Sep 22, 2015||Noo Inc.||Content preparation systems and methods for interactive video systems|
|US20020015055 *||Jun 26, 2001||Feb 7, 2002||Silicon Graphics, Inc.||Method and system for presenting three-dimensional computer graphics images using multiple graphics processing units|
|US20040218100 *||Apr 30, 2004||Nov 4, 2004||Staker Allan Robert||Interactive system and method for video compositing|
|US20060282781 *||Jun 10, 2005||Dec 14, 2006||Diamond Michael B||Using a graphics system to enable a multi-user computer system|
|U.S. Classification||345/629, 345/4|
|Aug 16, 2002||AS||Assignment|
Owner name: SILICON GRAPHICS, INC., CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BLYTHE, DAVID;FORAN, JAMES L.;REEL/FRAME:013202/0336;SIGNING DATES FROM 20020328 TO 20020422
|Aug 19, 2005||AS||Assignment|
Owner name: WELLS FARGO FOOTHILL CAPITAL, INC.,CALIFORNIA
Free format text: SECURITY AGREEMENT;ASSIGNOR:SILICON GRAPHICS, INC. AND SILICON GRAPHICS FEDERAL, INC. (EACH A DELAWARE CORPORATION);REEL/FRAME:016871/0809
Effective date: 20050412
|Oct 24, 2006||AS||Assignment|
Owner name: GENERAL ELECTRIC CAPITAL CORPORATION,CALIFORNIA
Free format text: SECURITY INTEREST;ASSIGNOR:SILICON GRAPHICS, INC.;REEL/FRAME:018545/0777
Effective date: 20061017
|Oct 18, 2007||AS||Assignment|
Owner name: MORGAN STANLEY & CO., INCORPORATED,NEW YORK
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:GENERAL ELECTRIC CAPITAL CORPORATION;REEL/FRAME:019995/0895
Effective date: 20070926