WO2006114898A1 - 3d image generation and display system - Google Patents

3d image generation and display system Download PDF

Info

Publication number
WO2006114898A1
WO2006114898A1 PCT/JP2005/008335 JP2005008335W WO2006114898A1 WO 2006114898 A1 WO2006114898 A1 WO 2006114898A1 JP 2005008335 W JP2005008335 W JP 2005008335W WO 2006114898 A1 WO2006114898 A1 WO 2006114898A1
Authority
WO
WIPO (PCT)
Prior art keywords
images
data
image
display
generating
Prior art date
Application number
PCT/JP2005/008335
Other languages
French (fr)
Inventor
Masahiro Ito
Original Assignee
Yappa Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yappa Corporation filed Critical Yappa Corporation
Priority to US11/912,669 priority Critical patent/US20080246757A1/en
Priority to BRPI0520196-9A priority patent/BRPI0520196A2/en
Priority to PCT/JP2005/008335 priority patent/WO2006114898A1/en
Priority to CA002605347A priority patent/CA2605347A1/en
Priority to AU2005331138A priority patent/AU2005331138A1/en
Priority to EP05737019A priority patent/EP1877982A1/en
Publication of WO2006114898A1 publication Critical patent/WO2006114898A1/en
Priority to NO20075929A priority patent/NO20075929L/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00Manipulating 3D models or images for computer graphics
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/003D [Three Dimensional] image rendering
    • G06T15/10Geometric effects
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/207Image signal generators using stereoscopic image cameras using a single 2D image sensor
    • H04N13/221Image signal generators using stereoscopic image cameras using a single 2D image sensor using the relative movement between cameras and objects
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/243Image signal generators using stereoscopic image cameras using three or more 2D image sensors
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/254Image signal generators using stereoscopic image cameras in combination with electromagnetic radiation sources for illuminating objects
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/302Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays
    • H04N13/305Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays using lenticular lenses, e.g. arrangements of cylindrical lenses
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/302Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays
    • H04N13/31Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays using parallax barriers
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/324Colour aspects
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/356Image reproducers having separate monoscopic and stereoscopic modes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/363Image reproducers using image projection screens

Definitions

  • the present invention relates to a 3D image generation and display system that generates a three-dimensional (3D) object for displaying various photographic images and computer graphics models in 3D, and for editing and processing the 3D objects for drawing and displaying 3D scenes in a Web browser.
  • 3D three-dimensional
  • 3D objects used in 3D displays.
  • One such technique that uses a 3D scanner for modeling and displaying 3D objects is the light-sectioningmethod (implementedbyproj ecting a slit of light) andthe likewell known in the art .
  • Fig. 13 (a) is a schematic diagram showing a conventional 3D modeling apparatus employing light sectioning.
  • ACCD camera captures images when a slit of light is projected onto an object from a light source. By scanning the entire object being measured while gradually changing the direction in which the light source projects the slit of light, an image such as that shown in Fig. 13 (b) is obtained. 3D shape data is calculated according to the triangulation method from the known positions of the light source andcamera. However, since the entireperiphery of the object cannot be rendered in three dimensions with the light-sectioning method, it is necessary to collect images around the entire periphery of the object by providing a plurality of cameras, as shown in Fig. 14, so that the object can be imaged with no hidden areas .
  • the 3D objects created through these methods must then be subjected to various effects applications and animation processes for displaying the 3D images according to the desired use, as well as various data processes required for displaying the objects three-dimensionally in a Web browser. For example, it is necessary to optimize the image by reducing the file size or the like to suit the quality of the communication line.
  • 3D image display is a liquid crystal panel or a display used in game consoles and the like to display 3D images in which objects appear to jump out of the screen.
  • This technique employs special glasses such as polarizing glasses with a different direction of polarization in the left and right lenses.
  • polarizing glasses with a different direction of polarization in the left and right lenses.
  • left and right images are captured from the same positions as when viewed with the left and right eyes, and polarization is used so that the left image is seen only with the left eye and the right image only with the right eye.
  • Other examples include devices that usemirrors orprisms .
  • these 3D image displays have the complication of requiring viewers to wear glasses and the like.
  • Fig. 15 is a schematic diagram showing this 3D image signal generator.
  • the 3D image signal generator includes a backlight 1 including light sources 12 disposed to the sides in a side lighting method; a lenticular lens 15 capable of moving in the front-to-rear direction; a diffuser 5 for slightly diffusing incident light; and an LCD 6 for displaying an image.
  • the LCD 6 has a structure well known in the art in which pixels P displaying each of the colors R, G, and B are arranged in a striped pattern.
  • the lenticular lens 15 makes the sub-pixel array on the LCD 6 viewed from a right eye 11 appear differently from a sub-pixel array viewed from a left eye 10.
  • the left eye 10 can only see sub-pixels of even columns 0, 2, 4, ..., while the right eye 11 can only see sub-pixels of odd columns 1, 3, 5, ....
  • the 3D image signal generator generates a 3D image signal from image signals for the left image and right image captured at the positions of the left and right eyes and supplies these signals to the LCD 6.
  • the stereoscopic display image 20 is generated by interleaving RGB signals from a left image 21 and a right image 22.
  • the 3D image signal generator configures rgb components of a pixel PO in the 3D image signal from the r and b components of the pixel PO in the left image signal and the g component of the pixel PO in the right image signal, and configures rgb components of a pixel Pl in the 3D image signal (center columns) from the g component of the pixel Pl in the left image signal and the r and b components of the pixel Pl in the right image signal.
  • rgb components of a k th (where k is 1, 2, ...) pixel in the 3D image signal are configured of the r and b components of the k th pixel in the left image signal and the g component of the k th pixel in the right image signal
  • the rgb components of the k+l th image pixel in the 3D image signal are configured of the g component of the k+l th pixel in the left image signal and the r andb components of the k+l th pixel in the right image signal.
  • the 3D image signals generated in this method can display a 3D image compressed to the same number of pixels in the original image.
  • a 3D image can be displayed.
  • the display can be switched between a 3D and 2D display by adjusting the position of the lenticular lens 15. While the example described above in Fig. 15 has the lenticular lens 15 arranged on the back surface of the LCD 6, a "stereoscopic image display device" disclosed in patent reference 2 (Japanese unexamined patent application publication No. Hll-72745) gives an example of a lenticular lens disposed on the front surface of an LCD. As shown in Fig.
  • the stereoscopic image display device has a parallax barrier (a lenticular lens is also possible) 26 disposed on the front surface of an LCD 25.
  • pixel groups 27R, 27G, and 27B are formed from pairs of pixels for the right eye (Rr, Gr, and Br) driven by image signals for the right eye, and pixels for the left eye (RL, GL, and BL) driven by image signals for the left eye.
  • a means for compressing and combining these signals is used to rearrange the R and L signals in an alternating pattern (R, L, R, L, ...) to form a single stereoscopic image, as shown in Fig. 20 (c) . Since the combinedright and left signals mustbe compressed by half, the actual signal for forming a single stereoscopic image is configured of pairs of image data in different colors for the left and right eyes, as shown in Fig. 20 (d) . In this example, the display is switched between 2D and 3D by switching the slit positions in the parallax barrier.
  • Patent reference 1 Japanese unexamined patent application publication No. H10-271533
  • Patent reference 2 Japanese unexamined patent application publication No. Hll-72745
  • the 3D scanning method illustrated in Figs. 13 and 14 uses a large volume of data and necessitates many computations, requiring a long time to generate the 3D object.
  • the device is complex and expensive. The device also requires special expensive software for applying various effects and animation to the 3D object.
  • a 3D image generation and display system that uses a 3D scanner employing a scanning table method for rotating the object, in place of the method of collecting photographic data through a plurality of cameras disposed around the periphery of the obj ect, in order to generate precise 3D objects based on a plurality of different images in a short amount of time and with a simple construction.
  • This 3D image generation and display system generates a Web-specific 3D object using commercial software to edit andprocess the major parts of the 3D obj ect in order to rapidly draw and display 3D scenes in a Web browser.
  • the format of the left and right parallax signals differs when the format of the display devices differ, as in the system for switching between 2D and 3D displays when using the same liquidcrystal panel by moving the lenticular lens shown in Fig. 15 and the system for fixing the parallax barrier shown in Fig. 19.
  • the format of the left and right parallax signals differs for all display devices having different formats, such as the various display panels, CRT screens, 3D shutter glasses, and projectors .
  • the format of the left and right parallax signals also differs when using different image signal formats, such as the VGA method or the method of interlacing video signals.
  • the left and right parallax signals are created from two photographic images taken by two digital cameras positioned to correspond to left and right eyes .
  • the format and method of generating left and right parallax data differs when the format of the original image data differs, such as when creating left and right parallax data directly using left and right parallax data created by photographing an object and character images created by computer graphics modeling or the like.
  • a 3D image generation and display system is configured of a computer system for generating three-dimensional (3D) objects "used to display 3D images in a Web browser, the 3D image generation and display system comprising 3D object generating means for creating 3D images from a plurality of different images and/or computer graphics modeling and generating a 3D object from these images that has texture and attribute data; 3D description file outputtingmeans for converting the format of the 3D object generated by the 3D object generating means andoutputtingthe dataas a 3D description£ile for displaying 3D images according to a 3D graphics descriptive language; 3D obj ect processingmeans for extracting a 3D obj ect from the 3D description file, setting various attribute data, editing and processing the 3D object to introduce animation or the like, and outputting the resulting data again as a 3D description file or as a temporary file for setting attributes; texture processing means for extracting textures from the 3D description file, editing and processing
  • executable file generatingmeans for generating an executable file comprising a Web page and one or a plurality of programs including scripts, plug-ins, and applets for drawing and displaying 3D scenes in a Web browser with stereoscopic images produced from a plurality of combined images assigned with a prescribed parall ax, based on the behavior data and the Web 3D objects generated, edited, and processed by the means described above.
  • 2 comprises a turntable on which an object is mounted! and rotated either horizontally or vertically; a digital camera for capturing images of an object mounted on the turntable and creating digital image files of the images; turntable controllingmeans for rotating the turntable to prescribed positions; photographing means using the digital camera to photograph an object set in prescribed positions by the turntable controlling means; successive image creating means for creating successively creating a plurality of image files using the turntable controlling means and the photographing means; and 3D object combining means for generating 3D images based on the plurality of image files cr-eated by the successive image creating means and generating a 3D object having texture and attribute data from the 3D images for di_splaying the images in 3D.
  • 3 generates 3D images according to a silhouette method that estimates the three-dimensional shape of an obj ectusing silhouette data from a plurality of images taken by a single cam-era around the entire periphery of the object as the object is rrotated on the turntable.
  • the 3D object generating means according to Claim 4 generates a single 3D image as a composite scene oUotained by combining various image data, including images taken b;y a camera, images produced by computer graphics modeling, images scanned by a scanner, handwritten images, image data stored on other storage media, and the like.
  • the executable file generating means comprises automatic left and right parallax data generating means for automatically generating left and right parrallax data for drawing and displaying stereoscopic images according to a rendering function based on right eye images and left eye images assigned a parallax from a prescribed camera position; parallax data compressing means for compressing each of the lefft and right parallax data generated by the automatic left and rigtit parallax data generatingmeans ; parallax data combining means fo r combining the compressed left and right parallax data; pairallax data expandingmeans for separating the combined left andri ⁇ rht parallax data into left and right sections and expanding the data to be displayed on a stereoscopic image displaying device; and display data converting means for converting the data to be displayed according to the angle of view (aspect ratio) of the stereoscopic image displaying device.
  • the automatic left and right pairallax data generatingmeans according to Claim 6 automatically generates left and right parallax data corresponding to a 3D image g-enerated by the 3D object generating means based on a virtual camera set by a rendering function.
  • the parallax data compressing means according to Claim 7 compresses pixel data for left and right pa.rallax data by skipping pixels.
  • CTR screen 8 employs at least one of a CRT screen, liquid crystal panel, plasma display, EL display, and projector.
  • the 3D image generation and display system of the present invention can configure a computer systemthat generates 3D objects tobe displayedon a 3Ddisplay.
  • the 3D image generation and display system has a simple construction employing a scanning table system to model an object placed on a scanning table by collecting images around the entire periphery of the object with a single camera as the turntable is rotated. Further, the 3D image generation and display system facilitates the generation of hig"li-quality 3D objects by taking advantage of common software sold commercially.
  • the 3D image generation and display system can also display animation in a Web browser by installing a special plug-in for drawing and displaying 3D scenes in a Web browser or by generating applets for effectively displaying 3D images in a Web browser.
  • the 3D image generation and display system can also constitute a display program capable of displaying ster&oscopic images according to LR parallax image data, 3D images of the kind that do not "jump out” at the viewer, and common 2D images on the same display device.
  • Fig. 1 is a flowchart showing steps in a process performed by a 3D image generation and display system according to a first embodiment of the present invention.
  • a 3D scanner described later is used to form a plurality of 3D images.
  • a 3D obj ect is generated from the 3D images and converted to the standard Virtual RealityModeling Language (VRML; a language for describing 3D graphics) format.
  • VRML Virtual RealityModeling Language
  • the converted 3D object in the oiatputted VRML file is subjected to various processes for producing a Web
  • 3D object and a program file that can be executed in a Web browser .
  • a 3D scanner of a 3D obj ect generating means employing a digital camera captures images of a real object, olotaining twenty-four 3D images taken at varying angles of 15 degr ⁇ ees, for example (SlOl) .
  • the 3D object generating means genera.tes a 3D object from these images and 3D description file outputti_ng means converts the 3D object temporarily to the VRML format (Sl_02) .
  • 3D ScanWare product name
  • a similarprogramcanbe used for crea.ti.ng 3D images, generating 3D objects, and producing VRML files.
  • the 3D object generated with a 3D authoring software (such as a software mentioned below) is extracted from the VRML file and subjected to various editing and processing by 3D oloject processing means (S103) .
  • the 3D object is saved again as a 3D description file in the VRML format or is temporarily s "tored in a storage device or area of memory as a temporary file for set tting attributes.
  • the number of fraicies or time can be set in key frame animation for moving an obj ect provided in the 3D scene at intervals of a certain number of frames.
  • Animation can also be created using such techniques as path animation and character studio for creating a path, such as a Nurbs CV curve, along which an object is to be moved.
  • path animation and character studio for creating a path, such as a Nurbs CV curve, along which an object is to be moved.
  • the user extracts texture images appli_ed to various objects in the VRML file, edits the texture images for color, texture mapping, or the like, reduces the number of colors, modifies the region and location/position where the texti ⁇ re is applied, or performs other processes, and saves the resulting data as a texture file (S104) .
  • Texture editing and processing can be done using commercial image editing software, such as Photoshop (product name) .
  • 3D effects applying means are used to extract various 3D objects from the VRML file and to use the extracted objects in combination with 3ds max or similar software and various plug-ins in order to process the 3D objects and apply various effects, such as lighting andmaterial properties.
  • the resulting data is either re-stored as a 3D description file in the VRML format or saved as a temporary file for applyingeffects (S105) .
  • the 3D objects have undergone processes to be displayed as animation on a Web page and processes for reducing the file size as a pre-process in the texture image process or "the like.
  • the following steps cover processes for reducing and optimizing the object size and file size in order to actually display the objects in a Web browser.
  • Web 3D object generating means extracts 3D objects, texture images, attributes, animation data, and other rendering" elements from the VRML and temporary files created during edd-ting and processing and generates Web 3D objects for displaying 3D images on the Web (S106) .
  • behavior data generating means generates behavior data as a scenario for dispLaying the Web 3D object as animation (S107) .
  • executable file generatingmeans generates an executable file in the form ofplug-in software for a Web browser or a program combining a Ja ⁇ "a Applet, Java Script, and the like to draw and display images in a W «eb browser based on the above data for displaying 3D images (SL08) .
  • the VRML format which is supported b>y most 3D software programs
  • the system can also optimize the image for use on the Web based on the transfer- rate of the communication line or, when displaying images on a_ Web browser of a local computer, can edit and process the images appropriately according to the display environment, thereby controlling image rendering to be effective and achieve optimal- quality in the display environment.
  • Fig.2 is a schematic diagram showing the 3D obj ect generating means of the 3D image generation and display system described above with reference to Fig. 1.
  • the Web 3D object generating means in Fig. 2 includes a. turntable 31 that supports an object 33 (corresponding to the "object” in the claims section and referred to as an "object” oar “real object” in this specification) and rotates 360 degrees foztr scanning the object 33; a background panel 32 of a single primary/ color, such as green or blue; a digital camera 34, such as a CCD ⁇ lighting 35; a table rotation controller 36 that rotates the turntable 31 through servo control; photographing means 37 four controlling and calibrating the digital camera 34 and lightincj 35, performing gamma correction andother image processing of image data and capturing images of the object 33; and successive image creating means 38 for controlling the angle of table rotation and sampling and collecting images at prescribed angles.
  • Thes ⁇ components constitute a 3D modeling device employing a scanning- table and a single camera for generating a series of images viewe ⁇ d from a plurality of angles. At this point, the images are modified according to needusing commercial editing software such as AutoCAID and STL (product names) .
  • a 3D object combining means 39 extract s silhouettes from the series of images and creates 3D images using a silhouette method or the like to estimate 3D shapes in order to generate 3D object data.
  • the camera is calibrated by calculating, for example, correlations between the world coordinate system, camera coordinate system, and image coordinate system.
  • the points in the image coordinate system are converted to points in the world coordinate system in order to process the images in software.
  • the successive image creating means 38 coordinates with the table rotation controller 36 to control the rotational angle of the turntable for a prescribed number of scans (scanning images every 10 degrees for 36 scans or every 5 degrees for 72 scans, for example) , while the photographing means 37 captures images of the object 33.
  • Silhouette data of the object 33 is acquired from the captured images by obtaining a background difference, which is the difference between images of the background panel 32 taken previously and the current camera image.
  • a silhouette image of the object is derived from the background difference and camera parameters obtained from calibration.
  • 3D modeling is then performed on the silhouette image by placing a cube having a recursive octal tree structure in a three-dimensional space, for example, and determining intersections in the silhouette of the object.
  • Fig. 3 is a flowchart that gives a more specific/concrete example — which is in accordance with steps in the process for converting 3D images shown in Fig. 1 — so that the steps shown in Fig. 1 can be better/further explained.
  • the process in Fig. 3 is implemented by a Java Applet that can display 3D images in a Web browser without installing a plug-in for a viewer, such as Live 3D.
  • all the data necessary for displaying interactive 3D scenes is provided on a Web server.
  • the 3D scenes are displayedwhen the server is accessed from a Web browser running on a client computer.
  • 3ds max or the like is used to modify motion, camera, lighting, and material properties and the like in the generated 3D objects.
  • the 3D objects or the entire scene is first converted to the VRML format (S202) .
  • the resulting VRML file is inputted into a 3DA system (S203; here, 3DA describes 3D images that are displayed as animation on a Web browser using a Java Applet, and the entire system including the authoring software for Web-related editing and processing is called a 3DA system) .
  • the 3D scene is customized, and data for rendering the image with the 3DA applet is provided for drawing and displaying the 3D scene in the Web browser (S205) .
  • All 3D scene data is compressed at one time and saved as a compressed 3DA file (S206) .
  • the 3DA system generates a tool bar file for interactive operations and an HTML file, where the HTML page reads the tool bar file into the Web browser, so that the tool bar file is executed, and that 3D scenes are displayed in a Web browser.
  • the new Web page (HTML document) includes an applet tag for calling the 3DA applet. Java Script code for accessing the 3DA applet may be added to the HTML document to improve operations and interactivity (S209) . All files required for displaying the 3D scene created as described above are transferred to the Web server.
  • These files include the Web page (HTML document) possessing the applet tag for calling the 3DA applet, a tool bar file for interactive operations as an option, texture image files, 3DA scene files, and the 3DA applet for drawing and displaying 3D scenes (S210) .
  • HTML document Web page
  • tool bar file for interactive operations as an option
  • texture image files 3DA scene files
  • 3DA applet for drawing and displaying 3D scenes
  • the Web browser When a Web browser subsequently connects to the Web server and requests the 3DA applet, the Web browser downloads the 3DA applet from the Web server and executes the applet (S211) . Once the 3DA applet has been executed, the applet displays a 3D scene with which the user can perform interactive operations, and the Web browser can continue displaying the 3D scene independently of the Web server (S212) .
  • a 3DA Java applet file is generated after converting the 3D objects to the Web-based VRML, and the Web browser downloads the 3DA file and 3DA applet.
  • a plug-in for a viewer such as Live 3D (product name) and process the VRML 3D description file directly.
  • a company can easily create a Web site using three-dimensional and moving displays of products for e-commerce and the like.
  • an e-commerce product the following description covers the starting of a commercial Web site for printers, such as that shown in Fig. 4.
  • the successive image creating means 38 sets the number of images to sample, so that the photographing means 37 captures thirty-six images assuming a sampling angle of 10 degrees
  • the 3D object combining means 39 calculates the background difference between the camera position and the background panel 32 that has been previously photographed and converts image data for each of the thirty-six images of the printer created by the successive image creating means 38 to world coordinates by coordinate conversion among world coordinates, camera coordinates, and image positions.
  • the silhouette method for extracting contours of the object is used to model the outer shape of the printer and generate a 3D obj ect of the printer .
  • This object is temporarily outputted as a VRML file. At this time, all 3D images to be displayed on the Web are created, including a rear operating screen, left and right side views, top and bottom views, a front operating screen, and the like.
  • the 3D object processing means, texture processing means, and 3D effects applying means extract the generated 3D image data from the VRML file, analyze relevant parts of the data, generate 3D objects, apply various attributes, perform animation processes, and apply various effects and other processes, such as lighting and surface formation through color, material, and texture mapping properties.
  • the resulting data is savedas a texture file, a temporary file for attributes, atemporary file for effects, and the like.
  • the behavior data generating means generates data required for movement in all 3D description files used on the printer Web site. Specifically, the behavior data generating means generates a file for animating the actual operating screen in the setup guide or the like.
  • the 3D scene data created above can be displayed in the Web browser. It is also possible to use a method for processing the 3D scene data in the Web browser only, without using a viewer.
  • a 3DA file for a Java applet is downloaded to the Web browser for drawing and displaying the 3D scene data extracted from the VRML file, as described above.
  • the user can operate a mouse to click on items in a setup guidemenudisplayed in thebrowser to display an animation sequence in 3D.
  • This animation may illustrate a series of operations that rotate a button 63 on a cover 62 of the printer 60 to detach the cover 62 and install a USB connector 66.
  • a 3D animation sequence will be played in which the entire printer is rotated to show the front surface thereof (not shown in the drawings) .
  • a top cover 61 of the printer 60 is opened, and a cartridge holder in the printer 60 moves to a center position. Black and color ink cartridges are inserted into the cartridge holder, and the top cover 61 is closed.
  • "Maintenance Screen” a 3D image is displayed in which all of the plastic covers have been removed to expose the inner mechanisms of the printer (not shown) . In this way, the user can clearly view spatial relationships among the driver module, scanning mechanism, ink cartridges, and the like in three dimensions, facilitating maintenance operations.
  • the 3D image generation and display system can be used for other applications, such as trying on apparel.
  • the 3D image generation and display system can enable the user to try on a suit from a women' s clothing store or the like.
  • the user can click on a suit worn by a model; change the size and color of the suit; view the modeled suit from the front, back, and sides; modify the shape, size, and color of the buttons; and even order the suit by e-mail.
  • Various merchandise such as sculptures or other fine art at auctions and everyday products, can also be displayed in three-dimensional images that are more realistic than two-dimensional images.
  • Fig. 5 is a schematic diagram showing a 3D image generation and display system according to a second embodiment of the present invention.
  • the second embodiment further expands the 3D image generation and display system to allow the 3D images generated and displayed on a Web page in the first embodiment to be displayed as stereoscopic images using other 3D display devices.
  • the 3D image generation and display system in Fig.5 includes a turntable-type 3D object generator 71 identical to the 3D object generating means of the first embodiment shown in Fig. 2.
  • This 3D object generator 71 produces a 3D image by combining images of an object taken with a single camera while the object is rotated on a turntable.
  • the 3D image generation and display system of the second embodiment also includes a multiple camera 3D object generator 72.
  • the 3D object generator 72 generates 3D objects by arranging a plurality of cameras from two stereoscopic cameras corresponding to the positions of left and right eyes to n cameras (while not particularly limited to any number, a more detailed image can be achievedwith a largernumber of cameras) arounda stationaryobject .
  • the 3D image generation and display system also includes a computer graphics modeling 3D obj ect generator 73 for generating a 3D obj ect while performing computer graphics modeling through the graphics interface of a program, such as 3ds max.
  • the 3D object generator 73 is a computer graphics modeler that can combine scenes with computer graphics, photographs, or other data.
  • 3D scene data is extracted from the VRML files using a Web authoring tool, such as YAPPA 3D Studio (product name) .
  • the authoring software is used to edit and process the 3D objects and textures; add animation; apply, set, and prooess other effects, such as camera and lighting effects; and gene-irate Web 3D objects and their behavior data for drawing and disp laying interactive 3D images in a Web browser.
  • An example for creating Web 3D files was described in S202-S210 of Fig. 3.
  • Means 75-79 are parts of the executable fJL Ie generatingmeans used in S108 of Fig. 1 that apply left and r-ight parallax data for displaying stereoscopic images.
  • a renderer 75 applies rendering functions to generate left and rig-ht parallax images (LR data) required for displaying stereoscopic images.
  • An LR data compressing/combining means 76 compresses the LR data generated by the renderer 75, rearranges the data in a. combining process and stores the data in a display frame buffer.
  • An LR data separating/expanding means 77 separates and expands the left and right data when displaying LR data.
  • a data converting means 78 configured of a down converter or the like adjusts the angle of view (aspect ratio and the like) for displaying stereoscopic images so that the LR data can be made compatible witra various 3D display devices.
  • a stereoscopic displaying means 79 displays stereoscopic images based on the LR data and. using a variety of display devices, such as a liquid crystal panel , CRT screen, plasma display, EL (electroluminescent) display, oar projector shutter type display glasses and includes a variety of display formats, such as the common VGA format used in personal computer displays and the like and video formats used for televisions .
  • the 3D obj ect generator 71 is identical to the 3D obj ect gen&rating means described in Fig . 1 .
  • the obj ect 33 for which a 3D image is to be formed is placed on the turntable 31 .
  • the table rotation controller 36 regulates rotations of the turntable 31 , while the digital camera 34 and lighting 35 are controlled t o take sample photographs by the photographing means 37 against a single-color screen, such as a blue screen (the background panel 32 ) as the background .
  • the successive image creating means 38 then performs a process to combine the sampled images .
  • the 3D obj ect combining means 39 extracts silhouettes ( contours ) of the obj ect and generates s a 3D obj ect using a silhouette method or the like to estimate the three-dimensional shape of the obj ect . This method is performed using the following equation, for example . Equation 1
  • Coordinate conversion is pezrformed using camera coordinates Pfp and world coordinates Sp oi a point P to convert three-dimensional coordinates at vertices o ⁇ the 3D images to the world coordinate system [x, y, z, r, g, b] .
  • a variety of modeling programs are used to model the resulting- coordinates.
  • the 3D data generated fromthis process is saved in an image database (not shown) .
  • the 3D object generator 72 is a system for capturing images of an object by placing a plurality of cameras aro ⁇ md the object. For example, as shown in Fig. 6, six cameras (firs t through sixth cameras) are disposed around an object . Acontrol computer obtains photographic data from the cameras via USB hubs and reproduces 3D images of the object in real-time on first and secondproj ectors .
  • the 3D object generator 72 is not limited to six cameras, but may capture images with any number of cameras.
  • the system generates 3D objects in the world coordinate system from tlkie plurality of overlappingphotographs obtained fromthese camera ⁇ and falls under the category of image-based rendering (IBR) . Hence, the construction and process of this system is considerably more complicated than that of the 3D object generator ⁇ 71. As with the 3D object generator 71, the generated data is saved ⁇ n the database.
  • the 3D object generator 73 focuses primarily on computer graphics modeling using modeling software, sucb_ as 3ds max and
  • the composite image scene can be displaced at a position in which the scene has been shifted 30 degrees from the front by setting the coordinates of the camera angle and position using [X, Y, Z, w] .
  • virtual cameras that can be created include a free camera that can be freely rotated and moved to any position, and a target camera that can be rotated around arx obj ect .
  • the user may do so by setting new properties.
  • the user can quickL y change the viewpoint with the touch of a button by selecting or switching among a group of about ten virtual lenses from WIDE ⁇ to TELE. Lighting settings may be changed in the same way with various functions that can be applied to the rendered image. AILl of the data generated is saved in the database.
  • LR data of parallax signals corresponding to the left and right eyes can be easily acquired using the cameraposition setting function of themodeling software programs described above .
  • the coordinates of the position of each camera are represented by a vector normal to tine object being modeled (a cellular telephone in this example), as shown in Fig.7 (a) .
  • the coordinate for the position of the camera is set to 0; the focusing direction of the camera to a vector OT; and a vector OU is set to the direction upward from the camera and orthogonal to the vector OT.
  • the positions of the left and right eyes (L, R) is calculated according to the following equation 2, where ⁇ is the inclination angle for the left and right eyes (L, R) and d is a distance to a convergence point P for a zero parallax between the left and right eyes. Equation 2
  • the method for calculating the positions described above is not limited to this method but may be any calculating method that achieves the same effects.
  • the coordinates [X, Y, Z, w] can be inputted directly using the method for studying the camera (virtual camera) position described above.
  • the user After setting the positions of the eyes (camera positions) found from the above-described methods in the camera function, the user selects "renderer” or the like in the tool bar of the window displaying the scene to convert and render the 3D scene as a two-dimensional image in order to obtain a left and right parallax image for a stereoscopic display.
  • LR data is not limited to use with composite image scenes, but can also be created for photographic images taken by the 3D object generators 71 and 72.
  • the photographic images can be rendered, saving image data of the obj ect taken around the entire periphery to obtain LR data for left and right parallax images.
  • LR data can easilybe created by rendering various composite scenes.
  • Cnr, Cng, Cnb, Pnr, Png, and Pnb represent the nth
  • LR data for left and right parallax images obtained through this rendering process is generated automatically by calculating coordinates of the camera positions and shadows based on light source data.
  • Various filtering processes are also performed simultaneously but will be omitted from this description.
  • an up/down converter or the like converts the image data to bit data and adjusts the aspect ratio before displaying the image .
  • Fig.8 is an explanatory diagram illustrating amethod of generating simple left and right parallax images.
  • LR data of a character "A" has been created for the left eye.
  • a parallax image for the right eye can be created as a mirror image of the LR data for the left eye simply by reversing the LR data for the left eye. This reversal can be calculated using the following equation 4. Equation 4
  • X represents the X coordinate
  • Y the Y coordinate
  • X' and Y' the new coordinates in the mirror image.
  • Rx and Ry are equal to -1.
  • LR data is inputted into the conventional display device shown in Fig. 19 to display 3D images.
  • the display device shown in Fig. 19 is a liquid crystal panel (LCD) used in a personal computer or the like and employs a VGA display system using a sequential display technique.
  • Fig. 9 is a block diagram showing a parallax image signal processing circuit.
  • a resulting LR composite signal is inputted into a separator 81.
  • the separator 81 performs the same process in reverse, rearranging the image data by separating the R and L rows, as shown in Fig. 20 (c) .
  • This data is uncompressed and expanded by expanders 82 and 83 and supplied to display drivers to adjust the aspect ratios and the like.
  • the drivers display the L signal to be seen only with the left eye and the R signal to be seen only with the right eye, achieving a stereoscopic display. Since the pixels skippedduring compression are lost and. cannot be reproduced, the image data is adjusted using interpolation and the like. This data canbe used on displays in notebook personal computers, liquid crystal panels, direct-view game consoles, and the like.
  • the signal format for the LR data in these cases has no particular restriction.
  • Web 3D authoring tools such as YAPPA 3D Studio are configured to convert image data to LR data according to a. Java applet process .
  • Operating buttons such as those shown in Fig-. 10 can be displayed on the screen of a Web browser by attachirxg a tool bar file to one of Java applets, and downloading the data. (3d scene data, Java applets, and HTML files) from a Web server to the Web browser via a network.
  • the user can manipulate the stereoscopic image displayed in the Web br-owser (a car in this case) to zoom in and out, move or rotate the image, and the like.
  • Theprocess details of the operations for zooming in andout, moving, rotating, and the like are expressed in a transformation matrix. For example, movement canbe representedbyequation 5below. Other operations can be similarly expressed. Equation 5
  • X' and Y' are the new coordinates
  • X and Y are the original coordinates
  • Dx and Dy are tr ⁇ e distances moved in the horizontal and vertical directions respectively.
  • the following example uses a liquid crystal panel (or a CRT screen or the like) as shown in Fig. 19 for placing back video signals.
  • a parallax barrier, lenticular sheet, or the like for displaying stereoscopic images is mounted on the front surface of the displaydevice .
  • the displayprocess will be describedusing the block diagram in Fig. 11 showing a signal processing circuit for parallax images.
  • LR data for left and right parallax images such as that shown in Figs. 20 (a) and 20 (b) generrated according to the automatic generating method of the present invention, is inputted into compressors 90 and 91, respectively.
  • the compressors 90 and 91 compress the images by skipp dng every other pixel in the video signal.
  • a combiner 92 combines and compresses the left and right LR data, as shown in Figs. 2O (c) and 20 (d).
  • a video signal configured of this combined LR data is either transferred to a receiver or recorded on and pLayed back on a recording medium, such as a DVD.
  • a separator 93 pe_rforms the same operation in reverse, separating the combined LR data into left and right signals, as shown in Figs. 20 (c) and 20 ⁇ d) .
  • Expanders 94 and 95 expand the left and right image data to their original form shown in Figs. 20 (a) and 20 (b) .
  • Stereoscopi_c images can be displayed on a display like that shown in Fig.19 because ttie display data is arranged with alternating left video data and rzLght video data across the horizontal scanning lines and in the oirder R, G, and B.
  • the R (red) signal is arranged as "RO (for left) RO (for right), R2 (for left) R2 (for right), R4 C for left) R4 (for right) .
  • the G (green) signal is arranged as ⁇ GO (left) GO (right), G2 (left) G2 (right), .
  • the B (blue) signal is arranged as "BO (left) BO (right) , B2 (left) B2 (right) .
  • a stereoscopic display can be achieved in the same wayusing shutter glasses, having liquid crystal shutters or the like, as ttie display device, by sorting the LR data for parallax image sig ⁇ nals into an odd field and even field and processing the two in synchronization.
  • Fig.12 is a schematic diagramof a home theater tha ⁇ fc includes aprojector screen 101, the surface of whichhas undergone ⁇ a.n optical treatment (such as an application of a silver metal coating) ; two projectors 106 and 107 disposed in front of the projector screen 101; and polarizing filters 108 and 109 disposed one in front of each of the projectors 106 and 107, respectively.
  • Each component of the home theater is controlled by a controller IO 3. If the projector 106 is provided for the right eye and the projector 107 for the left eye, the filter 109 is a type that polari_zes light vertically, while the filter 108 is a type that polarl_zes light horizontally.
  • the home theater also includes a 3D image recorder 104 that supports DVD or another medium (certainly the device may also generate images through modeling) , and a left and right parallax image generator 105 for automatically generating LR data with the display drivers of the present invention based on 3D image data inputted from the 3D image recorder 104.
  • the polarizing filters 108 and 109 project images through the polarizing filters 108 and 109, which polarize the images hLorizontally and vertically, respectively.
  • the viewer can see stereoscopic images since images projectedby the projector
  • the present invention is alsomore user-friendly, since differ ent stereoscopic display software, such as a stereo driver or the like, need not be provided for each different type of hardware, such as a_ personal computer, television, game console, liquid panel display, shutter glasses, and projectors.
  • Fig. 1 is a flowchart showing steps in a process performed by the 3D image generation and display system according to a first embodiment of the present invention
  • Fig.2 is a schematic diagram showing 3D object generating means of the 3D image generation and display system des cribed in Fig. 1;
  • Fig. 3 is a flowchart that shows a process from generation of 3Dobjects to drawinganddisplaying of 3D scenes in aWEBbrowser.
  • Fig. 4 is a perspective view of a printer as an example of a 3D object;
  • Fig.5 is a schematic diagram showing a 3D image generation and display system according to a second embodiment of true present invention
  • Fig. 6 is a schematic diagram showing a 3D image generator of Fig. 5 having 2-n cameras
  • Fig. 7 is an explanatory diagram illustrating a method of setting camera positions in the renderer of Fig. 5;
  • Fig.8 is an explanatory diagram illustrating a prrocess for creating simple stereoscopic images
  • Fig. 9 is a block diagram of an LR data processing circuit in a VGA display
  • Fig. 10 is an explanatory diagram illustrating operations for zooming in and out, moving, and rotating a 3D image
  • Fig. 11 is a block diagram showing an LR data processing circuit of a video signal type display
  • Fig.12 is a schematic diagram showing a stereoscopic display system employing projectors
  • Fig. 13 (a) is a schematic diagram of a conventional 3D modeling display device
  • Fig. 13 (b) is an explanatory diagram illustrating the creation of slit images
  • Fig.14 is a block diagram showing a conventi onal 3D modeling device employing a plurality of cameras
  • Fig. 15 is a schematic diagram of a conventional 3D image signal generator
  • Fig. 16 is an explanatory diagram showing LR data for the signal generator of Fig. 15;
  • Fig. 17 is an explanatory diagram illustr-ating a process for compressing the LR data in Fig. 16;
  • Fig. 18 is an explanatory diagram showj_ng a method of displaying LR data on the display device of Fig. 15;
  • Fig. 19 is a schematic diagram of another conventional stereoscopic image displaying device.
  • Fig.20 is an explanatory diagram showing LU data displayed on the display device of Fig. 19.

Abstract

A 3D image generation and display system facilitating the display of high-quality images in a Web browser comprises means for creating 3D images from a plurality of different images and computer graphics modeling and generating a 3D object from these images that has texture and attribute data; means for converting and outputting the 3D object as a 3D description file in a 3D graphics descriptive language; means for extracting a 3D object and textures from the 3D description file, setting various attribute data, and editing and processing the 3D object to introduce animation or the like and assigning various effects; means for generating various Web-based 3D objects from the 3D data files produced above that are compressed to be displayed in a Web browser and generating behavior data to display 3D scenes in a Web browser with animation; and means for generating an executable file comprising a Web page and Web-based programs such as scripts, plug-ins, and applets for drawing and displaying 3D scenes in a Web browser.

Description

DESCRIPTION
3D IMAGE GENERATION AND DISPLAY SYSTEM
TECHNICAL FIELD
The present invention relates to a 3D image generation and display system that generates a three-dimensional (3D) object for displaying various photographic images and computer graphics models in 3D, and for editing and processing the 3D objects for drawing and displaying 3D scenes in a Web browser.
BACKGROUND ART
There are various systems well known in the art for creating 3D objects used in 3D displays. One such technique that uses a 3D scanner for modeling and displaying 3D objects is the light-sectioningmethod (implementedbyproj ecting a slit of light) andthe likewell known in the art . Thismethodperforms 3Dmodeling using a CCD camera to capture points or lines of light projected onto an obj ect by a laser beam or other light source, and measuring the distance from the camera using the principles of triangulation .
Fig. 13 (a) is a schematic diagram showing a conventional 3D modeling apparatus employing light sectioning.
ACCD camera captures images when a slit of light is projected onto an object from a light source. By scanning the entire object being measured while gradually changing the direction in which the light source projects the slit of light, an image such as that shown in Fig. 13 (b) is obtained. 3D shape data is calculated according to the triangulation method from the known positions of the light source andcamera. However, since the entireperiphery of the object cannot be rendered in three dimensions with the light-sectioning method, it is necessary to collect images around the entire periphery of the object by providing a plurality of cameras, as shown in Fig. 14, so that the object can be imaged with no hidden areas .
Further, the 3D objects created through these methods must then be subjected to various effects applications and animation processes for displaying the 3D images according to the desired use, as well as various data processes required for displaying the objects three-dimensionally in a Web browser. For example, it is necessary to optimize the image by reducing the file size or the like to suit the quality of the communication line.
One type of 3D image display is a liquid crystal panel or a display used in game consoles and the like to display 3D images in which objects appear to jump out of the screen. This technique employs special glasses such as polarizing glasses with a different direction of polarization in the left and right lenses. In this 3D image displaying device, left and right images are captured from the same positions as when viewed with the left and right eyes, and polarization is used so that the left image is seen only with the left eye and the right image only with the right eye. Other examples include devices that usemirrors orprisms . However, these 3D image displays have the complication of requiring viewers to wear glasses and the like. Hence, 3D image displaying systems using lenticular lenses, a parallax barrier, or other devices that allow a 3D image to be seen without glasses have been developed and commercialized. One such device is a λΛ3D image signal generator" disclosed in Patent Reference 1 (Japanese unexamined patent application publication No. H10-271533) . This device improved the 3D image display disclosed in U.S. Patent 5,410,345
(April 25, 1995) by enabling the display of 3D images on a normal LCD system used for displaying two-dimensional images.
Fig. 15 is a schematic diagram showing this 3D image signal generator. The 3D image signal generator includes a backlight 1 including light sources 12 disposed to the sides in a side lighting method; a lenticular lens 15 capable of moving in the front-to-rear direction; a diffuser 5 for slightly diffusing incident light; and an LCD 6 for displaying an image. As shown in a stereoscopic display image 20 in Fig. 16, the LCD 6 has a structure well known in the art in which pixels P displaying each of the colors R, G, and B are arranged in a striped pattern. A single pixel Pk, where k=0-n, is configured of three sub-pixels for RGB arranged horizontally. The color of the pixel is displayed by mixing the three primary colors displayed by each sub-pixel in an additive process .
When displaying a 3D image with the backlight 1 shown in Fig. 15, the lenticular lens 15 makes the sub-pixel array on the LCD 6 viewed from a right eye 11 appear differently from a sub-pixel array viewed from a left eye 10. To describe this phenomenon based on the stereoscopic display image 20 of Fig. 16, the left eye 10 can only see sub-pixels of even columns 0, 2, 4, ..., while the right eye 11 can only see sub-pixels of odd columns 1, 3, 5, .... Hence, to display a 3D image, the 3D image signal generator generates a 3D image signal from image signals for the left image and right image captured at the positions of the left and right eyes and supplies these signals to the LCD 6.
As shown in Fig. 16, the stereoscopic display image 20 is generated by interleaving RGB signals from a left image 21 and a right image 22. With this method, the 3D image signal generator configures rgb components of a pixel PO in the 3D image signal from the r and b components of the pixel PO in the left image signal and the g component of the pixel PO in the right image signal, and configures rgb components of a pixel Pl in the 3D image signal (center columns) from the g component of the pixel Pl in the left image signal and the r and b components of the pixel Pl in the right image signal. With this interleaving process, normally rgb components of a kth (where k is 1, 2, ...) pixel in the 3D image signal are configured of the r and b components of the kth pixel in the left image signal and the g component of the kth pixel in the right image signal, and the rgb components of the k+lth image pixel in the 3D image signal are configured of the g component of the k+lth pixel in the left image signal and the r andb components of the k+lth pixel in the right image signal. The 3D image signals generated in this method can display a 3D image compressed to the same number of pixels in the original image. Since the left eye can only see sub-pixels in the LCD 6 displayed in even columns, while the right eye can only see sub-pixels displayed in odd columns, as shown in Fig. 18, a 3D image can be displayed. In addition, the display can be switched between a 3D and 2D display by adjusting the position of the lenticular lens 15. While the example described above in Fig. 15 has the lenticular lens 15 arranged on the back surface of the LCD 6, a "stereoscopic image display device" disclosed in patent reference 2 (Japanese unexamined patent application publication No. Hll-72745) gives an example of a lenticular lens disposed on the front surface of an LCD. As shown in Fig. 19, the stereoscopic image display device has a parallax barrier (a lenticular lens is also possible) 26 disposed on the front surface of an LCD 25. In this device, pixel groups 27R, 27G, and 27B are formed from pairs of pixels for the right eye (Rr, Gr, and Br) driven by image signals for the right eye, and pixels for the left eye (RL, GL, and BL) driven by image signals for the left eye. By arranging two left and right cameras to photograph an object at left and right viewpoints corresponding to the left and right eyes of a viewer, two parallax signals are created. The example in Figs. 20 (a) and 20 (b) show R and L signals created for the same color. A means for compressing and combining these signals is used to rearrange the R and L signals in an alternating pattern (R, L, R, L, ...) to form a single stereoscopic image, as shown in Fig. 20 (c) . Since the combinedright and left signals mustbe compressed by half, the actual signal for forming a single stereoscopic image is configured of pairs of image data in different colors for the left and right eyes, as shown in Fig. 20 (d) . In this example, the display is switched between 2D and 3D by switching the slit positions in the parallax barrier.
Patent reference 1: Japanese unexamined patent application publication No. H10-271533 Patent reference 2: Japanese unexamined patent application publication No. Hll-72745
DISCLOSURE OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION
However, the 3D scanning method illustrated in Figs. 13 and 14 uses a large volume of data and necessitates many computations, requiring a long time to generate the 3D object. In addition, the device is complex and expensive. The device also requires special expensive software for applying various effects and animation to the 3D object.
Therefore, it is one object of the present invention to provide a 3D image generation and display system that uses a 3D scanner employing a scanning table method for rotating the object, in place of the method of collecting photographic data through a plurality of cameras disposed around the periphery of the obj ect, in order to generate precise 3D objects based on a plurality of different images in a short amount of time and with a simple construction. This 3D image generation and display system generates a Web-specific 3D object using commercial software to edit andprocess the major parts of the 3D obj ect in order to rapidly draw and display 3D scenes in a Web browser.
In the stereoscopic image devices shown in Figs. 15-20, the format of the left and right parallax signals differs when the format of the display devices differ, as in the system for switching between 2D and 3D displays when using the same liquidcrystal panel by moving the lenticular lens shown in Fig. 15 and the system for fixing the parallax barrier shown in Fig. 19. In the same way, the format of the left and right parallax signals differs for all display devices having different formats, such as the various display panels, CRT screens, 3D shutter glasses, and projectors . The format of the left and right parallax signals also differs when using different image signal formats, such as the VGA method or the method of interlacing video signals.
Further, in the conventional technology illustrated in Figs . 15-20, the left and right parallax signals are created from two photographic images taken by two digital cameras positioned to correspond to left and right eyes . However, the format and method of generating left and right parallax data differs when the format of the original image data differs, such as when creating left and right parallax data directly using left and right parallax data created by photographing an object and character images created by computer graphics modeling or the like.
Therefore, it is another object of the present invention to provide a 3D image generation and display system for creating 3D images that generalize the format of left and right parallax signals where possible to create a common platform that can assimilate various input images and differences in signal formats of these input images, as well as differences in the various display devices, and for displaying these 3D images in a Web browser.
MEANS FOR SOLVING THE PROBLEMS
To attain these objects, a 3D image generation and display system according to Claim 1 is configured of a computer system for generating three-dimensional (3D) objects "used to display 3D images in a Web browser, the 3D image generation and display system comprising 3D object generating means for creating 3D images from a plurality of different images and/or computer graphics modeling and generating a 3D object from these images that has texture and attribute data; 3D description file outputtingmeans for converting the format of the 3D object generated by the 3D object generating means andoutputtingthe dataas a 3D description£ile for displaying 3D images according to a 3D graphics descriptive language; 3D obj ect processingmeans for extracting a 3D obj ect from the 3D description file, setting various attribute data, editing and processing the 3D object to introduce animation or the like, and outputting the resulting data again as a 3D description file or as a temporary file for setting attributes; texture processing means for extracting textures from the 3D description file, editing and processing the textures to reduce the number of colors and the like, and outputting the resulting data again as a 3D description file or as a texture file; 3D effects applying means for extracting a 3D object from the 3D description file, processing the 3D object and assigning various effects such as lighting and material properties, and outputting the resulting data again as a 3D description file or as a temporary file for assigning effects; Web 3D object generating means for extracting various elements required for rendering 3D images in a Web browser from the 3D description file, texture file, temporary file for setting attributes, and temporary file for assigning effects, and for generating various Web-based 3D objects having texture and attribute data that are compressed to be displayed in aWeb browser; behavior data generating means for generating behav^ior data to display 3D scenes in a Web browser with animation by controlling attributes of the 3D obj ects and assigning effects; and. executable file generatingmeans for generating an executable file comprising a Web page and one or a plurality of programs including scripts, plug-ins, and applets for drawing and displaying 3D scenes in a Web browser with stereoscopic images produced from a plurality of combined images assigned with a prescribed parall ax, based on the behavior data and the Web 3D objects generated, edited, and processed by the means described above.
Further, a 3D object generating means according to Claim
2 comprises a turntable on which an object is mounted! and rotated either horizontally or vertically; a digital camera for capturing images of an object mounted on the turntable and creating digital image files of the images; turntable controllingmeans for rotating the turntable to prescribed positions; photographing means using the digital camera to photograph an object set in prescribed positions by the turntable controlling means; successive image creating means for creating successively creating a plurality of image files using the turntable controlling means and the photographing means; and 3D object combining means for generating 3D images based on the plurality of image files cr-eated by the successive image creating means and generating a 3D object having texture and attribute data from the 3D images for di_splaying the images in 3D.
Further, the 3D object generating means according to Claim
3 generates 3D images according to a silhouette method that estimates the three-dimensional shape of an obj ectusing silhouette data from a plurality of images taken by a single cam-era around the entire periphery of the object as the object is rrotated on the turntable. Further, the 3D object generating means according to Claim 4 generates a single 3D image as a composite scene oUotained by combining various image data, including images taken b;y a camera, images produced by computer graphics modeling, images scanned by a scanner, handwritten images, image data stored on other storage media, and the like.
Further, the executable file generating means according to Claim 5 comprises automatic left and right parallax data generating means for automatically generating left and right parrallax data for drawing and displaying stereoscopic images according to a rendering function based on right eye images and left eye images assigned a parallax from a prescribed camera position; parallax data compressing means for compressing each of the lefft and right parallax data generated by the automatic left and rigtit parallax data generatingmeans ; parallax data combining means fo r combining the compressed left and right parallax data; pairallax data expandingmeans for separating the combined left andriςrht parallax data into left and right sections and expanding the data to be displayed on a stereoscopic image displaying device; and display data converting means for converting the data to be displayed according to the angle of view (aspect ratio) of the stereoscopic image displaying device.
Further, the automatic left and right pairallax data generatingmeans according to Claim 6 automatically generates left and right parallax data corresponding to a 3D image g-enerated by the 3D object generating means based on a virtual camera set by a rendering function. Further, the parallax data compressing means according to Claim 7 compresses pixel data for left and right pa.rallax data by skipping pixels.
Further, the stereoscopic display device according to Claim
8 employs at least one of a CRT screen, liquid crystal panel, plasma display, EL display, and projector.
Further, the stereoscopic display device according to Claim
9 displays stereoscopic images that a viewer can see when wearing stereoscopic glasses or displays stereoscopic images that a viewer can see when not wearing glasses.
EFFECTS OF THE INVENTION
The 3D image generation and display system of the present invention can configure a computer systemthat generates 3D objects tobe displayedon a 3Ddisplay. The 3D image generation and display systemhas a simple construction employing a scanning table system to model an object placed on a scanning table by collecting images around the entire periphery of the object with a single camera as the turntable is rotated. Further, the 3D image generation and display system facilitates the generation of hig"li-quality 3D objects by taking advantage of common software sold commercially.
, The 3D image generation and display system can also display animation in a Web browser by installing a special plug-in for drawing and displaying 3D scenes in a Web browser or by generating applets for effectively displaying 3D images in a Web browser.
The 3D image generation and display system can also constitute a display program capable of displaying ster&oscopic images according to LR parallax image data, 3D images of the kind that do not "jump out" at the viewer, and common 2D images on the same display device.
BEST MODE FOR CARRYING OUT THE INVENTION Next, a preferred embodiment of the present invention will be described while referring to the accompanying drawings.
Fig. 1 is a flowchart showing steps in a process performed by a 3D image generation and display system according to a first embodiment of the present invention. In the process of Fig. 1 described below, a 3D scanner described later is used to form a plurality of 3D images. A 3D obj ect is generated from the 3D images and converted to the standard Virtual RealityModeling Language (VRML; a language for describing 3D graphics) format. The converted 3D object in the oiatputted VRML file is subjected to various processes for producing a Web
3D object and a program file that can be executed in a Web browser .
First, a 3D scanner of a 3D obj ect generating means employing a digital camera captures images of a real object, olotaining twenty-four 3D images taken at varying angles of 15 degr~ees, for example (SlOl) . The 3D object generating means genera.tes a 3D object from these images and 3D description file outputti_ng means converts the 3D object temporarily to the VRML format (Sl_02) . 3D ScanWare (product name) or a similarprogramcanbe used for crea.ti.ng 3D images, generating 3D objects, and producing VRML files.
The 3D object generated with a 3D authoring software (such as a software mentioned below) is extracted from the VRML file and subjected to various editing and processing by 3D oloject processing means (S103) . The commercial product "3ds max"
(product name) or other software is used to analyze necessary areas of the 3D object to extract texture images, to set req~uired attributes for animationprocesses andgenerate various 3Dobj ects, and to setup various animation features according to need. .After undergoing editing and processing, the 3D object is saved again as a 3D description file in the VRML format or is temporarily s "tored in a storage device or area of memory as a temporary file for set tting attributes. In the animation settings, the number of fraicies or time can be set in key frame animation for moving an obj ect provided in the 3D scene at intervals of a certain number of frames. Animation can also be created using such techniques as path animation and character studio for creating a path, such as a Nurbs CV curve, along which an object is to be moved. Using t&xture processing means, the user extracts texture images appli_ed to various objects in the VRML file, edits the texture images for color, texture mapping, or the like, reduces the number of colors, modifies the region and location/position where the textiαre is applied, or performs other processes, and saves the resulting data as a texture file (S104) . Texture editing and processing can be done using commercial image editing software, such as Photoshop (product name) . 3D effects applying means are used to extract various 3D objects from the VRML file and to use the extracted objects in combination with 3ds max or similar software and various plug-ins in order to process the 3D objects and apply various effects, such as lighting andmaterial properties. The resulting data is either re-stored as a 3D description file in the VRML format or saved as a temporary file for applyingeffects (S105) . Inthe description thus far, the 3D objects have undergone processes to be displayed as animation on a Web page and processes for reducing the file size as a pre-process in the texture image process or "the like. The following steps cover processes for reducing and optimizing the object size and file size in order to actually display the objects in a Web browser.
Web 3D object generating means extracts 3D objects, texture images, attributes, animation data, and other rendering" elements from the VRML and temporary files created during edd-ting and processing and generates Web 3D objects for displaying 3D images on the Web (S106) . At the same time, behavior data generating means generates behavior data as a scenario for dispLaying the Web 3D object as animation (S107) . Finally, executable file generatingmeans generates an executable file in the form ofplug-in software for a Web browser or a program combining a Ja\^"a Applet, Java Script, and the like to draw and display images in a W«eb browser based on the above data for displaying 3D images (SL08) . By using the VRML format, which is supported b>y most 3D software programs, it is possible to edit and process 3D images using an all-purpose commercial software program. The system can also optimize the image for use on the Web based on the transfer- rate of the communication line or, when displaying images on a_ Web browser of a local computer, can edit and process the images appropriately according to the display environment, thereby controlling image rendering to be effective and achieve optimal- quality in the display environment.
Fig.2 is a schematic diagram showing the 3D obj ect generating means of the 3D image generation and display system described above with reference to Fig. 1. The Web 3D object generating means in Fig. 2 includes a. turntable 31 that supports an object 33 (corresponding to the "object" in the claims section and referred to as an "object" oar "real object" in this specification) and rotates 360 degrees foztr scanning the object 33; a background panel 32 of a single primary/ color, such as green or blue; a digital camera 34, such as a CCD^ lighting 35; a table rotation controller 36 that rotates the turntable 31 through servo control; photographing means 37 four controlling and calibrating the digital camera 34 and lightincj 35, performing gamma correction andother image processing of image data and capturing images of the object 33; and successive image creating means 38 for controlling the angle of table rotation and sampling and collecting images at prescribed angles. Thes^ components constitute a 3D modeling device employing a scanning- table and a single camera for generating a series of images viewe<d from a plurality of angles. At this point, the images are modified according to needusing commercial editing software such as AutoCAID and STL (product names) . A 3D object combining means 39 extract s silhouettes from the series of images and creates 3D images using a silhouette method or the like to estimate 3D shapes in order to generate 3D object data.
Next, the operations of the 3D image generation and display system will be described.
In the silhouette method, the camera is calibrated by calculating, for example, correlations between the world coordinate system, camera coordinate system, and image coordinate system. The points in the image coordinate system are converted to points in the world coordinate system in order to process the images in software.
After calibration is completed, the successive image creating means 38 coordinates with the table rotation controller 36 to control the rotational angle of the turntable for a prescribed number of scans (scanning images every 10 degrees for 36 scans or every 5 degrees for 72 scans, for example) , while the photographing means 37 captures images of the object 33. Silhouette data of the object 33 is acquired from the captured images by obtaining a background difference, which is the difference between images of the background panel 32 taken previously and the current camera image. A silhouette image of the object is derived from the background difference and camera parameters obtained from calibration. 3D modeling is then performed on the silhouette image by placing a cube having a recursive octal tree structure in a three-dimensional space, for example, and determining intersections in the silhouette of the object. Fig. 3 is a flowchart that gives a more specific/concrete example — which is in accordance with steps in the process for converting 3D images shown in Fig. 1 — so that the steps shown in Fig. 1 can be better/further explained. The process in Fig. 3 is implemented by a Java Applet that can display 3D images in a Web browser without installing a plug-in for a viewer, such as Live 3D. In this example, all the data necessary for displaying interactive 3D scenes is provided on a Web server. The 3D scenes are displayedwhen the server is accessed from a Web browser running on a client computer. Normally, after 3D objects are created, 3ds max or the like is used to modify motion, camera, lighting, and material properties and the like in the generated 3D objects. However, in the preferred embodiment, the 3D objects or the entire scene is first converted to the VRML format (S202) . The resulting VRML file is inputted into a 3DA system (S203; here, 3DA describes 3D images that are displayed as animation on a Web browser using a Java Applet, and the entire system including the authoring software for Web-related editing and processing is called a 3DA system) . The 3D scene is customized, and data for rendering the image with the 3DA applet is provided for drawing and displaying the 3D scene in the Web browser (S205) . All 3D scene data is compressed at one time and saved as a compressed 3DA file (S206) . The 3DA system generates a tool bar file for interactive operations and an HTML file, where the HTML page reads the tool bar file into the Web browser, so that the tool bar file is executed, and that 3D scenes are displayed in a Web browser. (S207) . The new Web page (HTML document) includes an applet tag for calling the 3DA applet. Java Script code for accessing the 3DA applet may be added to the HTML document to improve operations and interactivity (S209) . All files required for displaying the 3D scene created as described above are transferred to the Web server. These files include the Web page (HTML document) possessing the applet tag for calling the 3DA applet, a tool bar file for interactive operations as an option, texture image files, 3DA scene files, and the 3DA applet for drawing and displaying 3D scenes (S210) .
When a Web browser subsequently connects to the Web server and requests the 3DA applet, the Web browser downloads the 3DA applet from the Web server and executes the applet (S211) . Once the 3DA applet has been executed, the applet displays a 3D scene with which the user can perform interactive operations, and the Web browser can continue displaying the 3D scene independently of the Web server (S212) .
In the process described to this point, a 3DA Java applet file is generated after converting the 3D objects to the Web-based VRML, and the Web browser downloads the 3DA file and 3DA applet. However, rather than generating a 3DA file, it is of course possible to install a plug-in for a viewer, such as Live 3D (product name) and process the VRML 3D description file directly. With the 3D image generation and display system of the preferred embodiment, a company can easily create a Web site using three-dimensional and moving displays of products for e-commerce and the like. As an example of an e-commerce product, the following description covers the starting of a commercial Web site for printers, such as that shown in Fig. 4.
First, the company's product, a printer 60 as the object
33, is placed on the turntable 31 shown in Fig. 2 and rotated, while the photographing means 37 captures images at prescribed sampling angles. The successive image creating means 38 sets the number of images to sample, so that the photographing means 37 captures thirty-six images assuming a sampling angle of 10 degrees
(360 degrees/10 degrees = 36) . The 3D object combining means 39 calculates the background difference between the camera position and the background panel 32 that has been previously photographed and converts image data for each of the thirty-six images of the printer created by the successive image creating means 38 to world coordinates by coordinate conversion among world coordinates, camera coordinates, and image positions. The silhouette method for extracting contours of the object is used to model the outer shape of the printer and generate a 3D obj ect of the printer . This object is temporarily outputted as a VRML file. At this time, all 3D images to be displayed on the Web are created, including a rear operating screen, left and right side views, top and bottom views, a front operating screen, and the like.
Next, as described in Fig.1, the 3D object processing means, texture processing means, and 3D effects applying means extract the generated 3D image data from the VRML file, analyze relevant parts of the data, generate 3D objects, apply various attributes, perform animation processes, and apply various effects and other processes, such as lighting and surface formation through color, material, and texture mapping properties. The resulting data is savedas a texture file, a temporary file for attributes, atemporary file for effects, and the like. Next, the behavior data generating means generates data required for movement in all 3D description files used on the printer Web site. Specifically, the behavior data generating means generates a file for animating the actual operating screen in the setup guide or the like.
By installing a plug-in in the Web browser for a viewer, such as Live 3D, the 3D scene data created above can be displayed in the Web browser. It is also possible to use a method for processing the 3D scene data in the Web browser only, without using a viewer. In this case, a 3DA file for a Java applet is downloaded to the Web browser for drawing and displaying the 3D scene data extracted from the VRML file, as described above. When viewing the Web site created above displaying a 3D image of the printer, the user can operate a mouse to click on items in a setup guidemenudisplayed in thebrowser to display an animation sequence in 3D. This animation may illustrate a series of operations that rotate a button 63 on a cover 62 of the printer 60 to detach the cover 62 and install a USB connector 66.
When the user clicks on "Install Cartridge" in the menu, a 3D animation sequence will be played in which the entire printer is rotated to show the front surface thereof (not shown in the drawings) . A top cover 61 of the printer 60 is opened, and a cartridge holder in the printer 60 moves to a center position. Black and color ink cartridges are inserted into the cartridge holder, and the top cover 61 is closed. Further, if the user clicks on "Maintenance Screen," a 3D image is displayed in which all of the plastic covers have been removed to expose the inner mechanisms of the printer (not shown) . In this way, the user can clearly view spatial relationships among the driver module, scanning mechanism, ink cartridges, and the like in three dimensions, facilitating maintenance operations.
By displaying operating windows with 3D animation in this way, the user can look over products with the same sense of reality as when actually operating the printer in a retail store. While the above description is a simple example for viewing printer operations, the 3D image generation and display system can be used for other applications, such as trying on apparel. For example, the 3D image generation and display system can enable the user to try on a suit from a women' s clothing store or the like. The user can click on a suit worn by a model; change the size and color of the suit; view the modeled suit from the front, back, and sides; modify the shape, size, and color of the buttons; and even order the suit by e-mail. Various merchandise, such as sculptures or other fine art at auctions and everyday products, can also be displayed in three-dimensional images that are more realistic than two-dimensional images.
Next, a second embodiment of the present invention will be described while referring to the accompanying drawings.
Fig. 5 is a schematic diagram showing a 3D image generation and display system according to a second embodiment of the present invention. The second embodiment further expands the 3D image generation and display system to allow the 3D images generated and displayed on a Web page in the first embodiment to be displayed as stereoscopic images using other 3D display devices.
The 3D image generation and display system in Fig.5 includes a turntable-type 3D object generator 71 identical to the 3D object generating means of the first embodiment shown in Fig. 2. This 3D object generator 71 produces a 3D image by combining images of an object taken with a single camera while the object is rotated on a turntable. The 3D image generation and display system of the second embodiment also includes a multiple camera 3D object generator 72. Unlike the turntable-type 3D object generator 71, the 3D object generator 72 generates 3D objects by arranging a plurality of cameras from two stereoscopic cameras corresponding to the positions of left and right eyes to n cameras (while not particularly limited to any number, a more detailed image can be achievedwith a largernumber of cameras) arounda stationaryobject . The 3D image generation and display system also includes a computer graphics modeling 3D obj ect generator 73 for generating a 3D obj ect while performing computer graphics modeling through the graphics interface of a program, such as 3ds max. The 3D object generator 73 is a computer graphics modeler that can combine scenes with computer graphics, photographs, or other data.
After performing the processes of S103-S107 described in Fig. 1 of the first embodiment to save 3D objects produced by the 3D object generators 71-73 temporarily as general-purpose VRML files, 3D scene data is extracted from the VRML files using a Web authoring tool, such as YAPPA 3D Studio (product name) . The authoring software is used to edit and process the 3D objects and textures; add animation; apply, set, and prooess other effects, such as camera and lighting effects; and gene-irate Web 3D objects and their behavior data for drawing and disp laying interactive 3D images in a Web browser. An example for creating Web 3D files was described in S202-S210 of Fig. 3.
Means 75-79 are parts of the executable fJL Ie generatingmeans used in S108 of Fig. 1 that apply left and r-ight parallax data for displaying stereoscopic images. A renderer 75 applies rendering functions to generate left and rig-ht parallax images (LR data) required for displaying stereoscopic images. An LR data compressing/combining means 76 compresses the LR data generated by the renderer 75, rearranges the data in a. combining process and stores the data in a display frame buffer. An LR data separating/expanding means 77 separates and expands the left and right data when displaying LR data. A data converting means 78 configured of a down converter or the like adjusts the angle of view (aspect ratio and the like) for displaying stereoscopic images so that the LR data can be made compatible witra various 3D display devices. A stereoscopic displaying means 79 displays stereoscopic images based on the LR data and. using a variety of display devices, such as a liquid crystal panel , CRT screen, plasma display, EL (electroluminescent) display, oar projector shutter type display glasses and includes a variety of display formats, such as the common VGA format used in personal computer displays and the like and video formats used for televisions .
Next, the operations of the 3D image gerxeration and display system according to the second embodiment will be described. First, a 3D obj ect generating process perforzmed by the 3D obj ect generators 71-73 will be described briefly. The 3D obj ect generator 71 is identical to the 3D obj ect gen&rating means described in Fig . 1 . The obj ect 33 for which a 3D image is to be formed is placed on the turntable 31 . The table rotation controller 36 regulates rotations of the turntable 31 , while the digital camera 34 and lighting 35 are controlled t o take sample photographs by the photographing means 37 against a single-color screen, such as a blue screen (the background panel 32 ) as the background . The successive image creating means 38 then performs a process to combine the sampled images . Based on "the resulting composite image, the 3D obj ect combining means 39 extracts silhouettes ( contours ) of the obj ect and generates s a 3D obj ect using a silhouette method or the like to estimate the three-dimensional shape of the obj ect . This method is performed using the following equation, for example . Equation 1
Figure imgf000026_0001
Coordinate conversion (calibration) is pezrformed using camera coordinates Pfp and world coordinates Sp oi a point P to convert three-dimensional coordinates at vertices o ± the 3D images to the world coordinate system [x, y, z, r, g, b] . A variety of modeling programs are used to model the resulting- coordinates. The 3D data generated fromthis process is saved in an image database (not shown) .
The 3D object generator 72 is a system for capturing images of an object by placing a plurality of cameras aro αmd the object. For example, as shown in Fig. 6, six cameras (firs t through sixth cameras) are disposed around an object . Acontrol computer obtains photographic data from the cameras via USB hubs and reproduces 3D images of the object in real-time on first and secondproj ectors . The 3D object generator 72 is not limited to six cameras, but may capture images with any number of cameras. The system generates 3D objects in the world coordinate system from tlkie plurality of overlappingphotographs obtained fromthese camera ≤ and falls under the category of image-based rendering (IBR) . Hence, the construction and process of this system is considerably more complicated than that of the 3D object generator ~71. As with the 3D object generator 71, the generated data is saved ±n the database.
The 3D object generator 73 focuses primarily on computer graphics modeling using modeling software, sucb_ as 3ds max and
YAPPA 3D Studio that assigns "top," "left," "rzLght," "front,"
"perspective," and "camera" to each of four vie:-ws in a divided view port window, establishes a grid corresponding? to the vertices of the graphics in a display screen andmodels anima_ge using various objects, shapes, andother data stored in a library. Thesemodeling programs can combine computer graphics data withi photographs or image data created with the 3D object generators 71 and 72. This combining can easilybe implementedby adjusting th*.e camera' s angle of view, the aspect ratio for rendered images in a bitmap of photographic data and computer graphic data. A camera (virtual camera) can be created at any paint for setting or modifying the viewpoint of the combined sceme. For example, to change the camera position (user's viewpoint) that is set to the front by default to a position shifted 30 degrees left or right, the composite image scene can be displaced at a position in which the scene has been shifted 30 degrees from the front by setting the coordinates of the camera angle and position using [X, Y, Z, w] . Further, virtual cameras that can be created include a free camera that can be freely rotated and moved to any position, and a target camera that can be rotated around arx obj ect . When the user wishes to change the viewpoint of a composi te image scene or the like, the user may do so by setting new properties. With the lens functions and the like, the user can quickL y change the viewpoint with the touch of a button by selecting or switching among a group of about ten virtual lenses from WIDE ~to TELE. Lighting settings may be changed in the same way with various functions that can be applied to the rendered image. AILl of the data generated is saved in the database.
Next, the process for generating left and right parallax images with the renderer and LR data (parallax images) generating means 75 will be described. LR data of parallax signals corresponding to the left and right eyes can be easily acquired using the cameraposition setting function of themodeling software programs described above . A specific example for calculating the camerapositions forthe left andright eyes inthis caseisdescribed next with reference to Fig. 7. The coordinates of the position of each camera are represented by a vector normal to tine object being modeled (a cellular telephone in this example), as shown in Fig.7 (a) . Here, the coordinate for the position of the camera is set to 0; the focusing direction of the camera to a vector OT; and a vector OU is set to the direction upward from the camera and orthogonal to the vector OT. In order to achieve a stereoscopic display with positions for the left and right eyes, the positions of the left and right eyes (L, R) is calculated according to the following equation 2, where θ is the inclination angle for the left and right eyes (L, R) and d is a distance to a convergence point P for a zero parallax between the left and right eyes. Equation 2
OR OL =dtanθ dtanø
dtan θ (2)
Figure imgf000029_0001
Here,(0<d,0≤θ <180>
The method for calculating the positions described above is not limited to this method but may be any calculating method that achieves the same effects. For example, since the default camera position is set to the front, obviously the coordinates [X, Y, Z, w] can be inputted directly using the method for studying the camera (virtual camera) position described above.
After setting the positions of the eyes (camera positions) found from the above-described methods in the camera function, the user selects "renderer" or the like in the tool bar of the window displaying the scene to convert and render the 3D scene as a two-dimensional image in order to obtain a left and right parallax image for a stereoscopic display.
LR data is not limited to use with composite image scenes, but can also be created for photographic images taken by the 3D object generators 71 and 72. By setting coordinates [X, Y, Z, w] for camera positions (virtual cameras) corresponding to positions of the left and right eyes, the photographic images can be rendered, saving image data of the obj ect taken around the entire periphery to obtain LR data for left and right parallax images. It is also possible to create LR data from image data taken around the entire periphery of an object saved in the same way for a 3D object that is derived from computer graphics images and the like modeledby the 3D object generator 73. LR data can easilybe created by rendering various composite scenes. In the actual rendering process, coordinates for each vertex of polygons in the world coordinate system are converted to a two-dimensional screen coordinate system. Accordingly, a 3D/2D conversion is performed by a reverse conversion of equation 1 used to convert camera coordinates to three-dimensional coordinates. In addition to calculating the camera positions, it is necessary to calculate shadows (brightness) due to virtual light shining from a light source. For example, light source data Cnr, Cng, and Cnb accounting for material colors Mr, Mg, and Mb can be calculating using the following transformation matrix equation 3.
Equation 3
Figure imgf000031_0001
Here, Cnr, Cng, Cnb, Pnr, Png, and Pnb represent the nth
vertex.
LR data for left and right parallax images obtained through this rendering process is generated automatically by calculating coordinates of the camera positions and shadows based on light source data. Various filtering processes are also performed simultaneously but will be omitted from this description. In the display device, an up/down converter or the like converts the image data to bit data and adjusts the aspect ratio before displaying the image .
Next, a method for automatically generating simple LR data will be described as another example of the present invention. Fig.8 is an explanatory diagram illustrating amethod of generating simple left and right parallax images. As shown in the example of Fig. 8, LR data of a character "A" has been created for the left eye. If the object is symmetrical left to right, a parallax image for the right eye can be created as a mirror image of the LR data for the left eye simply by reversing the LR data for the left eye. This reversal can be calculated using the following equation 4. Equation 4
Figure imgf000031_0002
Here, X represents the X coordinate, Y the Y coordinate, and X' and Y' the new coordinates in the mirror image. Rx and Ry are equal to -1. This simple process is sufficiently practical when there are few changes in the image data, and can greatly reduce memory consumption and processing time. Next, an example of displaying actual 3D images on various display devices using the LR data found in the above process will be described.
For simplicity, this description will cover the case in which LR data is inputted into the conventional display device shown in Fig. 19 to display 3D images. The display device shown in Fig. 19 is a liquid crystal panel (LCD) used in a personal computer or the like and employs a VGA display system using a sequential display technique. Fig. 9 is a block diagram showing a parallax image signal processing circuit. When LR data automatically generated according to the present invention is supplied to this type of display device, the LR data for both left and right parallax images shown in Figs. 20 (a) and 20 (b) is inputted into a compressor/combiner 80. The compressor/combiner 80 rearranges the image data with alternating R and L data, as shown in Fig. 20 (c) , and compresses the image in half by skipping pixel, as shown in Fig. 20 (d) . A resulting LR composite signal is inputted into a separator 81. The separator 81 performs the same process in reverse, rearranging the image data by separating the R and L rows, as shown in Fig. 20 (c) . This data is uncompressed and expanded by expanders 82 and 83 and supplied to display drivers to adjust the aspect ratios and the like. The drivers display the L signal to be seen only with the left eye and the R signal to be seen only with the right eye, achieving a stereoscopic display. Since the pixels skippedduring compression are lost and. cannot be reproduced, the image data is adjusted using interpolation and the like. This data canbe used on displays in notebook personal computers, liquid crystal panels, direct-view game consoles, and the like. The signal format for the LR data in these cases has no particular restriction.
Web 3D authoring tools such as YAPPA 3D Studio are configured to convert image data to LR data according to a. Java applet process . Operating buttons such as those shown in Fig-. 10 can be displayed on the screen of a Web browser by attachirxg a tool bar file to one of Java applets, and downloading the data. (3d scene data, Java applets, and HTML files) from a Web server to the Web browser via a network. By selecting a button, the user can manipulate the stereoscopic image displayed in the Web br-owser (a car in this case) to zoom in and out, move or rotate the image, and the like. Theprocess details of the operations for zooming in andout, moving, rotating, and the like are expressed in a transformation matrix. For example, movement canbe representedbyequation 5below. Other operations can be similarly expressed. Equation 5
Figure imgf000033_0001
Here, X' and Y' are the new coordinates, X and Y are the original coordinates, and Dx and Dy are trαe distances moved in the horizontal and vertical directions respectively. Next, an example of displaying images on an interlaced type display, such as a television screen, will be desciribed. Various converters are commercially sold as display mea:ns in personal computers and the like for converting image data t o common TV and video images. This example uses such a converter to display stereoscopic images in a Web browser. The construction and operations of the converter itself will not be described.
The following example uses a liquid crystal panel (or a CRT screen or the like) as shown in Fig. 19 for placing back video signals. A parallax barrier, lenticular sheet, or the like for displaying stereoscopic images is mounted on the front surface of the displaydevice . The displayprocess will be describedusing the block diagram in Fig. 11 showing a signal processing circuit for parallax images. LR data for left and right parallax images, such as that shown in Figs. 20 (a) and 20 (b) generrated according to the automatic generating method of the present invention, is inputted into compressors 90 and 91, respectively. The compressors 90 and 91 compress the images by skipp dng every other pixel in the video signal. A combiner 92 combines and compresses the left and right LR data, as shown in Figs. 2O (c) and 20 (d). A video signal configured of this combined LR data is either transferred to a receiver or recorded on and pLayed back on a recording medium, such as a DVD. A separator 93 pe_rforms the same operation in reverse, separating the combined LR data into left and right signals, as shown in Figs. 20 (c) and 20 <d) . Expanders 94 and 95 expand the left and right image data to their original form shown in Figs. 20 (a) and 20 (b) . Stereoscopi_c images can be displayed on a display like that shown in Fig.19 because ttie display data is arranged with alternating left video data and rzLght video data across the horizontal scanning lines and in the oirder R, G, and B. For example, the R (red) signal is arranged as "RO (for left) RO (for right), R2 (for left) R2 (for right), R4 C for left) R4 (for right) ...." The G (green) signal is arranged as ^GO (left) GO (right), G2 (left) G2 (right), ...." The B (blue) signal is arranged as "BO (left) BO (right) , B2 (left) B2 (right) ...." Further, a stereoscopic display can be achieved in the same wayusing shutter glasses, having liquid crystal shutters or the like, as ttie display device, by sorting the LR data for parallax image sig~nals into an odd field and even field and processing the two in synchronization.
Next, a descriptionwillbe given for displayingste reoscopic images on a projector used for presentations or as a home theater or the like.
Fig.12 is a schematic diagramof a home theater tha~fc includes aprojector screen 101, the surface of whichhas undergone <a.n optical treatment (such as an application of a silver metal coating) ; two projectors 106 and 107 disposed in front of the projector screen 101; and polarizing filters 108 and 109 disposed one in front of each of the projectors 106 and 107, respectively. Each component of the home theater is controlled by a controller IO 3. If the projector 106 is provided for the right eye and the projector 107 for the left eye, the filter 109 is a type that polari_zes light vertically, while the filter 108 is a type that polarl_zes light horizontally. The type ofprojectormaybe aMLP (meridiam lossless packing) liquid crystal projector using a DMD (digital micromirror device) . The home theater also includes a 3D image recorder 104 that supports DVD or another medium (certainly the device may also generate images through modeling) , and a left and right parallax image generator 105 for automatically generating LR data with the display drivers of the present invention based on 3D image data inputted from the 3D image recorder 104. The aspect ratio of the LR data generated by the left and right parallax image generator
105 is adjusted by a down converter or the like and provided to the respective left and right projectors 106 and 107. The projectors 106 and 107 project images through the polarizing filters 108 and 109, which polarize the images hLorizontally and vertically, respectively. The viewer puts on polarizing glasses 102 having a vertically polarizing filter for the right eye and a horizontally polarizing filter for the left e^/e . Hence, when viewing the image projected on the projector screen 101, the viewer can see stereoscopic images since images projectedby the projector
106 can only be seen with the right eye and images projected by the projector 107 can only be seen with the left eye.
INDUSTRIAL APPLICABILITY
By using a Web browser for displaying 3D images in this way, only an electronic device having a browser is required, and not a special 3D image displaying device, and the 3D images can be supported on a variety of electronic devices . The present invention is alsomore user-friendly, since differ ent stereoscopic display software, such as a stereo driver or the like, need not be provided for each different type of hardware, such as a_ personal computer, television, game console, liquid panel display, shutter glasses, and projectors.
BRIEF DESCRIPTION OF THE DRAWINGS In the drawings :
Fig. 1 is a flowchart showing steps in a process performed by the 3D image generation and display system according to a first embodiment of the present invention; Fig.2 is a schematic diagram showing 3D object generating means of the 3D image generation and display system des cribed in Fig. 1;
Fig. 3 is a flowchart that shows a process from generation of 3Dobjects to drawinganddisplaying of 3D scenes in aWEBbrowser. Fig. 4 is a perspective view of a printer as an example of a 3D object;
Fig.5 is a schematic diagram showing a 3D image generation and display system according to a second embodiment of true present invention; Fig. 6 is a schematic diagram showing a 3D image generator of Fig. 5 having 2-n cameras;
Fig. 7 is an explanatory diagram illustrating a method of setting camera positions in the renderer of Fig. 5;
Fig.8 is an explanatory diagram illustrating a prrocess for creating simple stereoscopic images;
Fig. 9 is a block diagram of an LR data processing circuit in a VGA display; Fig. 10 is an explanatory diagram illustrating operations for zooming in and out, moving, and rotating a 3D image;
Fig. 11 is a block diagram showing an LR data processing circuit of a video signal type display; Fig.12 is a schematic diagram showing a stereoscopic display system employing projectors;
Fig. 13 (a) is a schematic diagram of a conventional 3D modeling display device;
Fig. 13 (b) is an explanatory diagram illustrating the creation of slit images;
Fig.14 is a block diagram showing a conventi onal 3D modeling device employing a plurality of cameras;
Fig. 15 is a schematic diagram of a conventional 3D image signal generator; Fig. 16 is an explanatory diagram showing LR data for the signal generator of Fig. 15;
Fig. 17 is an explanatory diagram illustr-ating a process for compressing the LR data in Fig. 16;
Fig. 18 is an explanatory diagram showj_ng a method of displaying LR data on the display device of Fig. 15;
Fig. 19 is a schematic diagram of another conventional stereoscopic image displaying device; and
Fig.20 is an explanatory diagram showing LU data displayed on the display device of Fig. 19.

Claims

1. A 3D image generation and display system configured of a computer system for generating three-dimensional (3D) objects used to display 3D images in a Web browser, the 3D image generation and display system comprising:
3D object generating means for creating 3D images from a plurality of different images and/or computer graphics modeling and generating a 3D object from these images that has texture and attribute data;
3D description file outputting means for converting the format of the 3D obj ect generated by the 3D obj ect generating means and outputting the data as a 3D description file for displaying 3D images according to a 3D graphics descriptive language;
3D object processing means for extracting a 3D object from the 3D description file, setting various attribute data, editing and processing the 3D object to introduce animation or the like, and outputting the resulting data again as a 3D description file or as a temporary file for setting attributes; texture processing means for extracting textures from the 3D description file, editing and processing the textures to reduce the number of colors and the like, and outputting the resulting data again as a 3D description file or as a texture file;
3D effects applying means for extracting a 3D object from the 3D description file, processing the 3D object and assigning various effects such as lighting and material properties, and outputting the resulting data again as a 3D description file or as a temporary file for assigning effects;
Web 3D object generating means for extracting various elements required for rendering 3D images in a Web browser from the 3D description file, texture file, temporary file for setting attributes, and temporary file for assigning effects, and for generating various Web-based 3D objects having texture and attribute data that are compressed to be displayed in a Web browser; behavior data generating means for generating behavior data to display 3D scenes in a Web browser with animation by controlling attributes of the 3D objects and assigning effects; and executable file generating means for generating an executable file comprising a Web page and one or a plurality of programs including scripts, plug-ins, and applets for drawing and displaying 3D scenes in a Web browser with stereoscopic images produced from a plurality of combined images assigned with a prescribed parallax, based on the behavior data and the Web 3D objects generated, edited, and processed by the means described above .
2. A 3D image generation and display system according to Claim 1, wherein the 3D object generating means comprises: a turntable on which an object is mounted and rotated either horizontally or vertically; a digital camera for capturing images of an object mounted on the turntable and creating digital image files of the images; turntable controlling means for rotating the turntable to prescribed positions; photographing means using the digital camera to photograph an obj ect set in prescribed positions by the turntable controlling means; successive image creating means for creating successively creating a plurality of image files using the turntable controlling means and the photographing means; and
3D object combining means for generating 3D images based on the plurality of image files created by the successive image creating means and generating a 3D object having texture and attribute data from the 3D images for displaying the images in 3D.
3. A 3D image generation and display system according to Claim 2, wherein the 3D object generating means generates 3D images according to a silhouette method that estimates the three-dimensional shape of an object using silhouette data from a plurality of images taken by a single camera around the entire periphery of the object as the object is rotated on the turntable.
4. A 3D image generation and display system according to Claim 1, wherein the 3D object generating means generates a single 3D image as a composite scene obtained by combining various image data, including images taken by a camera, images produced by computer graphics modeling, images scanned by a scanner, handwritten images, image data stored on other storage media, and the like.
5. A 3D image generation and display system according to Claim 1, wherein the executable file generating means comprises: automatic left and right parallax data generating means for automatically generating left and right parallax data for drawing and displaying stereoscopic images according to a rendering function based on right eye images and left eye images assigned a parallax from a prescribed camera position; parallax data compressing means for compressing each of the left and right parallax data generated by the automatic left and right parallax data generating means; parallax data combining means for combining the compressed left and right parallax data; parallax data expanding means for separating the combined left and right parallax data into left and right sections and expanding the data to be displayed on a stereoscopic image displaying device; and display data converting means for converting the data to be displayed according to the angle of view (aspect ratio) of the stereoscopic image displaying device.
6. A 3D image generation and display system according to Claim 5, wherein the automatic left and right parallax data generating means automatically generates left and right parallax data corresponding to a 3D image generated by the 3D object generating means based on a virtual camera set by a rendering function.
7. A 3D image generation and display system according to Claim 5, wherein the parallax data compressing means compresses pixel data for left and right parallax data by skipping pixels .
8. A 3D image generation and display system according to Claim 5, wherein the stereoscopic display device employs at least one of a CRT screen, liquid crystal panel, plasma display, EL display, and projector.
9. A 3D image generation and display system according to Claim 5, wherein the stereoscopic display device displays stereoscopic images that a viewer can see when wearing stereoscopic glasses or displays stereoscopic images that a viewer can see when not wearing glasses.
PCT/JP2005/008335 2005-04-25 2005-04-25 3d image generation and display system WO2006114898A1 (en)

Priority Applications (7)

Application Number Priority Date Filing Date Title
US11/912,669 US20080246757A1 (en) 2005-04-25 2005-04-25 3D Image Generation and Display System
BRPI0520196-9A BRPI0520196A2 (en) 2005-04-25 2005-04-25 3d image generation and display system
PCT/JP2005/008335 WO2006114898A1 (en) 2005-04-25 2005-04-25 3d image generation and display system
CA002605347A CA2605347A1 (en) 2005-04-25 2005-04-25 3d image generation and display system
AU2005331138A AU2005331138A1 (en) 2005-04-25 2005-04-25 3D image generation and display system
EP05737019A EP1877982A1 (en) 2005-04-25 2005-04-25 3d image generation and display system
NO20075929A NO20075929L (en) 2005-04-25 2007-11-19 3D image generation and screen system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2005/008335 WO2006114898A1 (en) 2005-04-25 2005-04-25 3d image generation and display system

Publications (1)

Publication Number Publication Date
WO2006114898A1 true WO2006114898A1 (en) 2006-11-02

Family

ID=35478817

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2005/008335 WO2006114898A1 (en) 2005-04-25 2005-04-25 3d image generation and display system

Country Status (7)

Country Link
US (1) US20080246757A1 (en)
EP (1) EP1877982A1 (en)
AU (1) AU2005331138A1 (en)
BR (1) BRPI0520196A2 (en)
CA (1) CA2605347A1 (en)
NO (1) NO20075929L (en)
WO (1) WO2006114898A1 (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2010008954A2 (en) * 2008-07-16 2010-01-21 Google Inc. Web-based graphics rendering system
CN102214303A (en) * 2010-04-05 2011-10-12 索尼公司 Information processing device, information processing method and program
US8294723B2 (en) 2008-11-07 2012-10-23 Google Inc. Hardware-accelerated graphics for web applications using native code modules
CN102917236A (en) * 2012-09-27 2013-02-06 深圳天珑无线科技有限公司 Single-camera based stereoscopic photographing method and digital photographing device
US8675000B2 (en) 2008-11-07 2014-03-18 Google, Inc. Command buffers for web-based graphics rendering
US9619858B1 (en) 2009-07-02 2017-04-11 Google Inc. Graphics scenegraph rendering for web applications using native code modules
KR20170132843A (en) * 2015-03-30 2017-12-04 알리바바 그룹 홀딩 리미티드 Image synthesis method and apparatus

Families Citing this family (88)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7747655B2 (en) 2001-11-19 2010-06-29 Ricoh Co. Ltd. Printable representations for time-based media
US7861169B2 (en) 2001-11-19 2010-12-28 Ricoh Co. Ltd. Multimedia print driver dialog interfaces
JP2005108230A (en) 2003-09-25 2005-04-21 Ricoh Co Ltd Printing system with embedded audio/video content recognition and processing function
US8077341B2 (en) 2003-09-25 2011-12-13 Ricoh Co., Ltd. Printer with audio or video receiver, recorder, and real-time content-based processing logic
US7864352B2 (en) 2003-09-25 2011-01-04 Ricoh Co. Ltd. Printer with multimedia server
US8274666B2 (en) * 2004-03-30 2012-09-25 Ricoh Co., Ltd. Projector/printer for displaying or printing of documents
DE102005017313A1 (en) * 2005-04-14 2006-10-19 Volkswagen Ag Method for displaying information in a means of transport and instrument cluster for a motor vehicle
JP2009134068A (en) * 2007-11-30 2009-06-18 Seiko Epson Corp Display device, electronic apparatus, and image processing method
US8233032B2 (en) * 2008-06-09 2012-07-31 Bartholomew Garibaldi Yukich Systems and methods for creating a three-dimensional image
KR101588666B1 (en) * 2008-12-08 2016-01-27 삼성전자주식회사 Display apparatus and method for displaying thereof
CN101939703B (en) * 2008-12-25 2011-08-31 深圳市泛彩溢实业有限公司 Hologram three-dimensional image information collecting device and method, reproduction device and method
US20110304692A1 (en) * 2009-02-24 2011-12-15 Hoe Jin Ha Stereoscopic presentation system
US20100225734A1 (en) * 2009-03-03 2010-09-09 Horizon Semiconductors Ltd. Stereoscopic three-dimensional interactive system and method
JP4919122B2 (en) * 2009-04-03 2012-04-18 ソニー株式会社 Information processing apparatus, information processing method, and program
JP5409107B2 (en) * 2009-05-13 2014-02-05 任天堂株式会社 Display control program, information processing apparatus, display control method, and information processing system
US9479768B2 (en) 2009-06-09 2016-10-25 Bartholomew Garibaldi Yukich Systems and methods for creating three-dimensional image media
US9030535B2 (en) * 2009-06-23 2015-05-12 Lg Electronics Inc. Shutter glasses, method for adjusting optical characteristics thereof, and 3D display system adapted for the same
JP5438412B2 (en) * 2009-07-22 2014-03-12 株式会社コナミデジタルエンタテインメント Video game device, game information display control method, and game information display control program
JP2011035592A (en) * 2009-07-31 2011-02-17 Nintendo Co Ltd Display control program and information processing system
US20130124148A1 (en) * 2009-08-21 2013-05-16 Hailin Jin System and Method for Generating Editable Constraints for Image-based Models
JP5405264B2 (en) * 2009-10-20 2014-02-05 任天堂株式会社 Display control program, library program, information processing system, and display control method
JP4754031B2 (en) * 2009-11-04 2011-08-24 任天堂株式会社 Display control program, information processing system, and program used for stereoscopic display control
KR101635567B1 (en) * 2009-11-12 2016-07-01 엘지전자 주식회사 Apparatus for displaying image and method for operating the same
KR101611263B1 (en) * 2009-11-12 2016-04-11 엘지전자 주식회사 Apparatus for displaying image and method for operating the same
US8854531B2 (en) 2009-12-31 2014-10-07 Broadcom Corporation Multiple remote controllers that each simultaneously controls a different visual presentation of a 2D/3D display
US8823782B2 (en) 2009-12-31 2014-09-02 Broadcom Corporation Remote control with integrated position, viewer identification and optical and audio test
US9247286B2 (en) * 2009-12-31 2016-01-26 Broadcom Corporation Frame formatting supporting mixed two and three dimensional video data communication
US8964013B2 (en) * 2009-12-31 2015-02-24 Broadcom Corporation Display with elastic light manipulator
US9185383B2 (en) 2010-01-07 2015-11-10 Thomson Licensing System and method for providing optimal display of video content
JP2013057697A (en) * 2010-01-13 2013-03-28 Panasonic Corp Stereoscopic image displaying apparatus
EP2355526A3 (en) 2010-01-14 2012-10-31 Nintendo Co., Ltd. Computer-readable storage medium having stored therein display control program, display control apparatus, display control system, and display control method
US8687044B2 (en) * 2010-02-02 2014-04-01 Microsoft Corporation Depth camera compatibility
JP5872185B2 (en) * 2010-05-27 2016-03-01 任天堂株式会社 Portable electronic devices
US9693039B2 (en) 2010-05-27 2017-06-27 Nintendo Co., Ltd. Hand-held electronic device
US8384770B2 (en) 2010-06-02 2013-02-26 Nintendo Co., Ltd. Image display system, image display apparatus, and image display method
US8633947B2 (en) 2010-06-02 2014-01-21 Nintendo Co., Ltd. Computer-readable storage medium having stored therein information processing program, information processing apparatus, information processing system, and information processing method
WO2011150466A1 (en) * 2010-06-02 2011-12-08 Fujifilm Australia Pty Ltd Digital kiosk
EP2395768B1 (en) 2010-06-11 2015-02-25 Nintendo Co., Ltd. Image display program, image display system, and image display method
US8581962B2 (en) * 2010-08-10 2013-11-12 Larry Hugo Schroeder Techniques and apparatus for two camera, and two display media for producing 3-D imaging for television broadcast, motion picture, home movie and digital still pictures
CA2711874C (en) 2010-08-26 2011-05-31 Microsoft Corporation Aligning animation state update and frame composition
JP4869430B1 (en) * 2010-09-24 2012-02-08 任天堂株式会社 Image processing program, image processing apparatus, image processing system, and image processing method
JP5739674B2 (en) 2010-09-27 2015-06-24 任天堂株式会社 Information processing program, information processing apparatus, information processing system, and information processing method
US8854356B2 (en) 2010-09-28 2014-10-07 Nintendo Co., Ltd. Storage medium having stored therein image processing program, image processing apparatus, image processing system, and image processing method
JP4917664B1 (en) * 2010-10-27 2012-04-18 株式会社コナミデジタルエンタテインメント Image display device, game program, and game control method
JP2012100129A (en) * 2010-11-04 2012-05-24 Jvc Kenwood Corp Image processing method and image processing apparatus
US8682107B2 (en) * 2010-12-22 2014-03-25 Electronics And Telecommunications Research Institute Apparatus and method for creating 3D content for oriental painting
US8842135B2 (en) * 2011-03-17 2014-09-23 Joshua Morgan Jancourtz Image editing system and method for transforming the rotational appearance of a subject
US8555204B2 (en) * 2011-03-24 2013-10-08 Arcoinet Advanced Resources, S.L. Intuitive data visualization method
US8473362B2 (en) * 2011-04-07 2013-06-25 Ebay Inc. Item model based on descriptor and images
US20130335437A1 (en) * 2011-04-11 2013-12-19 Vistaprint Technologies Limited Methods and systems for simulating areas of texture of physical product on electronic display
CN102789348A (en) * 2011-05-18 2012-11-21 北京东方艾迪普科技发展有限公司 Interactive three dimensional graphic video visualization system
RU2486608C2 (en) * 2011-08-23 2013-06-27 Федеральное государственное автономное образовательное учреждение высшего профессионального образования "Национальный исследовательский университет "МИЭТ" Device for organisation of interface with object of virtual reality
US20130154907A1 (en) * 2011-12-19 2013-06-20 Grapac Japan Co., Inc. Image display device and image display method
FR2986893B1 (en) * 2012-02-13 2014-10-24 Total Immersion SYSTEM FOR CREATING THREE-DIMENSIONAL REPRESENTATIONS FROM REAL MODELS HAVING SIMILAR AND PREDETERMINED CHARACTERISTICS
US20130293678A1 (en) * 2012-05-02 2013-11-07 Harman International (Shanghai) Management Co., Ltd. Virtual navigation system for video
KR20130133319A (en) * 2012-05-23 2013-12-09 삼성전자주식회사 Apparatus and method for authoring graphic user interface using 3d animations
US9163938B2 (en) * 2012-07-20 2015-10-20 Google Inc. Systems and methods for image acquisition
US9117267B2 (en) 2012-10-18 2015-08-25 Google Inc. Systems and methods for marking images for three-dimensional image generation
US10455219B2 (en) 2012-11-30 2019-10-22 Adobe Inc. Stereo correspondence and depth sensors
US9135710B2 (en) 2012-11-30 2015-09-15 Adobe Systems Incorporated Depth map stereo correspondence techniques
US10249052B2 (en) 2012-12-19 2019-04-02 Adobe Systems Incorporated Stereo correspondence model fitting
US9208547B2 (en) 2012-12-19 2015-12-08 Adobe Systems Incorporated Stereo correspondence smoothness tool
CA2903870C (en) 2013-03-15 2021-06-22 The Coca-Cola Company Display devices for advertising
US20150042757A1 (en) * 2013-08-09 2015-02-12 Makerbot Industries, Llc Laser scanning systems and methods
CN104424662B (en) * 2013-08-23 2017-07-28 三纬国际立体列印科技股份有限公司 Stereoscan device
KR101512084B1 (en) * 2013-11-15 2015-04-17 한국과학기술원 Web search system for providing 3 dimensional web search interface based virtual reality and method thereof
US20150138320A1 (en) * 2013-11-21 2015-05-21 Antoine El Daher High Accuracy Automated 3D Scanner With Efficient Scanning Pattern
TWI510052B (en) * 2013-12-13 2015-11-21 Xyzprinting Inc Scanner
US10200627B2 (en) 2014-04-09 2019-02-05 Imagination Technologies Limited Virtual camera for 3-D modeling applications
EP3175323A4 (en) * 2014-07-31 2018-07-18 Hewlett-Packard Development Company, L.P. Model data of an object disposed on a movable surface
JP6376887B2 (en) * 2014-08-08 2018-08-22 キヤノン株式会社 3D scanner, 3D scanning method, computer program, recording medium
US9761029B2 (en) * 2015-02-17 2017-09-12 Hewlett-Packard Development Company, L.P. Display three-dimensional object on browser
US9361553B1 (en) * 2015-03-26 2016-06-07 Adobe Systems Incorporated Structural integrity when 3D-printing objects
CN104715448B (en) * 2015-03-31 2017-08-08 天脉聚源(北京)传媒科技有限公司 A kind of image display method and device
US10205929B1 (en) * 2015-07-08 2019-02-12 Vuu Technologies LLC Methods and systems for creating real-time three-dimensional (3D) objects from two-dimensional (2D) images
US10013157B2 (en) * 2015-07-22 2018-07-03 Box, Inc. Composing web-based interactive 3D scenes using high order visual editor commands
US10620610B2 (en) * 2015-07-28 2020-04-14 Autodesk, Inc. Techniques for generating motion sculpture models for three-dimensional printing
KR101811696B1 (en) * 2016-01-25 2017-12-27 주식회사 쓰리디시스템즈코리아 3D scanning Apparatus and 3D scanning method
US10452227B1 (en) * 2016-03-31 2019-10-22 United Services Automobile Association (Usaa) System and method for data visualization and modification in an immersive three dimensional (3-D) environment
US10498741B2 (en) 2016-09-19 2019-12-03 Box, Inc. Sharing dynamically changing units of cloud-based content
EP3516644A4 (en) 2016-09-26 2020-05-06 The Coca-Cola Company Display device
US11254152B2 (en) * 2017-09-15 2022-02-22 Kamran Deljou Printed frame image on artwork
US10477186B2 (en) * 2018-01-17 2019-11-12 Nextvr Inc. Methods and apparatus for calibrating and/or adjusting the arrangement of cameras in a camera pair
US10248981B1 (en) 2018-04-10 2019-04-02 Prisma Systems Corporation Platform and acquisition system for generating and maintaining digital product visuals
US10902685B2 (en) * 2018-12-13 2021-01-26 John T. Daly Augmented reality remote authoring and social media platform and system
US10818090B2 (en) 2018-12-28 2020-10-27 Universal City Studios Llc Augmented reality system for an amusement ride
JP2021118479A (en) * 2020-01-28 2021-08-10 株式会社ジャパンディスプレイ Image processing apparatus and head-up display
CN111489428B (en) * 2020-04-20 2023-06-30 北京字节跳动网络技术有限公司 Image generation method, device, electronic equipment and computer readable storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6496183B1 (en) * 1998-06-30 2002-12-17 Koninklijke Philips Electronics N.V. Filter for transforming 3D data in a hardware accelerated rendering architecture
JP2003168132A (en) * 2001-12-03 2003-06-13 Yappa Corp Web 3d object creation system
EP1453011A1 (en) * 2001-10-11 2004-09-01 Yappa Corporation Web 3d image display system

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6437777B1 (en) * 1996-09-30 2002-08-20 Sony Corporation Three-dimensional virtual reality space display processing apparatus, a three-dimensional virtual reality space display processing method, and an information providing medium
US6879946B2 (en) * 1999-11-30 2005-04-12 Pattern Discovery Software Systems Ltd. Intelligent modeling, transformation and manipulation system
JP2001273520A (en) * 2000-03-23 2001-10-05 Famotik Ltd System for integrally displaying multimedia document

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6496183B1 (en) * 1998-06-30 2002-12-17 Koninklijke Philips Electronics N.V. Filter for transforming 3D data in a hardware accelerated rendering architecture
EP1453011A1 (en) * 2001-10-11 2004-09-01 Yappa Corporation Web 3d image display system
JP2003168132A (en) * 2001-12-03 2003-06-13 Yappa Corp Web 3d object creation system

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
JERN M ED - WOLTER F-E ET AL: "Thin vs. fat visualization client", COMPUTER GRAPHICS INTERNATIONAL, 1998. PROCEEDINGS HANNOVER, GERMANY 22-26 JUNE 1998, LOS ALAMITOS, CA, USA,IEEE COMPUT. SOC, US, 22 June 1998 (1998-06-22), pages 772 - 788, XP010291490, ISBN: 0-8186-8445-3 *
MATUSIK W ET AL: "3D TV: A SCALABLE SYSTEM FOR REAL-TIME ACQUISITION, TRANSMISSION, AND AUTOSTEREOSCOPIC DISPLAY OF DYNAMIC SCENES", COMPUTER GRAPHICS PROCEEDINGS, PROCEEDINGS OF SIGGRAPH ANNUAL INTERNATIONAL CONFERENCE ON COMPUTER GRAPHICS AND INTERACTIVE TECHNIQUES, XX, XX, 8 August 2004 (2004-08-08), pages 1 - 11, XP009059236 *
PATENT ABSTRACTS OF JAPAN vol. 2003, no. 10 8 October 2003 (2003-10-08) *
RAPOSO A B ET AL: "Working with remote VRML scenes through low-bandwidth connections", COMPUTER GRAPHICS AND IMAGE PROCESSING, 1997. PROCEEDINGS., X BRAZILIAN SYMPOSIUM ON CAMPOS DO JORDAO, BRAZIL 14-17 OCT. 1997, LOS ALAMITOS, CA, USA,IEEE COMPUT. SOC, US, 14 October 1997 (1997-10-14), pages 34 - 41, XP010248297, ISBN: 0-8186-8102-0 *
VAN DE WETERING H: "Javra : a simple, extensible Java package for VRML", COMPUTER GRAPHICS INTERNATIONAL 2001. PROCEEDINGS 3-6 JULY 2001, PISCATAWAY, NJ, USA,IEEE, 3 July 2001 (2001-07-03), pages 333 - 336, XP010552336, ISBN: 0-7695-1007-8 *

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8368705B2 (en) 2008-07-16 2013-02-05 Google Inc. Web-based graphics rendering system
WO2010008954A3 (en) * 2008-07-16 2010-04-15 Google Inc. Web-based graphics rendering system
WO2010008954A2 (en) * 2008-07-16 2010-01-21 Google Inc. Web-based graphics rendering system
US8723875B2 (en) 2008-07-16 2014-05-13 Google Inc. Web-based graphics rendering system
US8797339B2 (en) 2008-11-07 2014-08-05 Google Inc. Hardware-accelerated graphics for web applications using native code modules
US8675000B2 (en) 2008-11-07 2014-03-18 Google, Inc. Command buffers for web-based graphics rendering
US8294723B2 (en) 2008-11-07 2012-10-23 Google Inc. Hardware-accelerated graphics for web applications using native code modules
US9767597B1 (en) 2008-11-07 2017-09-19 Google Inc. Hardware-accelerated graphics for web application using native code modules
US10026211B2 (en) 2008-11-07 2018-07-17 Google Llc Hardware-accelerated graphics for web applications using native code modules
US9619858B1 (en) 2009-07-02 2017-04-11 Google Inc. Graphics scenegraph rendering for web applications using native code modules
US9824418B1 (en) 2009-07-02 2017-11-21 Google Llc Graphics scenegraph rendering for web applications using native code modules
US10026147B1 (en) 2009-07-02 2018-07-17 Google Llc Graphics scenegraph rendering for web applications using native code modules
CN102214303A (en) * 2010-04-05 2011-10-12 索尼公司 Information processing device, information processing method and program
CN102917236A (en) * 2012-09-27 2013-02-06 深圳天珑无线科技有限公司 Single-camera based stereoscopic photographing method and digital photographing device
CN102917236B (en) * 2012-09-27 2015-12-02 深圳天珑无线科技有限公司 A kind of solid picture-taking method based on single camera and digital camera
KR20170132843A (en) * 2015-03-30 2017-12-04 알리바바 그룹 홀딩 리미티드 Image synthesis method and apparatus
KR102105238B1 (en) 2015-03-30 2020-04-28 알리바바 그룹 홀딩 리미티드 Image synthesis method and apparatus

Also Published As

Publication number Publication date
AU2005331138A1 (en) 2006-11-02
US20080246757A1 (en) 2008-10-09
CA2605347A1 (en) 2006-11-02
BRPI0520196A2 (en) 2009-04-22
EP1877982A1 (en) 2008-01-16
NO20075929L (en) 2007-12-28

Similar Documents

Publication Publication Date Title
EP1877982A1 (en) 3d image generation and display system
US7643025B2 (en) Method and apparatus for applying stereoscopic imagery to three-dimensionally defined substrates
US4925294A (en) Method to convert two dimensional motion pictures for three-dimensional systems
CN103426163B (en) System and method for rendering affected pixels
EP1141893B1 (en) System and method for creating 3d models from 2d sequential image data
CN101189643A (en) 3D image forming and displaying system
US8471898B2 (en) Medial axis decomposition of 2D objects to synthesize binocular depth
US8957892B2 (en) Stereo composition based on multiple camera rigs
US20120182403A1 (en) Stereoscopic imaging
US20110157155A1 (en) Layer management system for choreographing stereoscopic depth
US20110216160A1 (en) System and method for creating pseudo holographic displays on viewer position aware devices
US20150002636A1 (en) Capturing Full Motion Live Events Using Spatially Distributed Depth Sensing Cameras
Inamoto et al. Virtual viewpoint replay for a soccer match by view interpolation from multiple cameras
JP2000215311A (en) Method and device for generating virtual viewpoint image
JP2000503177A (en) Method and apparatus for converting a 2D image into a 3D image
JP3524147B2 (en) 3D image display device
US20040179262A1 (en) Open GL
KR20080034419A (en) 3d image generation and display system
Yang et al. Toward the light field display: Autostereoscopic rendering via a cluster of projectors
CA2540538C (en) Stereoscopic imaging
JP7394566B2 (en) Image processing device, image processing method, and image processing program
JP2004334550A (en) Method for processing three-dimensional image
Ichikawa et al. Multimedia ambiance communication
Morvan et al. Handling occluders in transitions from panoramic images: A perceptual study
Kato et al. Interactive Visual Simulation in a Quasi-three-dimensional World based on the Structuralization of Images

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application
WWE Wipo information: entry into national phase

Ref document number: 2605347

Country of ref document: CA

NENP Non-entry into the national phase

Ref country code: DE

WWW Wipo information: withdrawn in national office

Ref document number: DE

WWE Wipo information: entry into national phase

Ref document number: 2005331138

Country of ref document: AU

WWE Wipo information: entry into national phase

Ref document number: 2005737019

Country of ref document: EP

Ref document number: 200580049839.3

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 1200702493

Country of ref document: VN

Ref document number: 1020077027432

Country of ref document: KR

NENP Non-entry into the national phase

Ref country code: RU

WWW Wipo information: withdrawn in national office

Ref document number: RU

WWP Wipo information: published in national office

Ref document number: 2005331138

Country of ref document: AU

WWP Wipo information: published in national office

Ref document number: 2005737019

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 11912669

Country of ref document: US

ENP Entry into the national phase

Ref document number: PI0520196

Country of ref document: BR

Kind code of ref document: A2