US20030007679A1 - Image synthesizing apparatus and method, position detecting apparatus and method, and supply medium - Google Patents

Image synthesizing apparatus and method, position detecting apparatus and method, and supply medium Download PDF

Info

Publication number
US20030007679A1
US20030007679A1 US10/032,875 US3287501A US2003007679A1 US 20030007679 A1 US20030007679 A1 US 20030007679A1 US 3287501 A US3287501 A US 3287501A US 2003007679 A1 US2003007679 A1 US 2003007679A1
Authority
US
United States
Prior art keywords
image
coordinates
projected
characteristic point
obtaining
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US10/032,875
Inventor
Mitsuharu Ohki
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to US10/032,875 priority Critical patent/US20030007679A1/en
Publication of US20030007679A1 publication Critical patent/US20030007679A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00Manipulating 3D models or images for computer graphics
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/003D [Three Dimensional] image rendering
    • G06T15/10Geometric effects
    • G06T15/20Perspective computation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/003D [Three Dimensional] image rendering

Definitions

  • the present invention relates to image synthesizing apparatus and method, position detecting apparatus and method, and a supply medium, particularly to image synthesizing apparatus and method, position detecting apparatus and method, and a supply medium for making it possible that the projected image of a predetermined object seems to be natural in continuous photographic images by synthesizing the projected image of the predetermined object with a plurality of continuous photographic images.
  • FIG. 1 shows a conventional image synthesizing method.
  • an existent plate 1 is picked up by an image pickup section 2 and balls 5 and 6 formed through CG are synthesized with the image of the plate 1 so that the balls 5 and 6 seem to be stationary.
  • the plate 1 on which patterns are drawn is first actually picked up by an image pickup section 2 .
  • the i-th pattern formed on the plate 1 is assumed as ai.
  • symbol i denotes an integer and also an identification number attached to each pattern ai.
  • the patterns ai are different from each other in color and shape and therefore, they can be distinguished between them.
  • the image pickup section 2 uses a stereophonic video camera constituted with a main video camera and a sub-video camera and is constituted so that three-dimensional information can be obtained from the parallax between an image picked up by the main video camera and an image picked up by the sub-video camera.
  • the plate 1 is picked up while moving the image pickup section 2 .
  • the time for the image pickup section 2 to start pickup is assumed as a first time and a predetermined time after the first time in the pickup time is assumed as a second time.
  • FIG. 1 shows the image pickup section 2 at the first time and the second time.
  • a frame image 1 A 1 obtained by picking up the plate 1 from a predetermined angle is obtained from the image pickup section 2 at the first time.
  • a frame image 1 A 2 different from the frame image 1 A 1 obtained at the first time is obtained at the second time.
  • projected images 5 A 1 and 6 B 1 of the balls 5 and 6 are synthesized with the image (image in FIG. 18) 1 A 1 picked up by the image pickup section 2 at the first time.
  • projected images 5 A 2 and 6 B 2 of the balls 5 and 6 are synthesized with the image (image in FIG. 19) 1 A 2 picked up by the image pickup section 2 at the second time at positions where the balls 5 and 6 seem to be stationary on the plate 1 .
  • a user designates a three-dimensional position (three-dimensional synthesis position)(X1A, Y1A, Z1A) on which the ball 5 is put and a three-dimensional position (three-dimensional synthesis position)(X1B, Y1B, Z1B) on which the ball 6 is put.
  • a synthesis position (two-dimensional position) on the image 1 A 1 corresponding to a three-dimensional synthesis position is obtained.
  • the two-dimensional position is obtained as a position obtained through perspective projection transform of the three-dimensional position. That is, when assuming the focal distance of the image pickup section 2 as f, the two-dimensional synthesis position of a projected image 5 A 1 on the image 1 A 1 at the first time corresponding to the three-dimensional synthesis position (X1A, Y1A, Z1A) is obtained as (X1A ⁇ f/Z1A, Y1A ⁇ f/Z1A).
  • the two-dimensional synthesis position of a projected image 6 B 1 on the image 1 A 1 at the first time corresponding to the three-dimensional synthesis position (X1B, Y1B, Z1B) is obtained as (X1B ⁇ f/Z1B, Y1B ⁇ f/Z1B).
  • the position of the image pickup section 2 at the second time is assumed as an origin O 2 and the horizontal, vertical, and depth directions of the image pickup section 2 are assumed as X-axis, Y-axis, and Z-axis. Then, the three-dimensional position (X2i, Y2i, Z2i) of every pattern ai of the plate 1 is obtained from the two-dimensional image 1 A 2 .
  • the patterns ai on the plate 1 can be distinguished between them, it is possible to identify that the i-th pattern ai in the first three-dimensional coordinate system restored from the image 1 A 1 at the first time corresponds to which pattern ai in the second three-dimensional system restored from the image 1 A 2 at the second time. Therefore, it is possible to make the coordinates (X1i, Y1i, Z1i) of the pattern ai at the first time correspond to the coordinates (X2i, Y2i, Z2i) of the pattern ai at the second time.
  • the second three-dimensional coordinate system can be obtained by applying predetermined rotational transform (hereafter, the function of the transform is assumed as R 1 ) and predetermined rectilinear transform (hereafter, the function of the transform is assumed as S 1 ) to the first three-dimensional coordinate system (though the function of the rectilinear transform is normally shown by T, it is shown by S because T is used as a variable showing time in this specification). Therefore, the relation shown by the following Equation is effected for each pattern ai.
  • predetermined rotational transform hereafter, the function of the transform is assumed as R 1
  • predetermined rectilinear transform hereafter, the function of the transform is assumed as S 1
  • the coordinate transform functions R 1 and S 1 can be obtained by substituting (X1i, Y1i, Z1i) and (X2i, Y2i, Z2i) of each pattern ai for the above Equation.
  • the restored position includes any error. Therefore, when using, for example, the first three-dimensional coordinate system as a criterion, the position of the i-th pattern ai is not exactly present at (X1i, Y1i, Z1i). Similarly, when using the second three-dimensional coordinate system as a criterion, the position of the i-th pattern ai is not exactly present at (X2i, Y2i, Z2i).
  • the three-dimensional position (X2A, Y2A, Z2A) of the ball 5 at the second time corresponding to the three-dimensional position (X1A, Y1A, Z1A) of the ball 5 at the first time is obtained by the following Equation (2).
  • Equation (3) the position (X2B, Y2B, Z2B) of the ball 6 at the second time corresponding to the three-dimensional position (X1B, Y1B, Z1B) of the ball 6 at the first time is obtained by the following Equation (3).
  • the position of the i-th pattern ai and the positions of the balls 5 and 6 at the first time are shown as (X1i, Y1i, Z1i), (X1A, Y1A, Z1A), and (X1B, Y1B, Z1B) in the first three-dimensional coordinate system.
  • the position of the i-th pattern ai and the positions of the balls 5 and 6 at the second time are shown as (X2i, Y2i, Z2i), (X2A, Y2A, Z2A), and (X2B, Y2B, Z2B) in the second three-dimensional coordinate system.
  • the relation between the positions of the i-th patterns ai and the positions of the balls 5 and 6 in these two coordinate systems can be shown by the following Equations (4), (5), and (6).
  • the transform functions R 1 and S 1 are obtained by using the least-squares method and therefore, the functions include errors.
  • the three-dimensional coordinates (obtained from a second image 1 A 2 ) of the pattern ai of the second image 1 A 2 do not accurately coincide with the coordinates obtained by applying the transform functions R 1 and S 1 to the three-dimensional coordinates (X1A, Y1A, Z1A) of a first image 1 A 3 .
  • the relation of projected images 5 A 1 and 6 B 1 to the pattern ai of the first image 1 A 1 at the first time is different from the relation of the projected images 5 A 2 and 6 B 2 to each pattern ai of the second image 1 A 2 at the second time.
  • the balls 5 and 6 seem to be unnatural because the balls 5 and 6 are lifted from the plate 1 on which the patterns ai are drawn.
  • the present invention is made to solve the above problems and its object is to provide an image synthesizing apparatus and method and a supply medium for making it possible that, when synthesizing the projected image of a predetermined object with a plurality of photographic images, a synthesized image seems to be natural by synthesizing the projected image of the predetermined object in accordance with a characteristic point nearby the synthesis position.
  • the image synthesizing apparatus of claim 1 comprises first means for obtaining a first characteristic point nearby a synthesis position for synthesizing an object among the characteristic points projected onto a first image, second means for obtaining first coordinates of a first characteristic point on a three-dimensional coordinate system corresponding to the first image, third means for obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to a second image among the characteristic points projected onto the second image, fourth means for obtaining coordinate transform functions of the first and second coordinates, fifth means for obtaining third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing an object with the first image, sixth means for obtaining fourth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and seventh means for synthesizing the projected image of the object at a position corresponding to the fourth coordinates of the second image.
  • the image synthesizing method of claim 8 comprises the first step of obtaining a first characteristic point nearby a synthesis position for synthesizing an object among the characteristic points projected onto a first image, the second step of obtaining first coordinates on the three-dimensional coordinate system of a first characteristic point corresponding to the first image, the third step of obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to a second image among the characteristic points projected onto the second image, the fourth step of obtaining coordinate transform functions of the first coordinates and the second coordinates, the fifth step of obtaining third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing an object with the first image, the sixth step of obtaining fourth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and the seventh step of synthesizing the projected image of an object at a position corresponding to the fourth coordinates of the second image.
  • the supply medium of claim 15 provides a computer program used for an image synthesizing apparatus for synthesizing the projected image of an object with at least first and second images and comprising the first step of obtaining a first characteristic point nearby a synthesis position for synthesizing an object among the characteristic points projected onto the first image, the second step of obtaining first coordinates of a first characteristic point on a three-dimensional coordinate system corresponding to the first image, the third step of obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, the fourth step of obtaining coordinate transform functions of the first and second coordinates, the fifth step of obtaining third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing an object with the first image, the sixth step of obtaining fourth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and the seventh step of synthe
  • the position detecting apparatus of claim 16 comprises first means for obtaining a first characteristic point nearby an object among the characteristic points projected onto a first image, second means for obtaining first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image, third means for obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, fourth means for obtaining third coordinates of the object on a three-dimensional coordinate system corresponding to the first image, fifth means for obtaining fourth coordinates of the object on a three-dimensional coordinate system corresponding to the second image, sixth means for obtaining coordinate transform functions of the first and second coordinates, seventh means for obtaining fifth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and eighth means for detecting the difference between the fourth coordinates and the fifth coordinates.
  • the position detecting method of claim 17 comprises the first step of obtaining a first characteristic point nearby an object among the characteristic points projected onto a first image, the second step of obtaining first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image, the third step of obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected-onto the second image, the fourth step of obtaining third coordinates of the object on a three-dimensional coordinate system corresponding to the first image, the fifth step of obtaining fourth coordinates of the object on a three-dimensional coordinate system corresponding to the second image, the sixth step of obtaining coordinate transform functions of the first and second coordinates, the seventh step of obtaining fifth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and the eighth step of detecting the difference between the fourth and fifth coordinates.
  • the supply medium of claim 18 provides a computer program used for a position detecting apparatus for detecting the positional relation of an object projected onto first and second images and comprising the first step of obtaining a first characteristic point nearby the object among the characteristic points projected onto the first image, the second step of obtaining first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image, the third step of obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, the fourth step of obtaining third coordinates of the object on a three-dimensional coordinate system corresponding to the first image, the fifth step of obtaining fourth coordinates of the object on a three-dimensional coordinate system corresponding to the second image, the sixth step of obtaining coordinate transform functions of the first and second coordinates, the seventh step of obtaining fifth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and the eighth step of detecting the
  • the image synthesizing apparatus of claim 19 comprises image-pickup-device-position computing means for computing the most-probable position of an image pickup device when photographing each of a plurality of photographic images, characteristic point position computing means for computing the most-probable position of the characteristic point of a first object, virtual projected position computing means for computing a virtual projected position to which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device present at the most-probable position, distortion value computing means for computing a distortion value in accordance with the difference between a virtual projected position and the position of a projected image of a characteristic point actually photographed on each of a plurality of photographic images, correcting means for correcting a position to which the projected image of a second object is set in accordance with the distortion value computed by the distortion value computing means, projected image computing means for computing the projected image of the second object when photographing the second object from an image pickup device at the most-probable position by assuming that the second object is located at the corrected position by
  • the image synthesizing method of claim 21 comprises the image-pickup-device-position computing step of computing the most-probable position of an image pickup device when photographing each of a plurality of photographic images, the characteristic point position computing step of computing the most-probable position of the characteristic point of a first object, the virtual projected position computing step of computing a virtual projected position to which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device present at the most-probable position, distortion value computing step of computing a distortion value in accordance with the difference between a virtual projected position and the position of the projected image of a characteristic point actually photographed on each of a plurality of photographic images, the correcting step of correcting a position to which the projected image of a second object is set in accordance with the distortion value computed in the distortion value computing step, the projected image computing step of computing the projected image of the second object when photographing the second object from an image pickup device at the most-probable position by assuming that the second object is located at
  • the supply medium of claim 23 provides a computer program used for an image synthesizing apparatus for synthesizing the projected image of a second object with each of a plurality of photographic images on which a first object photographed by an image pickup device is photographed by assuming that the second object is located at a predetermined position and comprising the image-pickup-device-position computing step of computing the most-probable position of the image pickup device when photographing each of a plurality of photographic images, the characteristic point position computing step of computing the most-probable position of the characteristic point of the first object, virtual projected position computing step of computing a virtual projected position to which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device present at the most-probable position, the distortion value computing step of computing a distortion value in accordance with the difference between a virtual projected position and the position of the projected image of the characteristic point actually photographed on each of a plurality of photographic images, the correcting step of correcting a position to which the projected image of a second
  • first means obtains a first characteristic point nearby a synthesis position for synthesizing the image of an object among the characteristic points projected onto a first image
  • second means obtains first coordinates of a first characteristic point on a three-dimensional coordinate system corresponding to the first image
  • third means obtains second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to a second image among the characteristic points projected onto the second image
  • fourth means obtains coordinate transform functions of the first and second coordinates
  • fifth means obtains third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing the object with the first image
  • sixth means obtains fourth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates
  • seventh means synthesizes the projected image of the object at a position corresponding to the fourth coordinates of the second image.
  • a first characteristic point nearby a synthesis position for synthesizing an object is obtained among the characteristic points projected onto a first image, first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image are obtained, second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image are obtained, coordinate transform functions of the first and second coordinates are obtained, third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing the object with the first image are obtained, and fourth coordinates on a three-dimensional coordinate system corresponding to the second image are obtained by applying the coordinate transform functions to the third coordinates to synthesize the projected image of the object at a position corresponding to the fourth coordinates of the second image.
  • first means obtains a first characteristic point nearby an object among the characteristic points projected onto a first image
  • second means obtains first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image
  • third means obtains second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image
  • fourth means obtains third coordinates of the object on a three-dimensional coordinate system corresponding to the first image
  • fifth means obtains fourth coordinates of the object on a three-dimensional coordinate system corresponding to the second image
  • sixth means obtains coordinate transform functions of the first and second coordinates
  • seventh means obtains fifth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates
  • eighth means detects the difference between the fourth and fifth coordinates.
  • a first characteristic point nearby an object is obtained among the characteristic points projected onto a first image
  • first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image are obtained
  • second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image are obtained among the characteristic points projected onto the second image
  • third coordinates of the object on a three-dimensional coordinate system corresponding to the first image are obtained
  • fourth coordinates of the object on a three-dimensional coordinate system corresponding to the second image are obtained
  • coordinate transform functions of the first and second coordinates are obtained
  • fifth coordinates on a three-dimensional coordinate system corresponding to the second image are obtained by applying the coordinate transform functions to the third coordinates to detect the difference between the fourth and fifth coordinates.
  • image-pickup-device-position computing means computes the most-probable position of an image pickup device when photographing each of a plurality of photographic images
  • characteristic point position computing means computes the most-probable position of the characteristic point of a first object
  • virtual projected position computing means computes a virtual projected position to which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device present at the most-probable position
  • distortion value computing means computes a distortion value in accordance with the difference between a virtual projected position and the position of the projected image of the characteristic point actually photographed on each of a plurality of photographic images
  • correcting means corrects a position to which the projected image of a second object is set in accordance with the distortion value computed by the distortion value computing means
  • projected image computing means computes the projected image of the second object when photographing the second object from an image pickup device present at the most-probable position by assuming that a second object is located at the position
  • the most-probable position of an image pickup device when photographing each of a plurality of photographic images is computed
  • the most-probable position of a characteristic point of a first object is computed
  • a virtual projected position to which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device set to the most-probable position is computed
  • a distortion value is computed in accordance with the difference between a virtual projected position and the position of the projected image of the characteristic point actually photographed on each of a plurality of photographic images
  • a position to which the projected image of a second object is set in accordance with the computed distortion value is corrected
  • the projected image of the second object when photographing the second object from an image pickup device present at the most-probable position is computed by assuming that the second object is located at a corrected position to synthesize the projected image of the computed second object with each photographic image.
  • FIG. 1 is an illustration showing a positional relation between the image pickup section 2 and the plate 1 of a conventional image synthesizing method
  • FIG. 2 is an illustration showing an image pickup example of the plate 1 of the image pickup section 2 in FIG. 1;
  • FIG. 3 is an illustration of another image pickup example of the plate 1 of the image pickup section 2 in FIG. 1;
  • FIG. 4 is an illustration showing a synthesizing example of the balls 5 and 6 in FIG. 1;
  • FIG. 5 is an illustration showing another synthesizing example of the balls 5 and 6 in FIG. 1;
  • FIG. 6 is an illustration showing a positional relation between the image pickup section 2 and the plate 1 in FIG. 1;
  • FIG. 7 is an illustration showing another positional relation between the image pickup section 2 and the plate 1 in FIG. 1;
  • FIG. 8 is a block diagram showing the structure of the first embodiment of an image synthesizing apparatus of the present invention.
  • FIG. 9 is a flowchart for explaining an operation of the embodiment in FIG. 8;
  • FIG. 10 is an illustration showing a positional relation between the image pickup section 2 and the plate 1 of the embodiment in FIG. 8;
  • FIG. 11 is an illustration showing another positional relation between the image pickup section 2 and the plate 1 of the embodiment in FIG. 8;
  • FIG. 12 is a flowchart for explaining another operation of the embodiment in FIG. 8;
  • FIG. 13 is an illustration showing a positional relation between the image pickup section 2 and the plate 1 of the embodiment in FIG. 8;
  • FIG. 14 is an illustration showing another positional relation between the image pickup section 2 and the plate 1 of the embodiment in FIG. 8;
  • FIG. 15 is an illustration for explaining a relation between an image pickup device and a world coordinate system
  • FIG. 16 is an illustration for explaining the projected image of a point based on world coordinates
  • FIG. 17 is a block diagram showing the structure of the second embodiment of an image synthesizing apparatus of the present invention.
  • FIG. 18 is a flowchart for explaining an operation of the embodiment in FIG. 17;
  • FIG. 19 is an illustration for explaining the processing for picking up an object with an image pickup device
  • FIG. 20 is an illustration showing an example of synthesizing a CG image with an actually picked-up image
  • FIG. 21 is an illustration for explaining the change of an actually picked-up image
  • FIG. 22 is an illustration for explaining the change of CG images.
  • FIG. 23 is an illustration for explaining the change of a synthesized image of a picked-up image with a CG image.
  • the image synthesizing apparatus of claim 1 comprises first means (e.g. step S 3 in FIG. 9) for obtaining a first characteristic point nearby a synthesis position for synthesizing an object among the characteristic points projected onto a first image, second means (e.g. step S 2 in FIG. 9) for obtaining first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image, third means (e.g. step S 6 in FIG. 9) for obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, fourth means (e.g. step S 7 in FIG.
  • step S 3 in FIG. 9 for obtaining coordinate transform functions of the first and second coordinates
  • fifth means e.g. step S 3 in FIG. 9 for obtaining third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing the object with the first image
  • sixth means e.g. step SB in FIG. 9 for obtaining fourth coordinates on a three-dimensional coordinate system corresponding to the second image by applying a coordinate transform function to the third coordinates
  • seventh means e.g. step S 10 in FIG. 9) for synthesizing the projected image of the object at a position corresponding to the fourth coordinates of the second image.
  • the position detecting apparatus of claim 16 comprises first means (e.g. step S 34 in FIG. 12) for obtaining a first characteristic point nearby an object among the characteristic points projected onto a first image, second means (e.g. step S 32 in FIG. 12) for obtaining first coordinates of a first characteristic point on a three-dimensional coordinate system corresponding to the first image, third means (e.g. step S 36 in FIG. 12) for obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, fourth means (e.g. step S 33 in FIG.
  • first means e.g. step S 34 in FIG. 12
  • second means e.g. step S 32 in FIG. 12
  • third means e.g. step S 36 in FIG. 12
  • fourth means e.g. step S 33 in FIG.
  • step S 12 for obtaining third coordinates of the object on a three-dimensional coordinate system corresponding to the first image
  • fifth means e.g. step S 37 in FIG. 12
  • sixth means e.g. step S 38 in FIG. 12
  • seventh means e.g. step S 39 in FIG. 12
  • eighth means e.g. step S 40 in FIG. 12 for detecting the difference between the fourth and fifth coordinates.
  • the image synthesizing apparatus of claim 19 comprises image-pickup-device-position computing means (e.g. step S 61 in FIG. 18) for computing the most-probable position of an image pickup device when photographing each of a plurality of photographic images, characteristic point position computing means (e.g. step S 61 in FIG. 18) for computing the most-probable position of the characteristic point of a first object, virtual projected position computing means (e.g. step S 61 in FIG. 18) for computing a virtual projected position on which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device set at the most-probable position, distortion value computing means (e.g. step S 63 in FIG.
  • image-pickup-device-position computing means e.g. step S 61 in FIG. 18
  • characteristic point position computing means e.g. step S 61 in FIG. 18
  • virtual projected position computing means e.g. step S 61 in FIG. 18
  • distortion value computing means e.g
  • correcting means e.g. step S 66 in FIG. 18
  • correcting means for correcting a position to which the projected image of a second object is set in accordance with the distortion value computed by the distortion value computing means
  • projected image computing means e.g. step S 67 in FIG. 18
  • synthesizing means e.g. step S 67 in FIG. 18 for synthesizing the projected image of the second object computed by the projected image computing means with each photographic image.
  • FIG. 8 An embodiment of an image synthesizing apparatus of the present invention is described below by referring to FIG. 8.
  • the image synthesizing apparatus of the present invention has an input section 11 to be operated by a user when the user inputs the synthesis position of a CG image (image of the ball 5 or 6 ).
  • the input section 11 is constituted with, for example, a keyboard, a touch panel, a mouse, and a remote controller.
  • a control section 12 controls each section in order to synthesize CG data. Moreover, the control section 12 has a memory 12 A in which a program for synthesizing the CG data is stored.
  • a computing section 13 performs various operations for synthesizing the CG data input from a CG data storing section 16 with the image data input from an image data accumulating section 14 . Moreover, the computing section 13 is constituted so as to output an image synthesized with the CG data to a reproducing section 18 .
  • the control section 12 and the computing section 13 are realized by, for example, a microcomputer.
  • An image pickup section 2 uses, for example, a stereophonic video camera constituted with a main video camera 2 A and a sub-video camera 2 B and is constituted so as to pickup a plate 1 with the main video camera 2 A and sub-video camera 2 B and output the picked-up image data to the image data accumulating section 14 .
  • the image data accumulating section 14 is controlled by the control section 12 , which accumulates the image data input from the image pickup section 2 and outputs the data to the computing section 13 .
  • the CG data storing section 16 is controlled by the control section 12 , which stores CG data (image data of the balls 5 and 6 ) to be synthesized with the image data picked up by the image pickup section 2 and outputs the CG data to the computing section 13 .
  • a display section 17 displays the image data picked up by the image pickup section 2 or the positional information of the plate 1 output from the computing section 13 .
  • a reproducing section 18 reproduces image data with which the CG data input from the computing section 13 is synthesized.
  • step S 1 a user operates the input section 11 to command it to start pickup of the plate 1 .
  • the input section 11 outputs a signal corresponding to the command to the control section 12 .
  • the control section 12 makes the image data accumulating section 14 accumulate the image data sent from the image pickup section 2 in accordance with the signal.
  • the image pickup section 2 (the main video camera 2 A and sub-video camera 2 B) picks up the plate 1 while being moved by the user and outputs the picked-up image data to the image data accumulating section 14 , and the image data is stored in the image data accumulating section 14 .
  • the number of images picked up by the image pickup section 2 is N.
  • the control section 12 makes the memory 12 A store the identification numbers k to use the memory 12 A as a counter for synthesis processing.
  • the control section 12 controls the image data accumulating section 14 in step S 2 to make the section 14 output the image data accumulated in the section 14 to the computing section 13 .
  • the computing section 13 obtains a characteristic point (e.g. pattern on the plate 1 , defect on the plate 1 , or other pattern) which extremely changes compared to a point to which a brightness signal or color-difference signal of the image data input from the image data accumulating section 14 is adjacent and therefore, the characteristic point can be separated from other part of the image data.
  • the i-th characteristic point (pattern) is assumed as ai.
  • symbol i denotes an integer which is an identification number provided for each pattern ai.
  • the computing section 13 compares the same patterns ai out of the image data input from the main video camera 2 A and the sub-video camera 2 B at the same time, calculates the parallax between the patterns, and obtains the three-dimensional position (coordinates) of every pattern (characteristic point) ai. Because the three-dimensional position is obtained by using the parallax between the cameras, this position is a position on three-dimensional coordinates using the camera (image pickup section 2 ) as an origin. That is, for example, the position in the three-dimensional coordinates shown in FIG. 10 to be described later is obtained.
  • the position of the image pickup section 2 at the first time is an origin O 1 and the horizontal direction, vertical direction, and depth direction of the image pickup section 2 are X-axis, Y-axis, and Z-axis.
  • the computing section 13 obtains the three-dimensional position (X1i, Y1i, Z1i) of the every pattern ai on the image 1 A 1 of the plate 1 at the first time.
  • step S 3 the computing section 13 outputs the position of every pattern ai and the first image 1 A 1 to the display section 17 to make the section 17 display the position of every pattern ai and the position of the first image 1 A 1 .
  • the user operates the input section 11 , and designates and inputs the three-dimensional position (X1A, Y1A, Z1A) of the ball 5 for synthesizing the projected image 5 A 1 of the ball 5 with the first image 1 A 1 and the three-dimensional position (X1B, Y1B, Z1B) of the ball 6 for synthesizing the projected image 6 B 1 of the ball 6 with the first image 1 A 1 based on the position information at the first time displayed in the display section 17 .
  • a predetermined number of patterns e.g. 12 patterns
  • step 64 is started to perform image synthesis processing. That is, the computing section 13 obtains a two-dimensional synthesis position on the first image 1 A 1 corresponding to a three-dimensional synthesis position.
  • the two-dimensional synthesis position is obtained by perspective-projection-transforming the three-dimensional synthesis position.
  • the two-dimensional position of the projected image 5 A 1 of the ball 5 on the image 1 A 1 at the first time corresponding to the three-dimensional position (X1A, Y1A, Z1A) is shown as (X1A ⁇ f/Z1A, Y1A ⁇ f/Z1A).
  • the position of the projected image 6 B 1 of the ball 6 on the two-dimensional image 1 A 1 at the first time corresponding to the three-dimensional position (X1B, Y1B, Z1B) is shown as (X1B ⁇ f/Z1B, Y1B ⁇ f/Z1B).
  • the control section 12 controls the CG data storing section 16 to output the CG data (image data for the balls 5 and 6 ) stored in the CG data storing section 16 to the computing section 13 . Then, the computing section 13 synthesizes the CG data at the obtained synthesis position.
  • step S 5 the control section 12 sets an image identification number k to 2.
  • the method for obtaining the position of the pattern ai is the same as the method for obtaining the position of the pattern ai of the first image 1 A 1 .
  • it is assumed that the positions of the patterns Aj and Bj (j 1, . . .
  • step S 7 the computing section 13 obtains the transform functions R 2 and S 2 by substituting (X(k ⁇ 1)Aj, Y(k ⁇ 1)Aj, Z(k ⁇ 1)Aj) and (XkAj, YkAj, ZkAj) for the above Equation (9) on each pattern Aj.
  • the computing section 13 obtains the transform functions R 3 and S 3 by substituting (X(k ⁇ 1)Bj, Y(k ⁇ 1)Bj, Z(k ⁇ 1)Bj) and (XkBj, YkBj, ZkBj) for the above Equation (10) on each pattern Bj.
  • the restored position includes any error. Therefore, when using, for example, the (k ⁇ 1)-th three-dimensional coordinate system as a criterion, the positions of the j-th patterns Aj and Bj are not accurately present at (X(k ⁇ 1)Aj, Y(k ⁇ 1)Aj, Z(k ⁇ 1)Aj) and (X(k ⁇ 1)Bj, Y(k ⁇ 1)Bj, Z(k ⁇ 1)Bj).
  • the positions of the j-th patterns Aj and Bj are not accurately present at (XkAj, YkAj, ZkAj) and (XkBj, YkBj, ZkBj).
  • the computing section 13 totalizes values obtained by squaring the magnitude of the three-dimensional vector (x4, y4, z4) in the following Equation (11) on all patterns Aj in accordance with the least-squares method to obtain (most-probable) transform functions R 2 and S 2 for minimizing the total value.
  • step S 8 the computing section 13 obtains the three-dimensional position (XkA, YkA, ZkA) of the ball 5 at the k-th time in accordance with the following Equation (12). This position is the position in the three-dimensional coordinate system shown in FIG. 11.
  • the computing section 13 totalizes values obtained by squaring three-dimensional vector (x5, y5, z5) on all patterns Bj in accordance with the least-squares method to obtain (most-probable) transform functions R 3 and S 3 for minimizing the totalized value as shown by the following Equation (13).
  • step S 8 the computing section 13 obtains the position (XkB, YkB, ZkB) of the projected image 6 B k of the ball 6 at the k-th time from the following Equation (14). This position is the position in the three-dimensional coordinate system shown in FIG. 11.
  • R 2 , S 2 , R 3 , and S 3 depend on the positional relation between the image pickup sections 2 at the (k ⁇ 1)-th time and the k-th time. That is, R 2 , S 2 /R 3 , and S 3 are functions of k, and R 2 equals R 2 (k), S 2 equals S 2 (k), R 3 equals R 3 (k), and S 3 equals S 3 (k).
  • step S 9 the computing section 13 obtains three-dimensional vectors (x6, y6, z6) between the ball 5 and the pattern ai on all patterns ai in accordance with the following Equation (15) and selects 12 patterns (in the portion enclosed by the frame 21 in FIG. 11) Aj in the k-th three-dimensional coordinate system in order of magnitudes of the vectors starting with a vector having the smallest magnitude (starting with a vector closest to the ball 5 ) as shown by the following Equation (15).
  • the computing section 13 obtains three-dimensional vectors (x7, y7, z7) between the ball 6 and the pattern ai on all patterns ai in accordance with the following Equation (16) and selects 12 patterns (in the portion enclosed by the frame 22 in FIG. 11) Bj in the k-th three-dimensional coordinate system in order of magnitudes of the vectors starting with a vector having the smallest magnitude (starting with a vector closest to the ball 6 ).
  • ( x7 , y7 , z7 ) ( Xki , Yki , Zki ) - ( XkB , YkB , ZkB ) ( 16 )
  • step S 10 the computing section 13 synthesizes the projected image 5 A k of the ball 5 at the position of (XkA ⁇ f/ZkA, YkA ⁇ f/ZkA) by using the relation of the perspective projection transform in the k-th image 1 A k . Moreover, when there is any object other than the projected image 5 A k in the k-th image 1 A k at the image pickup section- 2 side, the computing section does not synthesize the projected image 5 A k if it is behind the object.
  • the computing section 13 synthesizes the projected image 6 B k of the ball 6 at the position of (XkB ⁇ f/ZkB, YkB ⁇ f/ZkB) by using the relation of the perspective projection transform in the k-th image 1 A k . Also in this case, when there is any object other than the projected image 6 B k in the k-th image 1 A k at the image pickup section- 2 side, the computing section 13 does not synthesize the projected image 6 B k if it is behind the object.
  • a pattern nearby a synthesis position is selected without considering a pattern present far from the synthesis position (that is, a pattern not to be sensed by a person) to obtain the rotational transform and rectilinear transform for the pattern. Therefore, it is possible to synthesize an image so that the balls 5 and 6 seem to be stationary on the plate 1 .
  • the stationary balls 5 and 6 are synthesized with the plate 1 .
  • step S 31 the image pickup section 2 picks up the plate 1 while being moved by, for example, a user.
  • N the number of images picked up by the image pickup section 2 within a pickup time.
  • the control section 12 makes a memory 12 A store the identification numbers k and uses the memory 12 as a synthesis-processing counter.
  • the computing section 13 obtains the three-dimensional position (X1i, Y1i, Z1i) of every pattern (characteristic point) ai on the plate 1 at the first time in step S 32 . Moreover, the computing section 13 obtains the three-dimensional position (X1C, Y1C, Z1C) of the ball 31 and the three-dimensional position (X1D, Y1D, Z1D) of the ball 32 at the first time in step S 33 .
  • the computing section 13 obtains three-dimensional vectors (x8, y8, z8) between the ball 31 and the pattern ai on all patterns ai in step S 34 in accordance with the following Equation (17) and selects 12 patterns (in the portion enclosed by the frame 41 in FIG. 13) Cj in the first three-dimensional coordinate system in order of magnitudes of the vectors starting with a vector having the smallest magnitude (starting with a vector closest to the ball 31 ).
  • the computing section 13 obtains three-dimensional vectors (x9, y9, z9) between the ball 32 and the pattern ai on all patterns ai in accordance with the following Equation (18) and selects 12 patterns (in the portion enclosed by the frame 42 in FIG. 13) Dj in the first three-dimensional coordinate system in order of magnitudes of the vector starting with a vector having the smallest magnitude (starting with a vector closest to the ball 32 ).
  • step S 35 the control section 12 sets the identification number k of image data to 2.
  • the computing section 13 obtains the three-dimensional position (Xki, Yki, Zki) of the pattern ai in the k-th image 1 A k in step S 36 .
  • the computing section 13 obtains the position (XkC, YkC, ZkC) of the ball 31 and the position (XkD, YkD, ZkD) of the ball 32 at the k-th time in step S 37 .
  • predetermined rotational transform hereafter, the transform function of the rotational transform is assumed as R 4
  • predetermined rectilinear transform hereafter, the transform function of the rectilinear transform is assumed as S 4
  • the computing section 13 totalizes values obtained by squaring the magnitude of three-dimensional vector (x12, y12, z12) on all patterns Cj by the least-squares method in accordance with the following Equation (19) to obtain (most-probable) transform functions R 4 and S 4 for minimizing the total value in step S38 by considering that the restored position includes any error.
  • the computing section 13 obtains a position (XkC′, YkC′, ZkC′) when the ball 32 is stationary in accordance with the following Equation (20) in step S 39 by using the transform functions R 4 and S 4 thus obtained.
  • the three-dimensional vector (xkc, ykc, zkc) in the following Equation (21) shows the movement of the ball 31 between the (k ⁇ 1)-th time and the k-th time. This operation is performed in step S 40 .
  • the computing section 13 totalizes values obtained by squaring the magnitude of the three-dimensional vector (x13, y13, z13) in the following Equation (22) on all patterns Dj by the least-squares method to obtain (most-probable) transform functions R 5 and S 5 for minimizing the total value by considering that the restored position includes any error.
  • the computing section 13 obtains a position (XkD′, YkD′, ZkD′) when the ball 32 is stationary in accordance with the following Equation (23) by using the transform functions R 5 and S 5 thus obtained.
  • Equation (24) it is possible to obtain the three-dimensional vector (xkd, ykd, zkd) in the following Equation (24), that is, the movement of the ball 32 between the (k ⁇ 1)-th time and the k-th time.
  • the computing section 13 obtains three-dimensional vectors (x10, y10, z10) between the ball 31 and the pattern ai on all patterns ai in step S 41 as shown by the following Equation (25) and selects 12 patterns (in the portion enclosed by the frame 41 in FIG. 14) Cj in the k-th three-dimensional coordinate system in order of magnitudes of the vectors starting with a vector having the smallest magnitude (starting with a vector closest to the ball 31 ).
  • the computing section 13 obtains three-dimensional vectors (x11, y11, z11) between the ball 32 and the pattern ai on all patterns ai as shown by the following Equation (26) and selects 12 patterns (in the portion enclosed by the frame 42 in FIG. 14) Dj in the k-th three-dimensional coordinate system in order of magnitudes of the vectors starting with a vector having the smallest magnitude (starting with a vector closest to the ball 32 ).
  • step S 42 the control section 12 decides whether synthesis processing is completed on the whole image data picked up by the image pickup section 2 .
  • the computing section 13 starts step S 44 to make the display section 17 display movement values (xkc, ykc, zkc) and (xkd, ykd, zkd) at each time.
  • a characteristic point uses a two-dimensional object such as the pattern ai on the plate 1 .
  • a two-dimensional object such as the pattern ai on the plate 1 .
  • a characteristic point is selected through the operation by the computing section 13 .
  • a user selects the characteristic point by operating the input section 11 during processing.
  • 12 patterns (characteristic points) Aj and Bj nearby a synthesis position are selected.
  • a rotational transform function and a rectilinear transform function by selecting 20 patterns close to a synthesis position, providing a weight larger than that of 8 remaining patterns for 12 patterns closer to the synthesis position out of 20 patterns, and using the least-squares method.
  • the obtained rotational transform function and rectilinear transform function serve as a rotational transform function and a rectilinear transform function having a relatively large error for 8 relatively remote patterns and serve as a rotational transform function and a rectilinear transform function having a completely controlled error for the 12 patterns close enough.
  • This is favorable because this corresponds to the human-eye characteristic of recognizing the positional relation between characteristic points separate from each other to a certain extent (the above 8 patterns) at a relatively rough accuracy and recognizing the positional relation between characteristic points close enough to each other (the above 12 patterns) at a high accuracy.
  • the present invention does not restrict the number of characteristic points to 12 or 20.
  • the picked up plate 1 is synthesized with CG data so as to simplify the description.
  • the plate 1 is a real image photographed by a camera and a characteristic point in the plate 1 corresponds to a characteristic point such as an edge in the real image.
  • a characteristic point such as an edge in the real image.
  • a corner of the desk or stain (dust or the like) on the wall serves as a characteristic point.
  • a small ball formed through CG is synthesized with a photographic image.
  • eight vertexes of the rectangular parallelepiped are assumed as independent objects, each vertex is synthesized with a characteristic point nearby the vertex so that the positions coincide with each other, and sides and planes are synthesized at positions where these vertexes are connected by straight lines. That is, because vertexes are positioned, a person does not sense that the vertexes of the rectangular parallelepiped are displaced. However, sides and planes are not accurately positioned.
  • the synthesized image string when processing a photographic image string obtained by continuously photographing a first object and thereby, generating a new photographic image string showing as if a second object is present at the string, that is, when synthesizing the second object with the original photographic image string and thereby, generating a synthesized image string, the synthesized image string may be unnatural because a synthesis position is slightly displaced in each synthesized image.
  • a projected image at a certain time Th is noticed.
  • the projected position of a characteristic point when photographing the characteristic point from the obtained position and attitude (direction) of the image pickup device including errors but not showing true values at the time Th by assuming that the characteristic point is present at the position of the obtained characteristic point including errors but not showing a true value is referred to as a virtual projected position.
  • the displacement value (distortion value) is obtained.
  • the distortion value depends on the position. Moreover, because the distortion value depends on time, it is obtained for each time. In other words, the distortion value is the function of position and time.
  • the position (virtual position) of the object (second object) to be synthesized is determined by an operator in accordance with the relation between the position of the second object and the position of the above obtained characteristic point.
  • the virtual position of the second object determined by the operator is distorted by a value equal to the distortion value obtained above.
  • the projected image of the second object is obtained by assuming that the second object is present at the distorted position.
  • the position of the projected image of the second object coincides with that of the projected image of the characteristic point in the actually-photographed photographic image Ph.
  • Equations (30) to (32) are derived.
  • (XXhj, YYhj, ZZhj) denotes the position of the j-th point on a coordinate system based on the image pickup device at the time Th.
  • Equations (31) and (32) When solving the Equations (31) and (32) by substituting the value of (Uhj, Vhj) for the Equations (31) and (32), the indeterminacy of how to put the world coordinates remains. However, this is not important. For example, it is possible to make the position and attitude (direction) of the image pickup device at the first time T 1 coincide with the world coordinates. In other words, it is possible to solve the Equations (31) and (32) under the condition that R 1 is a zero matrix and S 1 is a zero vector.
  • FIG. 17 shows an example of the structure of the image synthesizing apparatus of the second embodiment.
  • a processing circuit 32 performs various types of operations in accordance with a program stored in a program memory 31 .
  • a data memory 33 is a memory for storing the data currently processed by the processing circuit 32 .
  • a frame memory 34 stores the image data to be displayed on an image display unit 35 .
  • An input unit 36 is constituted with, for example, a keyboard and a mouse and operated to input various commands.
  • An input/output terminal 37 is connected with a not-illustrated external unit to transfer data.
  • a bus line 38 is a bus for connecting these units each other.
  • step S 61 it is assumed to put (synthesize) a building (second object: shown as a building 91 in FIG. 20) formed through CG between two real buildings (first objects: shown as a building 71 and a building 81 in FIG. 19). Then, in step S 61 , the above two real buildings (first objects) are photographed while moving a video camera (image pickup device). The image data for the two buildings is stored in the data memory 33 through the bus line 38 from the input/output terminal 37 . In this case, because the position of the projected image of a building in each photographic image moves with passage of time, it is also necessary to displace the position of the projected image of the building (second object) to be synthesized in the photographic image. This state is shown in FIGS. 19 to 23 .
  • FIG. 21 shows photographic images P 1 , P 2 , . . . , Ph, . . . , PH taken by the video camera at the times T 1 , T 2 , . . . , Th, . . . , TH.
  • Symbols 71 - 1 , 71 - 2 , . . . , 71 -h, . . . , 71 -H denote projected images of the above real building 71 at various times.
  • Symbols 81 - 1 , 81 - 2 , . . . , 81 -h, . . . , 81 -H denote projected images of the above real building 81 at various times.
  • a characteristic point is a point at which brightness or color suddenly changes.
  • a characteristic point is a corner of a rectangular parallelepiped (e.g. building) or black point on a white plane (e.g. black dust attached to white wall).
  • corners 72 to 75 and 82 to 85 of the buildings 71 and 81 in FIG. 19 are first characteristic points.
  • the plane enclosed by the corners 72 to 75 is the front of the building 71 and the plane enclosed by the corners 82 to 85 is the front of the building 81 .
  • the position (U1j, V1j) of the projected image of the j-th characteristic point is obtained out of the first characteristic points (total of J characteristic points) appearing on the photographic image P 1 at the time T 1 .
  • the position (U2j, V2j) of the projected image of the j-th characteristic point is obtained out of the first characteristic points appearing on the photographic image P 2 at the time T 2 .
  • the obtained data is stored in the data memory 33 .
  • the obtained rotation matrix Rh and parallel displacement vector Sh show the position and attitude (direction) of the video camera at the time Th based on the world coordinates.
  • the obtained (Xj, Yj, Zj) shows the position of the j-th characteristic point based on the world coordinates.
  • the corner 75 (FIG. 19) which is one of the first characteristic points corresponds to the k-th characteristic point.
  • Symbol k denotes any numerical value of 1 to J.
  • the position (Xk, Yk, Zk) of the k-th characteristic point based on the world coordinates is obtained.
  • the position does not show a true value but errors are included.
  • the corner 75 is the corner at the front top right of the real building 71 .
  • the photographic image Ph at the time Th is a photographic image photographed when the video camera is moved from the origin of the world coordinates due to the rotation of the matrix Rh and the translation of the vector Sh. Therefore, the matrix Rh and the vector Sh are also obtained.
  • the above states ⁇ position and attitude (direction) shown by the rotation matrix Rh and parallel displacement vector Sh ⁇ do not show true values but errors are included.
  • the data for the position (Xj, Yj, Zj) is supplied to an image display unit 35 through the frame memory 34 and displayed in step S 62 .
  • an operator person performing the synthesis operation designates a position to which the building (second object) 91 formed through CG (position in a virtual three-dimensional real space) is set by operating the input unit 36 .
  • the operator determines the above operation while considering the positions of the two real buildings (first objects) 71 and 81 in a three-dimensional real space.
  • the determined positions are stored in the data memory 33 .
  • the building (second object) 91 formed through CG is put between the positions. That is, it is necessary to set the building (second object) 91 formed through CG so that the plane enclosed by the corners 75 , 74 , 83 , and 82 serves as the front of the building (second object) 91 formed through CG (so that the top left corner 92 and the bottom left corner 93 of the building 91 correspond to the top right corner 75 and the bottom right corner 74 of the building 71 and the top right corner 95 and bottom right corner 94 of the building 91 correspond to the top left corner 82 and the bottom left corner 83 of the building 81 ).
  • the position of the building (second object) 91 formed through CG is determined.
  • the building (second object) 91 formed through CG is virtually set so that the front top left of the building (second object) 91 formed through CG is brought to the position (Xk, Yk, Zk) based on the world coordinates (FIG. 20).
  • the front top left position of the building (second object) 91 formed through CG is (Xk, Yk, Zk).
  • the position (Xk, Yk, Zk) is the position of the corner 75 serving as the k-th characteristic point.
  • the real building 71 and the building (second object) 91 formed through CG are set so that the front top right of the building 71 contacts the front top left of the building 91 .
  • the photographic image Ph at the time Th includes the projected image of the corner 75 which is the k-th characteristic point.
  • the position of the projected image is shown as (Uhk, Vhk). Therefore, the following Equations (34) and (35) are effected.
  • Rh, Sh, Xk, Yk, and Zk are values obtained by minimizing the Equation (33).
  • the right and left sides of the Equations (34) and (35) have almost equal values. As previously described, however, they are not strictly equal to each other. However, the position (UUhk, VVhk) of the projected image at the front top left of the building 91 completely meets the following Equations (36) and (37).
  • Equations (34) and (35) are constant every time Th. Therefore, when a person continuously views an image synthesized every time, he feels as if the interval between the front top right of the real building 71 and the front top left of the building 91 formed through CG swings.
  • FIG. 23 shows synthesized images under an ideal state free from errors, (Uhk, Vhk) is equal to (UUhk, VVhk). In fact, however, (Uhk, Vhk) is not equal to (UUhk, VVhk).
  • the Rh, Sh, Xj, Yj, and Zj obtained in step S 61 and the position (Uhj, Vhj) of the projected image of the j-th characteristic point in the photographic image Ph do not completely meet the Equations (31) and (32). That is, the following Equations (38) and (39) are effected.
  • Equations (38) and (39) are effected.
  • the processing circuit 32 reads each time Th and the data for characteristic points from the data memory 33 in step S 63 , computes the following Equations (40) to (44) for the time Th and the characteristic point data, obtains the coordinates (XXhj, YYhj, ZZhj) and the displacement value ( ⁇ XXhj, ⁇ YYhj) of each characteristic point based on the video camera, and stores the coordinates and displacement value in the data memory 33 .
  • the corner 75 (k-th characteristic point) shown in FIG. 12 is considered below.
  • the position of the characteristic point becomes the position (XXhk, YYhk, ZZhk) shown by the following Equations (48) to (50) in the coordinate system based on the video camera at the time Th.
  • Vhk f ( YYhk/ZZhk ) (52)
  • the errors ( ⁇ XX, ⁇ YY) are computed in accordance with the Rh, Sh, Xk, Yk, and Zk obtained in step S 61 , measured values (Uhk, Vhk), and the Equations (48) to (50) and the Equations (53) and (54).
  • step S 62 It is determined in step S 62 to imaginarily set the corner 92 at the front top left of the building (second object) 91 formed through CG to the position (Xk, Yk, Zk) based on the world coordinates. That is, the corner 92 is set to the position (XXhk, YYhk, ZZhk) shown by the Equations (48) to (50) in the coordinate system based on the video camera at the time Th. The corner 92 is projected on the position (UUhk, VVhk) shown by the following Equations (55) and (56) in the photographic image at the time Th.
  • VVhk f ( YYhk/ZZhk ) (56)
  • Equations (48) to (50) and the Equations (55) and (56) are equivalent to the Equations (36) and (37).
  • the position (UUhk, Whk) shown by the Equations (55) and (56) is different from the position (Uhk, Vhk) of the projected image of the k-th actual characteristic point. This is a positional displacement becoming a problem.
  • the portion of the building (second object) 91 formed through the CG shown by the position (XXhk, YYhk, ZZhk) in the coordinate system based on the video camera at the time Th is displaced by the value shown by the Equations (53) and (54) before forming the projected image of the second object. That is, it is determined in step S 62 to set the corner 92 at the front top left of the building 91 to the position (XXhk, YYhk, ZZhk) in the coordinate system based on the video camera at the time Th.
  • the position is displaced by the value shown by the Equations (53) and (54) so that it is brought to (XXhk+ ⁇ XX, YYhk+ ⁇ YY, ZZhk).
  • the position of the corner 92 at the front top left of the building 91 is brought to the position shown by the following Equations (57) and (58).
  • VVhk f ( YYhk+ ⁇ YY )/ ZZhk (58)
  • step S 61 it is decided in step S 61 that the k-th characteristic point is present at the position (Xk, Yk, Zk) based on the world coordinates.
  • This position is the position (XXhk, YYhk, ZZhk) shown by the Equations (48) to (50) in the coordinate system based on the video camera at the time Th.
  • This value is displaced by ( ⁇ XX, ⁇ YY) shown by the Equations (53) and (54) from the position (Uhk, Vhk) of the projected image of the k-th characteristic point projected on the photographic image Ph at the time Th in a three-dimensional real space.
  • a projected image is formed by assuming that the projected image is present at a position displaced by ( ⁇ XX, ⁇ YY).
  • the position of the projected image coincides with that of the projected image of the actually-photographed object (corner 75 at the front top right of the building 71 ).
  • the corner 75 (k-th characteristic point) is described.
  • characteristic points other than the k-th characteristic point In the case of an object (or a part of the object) to be synthesized present at a position just coinciding with the first characteristic points (first, . . . , j-th characteristic points), it is necessary to compute ( ⁇ XX, ⁇ YY) which is the displacement value (distortion value) of a corresponding characteristic point and form the projected image of the object by assuming that the projected image is present at a position displaced by ( ⁇ XX, ⁇ YY), as described above.
  • step S 64 the processing circuit 32 reads the characteristic point (XXhj, YYhj, ZZhj) at each time Th from the data memory 33 , performs Delaunay division by using the characteristic point as a mother point, obtains a Delaunay convex polygon, and stores the polygon in the data memory 33 . That is, the position of the j-th characteristic point in the coordinate system based on the video camera at the time Th is assumed as (XXhj, YYhj, ZZhj). This position can be obtained from the Equations (40) to (42) by using the Rh, Sh, Xj, Yj, and Zj obtained in step S 61 .
  • the displacement value (distortion value) ( ⁇ XXhj, ⁇ YYhj) at (XXhj, YYhj, ZZhj) is obtained by using the Equations (43) and (44).
  • the suffix h and j are attached to clarify that XX, YY, and ZZ depend on the time Th and the j-th characteristic point out of the first characteristic points.
  • Computation of the Equations (43) and (44) is performed by substituting values obtained from the Equations (40) to (42) for XXhj, YYhj, and ZZhj and moreover, substituting a value measured from the actual photographic image Ph for Uhj and Vhj.
  • Obtaining the displacement value (distortion value) ( ⁇ XX, ⁇ YY) at every position (XX, YY, ZZ) through interpolation represents the fact of obtaining the displacement value (distortion value) at any position under the condition that the displacement value (distortion value) at (XXhj, YYhj, ZZhj) which is the position of the j-th characteristic point is ( ⁇ XXhj, ⁇ YYhj).
  • j 1, . . . , J.
  • the position in the three-dimensional real space described here represents a coordinate system based on the video camera at the time Th.
  • the Delaunay division is first performed in a three-dimensional real space. That is, division of Voronoi region is performed by assuming J characteristic points as mother points to perform the Delaunay division having a dual relation with the division of the Voronoi region.
  • Each Delaunay-divided region forms a convex polygon using mother points (that is, first characteristic points) as vertexes.
  • step S 63 and S 64 the displacement value (distortion value) at each of J first characteristic points is first obtained in accordance with the Equations (40) to (44) by using the Rh, Sh, Xj, Yj, and Zj obtained in step S 61 and the measured values (Uhk, Vhk). Then, the Delaunay convex polygon is obtained by assuming J characteristic points as mother points. For each position in each Delaunay convex polygon, the displacement value (distortion value) is obtained by weighting and averaging the above-obtained displacement values (distortion values) of vertexes (that is, first characteristic points).
  • the displacement value (distortion value) depends on the time Th and moreover, depends on a position in a three-dimensional real space. That is, the displacement value (distortion value) is the function of Th and also, the function of the position (XX, YY, ZZ) in the coordinate system based on the video camera.
  • the relation of the displacement value (distortion value) between the Th and (XX, YY, ZZ) is obtained in steps S 63 and S 64 .
  • Steps S 63 and S 64 are computation steps for obtaining the displacement value (distortion value) of each point in a three-dimensional real space at each time.
  • step S 65 the processing circuit 32 reads world-coordinates-based virtual positions (Xm, Ym, Zm) of all the points (assumed as M points) constituting the building (second object) 91 formed through CG from the data memory 3 , applies the rotation matrix Rh and parallel displacement vector Sh also read from the data memory 33 to the virtual positions, obtains the position (XXhm, YYhm, ZZhm) shown by the coordinate system based on the video camera at the time Th, and stores the position (XXhm, YYhm, ZZhm) in the data memory 33 .
  • m 1, . . . , M. That is, the following Equations (61) to (63) are computed.
  • step S 66 the processing circuit 32 obtains the displacement value (distortion value) ( ⁇ XXhm, ⁇ YYhm) obtained in steps S 63 and S 64 and corresponding to the position (XXhm, YYhm, ZZhm) obtained from the Equations (61) to (63) in accordance with the following Equations (64) and (65).
  • the ( ⁇ XXhm, ⁇ YYhm) is added to (XXhm, YYhm, ZZhm).
  • the added position (XXhm+ ⁇ XXhm, YYhm+ ⁇ YYhm, ZZhm) serves as the final position of the m-th point constituting the building (second object) 91 formed through CG.
  • the position is the coordinate system based on the video camera at the time Th.
  • VVhm f ( YYhm+ ⁇ YYhm )/ ZZhm ) (67)
  • the displacement value ( ⁇ XXhj, ⁇ YYhj) is obtained in accordance with the Equations (40) to (44) by assuming that a displacement value is present only in the directions of a two-dimensional plane (X, Y) based on the video camera.
  • a displacement value is present in Z direction. That is, it is possible to assume the minimum value of ( ⁇ XXhj, ⁇ YYhj, ⁇ ZZhj) meeting the following Equations (68) and (69) as a displacement value.
  • Vhj f ( YYhj+ ⁇ YYhj )/( ZZhj+ ⁇ ZZhj ) (69)
  • (XXhj, YYhj, ZZhj) in the above Equations can be obtained from the Equations (40) to (42).
  • the displacement value (distortion value) at (XXhj, YYhj, ZZhj) which is the position of the j-th characteristic point is equal to ( ⁇ XXhj, ⁇ YYhj, ⁇ ZZhj)
  • the position (UUhm, VVhm) of the projected image of the m-th point constituting the building (second object) 91 formed through CG is shown by the following Equations (70) and (71) instead of the Equations (66) and (67).
  • VVhm f ( YYhm+ ⁇ YYhm )/( ZZhm+ ⁇ ZZhm ) (71)
  • ( ⁇ XXhm, ⁇ YYhm, ⁇ ZZhm) denotes the displacement value (distortion value) at the position (XXhm, YYhm, ZZhm) obtained by the Equations (61) to (63).
  • a displacement value (distortion value) is added so that the virtual position of an object (second object) to be synthesized designated by an operator coincides with the position of a characteristic point of a photographic image actually photographed at the time for each time and each position in a three-dimensional real space. Therefore, the position of a synthesized image is not displaced.
  • the gist of the present invention is not restricted to synthesis of a CG image formed through CG with a plate having characteristic points but the gist includes every case related to synthesis of a CG image formed through CG with a real image.
  • supply media for supplying a computer program for performing the above processing to users it is possible to use not only recording media such as a magnetic disk, CD-ROM, and solid state memory but also communication media such as a network and satellite.
  • a coordinate transform function is obtained in accordance with a first characteristic point nearby a synthesis position for synthesizing an object and fourth coordinates on a three-dimensional coordinate system corresponding to a second image are obtained by applying the coordinate transform function to third coordinates so as to synthesize the projected image of the object at a position corresponding to the fourth coordinates of the second image. Therefore, it is possible that the projected image of the object seem to be natural in first and second images.
  • a coordinate transform function is obtained in accordance with a first characteristic point nearby an object and fifth coordinates on a three-dimensional coordinate system corresponding to a second image are obtained by applying the coordinate transform function to third coordinates so as to detect the difference between the fourth and fifth coordinates. Therefore, it is possible to accurately compute the positional relation of a third image in the first and second images.
  • An image synthesizing apparatus can be applied to the case of synthesizing a computer-graphics (CG) image with a real image photographed by a video camera.
  • CG computer-graphics

Abstract

When a CG image is synthesized with an actually photographic image (dynamic image), it is controlled that the synthesized image becomes unnatural. An image pickup section 2 picks up a plate 1 while moving. When synthesizing a ball formed through CG with a continuous image picked up by the image pickup section 2, how 12 patterns (in the portion enclosed by a frame 21) Aj (j=1, . . . , 12) present nearby a ball 5 are transformed in the continuous image picked up by the image pickup section 2 is shown by a rotational transform function and a rectilinear transform function to obtain the synthesis position of the projected image of the ball 5 in each image by using the rotational transform function and rectilinear transform function.

Description

    TECHNICAL FIELD
  • The present invention relates to image synthesizing apparatus and method, position detecting apparatus and method, and a supply medium, particularly to image synthesizing apparatus and method, position detecting apparatus and method, and a supply medium for making it possible that the projected image of a predetermined object seems to be natural in continuous photographic images by synthesizing the projected image of the predetermined object with a plurality of continuous photographic images. [0001]
  • BACKGROUND ART
  • Because the computer graphics (CG) art has been advanced in recent years, it is possible to form a real image almost same as an actually photographed image (taken on the spot) though the real image is artificially formed. In this case, the processing is frequently performed in which a predetermined object is continuously photographed (as a dynamic image) by a video camera and then, the CG image of the predetermined object is synthesized every image taken on the spot. [0002]
  • FIG. 1 shows a conventional image synthesizing method. In this case, it is assumed that an [0003] existent plate 1 is picked up by an image pickup section 2 and balls 5 and 6 formed through CG are synthesized with the image of the plate 1 so that the balls 5 and 6 seem to be stationary. In this case, the plate 1 on which patterns are drawn is first actually picked up by an image pickup section 2. In this case, the i-th pattern formed on the plate 1 is assumed as ai. In this case, symbol i denotes an integer and also an identification number attached to each pattern ai. The patterns ai are different from each other in color and shape and therefore, they can be distinguished between them. The image pickup section 2 uses a stereophonic video camera constituted with a main video camera and a sub-video camera and is constituted so that three-dimensional information can be obtained from the parallax between an image picked up by the main video camera and an image picked up by the sub-video camera. The plate 1 is picked up while moving the image pickup section 2.
  • In this case, the time for the [0004] image pickup section 2 to start pickup is assumed as a first time and a predetermined time after the first time in the pickup time is assumed as a second time. FIG. 1 shows the image pickup section 2 at the first time and the second time.
  • As shown in FIG. 2, a frame image [0005] 1A1 obtained by picking up the plate 1 from a predetermined angle is obtained from the image pickup section 2 at the first time. Similarly, as shown in FIG. 3, a frame image 1A2 different from the frame image 1A1 obtained at the first time is obtained at the second time.
  • Then, as shown in FIG. 4, projected images [0006] 5A1 and 6B1 of the balls 5 and 6 are synthesized with the image (image in FIG. 18) 1A1 picked up by the image pickup section 2 at the first time. Then, as shown in FIG. 5, projected images 5A2 and 6B2 of the balls 5 and 6 are synthesized with the image (image in FIG. 19) 1A2 picked up by the image pickup section 2 at the second time at positions where the balls 5 and 6 seem to be stationary on the plate 1.
  • In this case, a method of synthesizing the [0007] balls 5 and 6 formed through CG so that they seem to be stationary on the plate 1 is described below by referring to FIGS. 6 and 7.
  • Though positions for finally synthesizing projected images [0008] 5A and 6B are on the two-dimensional images 1A1 and 1A2, the synthesis positions are easily understood by considering them in a three-dimensional space. Therefore, three-dimensional data (coordinates) is restored from a two-dimensional frame image. Because the frame image is obtained by the stereophonic video camera, the above restoration is realized.
  • That is, as shown in FIG. 6, three-dimensional coordinates are assumed in which the position of the [0009] image pickup section 2 at the first time is an origin O1 and the horizontal, vertical, and depth directions of the image pickup section 2 are X-axis, Y-axis, and Z-axis. Then, the three-dimensional position (X1i, Y1i, Z1i) of every pattern ai of the plate 1 is first obtained from the two-dimensional frame image 1A1. Then, a user designates a three-dimensional position (three-dimensional synthesis position)(X1A, Y1A, Z1A) on which the ball 5 is put and a three-dimensional position (three-dimensional synthesis position)(X1B, Y1B, Z1B) on which the ball 6 is put.
  • Then, a synthesis position (two-dimensional position) on the image [0010] 1A1 corresponding to a three-dimensional synthesis position is obtained. The two-dimensional position is obtained as a position obtained through perspective projection transform of the three-dimensional position. That is, when assuming the focal distance of the image pickup section 2 as f, the two-dimensional synthesis position of a projected image 5A1 on the image 1A1 at the first time corresponding to the three-dimensional synthesis position (X1A, Y1A, Z1A) is obtained as (X1A×f/Z1A, Y1A×f/Z1A). Similarly, the two-dimensional synthesis position of a projected image 6B1 on the image 1A1 at the first time corresponding to the three-dimensional synthesis position (X1B, Y1B, Z1B) is obtained as (X1B×f/Z1B, Y1B×f/Z1B).
  • Similarly, as shown in FIG. 7, the position of the [0011] image pickup section 2 at the second time is assumed as an origin O2 and the horizontal, vertical, and depth directions of the image pickup section 2 are assumed as X-axis, Y-axis, and Z-axis. Then, the three-dimensional position (X2i, Y2i, Z2i) of every pattern ai of the plate 1 is obtained from the two-dimensional image 1A2.
  • As described above, because the patterns ai on the [0012] plate 1 can be distinguished between them, it is possible to identify that the i-th pattern ai in the first three-dimensional coordinate system restored from the image 1A1 at the first time corresponds to which pattern ai in the second three-dimensional system restored from the image 1A2 at the second time. Therefore, it is possible to make the coordinates (X1i, Y1i, Z1i) of the pattern ai at the first time correspond to the coordinates (X2i, Y2i, Z2i) of the pattern ai at the second time. Because these two coordinate systems view the same pattern ai from different angles, the second three-dimensional coordinate system can be obtained by applying predetermined rotational transform (hereafter, the function of the transform is assumed as R1) and predetermined rectilinear transform (hereafter, the function of the transform is assumed as S1) to the first three-dimensional coordinate system (though the function of the rectilinear transform is normally shown by T, it is shown by S because T is used as a variable showing time in this specification). Therefore, the relation shown by the following Equation is effected for each pattern ai.
  • (X2i, Y2i, Z2i)=(X1i, Y1i, Z1iR 1 +S 1
  • (Though the function of the rectilinear transform is normally shown by T, it is shown by S because T is used as a variable showing time in this specification.) [0013]
  • Therefore, the coordinate transform functions R[0014] 1 and S1 can be obtained by substituting (X1i, Y1i, Z1i) and (X2i, Y2i, Z2i) of each pattern ai for the above Equation.
  • However, when restoring the position (three-dimensional coordinates) of the pattern ai in a three-dimensional space from image data (image data for the images [0015] 1A1 and 1A2) in a two-dimensional space picked up at a certain time, the restored position includes any error. Therefore, when using, for example, the first three-dimensional coordinate system as a criterion, the position of the i-th pattern ai is not exactly present at (X1i, Y1i, Z1i). Similarly, when using the second three-dimensional coordinate system as a criterion, the position of the i-th pattern ai is not exactly present at (X2i, Y2i, Z2i).
  • Therefore, values obtained by squaring the magnitude of the three-dimensional vector (x1, y1, z1) in the following Equation (1) on every pattern ai by the least-squares method are totalized to obtain the (most-probable) transform functions R[0016] 1 and S1 for minimizing the total.
  • (x1,y1,z1)=(X2i,Y2i,Z2i)−{(X1i,Y1i,Z1i)·R1 +S 1}  (1)
  • By using the transform functions R[0017] 1 and S1 obtained as described above, the three-dimensional position (X2A, Y2A, Z2A) of the ball 5 at the second time corresponding to the three-dimensional position (X1A, Y1A, Z1A) of the ball 5 at the first time is obtained by the following Equation (2).
  • (X2A,Y2A,Z2A)=(X1A,Y1A,Z1AR 1 +S 1  (2)
  • Similarly, the position (X2B, Y2B, Z2B) of the [0018] ball 6 at the second time corresponding to the three-dimensional position (X1B, Y1B, Z1B) of the ball 6 at the first time is obtained by the following Equation (3).
  • (X2B,Y2B,Z2B)=(X1B,Y1B,Z1B)−R 1 +S 1  (3)
  • Then, by applying the relation of the perspective projection transform to the three-dimensional coordinates (X2A, Y2A, Z2A) and (X2B, Y2B, Z2B), the coordinates (X2AX f/Z2A, Y2A×f/Z2A) of the projected image 5A[0019] 2 and the coordinates (X2B×f/Z2B, Y2B×f/Z2B) of the projected image 6B2 on the two-dimensional image 1A2 picked up by the image pickup section 2 at the second time are obtained. By synthesizing the projected images 5A2 and 6B2 with the coordinate positions, a synthesized image must be seen so that the balls 5 and 6 are stationary for the plate 1 because the coordinate positions correspond to the coordinate positions of the projected images 5A1 and 6B1.
  • The position of the i-th pattern ai and the positions of the [0020] balls 5 and 6 at the first time are shown as (X1i, Y1i, Z1i), (X1A, Y1A, Z1A), and (X1B, Y1B, Z1B) in the first three-dimensional coordinate system. Moreover, the position of the i-th pattern ai and the positions of the balls 5 and 6 at the second time are shown as (X2i, Y2i, Z2i), (X2A, Y2A, Z2A), and (X2B, Y2B, Z2B) in the second three-dimensional coordinate system. In this case, the relation between the positions of the i-th patterns ai and the positions of the balls 5 and 6 in these two coordinate systems can be shown by the following Equations (4), (5), and (6).
  • (X2i,Y2i,Z2i)≠(X1i,Y1i,Z1i)·R1 +S 1  (4)
  • (X2A,Y2A,Z2A)=(X1A,Y1A,Z1A)·R1 +S 1  (5)
  • (X2B,Y2B,Z2B)=(X1B,Y1B,Z1BR 1 +S 1  (6)
  • That is, in the case of middle and bottom Equations among the above three Equations, because the left-side terms are obtained by operations of the right-side terms, the right and left terms are equal. However, as described above, the transform functions R[0021] 1 and S1 are obtained by using the least-squares method and therefore, the functions include errors. As a result, as shown in the top Equation, the three-dimensional coordinates (obtained from a second image 1A2) of the pattern ai of the second image 1A2 do not accurately coincide with the coordinates obtained by applying the transform functions R1 and S1 to the three-dimensional coordinates (X1A, Y1A, Z1A) of a first image 1A3.
  • Therefore, the relation of projected images [0022] 5A1 and 6B1 to the pattern ai of the first image 1A1 at the first time is different from the relation of the projected images 5A2 and 6B2 to each pattern ai of the second image 1A2 at the second time. When reproducing this image string, there is a problem that the balls 5 and 6 seem to be unnatural because the balls 5 and 6 are lifted from the plate 1 on which the patterns ai are drawn.
  • DISCLOSURE OF THE INVENTION
  • The present invention is made to solve the above problems and its object is to provide an image synthesizing apparatus and method and a supply medium for making it possible that, when synthesizing the projected image of a predetermined object with a plurality of photographic images, a synthesized image seems to be natural by synthesizing the projected image of the predetermined object in accordance with a characteristic point nearby the synthesis position. [0023]
  • Moreover, it is another object of the present invention to provide position detecting apparatus and method and a supply medium capable of correctly calculating the positional relation of a third image to first and second images. [0024]
  • Furthermore, it is still another object of the present invention to provide image synthesizing apparatus and method and a supply medium capable of controlling the fluctuation of a synthesized image by correcting the position of the projected image of a predetermined object in accordance with a distortion value when synthesizing the projected image of the object with a plurality of photographic images. [0025]
  • The image synthesizing apparatus of [0026] claim 1 comprises first means for obtaining a first characteristic point nearby a synthesis position for synthesizing an object among the characteristic points projected onto a first image, second means for obtaining first coordinates of a first characteristic point on a three-dimensional coordinate system corresponding to the first image, third means for obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to a second image among the characteristic points projected onto the second image, fourth means for obtaining coordinate transform functions of the first and second coordinates, fifth means for obtaining third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing an object with the first image, sixth means for obtaining fourth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and seventh means for synthesizing the projected image of the object at a position corresponding to the fourth coordinates of the second image.
  • The image synthesizing method of [0027] claim 8 comprises the first step of obtaining a first characteristic point nearby a synthesis position for synthesizing an object among the characteristic points projected onto a first image, the second step of obtaining first coordinates on the three-dimensional coordinate system of a first characteristic point corresponding to the first image, the third step of obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to a second image among the characteristic points projected onto the second image, the fourth step of obtaining coordinate transform functions of the first coordinates and the second coordinates, the fifth step of obtaining third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing an object with the first image, the sixth step of obtaining fourth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and the seventh step of synthesizing the projected image of an object at a position corresponding to the fourth coordinates of the second image.
  • The supply medium of claim 15 provides a computer program used for an image synthesizing apparatus for synthesizing the projected image of an object with at least first and second images and comprising the first step of obtaining a first characteristic point nearby a synthesis position for synthesizing an object among the characteristic points projected onto the first image, the second step of obtaining first coordinates of a first characteristic point on a three-dimensional coordinate system corresponding to the first image, the third step of obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, the fourth step of obtaining coordinate transform functions of the first and second coordinates, the fifth step of obtaining third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing an object with the first image, the sixth step of obtaining fourth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and the seventh step of synthesizing the projected image of an object at a position corresponding to the fourth coordinates of the second image. [0028]
  • The position detecting apparatus of [0029] claim 16 comprises first means for obtaining a first characteristic point nearby an object among the characteristic points projected onto a first image, second means for obtaining first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image, third means for obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, fourth means for obtaining third coordinates of the object on a three-dimensional coordinate system corresponding to the first image, fifth means for obtaining fourth coordinates of the object on a three-dimensional coordinate system corresponding to the second image, sixth means for obtaining coordinate transform functions of the first and second coordinates, seventh means for obtaining fifth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and eighth means for detecting the difference between the fourth coordinates and the fifth coordinates.
  • The position detecting method of [0030] claim 17 comprises the first step of obtaining a first characteristic point nearby an object among the characteristic points projected onto a first image, the second step of obtaining first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image, the third step of obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected-onto the second image, the fourth step of obtaining third coordinates of the object on a three-dimensional coordinate system corresponding to the first image, the fifth step of obtaining fourth coordinates of the object on a three-dimensional coordinate system corresponding to the second image, the sixth step of obtaining coordinate transform functions of the first and second coordinates, the seventh step of obtaining fifth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and the eighth step of detecting the difference between the fourth and fifth coordinates.
  • The supply medium of [0031] claim 18 provides a computer program used for a position detecting apparatus for detecting the positional relation of an object projected onto first and second images and comprising the first step of obtaining a first characteristic point nearby the object among the characteristic points projected onto the first image, the second step of obtaining first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image, the third step of obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, the fourth step of obtaining third coordinates of the object on a three-dimensional coordinate system corresponding to the first image, the fifth step of obtaining fourth coordinates of the object on a three-dimensional coordinate system corresponding to the second image, the sixth step of obtaining coordinate transform functions of the first and second coordinates, the seventh step of obtaining fifth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and the eighth step of detecting the difference between the fourth and fifth coordinates.
  • The image synthesizing apparatus of claim 19 comprises image-pickup-device-position computing means for computing the most-probable position of an image pickup device when photographing each of a plurality of photographic images, characteristic point position computing means for computing the most-probable position of the characteristic point of a first object, virtual projected position computing means for computing a virtual projected position to which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device present at the most-probable position, distortion value computing means for computing a distortion value in accordance with the difference between a virtual projected position and the position of a projected image of a characteristic point actually photographed on each of a plurality of photographic images, correcting means for correcting a position to which the projected image of a second object is set in accordance with the distortion value computed by the distortion value computing means, projected image computing means for computing the projected image of the second object when photographing the second object from an image pickup device at the most-probable position by assuming that the second object is located at the corrected position by the correcting means, and synthesizing means for synthesizing the projected image of the second object computed by the projected image computing means with each photographic image. [0032]
  • The image synthesizing method of claim 21 comprises the image-pickup-device-position computing step of computing the most-probable position of an image pickup device when photographing each of a plurality of photographic images, the characteristic point position computing step of computing the most-probable position of the characteristic point of a first object, the virtual projected position computing step of computing a virtual projected position to which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device present at the most-probable position, distortion value computing step of computing a distortion value in accordance with the difference between a virtual projected position and the position of the projected image of a characteristic point actually photographed on each of a plurality of photographic images, the correcting step of correcting a position to which the projected image of a second object is set in accordance with the distortion value computed in the distortion value computing step, the projected image computing step of computing the projected image of the second object when photographing the second object from an image pickup device at the most-probable position by assuming that the second object is located at the position corrected in the correcting step, and the synthesizing step of synthesizing the projected image of the second object computed in the projected image computing step with each photographic image. [0033]
  • The supply medium of claim 23 provides a computer program used for an image synthesizing apparatus for synthesizing the projected image of a second object with each of a plurality of photographic images on which a first object photographed by an image pickup device is photographed by assuming that the second object is located at a predetermined position and comprising the image-pickup-device-position computing step of computing the most-probable position of the image pickup device when photographing each of a plurality of photographic images, the characteristic point position computing step of computing the most-probable position of the characteristic point of the first object, virtual projected position computing step of computing a virtual projected position to which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device present at the most-probable position, the distortion value computing step of computing a distortion value in accordance with the difference between a virtual projected position and the position of the projected image of the characteristic point actually photographed on each of a plurality of photographic images, the correcting step of correcting a position to which the projected image of a second object is set in accordance with the distortion value computed in the distortion value computing step, projected image computing step of computing the projected image of the second object when photographing the second object from an image pickup device at the most-probable position by assuming that the second object is located at the position corrected in the correcting step, and the synthesizing step of synthesizing the projected image of the second object computed in the projected image computing step with each photographic image. [0034]
  • In the case of the image synthesizing apparatus of [0035] claim 1, first means obtains a first characteristic point nearby a synthesis position for synthesizing the image of an object among the characteristic points projected onto a first image, second means obtains first coordinates of a first characteristic point on a three-dimensional coordinate system corresponding to the first image, third means obtains second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to a second image among the characteristic points projected onto the second image, fourth means obtains coordinate transform functions of the first and second coordinates, fifth means obtains third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing the object with the first image, sixth means obtains fourth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and seventh means synthesizes the projected image of the object at a position corresponding to the fourth coordinates of the second image.
  • In the case of the image synthesizing method of [0036] claim 8 and the supply medium of claim 15, a first characteristic point nearby a synthesis position for synthesizing an object is obtained among the characteristic points projected onto a first image, first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image are obtained, second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image are obtained, coordinate transform functions of the first and second coordinates are obtained, third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing the object with the first image are obtained, and fourth coordinates on a three-dimensional coordinate system corresponding to the second image are obtained by applying the coordinate transform functions to the third coordinates to synthesize the projected image of the object at a position corresponding to the fourth coordinates of the second image.
  • In the case of the position detecting apparatus of [0037] claim 16, first means obtains a first characteristic point nearby an object among the characteristic points projected onto a first image, second means obtains first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image, third means obtains second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, fourth means obtains third coordinates of the object on a three-dimensional coordinate system corresponding to the first image, fifth means obtains fourth coordinates of the object on a three-dimensional coordinate system corresponding to the second image, sixth means obtains coordinate transform functions of the first and second coordinates, seventh means obtains fifth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and eighth means detects the difference between the fourth and fifth coordinates.
  • In the case of the position detecting method of [0038] claim 17 and the supply medium of claim 18, a first characteristic point nearby an object is obtained among the characteristic points projected onto a first image, first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image are obtained, second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image are obtained among the characteristic points projected onto the second image, third coordinates of the object on a three-dimensional coordinate system corresponding to the first image are obtained, fourth coordinates of the object on a three-dimensional coordinate system corresponding to the second image are obtained, coordinate transform functions of the first and second coordinates are obtained, and fifth coordinates on a three-dimensional coordinate system corresponding to the second image are obtained by applying the coordinate transform functions to the third coordinates to detect the difference between the fourth and fifth coordinates.
  • In the case of the image synthesizing apparatus of claim 19, image-pickup-device-position computing means computes the most-probable position of an image pickup device when photographing each of a plurality of photographic images, characteristic point position computing means computes the most-probable position of the characteristic point of a first object, virtual projected position computing means computes a virtual projected position to which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device present at the most-probable position, distortion value computing means computes a distortion value in accordance with the difference between a virtual projected position and the position of the projected image of the characteristic point actually photographed on each of a plurality of photographic images, correcting means corrects a position to which the projected image of a second object is set in accordance with the distortion value computed by the distortion value computing means, projected image computing means computes the projected image of the second object when photographing the second object from an image pickup device present at the most-probable position by assuming that a second object is located at the position corrected by the correcting means, and synthesizing means synthesizes the projected image of the second object computed by the projected image computing means with each photographic image. [0039]
  • In the case of the image synthesizing method of [0040] claim 21 and the supply medium of claim 23, the most-probable position of an image pickup device when photographing each of a plurality of photographic images is computed, the most-probable position of a characteristic point of a first object is computed, a virtual projected position to which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device set to the most-probable position is computed, a distortion value is computed in accordance with the difference between a virtual projected position and the position of the projected image of the characteristic point actually photographed on each of a plurality of photographic images, a position to which the projected image of a second object is set in accordance with the computed distortion value is corrected, and the projected image of the second object when photographing the second object from an image pickup device present at the most-probable position is computed by assuming that the second object is located at a corrected position to synthesize the projected image of the computed second object with each photographic image.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 is an illustration showing a positional relation between the [0041] image pickup section 2 and the plate 1 of a conventional image synthesizing method;
  • FIG. 2 is an illustration showing an image pickup example of the [0042] plate 1 of the image pickup section 2 in FIG. 1;
  • FIG. 3 is an illustration of another image pickup example of the [0043] plate 1 of the image pickup section 2 in FIG. 1;
  • FIG. 4 is an illustration showing a synthesizing example of the [0044] balls 5 and 6 in FIG. 1;
  • FIG. 5 is an illustration showing another synthesizing example of the [0045] balls 5 and 6 in FIG. 1;
  • FIG. 6 is an illustration showing a positional relation between the [0046] image pickup section 2 and the plate 1 in FIG. 1;
  • FIG. 7 is an illustration showing another positional relation between the [0047] image pickup section 2 and the plate 1 in FIG. 1;
  • FIG. 8 is a block diagram showing the structure of the first embodiment of an image synthesizing apparatus of the present invention; [0048]
  • FIG. 9 is a flowchart for explaining an operation of the embodiment in FIG. 8; [0049]
  • FIG. 10 is an illustration showing a positional relation between the [0050] image pickup section 2 and the plate 1 of the embodiment in FIG. 8;
  • FIG. 11 is an illustration showing another positional relation between the [0051] image pickup section 2 and the plate 1 of the embodiment in FIG. 8;
  • FIG. 12 is a flowchart for explaining another operation of the embodiment in FIG. 8; [0052]
  • FIG. 13 is an illustration showing a positional relation between the [0053] image pickup section 2 and the plate 1 of the embodiment in FIG. 8;
  • FIG. 14 is an illustration showing another positional relation between the [0054] image pickup section 2 and the plate 1 of the embodiment in FIG. 8;
  • FIG. 15 is an illustration for explaining a relation between an image pickup device and a world coordinate system; [0055]
  • FIG. 16 is an illustration for explaining the projected image of a point based on world coordinates; [0056]
  • FIG. 17 is a block diagram showing the structure of the second embodiment of an image synthesizing apparatus of the present invention; [0057]
  • FIG. 18 is a flowchart for explaining an operation of the embodiment in FIG. 17; [0058]
  • FIG. 19 is an illustration for explaining the processing for picking up an object with an image pickup device; [0059]
  • FIG. 20 is an illustration showing an example of synthesizing a CG image with an actually picked-up image; [0060]
  • FIG. 21 is an illustration for explaining the change of an actually picked-up image; [0061]
  • FIG. 22 is an illustration for explaining the change of CG images; and [0062]
  • FIG. 23 is an illustration for explaining the change of a synthesized image of a picked-up image with a CG image.[0063]
  • BEST MODE FOR CARRYING OUT THE INVENTION
  • An embodiment of the present invention is described below by referring to the accompanying drawings. [0064]
  • To clarify the corresponding relation between each means of the inventions of claims and the following embodiment, features of the present invention are described below by adding a corresponding embodiment (one example) into the parentheses after each means. [0065]
  • That is, the image synthesizing apparatus of claim [0066] 1 comprises first means (e.g. step S3 in FIG. 9) for obtaining a first characteristic point nearby a synthesis position for synthesizing an object among the characteristic points projected onto a first image, second means (e.g. step S2 in FIG. 9) for obtaining first coordinates of the first characteristic point on a three-dimensional coordinate system corresponding to the first image, third means (e.g. step S6 in FIG. 9) for obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, fourth means (e.g. step S7 in FIG. 9) for obtaining coordinate transform functions of the first and second coordinates, fifth means (e.g. step S3 in FIG. 9) for obtaining third coordinates on a three-dimensional coordinate system corresponding to the first image at a synthesis position for synthesizing the object with the first image, sixth means (e.g. step SB in FIG. 9) for obtaining fourth coordinates on a three-dimensional coordinate system corresponding to the second image by applying a coordinate transform function to the third coordinates, and seventh means (e.g. step S10 in FIG. 9) for synthesizing the projected image of the object at a position corresponding to the fourth coordinates of the second image.
  • The position detecting apparatus of claim [0067] 16 comprises first means (e.g. step S34 in FIG. 12) for obtaining a first characteristic point nearby an object among the characteristic points projected onto a first image, second means (e.g. step S32 in FIG. 12) for obtaining first coordinates of a first characteristic point on a three-dimensional coordinate system corresponding to the first image, third means (e.g. step S36 in FIG. 12) for obtaining second coordinates of a second characteristic point corresponding to the first characteristic point on a three-dimensional coordinate system corresponding to the second image among the characteristic points projected onto the second image, fourth means (e.g. step S33 in FIG. 12) for obtaining third coordinates of the object on a three-dimensional coordinate system corresponding to the first image, fifth means (e.g. step S37 in FIG. 12) for obtaining fourth coordinates of the object on a three-dimensional coordinate system corresponding to the second image, sixth means (e.g. step S38 in FIG. 12) for obtaining coordinate transform functions of the first and second coordinates, seventh means (e.g. step S39 in FIG. 12) for obtaining fifth coordinates on a three-dimensional coordinate system corresponding to the second image by applying the coordinate transform functions to the third coordinates, and eighth means (e.g. step S40 in FIG. 12) for detecting the difference between the fourth and fifth coordinates.
  • The image synthesizing apparatus of claim [0068] 19 comprises image-pickup-device-position computing means (e.g. step S61 in FIG. 18) for computing the most-probable position of an image pickup device when photographing each of a plurality of photographic images, characteristic point position computing means (e.g. step S61 in FIG. 18) for computing the most-probable position of the characteristic point of a first object, virtual projected position computing means (e.g. step S61 in FIG. 18) for computing a virtual projected position on which the most-probable position of a characteristic point is projected when performing photography by using an image pickup device set at the most-probable position, distortion value computing means (e.g. step S63 in FIG. 18) for computing a distortion value in accordance with the difference between a virtual projected position and the position of the projected image of a characteristic point actually photographed on each of a plurality of photographic images, correcting means (e.g. step S66 in FIG. 18) for correcting a position to which the projected image of a second object is set in accordance with the distortion value computed by the distortion value computing means, projected image computing means (e.g. step S67 in FIG. 18) for computing the projected image of the second object when photographing the second object from an image pickup device present at the most-probable position by assuming that the second object is located at the position corrected by the correcting means, and synthesizing means (e.g. step S67 in FIG. 18) for synthesizing the projected image of the second object computed by the projected image computing means with each photographic image.
  • Of course, the above description does not mean that it is restricted to the description of each means. [0069]
  • An embodiment of an image synthesizing apparatus of the present invention is described below by referring to FIG. 8. [0070]
  • As shown in FIG. 8, the image synthesizing apparatus of the present invention has an [0071] input section 11 to be operated by a user when the user inputs the synthesis position of a CG image (image of the ball 5 or 6). The input section 11 is constituted with, for example, a keyboard, a touch panel, a mouse, and a remote controller.
  • A [0072] control section 12 controls each section in order to synthesize CG data. Moreover, the control section 12 has a memory 12A in which a program for synthesizing the CG data is stored. A computing section 13 performs various operations for synthesizing the CG data input from a CG data storing section 16 with the image data input from an image data accumulating section 14. Moreover, the computing section 13 is constituted so as to output an image synthesized with the CG data to a reproducing section 18. The control section 12 and the computing section 13 are realized by, for example, a microcomputer.
  • An [0073] image pickup section 2 uses, for example, a stereophonic video camera constituted with a main video camera 2A and a sub-video camera 2B and is constituted so as to pickup a plate 1 with the main video camera 2A and sub-video camera 2B and output the picked-up image data to the image data accumulating section 14. The image data accumulating section 14 is controlled by the control section 12, which accumulates the image data input from the image pickup section 2 and outputs the data to the computing section 13.
  • The CG [0074] data storing section 16 is controlled by the control section 12, which stores CG data (image data of the balls 5 and 6) to be synthesized with the image data picked up by the image pickup section 2 and outputs the CG data to the computing section 13. A display section 17 displays the image data picked up by the image pickup section 2 or the positional information of the plate 1 output from the computing section 13. A reproducing section 18 reproduces image data with which the CG data input from the computing section 13 is synthesized.
  • Then, operations of the embodiment thus constituted are described below by referring to the flowchart in FIG. 9. [0075]
  • In step S[0076] 1, a user operates the input section 11 to command it to start pickup of the plate 1. The input section 11 outputs a signal corresponding to the command to the control section 12. The control section 12 makes the image data accumulating section 14 accumulate the image data sent from the image pickup section 2 in accordance with the signal.
  • The image pickup section [0077] 2 (the main video camera 2A and sub-video camera 2B) picks up the plate 1 while being moved by the user and outputs the picked-up image data to the image data accumulating section 14, and the image data is stored in the image data accumulating section 14. In this case, it is assumed that the number of images picked up by the image pickup section 2 is N. Moreover, it is assumed that the identification numbers k from 1 to N are provided for these images in time series and the time when the k-th (k=1, . . . , N) image 1Ak is formed is the k-th time. Furthermore, the control section 12 makes the memory 12A store the identification numbers k to use the memory 12A as a counter for synthesis processing.
  • When pickup is completed, the [0078] control section 12 controls the image data accumulating section 14 in step S2 to make the section 14 output the image data accumulated in the section 14 to the computing section 13. The computing section 13 obtains a characteristic point (e.g. pattern on the plate 1, defect on the plate 1, or other pattern) which extremely changes compared to a point to which a brightness signal or color-difference signal of the image data input from the image data accumulating section 14 is adjacent and therefore, the characteristic point can be separated from other part of the image data. In this case, the i-th characteristic point (pattern) is assumed as ai. In this case, symbol i denotes an integer which is an identification number provided for each pattern ai. Moreover, the computing section 13 compares the same patterns ai out of the image data input from the main video camera 2A and the sub-video camera 2B at the same time, calculates the parallax between the patterns, and obtains the three-dimensional position (coordinates) of every pattern (characteristic point) ai. Because the three-dimensional position is obtained by using the parallax between the cameras, this position is a position on three-dimensional coordinates using the camera (image pickup section 2) as an origin. That is, for example, the position in the three-dimensional coordinates shown in FIG. 10 to be described later is obtained.
  • For example, as shown in FIG. 10, it is assumed that the position of the [0079] image pickup section 2 at the first time is an origin O1 and the horizontal direction, vertical direction, and depth direction of the image pickup section 2 are X-axis, Y-axis, and Z-axis. The computing section 13 obtains the three-dimensional position (X1i, Y1i, Z1i) of the every pattern ai on the image 1A1 of the plate 1 at the first time.
  • Then, in step S[0080] 3, the computing section 13 outputs the position of every pattern ai and the first image 1A1 to the display section 17 to make the section 17 display the position of every pattern ai and the position of the first image 1A1. The user operates the input section 11, and designates and inputs the three-dimensional position (X1A, Y1A, Z1A) of the ball 5 for synthesizing the projected image 5A1 of the ball 5 with the first image 1A1 and the three-dimensional position (X1B, Y1B, Z1B) of the ball 6 for synthesizing the projected image 6B1 of the ball 6 with the first image 1A1 based on the position information at the first time displayed in the display section 17.
  • Then, the [0081] computing section 13 obtains the three-dimensional vector (x2, y2, z2) between the ball 5 and the pattern ai on every pattern ai in accordance with the following Equation (7) and selects a predetermined number of patterns (e.g. 12 patterns) (in the portion enclosed by the frame 21 in FIG. 10) Aj (j=1, . . . , 12) in order of magnitudes of the vectors starting with a vector having the smallest magnitude (starting with a vector closest to the ball 5).
  • (x2,y2,z2)=(X1i,Y1i,Z1i)−(X1A,Y1A,Z1A)  (7)
  • Then, the positions of the 12 selected patterns Aj (j=1, . . . , 12) are assumed as (X1Aj, Y1Aj, Z1Aj) (j=1, . . . , 12). [0082]
  • Similarly, the [0083] computing section 13 obtains three-dimensional vectors (x3, y3, z3) between the ball 6 and the pattern ai on all patterns ai in accordance with the following Equation (8) and selects 12 patterns (in the portion enclosed by the frame 22 in FIG. 10) Bj (j=1, . . . , 12) in order of magnitudes of the vectors starting with a vector having the smallest magnitude (starting with a vector closest to the ball 6).
  • (x3,y3,z3)=(X1i,Y1i,Z1i)−(X1B,Y1B,Z1B)  (8)
  • Then, the positions of the 12 selected patterns Bj (j=1, . . . , 12) are assumed as (X1Bj, Y1Bj, Z1Bj) (j=Thus, a characteristic point nearby a synthesis position is obtained and then, step [0084] 64 is started to perform image synthesis processing. That is, the computing section 13 obtains a two-dimensional synthesis position on the first image 1A1 corresponding to a three-dimensional synthesis position. The two-dimensional synthesis position is obtained by perspective-projection-transforming the three-dimensional synthesis position. That is, when assuming the focal distance of the image pickup section 2 as f, the two-dimensional position of the projected image 5A1 of the ball 5 on the image 1A1 at the first time corresponding to the three-dimensional position (X1A, Y1A, Z1A) is shown as (X1A×f/Z1A, Y1A×f/Z1A). Similarly, the position of the projected image 6B1 of the ball 6 on the two-dimensional image 1A1 at the first time corresponding to the three-dimensional position (X1B, Y1B, Z1B) is shown as (X1B×f/Z1B, Y1B×f/Z1B).
  • The [0085] control section 12 controls the CG data storing section 16 to output the CG data (image data for the balls 5 and 6) stored in the CG data storing section 16 to the computing section 13. Then, the computing section 13 synthesizes the CG data at the obtained synthesis position.
  • Then, in step S[0086] 5, the control section 12 sets an image identification number k to 2.
  • As shown in FIG. 11, it is assumed that the position of the [0087] image pickup section 2 at the k-th time is an origin Ok, and the horizontal direction, vertical direction, and depth direction of the image pickup section 2 are X-axis, Y-axis, and Z-axis. In step S6, the computing section 13 obtains the three-dimensional position (Xki, Yki, Zki) of the pattern ai (characteristic point) at the k-th (in this case, k=2) image 1Ak. The method for obtaining the position of the pattern ai is the same as the method for obtaining the position of the pattern ai of the first image 1A1.
  • As described above, because the patterns ai on the [0088] plate 1 can be separated from each other, it is possible to identify that patterns (nearby patterns) Aj and Bj (j=1, . . . , 12) in the (k−1)-th three-dimensional coordinate system restored by the (k−1)-th (in this case, 1st) image 1Ak−1 correspond to which patterns Aj and Bj (j=1, . . . , 12) in the k-th three-dimensional coordinate system restored by the k-th image 1Ak. In this case, it is assumed that the positions of the patterns Aj and Bj (j=1, . . . , 12) in the k-th three-dimensional coordinate system are (XkAj, YkAj, ZkAj) and (XkBj, YkBj, ZkBj) (j=1, . . . , 12).
  • The above two coordinate systems view the same patterns Aj and Bj from different angles. Therefore, in the (k−1)-th three-dimensional coordinate system, by applying predetermined rotational transform (hereafter, the transform function of the rotational transform is assumed as R[0089] 2) and predetermined rectilinear transform (hereafter, the transform function of the rectilinear transform is assumed as S2) to each pattern Aj in the (k−1)-th three-dimensional coordinate system, it is possible to obtain the k-th three-dimensional coordinate system. Therefore, the following Equation (9) is effected for each pattern Aj.
  • (XkAj,YkAj,ZkAj)=(X(k−1)Aj,Y(k−1)Aj,Z(k−1)AjR 2 +S 2  (9)
  • Therefore, in step S[0090] 7, the computing section 13 obtains the transform functions R2 and S2 by substituting (X(k−1)Aj, Y(k−1)Aj, Z(k−1)Aj) and (XkAj, YkAj, ZkAj) for the above Equation (9) on each pattern Aj.
  • Similarly, by applying predetermined rotational transform (hereafter, the transform function of the rotational transform is assumed as R[0091] 3) and predetermined rectilinear transform (hereafter, the transform function of the rectilinear transform is assumed as S3) to each pattern Bj in the (k−1)-th three-dimensional coordinate system, it is possible to obtain the k-th three-dimensional coordinate system. Therefore, the following Equation (10) is effected for each pattern Bj.
  • (XkBj,YkBj,ZkBj)=(X(k−1)Bj,Y(k−1)Bj,Z(k−1)Bj)R 3 +S 3  (10)
  • Therefore, the [0092] computing section 13 obtains the transform functions R3 and S3 by substituting (X(k−1)Bj, Y(k−1)Bj, Z(k−1)Bj) and (XkBj, YkBj, ZkBj) for the above Equation (10) on each pattern Bj.
  • As described above, when restoring the positions of the patterns Aj and Bj in a three-dimensional space from two-dimensional image data picked up at a certain time, the restored position includes any error. Therefore, when using, for example, the (k−1)-th three-dimensional coordinate system as a criterion, the positions of the j-th patterns Aj and Bj are not accurately present at (X(k−1)Aj, Y(k−1)Aj, Z(k−1)Aj) and (X(k−1)Bj, Y(k−1)Bj, Z(k−1)Bj). Moreover, when using the k-th three-dimensional coordinate system as a criterion, the positions of the j-th patterns Aj and Bj are not accurately present at (XkAj, YkAj, ZkAj) and (XkBj, YkBj, ZkBj). [0093]
  • Therefore, the [0094] computing section 13 totalizes values obtained by squaring the magnitude of the three-dimensional vector (x4, y4, z4) in the following Equation (11) on all patterns Aj in accordance with the least-squares method to obtain (most-probable) transform functions R2 and S2 for minimizing the total value.
  • (x4,y4,z4)=(XkAj,YkAj,ZkAj)−{(X(k−1)Aj,Y(k−1)Aj,Z(k−1)AjR 2 +S 2}  (11)
  • By using the transform functions R[0095] 2 and S2 obtained as described above, in step S8, the computing section 13 obtains the three-dimensional position (XkA, YkA, ZkA) of the ball 5 at the k-th time in accordance with the following Equation (12). This position is the position in the three-dimensional coordinate system shown in FIG. 11.
  • (XkA,YkA,ZkA) (X(k−1)A,Y(k−1)A,Z(k−1)AR 2 +S 2  (12)
  • Similarly, the [0096] computing section 13 totalizes values obtained by squaring three-dimensional vector (x5, y5, z5) on all patterns Bj in accordance with the least-squares method to obtain (most-probable) transform functions R3 and S3 for minimizing the totalized value as shown by the following Equation (13).
  • (x5,y5,z5)=(XkBj,YkBj,ZkBj)−{(X(k−1)Bj,Y(k−1)Bj,Z(k−1)BjR 3 +S 3}  (13)
  • By using the transform functions R[0097] 3 and S3 obtained as described above, in step S8, the computing section 13 obtains the position (XkB, YkB, ZkB) of the projected image 6Bk of the ball 6 at the k-th time from the following Equation (14). This position is the position in the three-dimensional coordinate system shown in FIG. 11.
  • (XkB,YkB,ZkB)=(X(k−1)B,Y(k−1)B,Z(k−1)BR 3 +S 3   (14)
  • In this connection, R[0098] 2, S2, R3, and S3 depend on the positional relation between the image pickup sections 2 at the (k−1)-th time and the k-th time. That is, R2, S2/R3, and S3 are functions of k, and R2 equals R2(k), S2 equals S2(k), R3 equals R3(k), and S3 equals S3(k).
  • Moreover, in step S[0099] 9, the computing section 13 obtains three-dimensional vectors (x6, y6, z6) between the ball 5 and the pattern ai on all patterns ai in accordance with the following Equation (15) and selects 12 patterns (in the portion enclosed by the frame 21 in FIG. 11) Aj in the k-th three-dimensional coordinate system in order of magnitudes of the vectors starting with a vector having the smallest magnitude (starting with a vector closest to the ball 5) as shown by the following Equation (15).
  • (x6,y6,z6)=(Xki,Yki,Zki)−(XkA,YkA,ZkA)  (15)
  • Moreover, the [0100] computing section 13 obtains three-dimensional vectors (x7, y7, z7) between the ball 6 and the pattern ai on all patterns ai in accordance with the following Equation (16) and selects 12 patterns (in the portion enclosed by the frame 22 in FIG. 11) Bj in the k-th three-dimensional coordinate system in order of magnitudes of the vectors starting with a vector having the smallest magnitude (starting with a vector closest to the ball 6). ( x7 , y7 , z7 ) = ( Xki , Yki , Zki ) - ( XkB , YkB , ZkB ) ( 16 )
    Figure US20030007679A1-20030109-M00001
  • Then, in step S[0101] 10, the computing section 13 synthesizes the projected image 5Ak of the ball 5 at the position of (XkA×f/ZkA, YkA×f/ZkA) by using the relation of the perspective projection transform in the k-th image 1Ak. Moreover, when there is any object other than the projected image 5Ak in the k-th image 1Ak at the image pickup section-2 side, the computing section does not synthesize the projected image 5Ak if it is behind the object.
  • Similarly, the [0102] computing section 13 synthesizes the projected image 6Bk of the ball 6 at the position of (XkB×f/ZkB, YkB×f/ZkB) by using the relation of the perspective projection transform in the k-th image 1Ak. Also in this case, when there is any object other than the projected image 6Bk in the k-th image 1Ak at the image pickup section-2 side, the computing section 13 does not synthesize the projected image 6Bk if it is behind the object.
  • Then, in step S[0103] 11, the control section 12 decides whether synthesis processing is completed on the whole image data picked up by the image pickup section 2. Unless the processing is completed (for k . . . N), the control section 12 starts step S12 to increment the identification number k of an image by 1 (k=k+1), and returns to step S4 to continue the synthesis processing. When the processing is completed (for k=N), the control section 12 starts step S13 and the computing section 13 outputs the image data undergoing the synthesis processing to the reproducing section 18 to make the section 18 reproduce the data (display the data).
  • For example, when positional relation between the patterns ai nearby the [0104] balls 5 and 6 (for example, 10 cm separate from the balls 5 and 6) is always the same, a person senses that the balls 5 and 6 are stationary. However, even if the positional relation between the patterns ai present far from the balls 5 and 6 (for example, 100 m separate from the balls 5 and 6) changes, the person does not sense the change of the positional relation.
  • In the case of the present invention, a pattern nearby a synthesis position is selected without considering a pattern present far from the synthesis position (that is, a pattern not to be sensed by a person) to obtain the rotational transform and rectilinear transform for the pattern. Therefore, it is possible to synthesize an image so that the [0105] balls 5 and 6 seem to be stationary on the plate 1.
  • In the case of the above embodiment, the [0106] stationary balls 5 and 6 are synthesized with the plate 1. However, it is also possible to synthesize the moving balls 5 and 6 with the plate 1. In this case, it is necessary to synthesize the projected images of the balls 5 and 6 at the synthesis position of the balls 5 and 6 to be synthesized in the k-th three-dimensional coordinate system obtained in step S8 in FIG. 9 by using values obtained by adding the three-dimensional vector showing each movement at the synthesis position as new positions of the balls 5 and 6.
  • Moreover, in the case of the above embodiment, relatively small objects such as the [0107] balls 5 and 6 are synthesized. However, it is also possible to synthesize a large object. For example, to synthesize a large rectangular parallelepiped, by designating eight vertexes of the rectangular parallelepiped, it is possible to obtain the sides for connecting the vertexes and a plane enclosed by the sides. The rectangular parallelepiped is synthesized by assuming these eight vertexes as independent objects. Because eight points are determined in each continuous image, it is possible to synthesize the rectangular parallelepiped by connecting these points by straight lines and using the points connected by the straight lines as sides and moreover, using the range enclosed by the sides as a plane.
  • In the case of the above embodiment, images of the [0108] balls 5 and 6 formed through CG are synthesized. However, it is also possible to obtain the positional relation between objects (objects to be actually photographed) actually present on the plate 1. Then, the operation is described below by referring to the flowchart in FIG. 12.
  • As shown in FIG. 13, it is assumed that the position of the [0109] image pickup section 2 at the first time is an origin O1 and the horizontal direction, vertical direction, and depth direction of the image pickup section 2 are X-axis, Y-axis, and Z-axis. Existent balls 31 and 32 move on the plate 1. In step S31, the image pickup section 2 picks up the plate 1 while being moved by, for example, a user. Similarly to the case of the above embodiment, it is assumed that the number of images picked up by the image pickup section 2 within a pickup time is N. Moreover, identification numbers k from 1 to N are provided for these images in time series and it is assumed that the time when the k-th (k=1, . . . , N) image 1Ak is formed is the k-th time. Moreover, the control section 12 makes a memory 12A store the identification numbers k and uses the memory 12 as a synthesis-processing counter.
  • When image pickup by the [0110] image pickup section 2 is completed, the computing section 13 obtains the three-dimensional position (X1i, Y1i, Z1i) of every pattern (characteristic point) ai on the plate 1 at the first time in step S32. Moreover, the computing section 13 obtains the three-dimensional position (X1C, Y1C, Z1C) of the ball 31 and the three-dimensional position (X1D, Y1D, Z1D) of the ball 32 at the first time in step S33.
  • The [0111] computing section 13 obtains three-dimensional vectors (x8, y8, z8) between the ball 31 and the pattern ai on all patterns ai in step S34 in accordance with the following Equation (17) and selects 12 patterns (in the portion enclosed by the frame 41 in FIG. 13) Cj in the first three-dimensional coordinate system in order of magnitudes of the vectors starting with a vector having the smallest magnitude (starting with a vector closest to the ball 31).
  • (x8,y8,z8)=(X1i,Y1i,Z1i)−(X1C,Y1C,Z1C)  (17)
  • Moreover, the positions of the 12 selected patterns Cj (j=1, . . . , 12) are assumed as (X1Cj, Y1Cj, Z1Cj) (j=1, . . . , 12). [0112]
  • Similarly, the [0113] computing section 13 obtains three-dimensional vectors (x9, y9, z9) between the ball 32 and the pattern ai on all patterns ai in accordance with the following Equation (18) and selects 12 patterns (in the portion enclosed by the frame 42 in FIG. 13) Dj in the first three-dimensional coordinate system in order of magnitudes of the vector starting with a vector having the smallest magnitude (starting with a vector closest to the ball 32).
  • (x9,y9,z9)=(X1i,Y1i,Z1i)−(X1D,Y1D,Z1D)  (18)
  • Then, positions of the 12 selected patterns Dj (j=1, . . . , 12) are assumed as (X1Dj, Y1Dj, Z1Dj) (j=1, . . . , 12). [0114]
  • Then, in step S[0115] 35, the control section 12 sets the identification number k of image data to 2.
  • Similarly, as shown in FIG. 14, it is assumed that the position of the [0116] image pickup section 2 at the k-th time (in this case, k=2) is an origin Ok, and the horizontal direction, vertical direction, and depth direction of the image pickup section 2 are X-axis, Y-axis, and Z-axis. Moreover, the computing section 13 obtains the three-dimensional position (Xki, Yki, Zki) of the pattern ai in the k-th image 1Ak in step S36.
  • As described above, the patterns ai on the [0117] plate 1 can be separated from each other, it is possible to identify that the patterns Cj and Dj (j=1, . . . , 12) in the (k−1)-th three-dimensional coordinate system restored from the (k−1)-th image 1Ak−1 correspond to which patterns Cj and Dj (j=1, . . . , 12) in the k-th three-dimensional coordinate system restored from the k-th image 1Ak. In this case, positions of the patterns Cj and Dj (j=1, . . . , 12) in the k-th three-dimensional coordinate system are assumed as (XkCj, YkCj, ZkCj) and (XkDj, YkDj, ZkDj) (j=1, . . . , 12).
  • The [0118] computing section 13 obtains the position (XkC, YkC, ZkC) of the ball 31 and the position (XkD, YkD, ZkD) of the ball 32 at the k-th time in step S37.
  • By applying predetermined rotational transform (hereafter, the transform function of the rotational transform is assumed as R[0119] 4) and predetermined rectilinear transform (hereafter, the transform function of the rectilinear transform is assumed as S4) to the pattern Cj in the (k−1)-th three-dimensional coordinate system, it is possible to obtain the k-th three-dimensional coordinate system. Moreover, to restore the position of the pattern Cj in a three-dimensional space, the computing section 13 totalizes values obtained by squaring the magnitude of three-dimensional vector (x12, y12, z12) on all patterns Cj by the least-squares method in accordance with the following Equation (19) to obtain (most-probable) transform functions R4 and S4 for minimizing the total value in step S38 by considering that the restored position includes any error.
  • (x12,y12,z12)=(XkCj,YkCj,ZkCj)−{(X(k−1)Cj,Y(k−1)Cj,Z(k−1)CjR 4 +S 4}  (19)
  • The [0120] computing section 13 obtains a position (XkC′, YkC′, ZkC′) when the ball 32 is stationary in accordance with the following Equation (20) in step S39 by using the transform functions R4 and S4 thus obtained.
  • (XkC′,YkC′,ZkC′) (X(k−1)C,Y(k−1)C,Z(k−1)CR 4 +S 4   (20)
  • Therefore, the three-dimensional vector (xkc, ykc, zkc) in the following Equation (21) shows the movement of the [0121] ball 31 between the (k−1)-th time and the k-th time. This operation is performed in step S40.
  • (xkc,ykc,zkc)=(XkC,YkC,ZkC)−(XkC′,YkC′,ZkC′)  (21)
  • Similarly, by applying predetermined rotational transform (hereafter, the transform function is assumed as R[0122] 5) and predetermined rectilinear transform (hereafter, the transform function is assumed as S5) to the pattern Dj in the (k−1)-th three-dimensional coordinate system, it is possible to obtain the k-th three-dimensional coordinate system. To restore the position of the pattern Dj in a three-dimensional space, the computing section 13 totalizes values obtained by squaring the magnitude of the three-dimensional vector (x13, y13, z13) in the following Equation (22) on all patterns Dj by the least-squares method to obtain (most-probable) transform functions R5 and S5 for minimizing the total value by considering that the restored position includes any error.
  • (x13,y13,z13)=(XkDj,YkDj,ZkDj)−{(X(k−1)Dj,Y(k−1)Dj,Z(k−1)Dj)·R5 +S 5}  (22)
  • The [0123] computing section 13 obtains a position (XkD′, YkD′, ZkD′) when the ball 32 is stationary in accordance with the following Equation (23) by using the transform functions R5 and S5 thus obtained.
  • (XkD′,YkD′,ZkD′)=(X(k−1)D,Y(k−1)D,Z(k−1)D)·R5 +S 5  (23)
  • Therefore, it is possible to obtain the three-dimensional vector (xkd, ykd, zkd) in the following Equation (24), that is, the movement of the [0124] ball 32 between the (k−1)-th time and the k-th time.
  • (xkd,ykd,zkd)=(XkD,YkD,ZkD)−(XkD′,YkD′,ZkD′)  (24)
  • Then, the [0125] computing section 13 obtains three-dimensional vectors (x10, y10, z10) between the ball 31 and the pattern ai on all patterns ai in step S41 as shown by the following Equation (25) and selects 12 patterns (in the portion enclosed by the frame 41 in FIG. 14) Cj in the k-th three-dimensional coordinate system in order of magnitudes of the vectors starting with a vector having the smallest magnitude (starting with a vector closest to the ball 31).
  • (x10,y10,z10)=(Xki,Yki,Zki)−(XkC,YkC,ZkC)  (25)
  • Moreover, the [0126] computing section 13 obtains three-dimensional vectors (x11, y11, z11) between the ball 32 and the pattern ai on all patterns ai as shown by the following Equation (26) and selects 12 patterns (in the portion enclosed by the frame 42 in FIG. 14) Dj in the k-th three-dimensional coordinate system in order of magnitudes of the vectors starting with a vector having the smallest magnitude (starting with a vector closest to the ball 32).
  • (x11,y11,z11)=(Xki,Yki,Zki)−(XkD,YkD,ZkD)  (26)
  • Then, in step S[0127] 42, the control section 12 decides whether synthesis processing is completed on the whole image data picked up by the image pickup section 2. When the processing is not completed (for k . . . N), the control section 12 starts step S43 to increment the identification number k of an image by 1 (k=k +1), and returns to step S36 to continue the synthesis processing. When the processing is completed (for k=N), the computing section 13 starts step S44 to make the display section 17 display movement values (xkc, ykc, zkc) and (xkd, ykd, zkd) at each time.
  • In the case of the above first embodiment, a characteristic point uses a two-dimensional object such as the pattern ai on the [0128] plate 1. However, it is also possible to use, for example, a side of a rectangular parallelepiped three-dimensionally arranged.
  • Moreover, in the case of the first embodiment, a characteristic point is selected through the operation by the [0129] computing section 13. However, it is also possible that, for example, a user selects the characteristic point by operating the input section 11 during processing.
  • Furthermore, in the case of the first embodiment, [0130] 12 patterns (characteristic points) Aj and Bj nearby a synthesis position are selected. However, for example, it is also possible to obtain a rotational transform function and a rectilinear transform function by selecting 20 patterns close to a synthesis position, providing a weight larger than that of 8 remaining patterns for 12 patterns closer to the synthesis position out of 20 patterns, and using the least-squares method.
  • The obtained rotational transform function and rectilinear transform function serve as a rotational transform function and a rectilinear transform function having a relatively large error for 8 relatively remote patterns and serve as a rotational transform function and a rectilinear transform function having a completely controlled error for the 12 patterns close enough. This is favorable because this corresponds to the human-eye characteristic of recognizing the positional relation between characteristic points separate from each other to a certain extent (the above 8 patterns) at a relatively rough accuracy and recognizing the positional relation between characteristic points close enough to each other (the above 12 patterns) at a high accuracy. Of course, the present invention does not restrict the number of characteristic points to 12 or 20. [0131]
  • In the case of the first embodiment, the picked up [0132] plate 1 is synthesized with CG data so as to simplify the description. Actually, however, the plate 1 is a real image photographed by a camera and a characteristic point in the plate 1 corresponds to a characteristic point such as an edge in the real image. For example, when there is a desk or wall in a photographic image, a corner of the desk or stain (dust or the like) on the wall serves as a characteristic point.
  • For example, it is assumed to put a vase formed through CG on a real-image desk obtained by photographing an actual room in which a real desk is present by a camera. First, it is possible to obtain three-dimensional information (characteristic point) from a corner of the desk. Moreover, a corner of a window frame of the room or a corner of a fluorescent lamp on the ceiling of the room is detected as three-dimensional information (characteristic point). Furthermore, when putting the vase formed through CG on the desk, the vase formed through CG is synthesized with the real image in accordance with the three-dimensional information obtained from a characteristic point (e.g. the corner of the desk or the like) present nearby the position where the vase is put. In this case, however, the three-dimensional information obtained from a remote characteristic point (e.g. the corner of the window frame of the room or the corner of the fluorescent lamp on the ceiling of the room) is ignored. [0133]
  • In the case of the above embodiment, a small ball formed through CG is synthesized with a photographic image. Moreover, to synthesize a large rectangular parallelepiped, eight vertexes of the rectangular parallelepiped are assumed as independent objects, each vertex is synthesized with a characteristic point nearby the vertex so that the positions coincide with each other, and sides and planes are synthesized at positions where these vertexes are connected by straight lines. That is, because vertexes are positioned, a person does not sense that the vertexes of the rectangular parallelepiped are displaced. However, sides and planes are not accurately positioned. [0134]
  • As a result, when processing a photographic image string obtained by continuously photographing a first object and thereby, generating a new photographic image string showing as if a second object is present at the string, that is, when synthesizing the second object with the original photographic image string and thereby, generating a synthesized image string, the synthesized image string may be unnatural because a synthesis position is slightly displaced in each synthesized image. [0135]
  • That is, when obtaining the position and attitude (direction) of an image pickup device in a three-dimensional real space and the position of a characteristic point in a three-dimensional real space at each time, these obtained values do not show true values but they include errors. [0136]
  • A projected image at a certain time Th is noticed. The projected position of a characteristic point when photographing the characteristic point from the obtained position and attitude (direction) of the image pickup device including errors but not showing true values at the time Th by assuming that the characteristic point is present at the position of the obtained characteristic point including errors but not showing a true value is referred to as a virtual projected position. [0137]
  • In fact, the position of the characteristic point appearing on a photographic image Ph photographed at the time Th is slightly displaced from the virtual projected position. [0138]
  • Therefore, in the case of the second embodiment to be described below, the displacement value (distortion value) is obtained. The distortion value depends on the position. Moreover, because the distortion value depends on time, it is obtained for each time. In other words, the distortion value is the function of position and time. [0139]
  • The position (virtual position) of the object (second object) to be synthesized is determined by an operator in accordance with the relation between the position of the second object and the position of the above obtained characteristic point. [0140]
  • Similarly to the case of the first embodiment, when obtaining the projected image of the second object from the obtained position and attitude (direction) of the image pickup device at the time when assuming that the second object is present at this virtual position, it is displaced from the projected position of a characteristic point in the actually-photographed photographic image Ph. [0141]
  • In the case of the second embodiment, the virtual position of the second object determined by the operator is distorted by a value equal to the distortion value obtained above. Moreover, the projected image of the second object is obtained by assuming that the second object is present at the distorted position. Thus, the position of the projected image of the second object coincides with that of the projected image of the characteristic point in the actually-photographed photographic image Ph. [0142]
  • Then, geometry about general computer vision is described below before specifically describing the second embodiment. [0143]
  • It is assumed that the position and attitude (direction) of an image pickup device (corresponding to the [0144] image pickup section 2 in FIG. 8) is present at a position moved from the origin of world coordinates due to the rotation of a matrix R and the translation of a vector S. In this case, a point at a position (X, Y, Z) on the basis of the world coordinates has coordinates (XX, YY, ZZ) shown in the following Equation (27) in the case of a coordinate system based on the above image pickup device. This state is shown in FIG. 15. [ XX YY ZZ ] = R [ X Y Z ] - RS = [ R11 R12 R13 R21 R22 R23 R31 R32 R33 ] [ X Y Z ] - [ R11 R12 R13 R21 R22 R23 R31 R32 R33 ] [ S1 S2 S3 ] ( 27 )
    Figure US20030007679A1-20030109-M00002
  • Therefore, the position (U, V) of the projected image of the above point in the photographic image photographed by the above image pickup device is shown by the following Equations (28) and (29). This state is shown in FIG. 16. [0145] U = f XX YY = f R11 · X + R12 · Y + R13 · Z + S1 R31 · X + R32 · Y + R33 · Z + S3 ( 28 ) V = f YY ZZ = f R21 · X + R22 · Y + R23 · Z + S2 R31 · X + R32 · Y + R33 · Z + S3 Where , [ S1 S2 S3 ] = - RS ( 29 )
    Figure US20030007679A1-20030109-M00003
  • Symbol f in the Equations (28) and (29) denotes the focal distance of the image pickup device. [0146]
  • Then, points in a plurality of photographic images and points in the three-dimensional real spaces are assumed. [0147]
  • A plurality of photographic images are obtained by performing photography while moving the image pickup device. It is assumed that photography is performed at H times such as T[0148] 1, T2, T3, . . . , TH. Moreover, it is assumed that a photographic image photographed at the time Th is Ph. Furthermore, it is assumed that the position and attitude (direction) of the image pickup device at the time Th is moved from the origin of the world coordinates due to rotation matrix Rh and parallel displacement vector Sh. In this case, h=1, . . . , H.
  • Furthermore, it is assumed that each of the positions of the points (J points) in the three-dimensional real spaces is (Xj, Yj, Zj) on the basis of the world coordinates. That is, it is assumed that the position of the j-th point is (Xj, Yj, Zj). In this case, j=Furthermore, it is assumed that each point is stationary in a three-dimensional real space. [0149]
  • Furthermore, it is assumed that the position of the projected image of the j-th point appearing on the photographic image Ph is (Uhj, Vhj). [0150]
  • Because of the same as the case of the Equations (27) to (29), the following Equations (30) to (32) are derived. [0151] [ XXhj YYhj ZZhj ] = Rh [ Xj Yj Zj ] - Rh Sh = [ Rh11 Rh12 Rh13 Rh21 Rh22 Rh23 Rh31 Rh32 Rh33 ] [ Xj Yj Zj ] - [ R11 R12 R13 R21 R22 R23 R31 R32 R33 ] [ Sh1 Sh2 Sh3 ] Where , j = 1 , , J , and h = 1 , , H . ( 30 ) Uhj = f Rh11 · Xj + Rh12 · Yj + Rh13 · Zj + Sh1 Rh31 · Xj + Rh32 · Yj + Rh33 · Zj + Sh3 ( 31 ) Vhj = f Rh21 · Xj + Rh22 · Yj + Rh23 · Zj + Sh2 Rh31 · Xj + Rh32 · Yj + Rh33 · Zj + Sh3 Where , [ Sh1 Sh2 Sh3 ] = - Rh Sh , j = 1 , , J , and h = 1 , , H . ( 32 )
    Figure US20030007679A1-20030109-M00004
  • In this case, (XXhj, YYhj, ZZhj) denotes the position of the j-th point on a coordinate system based on the image pickup device at the time Th. [0152]
  • It is assumed that the position and attitude (direction) of the image pickup device and the position (Xj, Yj, Zj) of the above point in a three-dimensional real space at each time are unknown. In this case, it is possible to obtain these unknown values from the photographic image Ph (h=1,. . . , H) at each time. That is, the position (Uhj, Vhj) of the projected image of the j-th point appearing on the photographic image Ph can be obtained by checking the image Ph. It is possible to obtain unknown parameters (rotation matrix Rh, parallel displacement vector Sh, and Xj, Yj, Zj) by substituting the obtained position (Uhj, Vhj) for the Equations (31) and (32). Where, h=1, . . . , H, j=1, . . . , J. [0153]
  • When solving the Equations (31) and (32) by substituting the value of (Uhj, Vhj) for the Equations (31) and (32), the indeterminacy of how to put the world coordinates remains. However, this is not important. For example, it is possible to make the position and attitude (direction) of the image pickup device at the first time T[0154] 1 coincide with the world coordinates. In other words, it is possible to solve the Equations (31) and (32) under the condition that R1 is a zero matrix and S1 is a zero vector.
  • Moreover, in the case of an image pickup device using a wide angle lens (fish-eye lens), a photographic image taken by the device may be distorted. In this case, it is necessary to previously measure a distortion value (to previously calibrate a camera). By multiplying a distorted photographic image by the inverse number of the distortion value, it is possible to form an undistorted photographic image. Moreover, by using the above method for the undistorted photographic image, it is possible to obtain the position and attitude (direction) of the image pickup device and the position (Xj, Yj, Zj) of the above point in a three-dimensional real space at each time from the Equations (31) and (32). [0155]
  • By solving the Equations (31) and (32), it is possible to obtain the position and attitude (direction) of the image pickup device in a three-dimensional real space and the position of the above point in the three-dimensional real space at each time. However, these obtained values do not show true values. As described above, errors are always included in measured data in general. The position and attitude (direction) of the image pickup device and the position (Xj, Yj, Zj) of the j-th point at each time (Th) are used as unknown parameters and these unknown parameters are obtained from observed data (Uhj, Vhj) in accordance with the Equations (31) and (32). In this case, because errors are included in the observed data (Uhj, Vhj), errors are also included in the position and attitude (direction) of the image pickup device in a three-dimensional real space and the position of the above point in the three-dimensional real space obtained by solving the Equations (31) and (32). [0156]
  • FIG. 17 shows an example of the structure of the image synthesizing apparatus of the second embodiment. In FIG. 17, a [0157] processing circuit 32 performs various types of operations in accordance with a program stored in a program memory 31. A data memory 33 is a memory for storing the data currently processed by the processing circuit 32. A frame memory 34 stores the image data to be displayed on an image display unit 35. An input unit 36 is constituted with, for example, a keyboard and a mouse and operated to input various commands. An input/output terminal 37 is connected with a not-illustrated external unit to transfer data. A bus line 38 is a bus for connecting these units each other.
  • Then, operations of the image synthesizing apparatus of the second embodiment are described below by referring to the flowchart in FIG. 18. The processing in FIG. 18 is mainly performed by the [0158] processing circuit 32.
  • It is assumed to put (synthesize) a building (second object: shown as a [0159] building 91 in FIG. 20) formed through CG between two real buildings (first objects: shown as a building 71 and a building 81 in FIG. 19). Then, in step S61, the above two real buildings (first objects) are photographed while moving a video camera (image pickup device). The image data for the two buildings is stored in the data memory 33 through the bus line 38 from the input/output terminal 37. In this case, because the position of the projected image of a building in each photographic image moves with passage of time, it is also necessary to displace the position of the projected image of the building (second object) to be synthesized in the photographic image. This state is shown in FIGS. 19 to 23.
  • That is, as shown in FIG. 19, it is assumed that the position of the video camera (image pickup device) is [0160] 61-1 at the first time T1. At the next time T2, the video camera moves to the position 61-2. Similarly, at the time Th (h=3, 4, . . . , H), the position of the video camera moves to 61-h. Thereby, the above two real buildings (first objects) 71 and 81 are picked up. It is the final object to synthesize the second object (building 91) between the buildings 71 and 81, that is, obtain the same synthesized image as the photographic image just obtained by photographing the three buildings 71, 91, and 81 shown in FIG. 20 while moving the video camera from the position 61-1 to the position 61-H.
  • FIG. 21 shows photographic images P[0161] 1, P2, . . . , Ph, . . . , PH taken by the video camera at the times T1, T2, . . . , Th, . . . , TH. Symbols 71-1, 71-2, . . . , 71-h, . . . , 71-H denote projected images of the above real building 71 at various times. Symbols 81-1, 81-2, . . . , 81-h, . . . , 81-H denote projected images of the above real building 81 at various times.
  • Then, the position and attitude (direction) of the video camera in a three-dimensional real space and positions of J characteristic points (first characteristic points) of the [0162] first objects 71 and 81 in the three-dimensional real space when photographing each photographic image Ph (FIG. 21) at each time are obtained. As described above, a characteristic point is a point at which brightness or color suddenly changes. For example, a characteristic point is a corner of a rectangular parallelepiped (e.g. building) or black point on a white plane (e.g. black dust attached to white wall). Specifically, corners 72 to 75 and 82 to 85 of the buildings 71 and 81 in FIG. 19 are first characteristic points. In this connection, the plane enclosed by the corners 72 to 75 is the front of the building 71 and the plane enclosed by the corners 82 to 85 is the front of the building 81.
  • That is, the position (U1j, V1j) of the projected image of the j-th characteristic point is obtained out of the first characteristic points (total of J characteristic points) appearing on the photographic image P[0163] 1 at the time T1. Moreover, the position (U2j, V2j) of the projected image of the j-th characteristic point is obtained out of the first characteristic points appearing on the photographic image P2 at the time T2. Similarly, the position (Uhj, Vhj) of the projected image of the j-th characteristic point is obtained out of the first characteristic points appearing on the photographic image Ph at the time Th (h=3, . . . , H).
  • Furthermore, rotation matrix Rh, parallel displacement vector Sh, and (Xj, Yj, Zj) are obtained for h=1, . . . , H and j=1, . . . , J by substituting the position (Uhj, Vhj) for the following Equation (33). The obtained data is stored in the [0164] data memory 33.
  • [Equation 33] [0165] h = 1 H j = 1 J { [ Uhj - f Rh11 · Xj + Rh12 · Yj + Rh13 · Zj + Sh1 Rh31 · Xj + Rh32 · Yj + Rh33 · Zj + Sh3 ] 2 + [ Vhj - f Rh21 · Xj + Rh22 · Yj + Rh23 · Zj + Sh2 Rh31 · Xj + Rh32 · Yj + Rh33 · Zj + Sh3 ] 2 } Where , [ R11 R12 R13 R21 R22 R23 R31 R32 R33 ] = Rh , [ Sh1 Sh2 Sh3 ] = - Rh Sh ( 33 )
    Figure US20030007679A1-20030109-M00005
  • The obtained rotation matrix Rh and parallel displacement vector Sh show the position and attitude (direction) of the video camera at the time Th based on the world coordinates. The obtained (Xj, Yj, Zj) shows the position of the j-th characteristic point based on the world coordinates. [0166]
  • Thus, it is possible to obtain the position and attitude (direction) of the video camera in a three-dimensional real space and positions of the characteristic points (first characteristic points) of the [0167] buildings 71 and 81 serving as first object in the three-dimensional real space when photographing each of the photographic images Ph (FIG. 21) at each time Th.
  • It is assumed that the corner [0168] 75 (FIG. 19) which is one of the first characteristic points corresponds to the k-th characteristic point. Symbol k denotes any numerical value of 1 to J. In accordance with the processing in step S61, the position (Xk, Yk, Zk) of the k-th characteristic point based on the world coordinates is obtained. Of course, the position does not show a true value but errors are included. In this connection, the corner 75 is the corner at the front top right of the real building 71.
  • Moreover, the photographic image Ph at the time Th is a photographic image photographed when the video camera is moved from the origin of the world coordinates due to the rotation of the matrix Rh and the translation of the vector Sh. Therefore, the matrix Rh and the vector Sh are also obtained. Of course, the above states {position and attitude (direction) shown by the rotation matrix Rh and parallel displacement vector Sh} do not show true values but errors are included. [0169]
  • Thus, because the positions of the two real buildings (first objects) [0170] 71 and 81 in a three-dimensional real space are obtained through the processing in step S61, the data for the position (Xj, Yj, Zj) is supplied to an image display unit 35 through the frame memory 34 and displayed in step S62. By viewing the indication, an operator (person performing the synthesis operation) designates a position to which the building (second object) 91 formed through CG (position in a virtual three-dimensional real space) is set by operating the input unit 36. The operator determines the above operation while considering the positions of the two real buildings (first objects) 71 and 81 in a three-dimensional real space. The determined positions are stored in the data memory 33.
  • Specifically, as shown in FIG. 20, because the positions of the [0171] corners 72 to 75 and the corners 82 to 85 serving as first characteristic points are obtained in the world coordinates, the building (second object) 91 formed through CG is put between the positions. That is, it is necessary to set the building (second object) 91 formed through CG so that the plane enclosed by the corners 75, 74, 83, and 82 serves as the front of the building (second object) 91 formed through CG (so that the top left corner 92 and the bottom left corner 93 of the building 91 correspond to the top right corner 75 and the bottom right corner 74 of the building 71 and the top right corner 95 and bottom right corner 94 of the building 91 correspond to the top left corner 82 and the bottom left corner 83 of the building 81). Thus, the position of the building (second object) 91 formed through CG is determined.
  • That is, the building (second object) [0172] 91 formed through CG is virtually set so that the front top left of the building (second object) 91 formed through CG is brought to the position (Xk, Yk, Zk) based on the world coordinates (FIG. 20). In other words, the front top left position of the building (second object) 91 formed through CG is (Xk, Yk, Zk). In this connection, the position (Xk, Yk, Zk) is the position of the corner 75 serving as the k-th characteristic point. Thereby, the real building 71 and the building (second object) 91 formed through CG are set so that the front top right of the building 71 contacts the front top left of the building 91.
  • Then, the corner [0173] 75 (k-th characteristic point) which is one of the first characteristic points is considered below. The photographic image Ph at the time Th includes the projected image of the corner 75 which is the k-th characteristic point. The position of the projected image is shown as (Uhk, Vhk). Therefore, the following Equations (34) and (35) are effected.
  • Uhk≠f(Rh11·Xk+Rh12·Yk+Rh13·Zk+Sh1′)/(Rh31·Xk+Rh32·Yk+Rh33·Zk+Sh3′)  (34)
  • Vhk≠f(Rh21·Xk+Rh22·Yk+Rh23·Zk+Sh2′)/(Rh31·Xk+Rh32+Yk+Rh33·Zk+Sh3′)  (35)
  • In this case, Rh, Sh, Xk, Yk, and Zk are values obtained by minimizing the Equation (33). The right and left sides of the Equations (34) and (35) have almost equal values. As previously described, however, they are not strictly equal to each other. However, the position (UUhk, VVhk) of the projected image at the front top left of the [0174] building 91 completely meets the following Equations (36) and (37). UUhk = f Rh11 · Xk + Rh12 · Yk + Rh13 · Zk + Sh1 Rh31 · Xk + Rh32 · Yk + Rh33 · Zk + Sh3 ( 36 ) VVhk = f Rh21 · Xk + Rh22 · Yk + Rh23 · Zk + Sh2 Rh31 · Xk + Rh32 · Yk + Rh33 · Zk + Sh3 Where , [ R11 R12 R13 R21 R22 R23 R31 R32 R33 ] = Rh , [ Sh1 Sh2 Sh3 ] = - Rh Sh ( 37 )
    Figure US20030007679A1-20030109-M00006
  • That is, as shown in FIG. 22, it is considered to photograph the building (second object) [0175] 91 formed through CG with the video camera by assuming that the front top left of the building 91 is present at the position (Xk, Yk, Zk) based on the world coordinates. The position and attitude (direction) of the video camera are shown by the rotation matrix Rh and parallel displacement vector Sh at the time Th. Therefore, the position of the projected image at the front top left of the building 91 results in the position (UUhk, VVhk) in the photographic image at the time Th shown by the Equations (36) and (37). Where, h=1, . . . , H.
  • As a result, (Uhk, Vhk) is not equal to (UUhk, VVhk). This Equation represents the following. [0176]
  • It is assumed that an operator arranges the [0177] real building 71 and the building 91 formed through CG so that the front top right of the building 71 contacts the front top left of the building 91, forms the projected image of the virtual building 91, and synthesizes an actually-photographed photographic image. The projected image at the front top right of the building 71 at the time Th is located at (Uhk, Vhk) (FIG. 21). Moreover, the projected image at the front top left of the artificially-formed building 91 is located at (UUhk, VVhk) (FIG. 22). Therefore, the projected image at the front top right of the real building 71 at the time Th does not coincide with the projected image at the front top left of the virtual building 91. When a person views the synthesized image of them, he does not sense that the state is photographed in which the front top right of the real building 71 contacts the front top left of the building 91 formed through CG.
  • Moreover, it is not guaranteed that the difference between the right and left sides of the Equations (34) and (35) is constant every time Th. Therefore, when a person continuously views an image synthesized every time, he feels as if the interval between the front top right of the [0178] real building 71 and the front top left of the building 91 formed through CG swings.
  • Because FIG. 23 shows synthesized images under an ideal state free from errors, (Uhk, Vhk) is equal to (UUhk, VVhk). In fact, however, (Uhk, Vhk) is not equal to (UUhk, VVhk). [0179]
  • Therefore, the Rh, Sh, Xj, Yj, and Zj obtained in step S[0180] 61 and the position (Uhj, Vhj) of the projected image of the j-th characteristic point in the photographic image Ph do not completely meet the Equations (31) and (32). That is, the following Equations (38) and (39) are effected. Where, h=1, . . . , H, j=1, . . . , J. Uhj f Rh11 · Xj + Rh12 · Yj + Rh13 · Zj + Sh1 Rh31 · Xj + Rh32 · Yj + Rh33 · Zj + Sh3 ( 38 ) Vhj f Rh21 · Xj + Rh22 · Yj + Rh23 · Zj + Sh2 Rh31 · Xj + Rh32 · Yj + Rh33 · Zj + Sh3 ( 39 )
    Figure US20030007679A1-20030109-M00007
    Where , [ R11 R12 R13 R21 R22 R23 R31 R32 R33 ] = Rh , [ Sh1 Sh2 Sh3 ] = - Rh Sh
    Figure US20030007679A1-20030109-M00008
  • Thus, the [0181] processing circuit 32 reads each time Th and the data for characteristic points from the data memory 33 in step S63, computes the following Equations (40) to (44) for the time Th and the characteristic point data, obtains the coordinates (XXhj, YYhj, ZZhj) and the displacement value (δXXhj, δYYhj) of each characteristic point based on the video camera, and stores the coordinates and displacement value in the data memory 33.
  • XXhj=Rh11·Xj+Rh12·Yj+Rh13·Zj+Sh1′  (40)
  • YYhj=Rh21·Xj+Rh22·Yj+Rh23·Zj+Sh2′  (41)
  • ZZhj=Rh31·Xj+Rh32·Yj+Rh33·Zj+Sh3′  (42) Where , [ R11 R12 R13 R21 R22 R23 R31 R32 R33 ] = Rh , [ Sh1 Sh2 Sh3 ] = - Rh Sh
    Figure US20030007679A1-20030109-M00009
  • δXXhj=(Uhj−ZZhj)/f−XXhj  (43)
  • δYYhj=(Vhj−ZZhj)/f−YYhj  (44)
  • That is, the position (X, Y, Z) based on the world coordinates is transformed into the position (XXh, YYh, ZZh) in a coordinate system based on the video camera at the time Th in accordance with the transform formulas shown in the following Equations (45) to (47). [0182]
  • XXh=Rh11·X+Rh12·Y+Rh13−Z+Sh1′  (45)
  • YYh=Rh21·X+Rh22·Y+Rh23·Z+Sh2′  (46)
  • ZZh=Rh31·X+Rh32·Y+Rh33·Z+Sh3′  (47)
  • The point located at (XXh, YYh, ZZh) is projected on the position (f·XXh/ZZh, f·YYh/ZZh) in the photographic image Ph. [0183]
  • The corner [0184] 75 (k-th characteristic point) shown in FIG. 12 is considered below. The position of the characteristic point becomes the position (XXhk, YYhk, ZZhk) shown by the following Equations (48) to (50) in the coordinate system based on the video camera at the time Th.
  • XXhk=Rh11·Xk+Rh12·Yk+Rh13·Zk+Sh1′  (48)
  • YYhk=Rh21·Xk+Rh22·Yk+Rh23·Zk+Sh2′  (49)
  • ZZhk=Rh31·Xk+Rh32·Yk+Rh33·Zk+Sh3′  (50)
  • Therefore, if an ideal photographing state free from errors is set, the position (XXhk, YYhk, ZZhk) and the position (Uhk, Vhk) of the projected image of the k-th characteristic point in the photographic image Ph have the relation between the following Equations (51) and (52). [0185]
  • Uhk=f(XXhk/ZZhk)  (51)
  • Vhk=f(YYhk/ZZhk)  (52)
  • However, because errors are actually present, the equal signs shown in the Equations (51) and (52) are not effected. That is, errors equivalent to (δXX, δYY) shown in the following Equations (53) and (54) are produced. [0186]
  • δXX=(Uhk·ZZhk)/f−XXhk  (53)
  • δYY=(Vhk·ZZhk)/f−YYhk  (54)
  • Therefore, the errors (δXX, δYY) are computed in accordance with the Rh, Sh, Xk, Yk, and Zk obtained in step S[0187] 61, measured values (Uhk, Vhk), and the Equations (48) to (50) and the Equations (53) and (54).
  • It is determined in step S[0188] 62 to imaginarily set the corner 92 at the front top left of the building (second object) 91 formed through CG to the position (Xk, Yk, Zk) based on the world coordinates. That is, the corner 92 is set to the position (XXhk, YYhk, ZZhk) shown by the Equations (48) to (50) in the coordinate system based on the video camera at the time Th. The corner 92 is projected on the position (UUhk, VVhk) shown by the following Equations (55) and (56) in the photographic image at the time Th.
  • UUhk=f(XXhk/ZZhk)  (55)
  • VVhk=f(YYhk/ZZhk)  (56)
  • In this connection, the Equations (48) to (50) and the Equations (55) and (56) are equivalent to the Equations (36) and (37). The position (UUhk, Whk) shown by the Equations (55) and (56) is different from the position (Uhk, Vhk) of the projected image of the k-th actual characteristic point. This is a positional displacement becoming a problem. [0189]
  • Therefore, in the case of the second embodiment, the portion of the building (second object) [0190] 91 formed through the CG shown by the position (XXhk, YYhk, ZZhk) in the coordinate system based on the video camera at the time Th is displaced by the value shown by the Equations (53) and (54) before forming the projected image of the second object. That is, it is determined in step S62 to set the corner 92 at the front top left of the building 91 to the position (XXhk, YYhk, ZZhk) in the coordinate system based on the video camera at the time Th. However, the position is displaced by the value shown by the Equations (53) and (54) so that it is brought to (XXhk+δXX, YYhk+δYY, ZZhk). Thus, by displacing the position of the corner 92 at the front top left of the building 91, the position of the projected image of the corner 92 at the front top left of the artificially-formed building 91 at the time Th is brought to the position shown by the following Equations (57) and (58).
  • UUhk=f(XXhk+δXX)/ZZhk  (57)
  • VVhk=f(YYhk+δYY)/ZZhk  (58)
  • The position (UUhk, VVhk) is just the same as the position (Uhk, Vhk) of the projected image of the corner [0191] 75 (k-th characteristic point) because the following Equations (59) and (60) are effected. UUhk = f ( XXhk + δXX ) / ZZhk ) = f { XXhk + ( Uhk · ZZhk / f ) - XXhk } / ZZhk = Uhk ( 59 ) VVhk = f ( YYhk + δYY ) / ZZhk ) = f { YYhk + ( Vhk · ZZhk / f ) - YYhk } / ZZhk = Vhk ( 60 )
    Figure US20030007679A1-20030109-M00010
  • In short, it is decided in step S[0192] 61 that the k-th characteristic point is present at the position (Xk, Yk, Zk) based on the world coordinates. This position is the position (XXhk, YYhk, ZZhk) shown by the Equations (48) to (50) in the coordinate system based on the video camera at the time Th. This value is displaced by (δXX, δYY) shown by the Equations (53) and (54) from the position (Uhk, Vhk) of the projected image of the k-th characteristic point projected on the photographic image Ph at the time Th in a three-dimensional real space. Therefore, to set the object (corner 92 at the front top left of the building 91) to be synthesized to the position (XXhk, YYhk, ZZhk) in the coordinate system based on the video camera at the time Th, a projected image is formed by assuming that the projected image is present at a position displaced by (δXX, δYY). Thereby, the position of the projected image coincides with that of the projected image of the actually-photographed object (corner 75 at the front top right of the building 71).
  • In the case of the above specific example, the corner [0193] 75 (k-th characteristic point) is described. The same is true for characteristic points other than the k-th characteristic point. In the case of an object (or a part of the object) to be synthesized present at a position just coinciding with the first characteristic points (first, . . . , j-th characteristic points), it is necessary to compute (δXX, δYY) which is the displacement value (distortion value) of a corresponding characteristic point and form the projected image of the object by assuming that the projected image is present at a position displaced by (δXX, δYY), as described above.
  • However, when the first characteristic points are not present at the position of the object (or a part of the object) to be synthesized, it is necessary to obtain the displacement value (distortion value) (δXX, δYY) through interpolation from a nearby first characteristic point. [0194]
  • Therefore, in step S[0195] 64, the processing circuit 32 reads the characteristic point (XXhj, YYhj, ZZhj) at each time Th from the data memory 33, performs Delaunay division by using the characteristic point as a mother point, obtains a Delaunay convex polygon, and stores the polygon in the data memory 33. That is, the position of the j-th characteristic point in the coordinate system based on the video camera at the time Th is assumed as (XXhj, YYhj, ZZhj). This position can be obtained from the Equations (40) to (42) by using the Rh, Sh, Xj, Yj, and Zj obtained in step S61.
  • Then, the displacement value (distortion value) (δXXhj, δYYhj) at (XXhj, YYhj, ZZhj) is obtained by using the Equations (43) and (44). In this case, the suffix h and j are attached to clarify that XX, YY, and ZZ depend on the time Th and the j-th characteristic point out of the first characteristic points. Computation of the Equations (43) and (44) is performed by substituting values obtained from the Equations (40) to (42) for XXhj, YYhj, and ZZhj and moreover, substituting a value measured from the actual photographic image Ph for Uhj and Vhj. [0196]
  • Obtaining the displacement value (distortion value) (δXX, δYY) at every position (XX, YY, ZZ) through interpolation represents the fact of obtaining the displacement value (distortion value) at any position under the condition that the displacement value (distortion value) at (XXhj, YYhj, ZZhj) which is the position of the j-th characteristic point is (δXXhj, δYYhj). In this case, j=1, . . . , J. Moreover, the position in the three-dimensional real space described here represents a coordinate system based on the video camera at the time Th. [0197]
  • To perform interpolation, the Delaunay division is first performed in a three-dimensional real space. That is, division of Voronoi region is performed by assuming J characteristic points as mother points to perform the Delaunay division having a dual relation with the division of the Voronoi region. Each Delaunay-divided region forms a convex polygon using mother points (that is, first characteristic points) as vertexes. Because an optional point (XX, YY, ZZ) in a three-dimensional real space is included in a Delaunay convex polygon (Delaunay-divided region), values obtained by weighting and averaging the displacement values (distortion value) (δXXhj, δYYhj) of the vertexes of the Delaunay convex polygon (first characteristic points in which the displacement values (distortion values) are already obtained) so as to be inversely proportional to the distance from the point (XX, YY, ZZ) up to each vertex of the polygon is used as the displacement value (distortion value) of the point (XX, YY, ZZ). Thus, it is possible to obtain the displacement value (distortion value) at a position other than J first characteristic points. [0198]
  • The Voronoi region and Delaunay division are described in detail in the description part (p. 1064) of computation geometry of “GENDAI SURIKAGAKU JITEN (transliterated)(issued by Maruzen Co., Ltd.).” Thus, in steps S[0199] 63 and S64, the displacement value (distortion value) at each of J first characteristic points is first obtained in accordance with the Equations (40) to (44) by using the Rh, Sh, Xj, Yj, and Zj obtained in step S61 and the measured values (Uhk, Vhk). Then, the Delaunay convex polygon is obtained by assuming J characteristic points as mother points. For each position in each Delaunay convex polygon, the displacement value (distortion value) is obtained by weighting and averaging the above-obtained displacement values (distortion values) of vertexes (that is, first characteristic points).
  • In this connection, the displacement value (distortion value) depends on the time Th and moreover, depends on a position in a three-dimensional real space. That is, the displacement value (distortion value) is the function of Th and also, the function of the position (XX, YY, ZZ) in the coordinate system based on the video camera. The relation of the displacement value (distortion value) between the Th and (XX, YY, ZZ) is obtained in steps S[0200] 63 and S64. Steps S63 and S64 are computation steps for obtaining the displacement value (distortion value) of each point in a three-dimensional real space at each time.
  • It is important for the computation steps for obtaining the displacement value (distortion value) of each point to obtain the displacement value (distortion value) at an optional point under the condition that the displacement value (distortion value) at (XXhj, YYhj, ZZhj) which is the position of the j-th characteristic point is equal to (δXXhj, δYYhj). It is also possible to use other method instead of the Delaunay division under the above condition. [0201]
  • In step S[0202] 62, a position (virtual position based on world coordinates) to which the building (second object) 91 formed through CG is set is obtained. Moreover, in step S61, the rotation matrix Rh and parallel displacement vector Sh showing the position and attitude (direction) of the video camera at each time Th (h=1, . . . , H) are obtained.
  • Therefore, in step S[0203] 65, the processing circuit 32 reads world-coordinates-based virtual positions (Xm, Ym, Zm) of all the points (assumed as M points) constituting the building (second object) 91 formed through CG from the data memory 3, applies the rotation matrix Rh and parallel displacement vector Sh also read from the data memory 33 to the virtual positions, obtains the position (XXhm, YYhm, ZZhm) shown by the coordinate system based on the video camera at the time Th, and stores the position (XXhm, YYhm, ZZhm) in the data memory 33. In this case, m=1, . . . , M. That is, the following Equations (61) to (63) are computed.
  • XXhm=Rh11·Xm+Rh12·Ym+Rh13·Zm+Sh1′  (61)
  • YYhm=Rh21·Xm+Rh22·Ym+Rh23·Zm+Sh2′  (62)
  • ZZhm=Rh31·Xm+Rh32·Ym+Rh33·Zm+Sh3′  (63)
  • Then, in step S[0204] 66, the processing circuit 32 obtains the displacement value (distortion value) (δXXhm, δYYhm) obtained in steps S63 and S64 and corresponding to the position (XXhm, YYhm, ZZhm) obtained from the Equations (61) to (63) in accordance with the following Equations (64) and (65). δ XXhm = j δ XXhj DISTANCE BETWEEN ( XXhm , YYhm , ZZhm ) AND ( XXhj , YYhj , ZZhj ) j 1 DISTANCE BETWEEN ( XXhm , YYhm , ZZhm ) AND ( XXhj , YYhj , ZZhj ) ( 64 ) δ YYhm = j δ YYhj DISTANCE BETWEEN ( XXhm , YYhm , ZZhm ) AND ( XXhj , YYhj , ZZhj ) j 1 DISTANCE BETWEEN ( XXhm , YYhm , ZZhm ) AND ( XXhj , YYhj , ZZhj ) ( 65 )
    Figure US20030007679A1-20030109-M00011
  • Then, the (δXXhm, δYYhm) is added to (XXhm, YYhm, ZZhm). The added position (XXhm+δXXhm, YYhm+δYYhm, ZZhm) serves as the final position of the m-th point constituting the building (second object) [0205] 91 formed through CG. Of course, the position is the coordinate system based on the video camera at the time Th.
  • Then, the position (UUhm, VVhm) of the projected image of the m-th point constituting the building(second object) [0206] 91 formed through CG is obtained. That is, the following Equations (66) and (67) are computed.
  • UUhm=f(XXhm+δXXhm)/ZZhm)  (66)
  • VVhm=f(YYhm+δYYhm)/ZZhm)  (67)
  • The Equations (61) to (63) and the Equations (66) and (67) are computed on every m=1, . . . , M. Thereby, it is possible to compute the projected image of the building (second object) [0207] 91 formed through CG.
  • In the case of this embodiment, the displacement value (δXXhj, δYYhj) is obtained in accordance with the Equations (40) to (44) by assuming that a displacement value is present only in the directions of a two-dimensional plane (X, Y) based on the video camera. However, it is also possible to assume that a displacement value is present in Z direction. That is, it is possible to assume the minimum value of (δXXhj, δYYhj, δZZhj) meeting the following Equations (68) and (69) as a displacement value. [0208]
  • Uhj=f(XXhj+δXXhj)/(ZZhj+δZZhj)  (68)
  • Vhj=f(YYhj+δYYhj)/(ZZhj+δZZhj)  (69)
  • However, (XXhj, YYhj, ZZhj) in the above Equations can be obtained from the Equations (40) to (42). In this case, by obtaining the displacement value of an optional position under the condition that the displacement value (distortion value) at (XXhj, YYhj, ZZhj) which is the position of the j-th characteristic point is equal to (δXXhj, δYYhj, δZZhj), it is possible to obtain the displacement value at each position. The position (UUhm, VVhm) of the projected image of the m-th point constituting the building (second object) [0209] 91 formed through CG is shown by the following Equations (70) and (71) instead of the Equations (66) and (67).
  • UUhm=f(XXhm+δXXhm)/(ZZhm+δZZhm)  (70)
  • VVhm=f(YYhm+δYYhm)/(ZZhm+δZZhm)  (71)
  • In the above Equations, (δXXhm, δYYhm, δZZhm) denotes the displacement value (distortion value) at the position (XXhm, YYhm, ZZhm) obtained by the Equations (61) to (63). [0210]
  • Finally, by synthesizing the projected image of the building (second object) [0211] 91 formed through CG with the photographic image Ph, it is possible to obtain a synthesized image to be finally obtained at the time Th. By performing the above operation for all times of h=1, . . . , H, it is possible to obtain synthesized images from the time T1 up to the time TH.
  • By using the technique of the second embodiment, a displacement value (distortion value) is added so that the virtual position of an object (second object) to be synthesized designated by an operator coincides with the position of a characteristic point of a photographic image actually photographed at the time for each time and each position in a three-dimensional real space. Therefore, the position of a synthesized image is not displaced. [0212]
  • Moreover, it is possible to determine the displacement value (distortion value) at every position in a three-dimensional space and a displacement value (distortion value) is added to every portion constituting a second object without distinguishing between vertexes, sides, planes of an object (second object) to be synthesized. Therefore, there is no problem on alignment of sides and planes described in the first embodiment. [0213]
  • Moreover, various modifications and applications can be considered other than the above description as long as they are not deviated from the gist of the present invention. Therefore, the gist of the present invention is not restricted to synthesis of a CG image formed through CG with a plate having characteristic points but the gist includes every case related to synthesis of a CG image formed through CG with a real image. [0214]
  • As supply media for supplying a computer program for performing the above processing to users, it is possible to use not only recording media such as a magnetic disk, CD-ROM, and solid state memory but also communication media such as a network and satellite. [0215]
  • As described above, according to the image synthesizing apparatus of [0216] claim 1, the image synthesizing method of claim 8, and the supply medium of claim 15, a coordinate transform function is obtained in accordance with a first characteristic point nearby a synthesis position for synthesizing an object and fourth coordinates on a three-dimensional coordinate system corresponding to a second image are obtained by applying the coordinate transform function to third coordinates so as to synthesize the projected image of the object at a position corresponding to the fourth coordinates of the second image. Therefore, it is possible that the projected image of the object seem to be natural in first and second images.
  • According to the position detecting apparatus of [0217] claim 16, the position detecting method of claim 17, and the supply medium of claim 18, a coordinate transform function is obtained in accordance with a first characteristic point nearby an object and fifth coordinates on a three-dimensional coordinate system corresponding to a second image are obtained by applying the coordinate transform function to third coordinates so as to detect the difference between the fourth and fifth coordinates. Therefore, it is possible to accurately compute the positional relation of a third image in the first and second images.
  • According to the image synthesizing apparatus of claim 19, the image synthesizing method of [0218] claim 21, and the supply medium of claim 23, a position to which the projected image of a second object is set is corrected in accordance with a distortion value. Therefore, it is possible to control swinging of a synthesized image.
  • INDUSTRIAL APPLICABILITY
  • An image synthesizing apparatus can be applied to the case of synthesizing a computer-graphics (CG) image with a real image photographed by a video camera. [0219]

Claims (23)

1. An image synthesizing apparatus for synthesizing the projected image of an object with at least first and second images, the apparatus comprising:
first means for obtaining a first characteristic point nearby a synthesis position for synthesizing said object among the characteristic points projected onto said first image;
second means for obtaining first coordinates of said first characteristic point on a three-dimensional coordinate system corresponding to said first image;
third means for obtaining second coordinates of a second characteristic point corresponding to said first characteristic point on a three-dimensional coordinate system corresponding to said second image among the characteristics points projected onto said second image;
fourth means for obtaining coordinate transform functions of said first coordinates and said second coordinates;
fifth means for obtaining third coordinates on the three-dimensional coordinate system of said synthesis position for synthesizing said object with said first image corresponding to said first image;
sixth means for obtaining fourth coordinates on a three-dimensional coordinate system corresponding to said second image by applying said coordinate transform functions to said third coordinates; and
seventh means for synthesizing the projected image of said object at a position corresponding to said fourth coordinates of said second image.
2. The image synthesizing apparatus according to claim 1,
wherein said coordinate transform functions are functions related to rotational transform and rectilinear transform.
3. The image synthesizing apparatus according to claim 1, wherein
said sixth means obtains said fourth coordinates in accordance with values obtained by adding first three-dimensional vector to said third coordinates,
said seventh means synthesizes the projected image of said object at a position corresponding to values obtained by adding second three-dimensional vector to said fourth coordinates.
4. The image synthesizing apparatus according to claim 3,
wherein at least one of said first and second three-dimensional vectors is zero vector.
5. The image synthesizing apparatus according to claim 1, wherein said fourth means provides a weight larger than that of a fourth characteristic point farther from a synthesis position for synthesizing said object for a third characteristic point closer to the synthesis position out of said first characteristic points and thereby, obtains coordinate transform functions of said first and second coordinates.
6. The image synthesizing apparatus according to claim 1, wherein said first characteristic points are obtained from points in which at least one of a brightness signal and a color difference signal greatly changes compared to adjacent points.
7. The image synthesizing apparatus according to claim 1, wherein
said first image is an image obtained by actually photographing a predetermined object at the first time,
said second image is an image obtained by actually photographing said object at the second time later than said first time, and
said object is formed through computer graphics.
8. An image synthesizing method for synthesizing the projected image of an object with first and second images, the method comprising:
the first step of obtaining a first characteristic point nearby a synthesis position for synthesizing said object among the characteristic points projected onto said first image;
the second step of obtaining first coordinates on the three-dimensional coordinate system of said first characteristic point corresponding to said first image;
the third step of obtaining second coordinates of a second characteristic point corresponding to said first characteristic point on a three-dimensional coordinate system corresponding to said second image among the characteristic points projected onto said second image;
the fourth step of obtaining coordinate transform functions of said first and second coordinates;
the fifth step of obtaining third coordinates on the three-dimensional coordinate system of said synthesis position for synthesizing said object with said first image corresponding to said first image;
the sixth step of obtaining fourth coordinates on a three-dimensional coordinate system corresponding to said second image by applying said coordinate transform functions to said third coordinates; and
the seventh step of synthesizing the projected image of said object at a position corresponding to the fourth coordinates of said second image.
9. The image synthesizing method according to claim 8, wherein said coordinate transform functions are functions related to rotational transform and rectilinear transform.
10. The image synthesizing method according to claim 8, wherein
said fourth coordinates are obtained in accordance with values obtained by adding first three-dimensional vector to said third coordinates in said sixth step, and
the projected image of said object is synthesized at a position corresponding to values obtained by adding second three-dimensional vector to said fourth coordinates in said seventh step.
11. The image synthesizing method according to claim 10, wherein at least one of said first and second three-dimensional vectors is zero vector.
12. The image synthesizing method according to claim 8, wherein coordinate transform functions of said first and second coordinates are obtained by providing a weight larger than that of fourth characteristic points farther from a synthesis position for synthesizing said object for third characteristic points closer to the synthesis position among said first characteristic points in said fourth step.
13. The image synthesizing method according to claim 8, wherein said first characteristic points are obtained from points in which at least one of a brightness signal and a color difference signal greatly changes compared to adjacent points.
14. The image synthesizing method according to claim 8, wherein
said first image is an image obtained by actually photographing a predetermined object at the first time,
said second image is an image obtained by actually photographing said object at the second time later than said first time; and
said object is an object formed through computer graphics.
15. A supply medium for supplying a computer program used for an image synthesizing apparatus for synthesizing the projected image of an object with at least first and second images, the computer program comprising:
the first step of obtaining first characteristic points nearby a synthesis position for synthesizing said object among the characteristic points projected onto said first image;
the second step of obtaining first coordinates of said first characteristic point on a three-dimensional coordinate system corresponding to said first image;
the third step of obtaining second coordinates of a second characteristic point corresponding to said first characteristic point on a three-dimensional coordinate system corresponding to said second image among the characteristic points projected onto said second image;
the fourth step of obtaining coordinate transform functions of said first coordinates and said second coordinates;
the fifth step of obtaining third coordinates of said synthesis position for synthesizing said object with said first image on a three-dimensional coordinate system corresponding to said first image;
the sixth step of obtaining fourth coordinates on a three-dimensional coordinate system corresponding to said second image by applying said coordinate transform functions to said third coordinates; and
the seventh step of synthesizing the projected image of said object at a position corresponding to said fourth coordinates of said second image.
16. A position detecting apparatus for detecting the positional relation between objects projected onto a first image and a second image, the apparatus comprising:
first means for obtaining a first characteristic point nearby said object among the characteristic points projected onto said first image;
second means for obtaining first coordinates of said first characteristic point on a three-dimensional coordinate system corresponding to said first image;
third means for obtaining second coordinates of a second characteristic point corresponding to said first characteristic point on a three-dimensional coordinate system corresponding to said second image among the characteristic points projected onto said second image;
fourth means for obtaining third coordinates of said object on a three-dimensional coordinate system corresponding to said first image;
fifth means for obtaining fourth coordinates of said object on a three-dimensional coordinate system corresponding to said second image;
sixth means for obtaining coordinate transform functions of said first coordinates and said second coordinates;
seventh means for obtaining fifth coordinates on a three-dimensional coordinate system corresponding to said second image by applying said coordinate transform functions to said third coordinates; and
eighth means for detecting the difference between said fourth coordinates and said fifth coordinates.
17. A position detecting method for detecting the positional relation between objects projected onto a first image and a second image, the method comprising;
the first step of obtaining a first characteristic point nearby said object among the characteristic points projected onto said first image;
the second step of obtaining first coordinates of said first characteristic point on a three-dimensional coordinate system corresponding to said first image;
the third step of obtaining second coordinates of a second characteristic point corresponding to said first characteristic point on a three-dimensional coordinate system corresponding to said second image among the characteristic points projected onto said second image;
the fourth step of obtaining third coordinates of said object on a three-dimensional coordinate system corresponding to said first image;
the fifth step of obtaining fourth coordinates of said object on a three-dimensional coordinate system corresponding to said second image;
the sixth step of obtaining coordinate transform functions of said first coordinates and said second coordinates;
the seventh step of obtaining fifth coordinates on a three-dimensional coordinate system corresponding to said second image by applying said coordinate transform functions to said third coordinates; and
the eighth step of detecting the difference between said fourth coordinates and said fifth coordinates.
18. A supply medium for supplying a computer program used for a position detecting apparatus for detecting the positional relation between objects projected onto a first image and a second image, the computer program comprising:
the first step of obtaining a first characteristic point nearby said object among the characteristic points projected onto said first image;
the second step of obtaining first coordinates of said first characteristic point on a three-dimensional coordinate system corresponding to said first image;
the third step of obtaining second coordinates of a second characteristic point corresponding to said first characteristic point on a three-dimensional coordinate system corresponding to said second image among the characteristic points projected onto said second image;
the fourth step of obtaining third coordinates of said object on a three-dimensional coordinate system corresponding to said first image;
the fifth step of obtaining fourth coordinates of said object on a three-dimensional coordinate system corresponding to said second image;
the sixth step of obtaining coordinate transform functions of said first coordinates and said second coordinates;
the seventh step of obtaining fifth coordinates on a three-dimensional coordinate system corresponding to said second image by applying said coordinate transform functions to said third coordinates; and
the eighth step of detecting the difference between said fourth coordinates and said fifth coordinates.
19. An image synthesizing apparatus for synthesizing the projected image of a second object by assuming that said second object is present at a predetermined position of each of a plurality of photographic images on which a first object photographed by an image pickup device is photographed, the apparatus comprising:
image-pickup-device-position computing means for computing the most-probable position of said image pickup device when photographing each of said photographic images;
characteristic point position computing means for computing the most-probable position of a characteristic point of said first object;
virtual projected position computing means for computing a virtual projected position on which the most-probable position of said characteristic point is projected when performing photography by using said image pickup device present at said most-probable position;
distortion value computing means for computing a distortion value in accordance with the difference between said virtual projected position and the position of the projected image of said characteristic point actually photographed on each of said photographic images;
correcting means for correcting a position to which the projected image of said second object is set in accordance with the distortion value computed by said distortion value computing means;
projected image computing means for computing the projected image of said second object when photographing said second object from said image pickup device at the most-probable position by assuming that said second object is present at the position corrected by said correcting means; and
synthesizing means for synthesizing the projected image of said second object computed by said projected image computing means with each of said photographic images.
20. The image synthesizing apparatus according to claim 19, wherein said distortion value computing means computes said distortion value through interpolation in accordance with said characteristic point.
21. An image synthesizing method for synthesizing the projected image of said second object with each of a plurality of photographic images on which a first object is photographed by an image pickup device by assuming that a second object is present at a predetermined position, the method comprising:
the image-pickup-device-position computing step for computing the most-probable position of said image pickup device when photographing each of said photographic images;
the characteristic point position computing step of computing the most-probable position of a characteristic point of said first object;
the virtual projected position computing step for computing a virtual projected position on which the most-probable position of said characteristic point is projected when performing photography by using said image pickup device set at said most-probable position;
the distortion value computing step of computing a distortion value in accordance with the difference between said virtual projected position and the position of the projected image of said characteristic point actually photographed on each of said photographic images;
the correcting step of correcting a position to which the projected image of said second object is set in accordance with the distortion value computed by said distortion value computing step;
the projected image computing step for computing the projected image of said second object when photographing said second object from said image pickup device at the most-probable position by assuming that said second object is located at the position corrected in said correcting step; and
the synthesizing step of synthesizing the projected image of said second object computed in said projected image computing step with each of said photographic images.
22. The image synthesizing method according to claim 21, wherein said distortion value is computed through interpolation in accordance with said characteristic point in said distortion value computing step.
23. A supply medium for supplying a computer program used for an image synthesizing apparatus for synthesizing the projected image of a second object with each of a plurality of photographic images on which a first object photographed by an image pickup device is photographed by assuming that the second object is present at a predetermined position, the computer program comprising:
the image-pickup-device-position computing step of computing the most-probable position of said image pickup device when photographing each of said photographic images;
characteristic point position computing step of computing the most-probable position of a characteristic point of said first object;
virtual projected position computing step of computing a virtual projected position to which the most-probable position of said characteristic point is projected when performing photography by using said image pickup device set at the most-probable position;
the distortion value computing step of computing a distortion value in accordance with the difference between said virtual projected position and the position of the projected image of said characteristic point actually photographed on each of said photographic images;
the correcting step of correcting a position to which the projected image of said second object is set in accordance with the distortion value computed in said distortion value computing step;
the projected image computing step of computing the projected image of said second object when photographing said second object from said image pickup device at the most-probable position by assuming that said second object is present at the position corrected in said correcting step; and
synthesizing step of synthesizing the projected image of said second object computed in said projected image computing step with each of said photographic images.
US10/032,875 1997-03-14 2001-12-27 Image synthesizing apparatus and method, position detecting apparatus and method, and supply medium Abandoned US20030007679A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US10/032,875 US20030007679A1 (en) 1997-03-14 2001-12-27 Image synthesizing apparatus and method, position detecting apparatus and method, and supply medium

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
JP6054197 1997-03-14
JPP09-060541 1997-03-14
PCT/JP1998/001053 WO1998041951A1 (en) 1997-03-14 1998-03-13 Image synthesizing device and method, position detecting device and method, and supply medium
US09/178,092 US6404913B1 (en) 1997-03-14 1998-10-23 Image synthesizing apparatus and method, position detecting apparatus and method, and supply medium
US10/032,875 US20030007679A1 (en) 1997-03-14 2001-12-27 Image synthesizing apparatus and method, position detecting apparatus and method, and supply medium

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US09/178,092 Continuation US6404913B1 (en) 1997-03-14 1998-10-23 Image synthesizing apparatus and method, position detecting apparatus and method, and supply medium

Publications (1)

Publication Number Publication Date
US20030007679A1 true US20030007679A1 (en) 2003-01-09

Family

ID=13145265

Family Applications (2)

Application Number Title Priority Date Filing Date
US09/178,092 Expired - Fee Related US6404913B1 (en) 1997-03-14 1998-10-23 Image synthesizing apparatus and method, position detecting apparatus and method, and supply medium
US10/032,875 Abandoned US20030007679A1 (en) 1997-03-14 2001-12-27 Image synthesizing apparatus and method, position detecting apparatus and method, and supply medium

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US09/178,092 Expired - Fee Related US6404913B1 (en) 1997-03-14 1998-10-23 Image synthesizing apparatus and method, position detecting apparatus and method, and supply medium

Country Status (6)

Country Link
US (2) US6404913B1 (en)
EP (1) EP0910046A1 (en)
KR (1) KR20000011022A (en)
CA (1) CA2254344A1 (en)
IL (1) IL127028A0 (en)
WO (1) WO1998041951A1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060034482A1 (en) * 2004-04-28 2006-02-16 Laurent Blonde Apparatus and method for displaying images
US20080088526A1 (en) * 2006-10-17 2008-04-17 Tatiana Pavlovna Kadantseva Method And Apparatus For Rendering An Image Impinging Upon A Non-Planar Surface
KR100928659B1 (en) * 2003-05-30 2009-11-27 스파이더 네비게이션즈 엘.엘.씨. Partially coherent constellations for multiple-antenna systems
US20110280473A1 (en) * 2009-02-09 2011-11-17 Hisashi Shiba Rotation estimation device, rotation estimation method, and record medium

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE19832974A1 (en) * 1998-07-22 2000-01-27 Siemens Ag Arrangement for generating virtual industrial system model compares system component information with real system image data to identify components in image data
US6473536B1 (en) * 1998-09-18 2002-10-29 Sanyo Electric Co., Ltd. Image synthesis method, image synthesizer, and recording medium on which image synthesis program is recorded
WO2000060571A1 (en) * 1999-04-02 2000-10-12 Massachusetts Institute Of Technology Haptic interface system for collision detection and applications therefore
US7084869B2 (en) * 2000-03-31 2006-08-01 Massachusetts Institute Of Technology Methods and apparatus for detecting and correcting penetration between objects
US20070058846A1 (en) * 2005-05-10 2007-03-15 Satoru Ohishi Three-dimensional image processing apparatus, three-dimensional image processing method and control program used in three-dimensional image processing apparatus
US8068665B2 (en) * 2005-05-10 2011-11-29 Kabushiki Kaisha Toshiba 3D-image processing apparatus, 3D-image processing method, storage medium, and program
US20070216711A1 (en) * 2006-03-14 2007-09-20 Microsoft Corporation Microsoft Patent Group Abstracting transform representations in a graphics API
JP2008021092A (en) * 2006-07-12 2008-01-31 Fanuc Ltd Simulation apparatus of robot system
CN101872277A (en) 2009-04-22 2010-10-27 介面光电股份有限公司 Three-dimensional imaging touch control device
US20100283836A1 (en) * 2009-05-08 2010-11-11 Jtouch Corporation Stereo imaging touch device
TW201104494A (en) 2009-07-20 2011-02-01 J Touch Corp Stereoscopic image interactive system
JP5785753B2 (en) * 2011-03-25 2015-09-30 京セラ株式会社 Electronic device, control method, and control program
US10368059B2 (en) * 2015-10-02 2019-07-30 Atheer, Inc. Method and apparatus for individualized three dimensional display calibration

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0628449A (en) * 1992-07-08 1994-02-04 Matsushita Electric Ind Co Ltd Image synthesizing device
JPH08101924A (en) * 1994-09-30 1996-04-16 Hitachi Ltd Picture composition method
US5727093A (en) * 1995-02-07 1998-03-10 Canon Kabushiki Kaisha Image processing method and apparatus therefor
KR100414629B1 (en) * 1995-03-29 2004-05-03 산요덴키가부시키가이샤 3D display image generation method, image processing method using depth information, depth information generation method
US5825483A (en) * 1995-12-19 1998-10-20 Cognex Corporation Multiple field of view calibration plate having a reqular array of features for use in semiconductor manufacturing

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100928659B1 (en) * 2003-05-30 2009-11-27 스파이더 네비게이션즈 엘.엘.씨. Partially coherent constellations for multiple-antenna systems
US20060034482A1 (en) * 2004-04-28 2006-02-16 Laurent Blonde Apparatus and method for displaying images
US7593541B2 (en) * 2004-04-28 2009-09-22 Thomson Licensing Apparatus and method for displaying images
US20080088526A1 (en) * 2006-10-17 2008-04-17 Tatiana Pavlovna Kadantseva Method And Apparatus For Rendering An Image Impinging Upon A Non-Planar Surface
US7873233B2 (en) * 2006-10-17 2011-01-18 Seiko Epson Corporation Method and apparatus for rendering an image impinging upon a non-planar surface
US20110280473A1 (en) * 2009-02-09 2011-11-17 Hisashi Shiba Rotation estimation device, rotation estimation method, and record medium
EP2395318A4 (en) * 2009-02-09 2017-08-09 Nec Corporation Rotation estimation device, rotation estimation method, and storage medium

Also Published As

Publication number Publication date
IL127028A0 (en) 1999-09-22
US6404913B1 (en) 2002-06-11
KR20000011022A (en) 2000-02-25
WO1998041951A1 (en) 1998-09-24
CA2254344A1 (en) 1998-09-24
EP0910046A1 (en) 1999-04-21

Similar Documents

Publication Publication Date Title
US20030007679A1 (en) Image synthesizing apparatus and method, position detecting apparatus and method, and supply medium
US11875537B2 (en) Multi view camera registration
CA3103844C (en) Method for reconstructing three-dimensional space scene based on photographing
CN110568447B (en) Visual positioning method, device and computer readable medium
US6529626B1 (en) 3D model conversion apparatus and method
US6081273A (en) Method and system for building three-dimensional object models
US9883163B2 (en) Method and system for determining camera parameters from a long range gradient based on alignment differences in non-point image landmarks
US6765569B2 (en) Augmented-reality tool employing scene-feature autocalibration during camera motion
US5982378A (en) System and method for modeling a three dimensional object
JP6764533B2 (en) Calibration device, chart for calibration, chart pattern generator, and calibration method
US6816187B1 (en) Camera calibration apparatus and method, image processing apparatus and method, program providing medium, and camera
Goshtasby Correction of image deformation from lens distortion using bezier patches
CN110505463A (en) Based on the real-time automatic 3D modeling method taken pictures
US5790713A (en) Three-dimensional computer graphics image generator
JPH03505924A (en) Method and system for automatically determining the position and orientation of an object in three-dimensional space
CN101563709A (en) Calibrating a camera system
JPH07294215A (en) Method and apparatus for processing image
JP2000011172A (en) Method and device for generating virtual environment and recording medium recording virtual environment generation program
JP2005507109A (en) Tracking system expandable with automatic line calibration
Thomas et al. Dealing with noise in multiframe structure from motion
WO1997026758A1 (en) Method and apparatus for insertion of virtual objects into a video sequence
KR101189167B1 (en) The method for 3d object information extraction from single image without meta information
CN113048985A (en) Camera relative motion estimation method under known relative rotation angle condition
JPH10320586A (en) Device and method for composting image, device and method for detecting position, and providing medium
JP2000331151A (en) Image processing method, image distortion correction processor and storage medium recording program of the processor

Legal Events

Date Code Title Description
STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION