US 20090067728 A1 Abstract An image matching method includes setting first and second lattices to first and second images respectively, computing potential force to each second lattice point of the second lattice by a gradient of an image correlation potential energy based on a position of each first lattice point and pixel thereof and a position of the second lattice point and pixel thereof, computing elasticity force to the second lattice point from elasticity energy between the second and adjacent lattice points, computing frictional force occurring at the second lattice point, performing a numerical analysis of an equation of motion regarding the second lattice point and based on the potential force, elasticity force and frictional force to obtain a convergence state of the second lattice points, and adding a new lattice point between an adjacent lattice point pair of second lattice points according to a distance between the adjacent lattice point pair.
Claims(25) 1-5. (canceled)6. An image interpolation method for interpolating an interpolation image between a first reference image and a second reference image using the first reference image, the second reference image and a third reference image which are arranged in time series, the method comprising:
computing, using a gradient of an image correlation potential energy obtained by a position of each first lattice point of a plurality of first lattice points of a first lattice on a first reference image and pixel information of the each first lattice point and a position of each second lattice point of a plurality of second lattice points of a second lattice on a second reference image, which correspond to each of the first lattice points, and pixel data of the each second lattice point, potential force applied to the each second lattice point; computing elasticity force applied to the each second lattice point from an elasticity energy between an adjacent lattice point pair of the second lattice points; computing frictional force occurring at the each second lattice point; performing a numerical analysis of an equation of motion regarding the second lattice points based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the second lattice points; detecting an occlusion region based on distribution of the positions which the second lattice points exist in the convergence state of the second lattice points; detecting mapping from the first reference image to the third reference image by an image matching method; and producing an interpolation image using corresponding relation of pixels of the first reference image and the second reference image which is determined by the second lattice points, the corresponding relation of pixels of the second reference image and the third reference image and information regarding the occlusion region. 7. The image interpolation method according to mapping fourth lattice points of a third lattice set on a third reference image to third lattice points of the second lattice on the second reference image respectively; computing potential force applied to one fourth lattice point of the fourth lattice points by a gradient of an image correlation potential energy obtained by a position of corresponding one third lattice point of the third lattice points and a position of the one fourth lattice point; computing elasticity force applied to the one fourth lattice point from an elasticity energy between the one fourth lattice point and the fourth lattice points adjacent to the one fourth lattice point, computing frictional force occurring at the one fourth lattice point, subjecting a equation of motion regarding the fourth lattice points and based on the potential force, the elasticity force and the frictional force to a numerical analysis to obtain a convergence state of the fourth lattice point, and obtaining corresponding relation of pixels of the second reference image and the third reference image from relation between the third lattice point and the fourth lattice point. 8. The image interpolation method according to dividing the second lattice in the convergence state of the second lattice point into a plurality of regions each having the second lattice points as vertexes, computing occlusion reliability of each of the divided regions, and detecting one region of the regions, which has the occlusion reliability greater than a threshold as the occlusion region, and the producing the interpolation image includes: obtaining a pixel value of each pixel in the occlusion region by weighting a first pixel value deriving from corresponding relation of pixels of the first reference image and the second reference image and a second pixel value derived from corresponding relation of pixels of the second reference image and the third reference image according to the occlusion reliability and combining them. 9. An image interpolation method for interpolating, using a first reference image, a second reference image, a third reference image and a fourth reference image which are arranged in time series, an interpolation image between the second reference image and the third reference image, the method comprising:
mapping, to each first lattice point of first lattice points of a first lattice on the second reference image, each second lattice point of second lattice points of a second lattice on the third reference image; computing potential force applied to the each second lattice point of the second lattice points by a gradient of an image correlation potential energy based on a position of the each first lattice point and image information of the each first lattice point and a position of the each second lattice point and image information of the each second lattice point; computing elasticity force applied to the each second lattice point from an elasticity energy between the each second lattice point and each of the second lattice points adjacent to the each second lattice point; computing frictional force occurring at the each second lattice point; performing a numerical analysis of an equation of motion regarding the second lattice points and based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the second lattice points; and detecting a first occlusion region based on distribution of the second lattice points in the convergence state of the second lattice points, mapping each fourth lattice point of fourth lattice points of the first lattice on the second reference image to each third lattice point of third lattice points of the second lattice on the third reference image, computing potential force applied to the each fourth lattice point by a gradient of an image correlation potential energy based on a position of the each third lattice point and image information of the each third lattice point and a position of the each fourth lattice point and image information of the each fourth lattice point, computing elasticity force applied to the each fourth lattice point from an elasticity energy from the each fourth lattice point to the fourth lattice points adjacent to the one fourth lattice point, computing frictional force occurring at the fourth lattice points, performing a numerical analysis of an equation of motion regarding the fourth lattice points and based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the fourth lattice points; and detecting a second occlusion region based on distribution of the fourth lattice points in the convergence state of the fourth lattice points; performing first image matching for obtaining corresponding relation of pixels of the first reference image and the second reference image, performing a second image matching for obtaining corresponding relation of pixels of the third reference image and the fourth reference image; and generating an interpolation image using corresponding relation of pixels of the second reference image and the reference image, which is settled by the second lattice points, corresponding relation of pixels of the third reference image and the second reference image, which is settled by the fourth lattice points, corresponding relation of pixels of the second reference image and the first reference image, corresponding relation of pixels of the third reference image and the fourth reference image, information of the first occlusion region, and information of the second occlusion region. 10. The image interpolation method according to the detecting the first occlusion region includes dividing the second lattice of the second lattice points in the convergence state into a plurality of first evaluation regions each having the second lattice points as vertexes, computing occlusion reliability of each of the first evaluation regions; detecting one of the first evaluation regions that has occlusion reliability greater than a threshold as a first occlusion region; the detecting the second occlusion region includes dividing the fourth lattice in the convergence state of the fourth lattice points into a plurality of second evaluation regions each having the fourth lattice points as vertexes, computing occlusion reliability of each of the second evaluation regions, detecting one of the second evaluation regions that has occlusion reliability greater than a threshold as a second occlusion region; and the generating the interpolation image includes: acquiring a pixel value of each pixel in the first occlusion region by weighting a first pixel and a second pixel according to the occlusion reliability and combining them, the first pixel value being derived from corresponding relation of pixels of the second reference image and the third reference image and the second pixel value being derived from corresponding relation of pixels of the second reference image and the first reference image; and acquiring a pixel value of each pixel in the second occlusion region by weighting a third pixel and a fourth pixel according to the occlusion reliability and combining them, the third pixel value being derived from corresponding relation of pixels of the second reference image and the third reference image and the fourth pixel value being derived from corresponding relation of pixels of the third reference image and the fourth reference image. 11. The image interpolation method according to 12. The image interpolation method according to 13. The image interpolation method according to 14. The image interpolation method according to 15. The image interpolation method according to computing surface reliability of the second reference image based on distribution of the second lattice points in the convergence state; computing surface reliability of the third reference image based on distribution of the fourth lattice points in the convergence state; generating a first intermediate interpolation image based on distribution of the second lattice points in the convergence state; generating a second intermediate interpolation image based on corresponding relation of pixels of the first reference image and the second reference image and the first occlusion region; generating a third intermediate interpolation image based on distribution of the fourth lattice points in the convergence state; generating a fourth intermediate interpolation image based on corresponding relation of pixels of the third reference image and the fourth reference image and the second occlusion region, and wherein the generating the interpolation image includes generating the interpolation image using the surface reliability of the third reference image, the surface reliability of the second reference image, the first intermediate interpolation image, the second intermediate interpolation image, the third intermediate interpolation image and the fourth intermediate interpolation image. 16. The image interpolation method according to 17. The image interpolation method according to 18. The image interpolation method according to 19. The image interpolation method according to 20. The image interpolation method according to 21. The image interpolation method according to 22. The image interpolation method according to 23. The image interpolation method according to the first image matching includes performing image matching using the first reference image as an original image and the second reference image as a target image; the second image matching includes performing image matching using the third reference image as a target image and the fourth reference image as a target image; and each of the first image matching and the second image matching includes: computing potential gradient force due to the gradient of the image correlation potential energy based on pixels of the original image in an initial position of one of lattice points set to the original image and pixels of the target image in a current position of the one of the lattice points; computing elasticity force due to an elasticity energy between one of the lattice points and the lattice points adjacent to the one of the lattice points; computing frictional force to act on the one of the lattice points, solving a equation of motion numerically by a discrete variable method to obtain a convergence state of each of the lattice points, the equation of motion regarding the lattice points and including the potential gradient force, the elasticity force and the frictional force; and performing matching for obtaining corresponding relation between the original image and the target image using the initial position and the position in the convergence state. 24. The image interpolation method according to 25. The image interpolation method according to 26. The image interpolation method according to 27. The image interpolation method according to the computing the surface reliability of the second reference image and the computing surface reliability of the third reference image each include obtaining a back side reliability that is a complement set of surface reliability and detecting, as a back side region, a region having the back side reliability greater than a given value; performing matching using a first back side region detected by the computing the surface reliability of the second reference image and a second back side region detected by the computing the surface reliability of the third reference image; and detecting, as a surface region, a region on which the first back side region and the second back side region correctly correspond to each other. 28. An image interpolation method for interpolating, using a first reference image, a second reference image and a third reference image and a fourth reference image, which are arranged in time series, an interpolation image between the second reference image and the third reference image, the method comprising:
mapping first lattice points of a first lattice on the second reference image onto second lattice points of a second lattice on the third reference image, respectively; computing potential force applied to each second lattice point of the second lattice points by a gradient of an image correlation potential energy based on a position of each first lattice point of the first lattice points that corresponds to the each second lattice point and image information the each first lattice point and a position of the each second lattice point and image information of the each second lattice point; computing elasticity force applied to the each second lattice point from an elasticity energy from the each second lattice point to the second lattice points adjacent to the each second lattice point; computing frictional force occurring at the each second lattice point, performing a numerical analysis of an equation of motion regarding the second lattice points and based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the second lattice points; detecting a first occlusion region based on distribution of the second lattice points in the convergence state thereof; mapping, third lattice points of the second lattice on the third reference image, onto fourth lattice points of the first lattice on the second reference image respectively, computing potential force applied to each fourth lattice point of the fourth lattice points by a gradient of an image correlation potential energy based on a position of each third lattice point of the third lattice points and image information thereof and a position of the each fourth lattice point and image information of the each fourth lattice point; computing elasticity force applied to the reference fourth lattice point from an elasticity energy from the each fourth lattice point to the fourth lattice points adjacent to the each fourth lattice point; computing frictional force occurring at the each fourth lattice point; performing a numerical analysis of an equation of motion regarding the fourth lattice points and based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the fourth lattice points; and detecting a second occlusion region based on distribution of the fourth lattice points in the convergence state thereof; mapping, fifth lattice points of the second lattice on the third reference image, onto sixth lattice points of the first lattice on the second reference image respectively, computing potential force applied to each sixth lattice point of the sixth lattice points by a gradient of an image correlation potential energy based on a position of each fifth lattice point of the fifth lattice points and a position of the each sixth lattice point; computing elasticity force applied to the each sixth lattice point by an elasticity energy from the each sixth lattice point to the sixth lattice points adjacent to the each sixth lattice point; computing frictional force occurring at the each sixth lattice point; performing a numerical analysis of an equation of motion regarding the sixth lattice points and based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the sixth lattice points; and mapping, seventh lattice points of the second lattice on the third reference image, onto eighth points of the first lattice on the second reference image respectively, computing potential force applied to each eighth lattice point of the eight lattice points by a gradient of an image correlation potential energy based on a position of each seventh lattice point of the seventh lattice points and a position of the each eighth lattice point; computing elasticity force applied to the each eighth lattice point by an elasticity energy from the each eighth lattice point to the eighth lattice points adjacent to the each eighth lattice point; computing frictional force occurring at the each eighth lattice point; performing a numerical analysis of an equation of motion regarding the eighth lattice points and based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the eighth lattice points; and generating an interpolation image using corresponding relation settled by the each second lattice point, corresponding relation settled by the each fourth lattice point, corresponding relation settled by the each sixth lattice point, corresponding relation settled by the each eighth lattice point, information of the first occlusion region, and information of the each second occlusion region. 29-35. (canceled) Description This is a division of application Ser. No. 11/355,144, filed Feb. 16, 2006, which is incorporated herein by reference in its entirety. This application is based upon and claims the benefit of priority from prior Japanese Patent Applications No. 2005-039623, filed Feb. 16, 2005; and No. 2005-336291, filed Nov. 21, 2005, the entire contents of both of which are incorporated herein by reference. 1. Field of the Invention The present invention relates to an image matching method for matching images by detecting corresponding points from two images, and an image interpolation method for interpolating the image using the same. 2. Description of the Related Art A technique of image matching to obtain association between one image and the other image is a basic technique in many technical fields such as motion detection, stereo matching, image morphing, image recognition, and video encoding. According to Murat Tekalp, “Digital Video Processing”, Prentice Hall, 1995, a technique of image matching can be sorted to four methods. In other words, there are an optical flow method, a block data base method, a gradient method, and a Bayesian method. The optical flow method is a method for deriving an optical flow type equation defining that “a change of luminance is constant” and obtaining a flow using the optical flow equation as condition of constraint. The block data base method is a method for obtaining a motion by block-by-block template matching. The gradient method is a method for performing matching in a direction in which luminance gradient of an image decreases. The Bayesian method is a method for obtaining plausible matching stochastically. Japanese patent No. 2927350 discloses an image matching method using a multiple resolution filter. This method is an image matching method for generating a pyramid of a plurality of multi-resolution images by means of a plurality of multiple resolution filters, and permitting an image matching from a high speed motion to a low speed motion with a high robustness by subjecting the image pyramid to a matching process sequentially from the top thereof. A method for obtaining a smooth mapping relation is mentioned in Japanese patent No. 2927350. However, there is a problem that constraint condition to make the mapping smooth acts on a discontinuous part of motion, resulting in that the discontinuous mapping cannot be expressed. This is a trade off to occur always in a conventional image matching method. There is a problem that if the mapping is smoothed, it comes to be difficult to express discontinuity, and if the discontinuity is expressed, the smoothness becomes impaired. It is an object of the present invention to provide an image matching method that even if the mapping is smoothed, discontinuity is not impaired, and even if discontinuity is expressed, smoothness is obtained, and an image interpolation method using the same. An aspect of the present invention provides an image matching method for detecting a mapping from a first image to a second image, comprising: setting a first lattice having a plurality of first lattice points to a first image; setting a second lattice having a plurality of second lattice points, each second point of the second lattice points of corresponding to each first lattice point of the first lattice points, to a second image; computing potential force applied to the each second lattice point by a gradient of an image correlation potential energy based on a position of the each first lattice point, pixel data of the each first lattice points, a position of the each second lattice point and pixel data of the each second lattice point; computing elasticity force applied to the each second lattice point from an elasticity energy between the each second lattice point and adjacent lattice points on the second lattice which are adjacent to the each second lattice point; computing frictional force occurring at the each second lattice point; performing a numerical analysis of an equation of motion regarding the second lattice points and based on the potential force, the elasticity force and the frictional force to obtain a convergence state of the second lattice points; and adding a new lattice point between an adjacent lattice point pair of the second lattice points according to a distance between the adjacent lattice point pair while the numerical analysis of the equation of motion is performed. There will be described the present embodiment according to a block diagram of According to Explaining the above referring to A lattice point addition will be explained. In The image matching and lattice point addition are explained in detail hereinafter. In a fundamental image matching, the following model is thought as a source image. A continuous image model is expressed as shown in equation 1.
This is a continuous image model of a real number basis. Because a digital image is regarded as a source image, the following sampling image model obtained by sampling the above model is used. This sampling image model is expressed by the following equation 2.
where V is a sampling matrix, coupling the lattice space nεΛ However, in the case of this formulation, the image value of the same object is assumed not to vary according to a time. Because the motion vector d is a real number, it should be noted that the right-hand side uses a designator of a continuous image model (equation 1). Because matching between two images, i.e., the source image and reference image is conceived, the following equivalent equation 4 is established. k The motion vector is simplified to be d:R The point decided by the mapping g is defined as y=g(x). yεR As described above, because y=g(x)=g(Vn), this is redefined to yn=g(x) to make one-to-one correspondence understandable. The equation 5 of the image matching problem results in a problem searching for ynopt satisfying the following equation 6. Dynamics is introduced to the point yn to solve the equation 6 of the image matching problem. In other words, the image matching problem results in a problem solving a dynamical system concerning the point yn. The point yn moves in a state satisfying the equation 6 as considering a relation with surrounding points to converge in an equilibrium state. It is assumed that the image matching is completed by the equilibrium state. This state is expressed as shown in A new time axis τεR is introduced to the point y, and the function yn (T) is defined. Assuming that an initial value is same as a rectangular lattice x as the following equation 7. When the dynamics converge
is defined. Because a new time axis is introduced, the derivation concerning a time can be defined by the following equation 8.
The ordinal dynamics is described by the following ordinary differential equation 9. Fε R A force applied to the point yn(τ) is conceived. At first a force due to a potential energy becoming a force making the dynamics drive is considered. This is a force for moving the point yn(τ) to a state satisfying the equation 6. The equation 5 is transformed to the following equation 10. It is difficult due to noise components included in the image to search for the point which satisfies the equation 10 closely. Consequently, the energy function as expressed by the following equation 11 is conceived. The point which this energy function Eu becomes smallest is searched for. If the principle of steepest descent method is used, the point can arrives at the local minimum by coming down in a direction of the steepest descent of the energy function Eu around the point yn(T). Accordingly, the gradient in the direction of this steepest descent is defined as a force to the point yn(τ). Since the energy function Eu is conceivable as correlation between images, this force is assumed to be a force Fu due to an image correlation potential energy. Although various methods of computing a gradient in a direction of the steepest descent are conceivable, the following method is adopted in the present embodiment. A differential value between the source image and the reference image is shown in
If a vector in the steepest descent direction is obtained by local optimization, normalized, and multiplied by a magnitude of gradient, the following equation 15 is provided. This equation shows a force due to an image correlation potential energy (square error energy).
It is possible to use a force due to the image correlation potential energy (absolute value difference error energy) of the following equation 17 wherein the energy function is defined by the following equation 16 in view of easy handling in mounting.
Subsequently, a force describing a relation of the point with the peripheral points is conceived. The image to be subjected to matching is assumed to be an image obtained by reflecting a three-dimensional space to a two-dimensional plane. When an object on the three-dimensional space is assumed to be a rigid body, the surface of the rigid body is observed as an object on the two-dimensional image. If the object on three-dimensional space is observed on the source image and reference image, in this time, the phase of the object observed on each image is maintained at a high probability. As shown in
If a spring constant (elasticity constant) is supposed to be k, the resilience of the spring applied to the point yn(τ) is a spring force expressed by an equation 19. The elasticity constant is a balancer between an image correlation energy and an elasticity energy. If the elastic constant has a large value, the lattice space is hard to be deformed, and thus a result is stable. However, the adequateness to the image becomes bad. If the elastic constant has a small value, the lattice space is easy to be deformed, and thus adequateness to the image is improved. However, consequence becomes too flexible. Consequently, this parameter is given from experience for the present. Because the behavior is not sensitive to the value of this parameter, a certain constant value is given fixedly basically.
When four points surrounding the lattice point are connected, a four-point connection spring model as expressed by the following equation 20 is obtained.
At last, a force making a stored energy dissipate is conceived. Energy is saved if a force applied to the point yn (τ) is only Fu, Fk, so that a system becomes an oscillating stationary state. Consequently, a force making the saved energy dissipate is introduced. This force can use a frictional force. When a speed can be near to a constant value, the frictional force can be expressed by the following equation 21. If the above forces are collected, the equation of motion is expressed by the following equation 22. Equation of Motion Because the force Fu due to the image correlation potential energy is not solved for in analysis, the ordinary differential equation 22 is not solved for in analysis. Accordingly, it is difficult to take the limit inτ→∞ of a system. Consequently, a convergence state of the system is estimated by computing an interval of t=(0,T) by a numerical analysis, in view of a sufficiently longer time T required for convergence of the system. The ordinary differential equation provides a solution uniquely by a numerical analysis if an initial value is determined. Generally, this is called an initial value problem of the ordinary differential equation. There are many numerical solutions for solving this problem, for example, a Euler method, a Runge Kutta method, a Bulirsch-Soer method, a predictor-corrector method, an implicit Runge Kutta method. The Runge Kutta method is most famous and has a high use frequent. However, the equation 22 is hard to fit a complicated numerical solution because it has a dimension for an image size. Consequently, the Euler method that implementation is the simplest is adopted in this embodiment. Because the Euler method is a numerical solution for the ordinary differential equation of first-order, at first the formula 22 is converted into a one-time ordinary differential equation. Then, the variable is transformed as expressed by the following equation 23.
This transformation of variable is substituted for the equation 22 to make the following equation 24. Converted equation of motion
where ψ The scheme of the Euler method for the ordinary differential equation 25 is expressed by the following equation 26. This makes a solution progress from t(n) to t
The above image matching is collected as algorithm as follows. Image Matching Algorithm 1. Initial value is set to
2. A lattice point addition step (described below). 3. The value of time T
in the time τ 4. T 5. If the relation is T Accordingly, the solution y The lattice point addition step is explained hereinafter. This is the above process If there are adjacent deformation lattice points y
to open both points greatly is to generate a large spring energy therebetween and occurs a large force. If y
belong to different objects respectively, nothing should exist between y
originally. This conflicts with that large spring energy is accumulated between y
This is shown in
so that it is not correct to connect with a spring between y
It is preferable for solving this contradiction to add a new lattice point between the lattice points y
as shown in A heterogeneous lattice point space shown in 1. When N is assumed to be a neighbor lattice space, l 2. Lattice points
are added. The processing returns to the image matching step after having added the lattice point. When adding the lattice point as described above, it is to introduce the heterogeneous lattice point space, whereby computation becomes complicated. Consequently, the lattice point addition model was analyzed. As a result, it is found that the lattice points connected to each other by the spring I of a spring constant k are subjected to a force of kI in the normal spring model as shown in The above explanation describes balance between the forces of the springs in the equilibrium state thereof and does not describe on the dynamic behavior of the spring. In other words, it simply describes that the spring forces are equivalent in static. It is equivalent to making a spring constant 1/(n+1) times to add n lattice points to the spring. It is the lattice point addition algorithm to add one lattice point if the length of the spring doubles, two lattice points if it is three times, three lattice points if it is four times, . . . . This can be simplified by setting the spring constant to ˝ if the length of the spring doubles, setting the spring constant to ⅓ if it is three times, setting the spring constant to Ľ if it is four times, . . . . A spring model of such a variable type can be expressed as follows. A spring model of a simplified model lattice point addition
where The following spring model can be used elsewhere. A linear function spring model varying in linear.
A logistic function spring model
where α>0, ⊖>0 are constants. In other words, the simplified lattice point addition algorithm uses a spring model of the equation 28. More specifically, the spring model of the equation 28 may be employed in the image matching algorithm. An effect was confirmed by a test pattern which a round object moves in parallel. As a result, when the lattice point addition is not done, a result shown in As described above, the present embodiment can provide a highly precise image matching method that can express discontinuity of movement by an algorithm adding a lattice point. (Single Direction Occlusion Model) The present embodiment is explained according to the block diagram of When an interpolation frame is generated, there is an occlusion problem making it difficult to generate an interpolation frame of high quality. The present embodiment provides a method for generating an interpolation frame of high quality by interpolating an occlusion region adequately. Schematically, this method performs an ordinal image matching (surface matching) in a time interval t An image signal of a motion picture is assumed to be divided into a plurality of frames, and subjected to processing. In this time, each frame is stored in a frame memory An occlusion region must be detected adequately for the occlusion region to be interpolated adequately. The ground that can be detected by the present embodiment adequately are explained referring to The lattice points on the rear part of the round object, that is, lattice points xn, xn+(1,0)T and yn, yn+(1,0)T are noted. Since the round object moves, if it is subjected to matching, an interval between the lattice points yn, yn+(1,0)T should expand greatly. In other words, a spring between the lattice points yn, yn+(1,0)T is expanded. When the uncovered occlusion region appears by movement of the object as described above, it is found that the spring always expands. In other words, the part that the spring expands greatly is an occlusion region. The superscript “T” expresses transposition. In the above example, the distance between the lattice points is used as an evaluation value, and the occlusion is determined when the distance is greater than a reference value. The occlusion can be detected based on distribution of lattice points. Density of distribution of the lattice points decreases on the occlusion region because the spring expands thereon. Therefore, the density may be used as an evaluation value for determining occlusion. In this case, the density lower than the reference is detected as occlusion. For example, the reference frame t+2lΔt is divided into plural equal rectangular regions, and the lattice points in each region are counted. The number of lattice points is a quantity corresponding to the density of distribution of the lattice points. Otherwise, the occlusion can be detected using an area of a region surrounded by the lattice points instead of the density. The area of the region surrounded by the lattice points increases on the occlusion region because the spring expands thereon and thus the distance between the lattice points increases. Therefore, the area can be used as an evaluation value for determining occlusion. In this case, the region having an area greater than the reference is detected as occlusion. For example, an area of a triangular region having the lattice points as apices and including no other lattice points, or an area of a rectangular region having the lattice points as apices and including no other lattice points is used. It can be understood from the above that the occlusion region is detected and the region corresponding to the occlusion region is subjected to matching. Thereafter, these results may be combined for generating an interpolation frame. The present embodiment will be described referring to the block circuit of In an image matching step S In an image matching step S A conventional image matching method can be used in the occlusion matching module. In other words, a conventional motion estimation method mentioned in Japanese patent No. 2927350, for example, a block matching algorithm, an optical flow estimating method, a gradient method, a bays method can be used. The block matching algorithm is described as an example. The block matching algorithm is a method for assigning the same motion vector to the lattice points in the block with assumption that movements in a block are the same. The whole screen is divided into plural (ordinarily, rectangular) blocks each including lattice points. In this example, a rectangular block is adopted. A point on the upper left of the block is assumed to be a base point XεΛ
where ΩεΛ Therefore, the mapping is obtained by the following equation 33. The occlusion reliability computation step S In the present embodiment, in view of a problem how an interpolation which makes the occlusion region correspond to the pixels of the motion image in units of pixel well is done, there is used, for example, a triangular area surrounded by the lattice points yn, yn+(0,−1),s□Ayn+(−1,−1),s as shown in
Similarly, the area of the lower triangular region is computed by the following equation 35.
Because the area of each of the triangular region xn, xn+(0,−1)T, xn+(−1,−1)T and the triangular region xn, xn+(−1,0)T, xn+(−1,−1)T is ˝, if the ratio between the triangular regions is computed, the ratio between the areas of the triangular regions is expressed by the following equations 36 and 37. The superscript “T” expresses a transpose. A probability distribution is acquired from these triangular ratios. If the length of the spring exceeds a certain constant value, it is assumed to be occlusion. In this case, if the triangular ratio exceeds a certain constant value, it is determined to be occlusion. A sigmoid function according to the following equation 38 as shown in
where α indicates a constant representing inclination and β indicates coordinates of the center of the inclination. The triangle in the lattice point n is assumed to be R A motion compensation step is done by a motion compensation unit The interpolation frame generation step S In the Digital Image Warping, a motion vector is obtained for each point of a rectangle. The rectangle is divided into two triangles, and an affine transformation is computed on the basis of three vertexes for each triangle (as described in J. Nieweglowski, T. George Campbell, P. Haavisto, □gA Novel Video Coding Sheme Based on Temporal Prediction Using Digital Image Warping□h, IEEE Transactions on, Volume: 39 Issue: 3, Aug. 1993, the entire contents of which are incorporated herein by reference). A triangular image to be interpolated by the affine transformation is obtained (as described in J. Nieweglowski, P. Haavisto, □gMotion Compensated Video Sequence Interpolation Using Digital Image Warping□h, IEEE Acoustics, Speech, and Signal Processing, vol. 5, p 205-208, 1994, the entire contents of which are incorporated herein by reference). As shown in Motion Compensated Video Sequence Interpolation Using Digital Image Warping, the affine transformation from a triangle having vertexes V={(u
It can be computed using the affine transformation that a point (u,v) An interpolation frame is obtained by the affine transformation. A time position of the reference frame t is set to 0, and a time position of the reference frame t+lΔtk is set to 1. A time position of the interpolation frame is assumed to beΔt A triangle having vertexes {g(x Affine transformation from the triangle W to the triangle X and affine transformation from the triangle W to the triangle are obtained by the equation 40, and a pixel value corresponding to the point w is obtained by the former and latter affine transformations. In other words, the point in the triangle which corresponds to the point w can be computed by the following equation 42: Similarly, the point in the triangle which corresponds to the point w can be computed by the following equation 43. If a weight average of both frames is obtained according to the time position, a pixel value of the interpolation frame can be computed by the interpolation frame obtained by the following equation 44 and the mapping g(;t;lΔt). An interpolation frame based on the mapping g(;t;lΔt) is generated by doing this computation for all triangles in a screen. An occlusion reliability map on the interpolation frame is generated by the occlusion reliability map as follows. The occlusion reliability map on the interpolation frame according to the mapping g(;t;lΔt) is expressed by the following equation 45. Similarly, an image correlation reliability map due to the mapping g(;t;lΔt) is generated based on the equation 46 in the image correlation reliability computation step described below. Because the occlusion mapping g(;t+lΔt;lΔt) produced by occlusion matching is due to matching from the reference frame t+lΔt to the reference frame t+2lΔt, the occlusion mapping is subjected to linear transformation based on the following equation 47 to generate a matching image between the reference frame t+lΔt and the reference frame t. The frame is assumed to be −lΔt because a direction of the image is inversed. When the occlusion mapping g(;t+lΔt;lΔt) is subjected to linear transformation according to Δti, it becomes an image based on the following equation 48. Correspondence between the points is shown in Similarly, a point in the triangle Yo corresponding to the point w is computed according to the following equation 50. If a weight average of both frames is computed according to a time position, a pixel value of an interpolation frame due to the occlusion mapping g(;t+lΔt;lΔt) can be computed by the following equation 51. If this processing is executed for all triangles in a screen, the interpolation frame due to the occlusion mapping g(;t+lΔt;lΔt) is generated. An image correlation reliability map due to the occlusion mapping g(;t+lΔt;lΔt) is generated based on the following equation 52 in the image correlation reliability computation step S Because the interpolation frame due to the mapping g(;t;lΔt) and the interpolation frame due to the occlusion mapping g(;t+lΔt;lΔt) can be generated by the above processing, these are combined by a reliability map. The probability of reliability given by the image correlation and the probability of reliability given by the occlusion are independent from each other. Therefore, the probability that the occlusion matching is adopted is expressed by the following equation 52 according to the multiplication theorem of probability. The probability that the ordinary matching is adopted is obtained by the following equation 54. The transparency to be used for combination is computed from these probabilities by the following equation 55.
The interpolation frame due to the mapping and the interpolation frame due to the occlusion mapping are combined using this transparency and based on the following equation 56. The interpolation frame Si, p(w,t+lΔt The image correlation reliability computation step S
This is a model that the reliability increases with a decrease of the differential value of the image. However, correlation between the images and a differential value between the pixels have no linear correlation generally. In other words, if the differential value between the pixels is small, the correlation between the images may be high. However, if the differential value between the pixels exceeds a certain constant value, the correlation between the images descends irrespective of the differential value between the pixels. The sigmoid function as shown in
where α>0, β>0 are constants. As described above, the present embodiment permits generation of the interpolation frame of high quality, which corresponds to a uncovered occlusion region. (Bidirectional Occlusion Model) The present embodiment is explained according to the block diagram of When the interpolation frame is generated, it is difficult owing to occlusion problem to generate an interpolation frame of high quality. The second embodiment provides a method dealing with uncovered occlusion. The present embodiment can deal with covered occlusion as well as uncovered occlusion, by matching in bi-direction of forward and backward with respect to a time axis. The image signal of the motion image is divided into frames, and subjected to processing. In this time, each frame is stored in a frame memory As explained in the second embodiment, a expansion of a spring corresponds to a uncovered occlusion region as shown in In an image matching step S In an occlusion matching step S An occlusion reliability computation step S The motion compensation step comprises an interpolation frame generation step S The interpolation frame obtained by the forward matching is similar to the second embodiment. In other words, the interpolation frame and reliability map are obtained by the equations 26, 27, 28, 29, 30. The interpolation frames are changed in subscript for identification as follows: A time position of the interpolation frame to be interpolated is assumed to be Δt Assuming that a triangle having vertexes {gi(x A pixel value of a point wεR Similarly, a point in the triangle corresponding to the point w can be computed according to the following equation 61. When a weight average of both frames is computed according to the time position, a pixel value of an interpolation frame due to the backward matching mapping g(;t+lΔt; −lΔt) can be computed by the following equation 62. The interpolation frame due to the backward matching mapping g(;t+lΔt; −lΔt) is generated by doing this processing on all triangles in a screen. A backward occlusion reliability map on an interpolation frame is generated by a backward occlusion reliability map P(y|g(;t+lΔt;−lΔt) as described hereinafter. In other words, a backward occlusion reliability map on an interpolation frame due to the backward matching mapping g(;t+lΔt; −lΔt) is obtained by the following equation. Similarly, a backward image correlation reliability map is generated by an image correlation reliability computation step. In other words, the backward image correlation reliability map due to the backward matching mapping g(;t+lΔt;−lΔt) is obtained by the following equation 64. Because the backward occlusion mapping g(;t; −lΔt) due to the backward occlusion matching corresponds to matching from the reference frame t to the reference frame t−lΔt, linear transformation defined by go(x,t;lΔt)=x−d(x,t;−lΔt) is subjected to the backward occlusion mapping to generate a mapping between the reference frame t+lΔt and the reference frame t. Since the direction of the mapping is reversed, it is written into +lΔt. When the backward occlusion mapping g (;t;−lΔt) is subjected to linear transform according to Δti, the backward occlusion mapping is converted into an image expressed by the equation 65. where a triangle having vertexes {gi,o (x A point in the triangle Xo which corresponds to the point w is computed by the following equation 66. Similarly, a point in the triangle Yo which corresponds to the point w is computed by the following equation 67. If a weight average of both frames is computed according to the time position, a pixel value of an interpolation frame due to the backward occlusion mapping g (;t;lΔt) is computed by the following equation 68. An interpolation frame due to the backward occlusion mapping g(;t;lΔt) is generated by being done on all triangles in a screen. A backward image correlation reliability map is generated by an image correlation reliability computation step. In other words, a backward image correlation reliability map due to the backward occlusion mapping g (;t;lΔt) is generated based on the following equation 69. Because an interpolation frame due to the forward matching mapping q(;t;lΔt) and an interpolation frame due to the forward occlusion mapping g(;t+lΔt;lΔt), and an interpolation frame due to the backward matching mapping g(;+lΔt; −lΔt) and an interpolation frame due to the backward occlusion mapping g(;t; −lΔt) can be generated, these interpolation frames are combined by a reliability map. probability of reliability given by image correlation and probability of the reliability given by occlusion are independent from each other. Therefore, the probability that the forward occlusion matching is adopted can be expressed by the following equation 70 according to multiplication theorem of probability. The probability that the forward matching is adopted can be expressed by the following equation 71: The probability that the backward occlusion matching is adopted can be expressed by the following equation 72. The probability that the backward matching is adopted can be expressed by the following equation 73: The transparency used for combination of the interpolation frames is computed from these probability based on the following equation 74:
Using this transparency the interpolation frames are combined based on the reliability according to the following equation 75. The interpolation frame Si, p (w,t+lΔti) can be generated by doing on the whole screen the above-mentioned processing. An image matching apparatus of the present embodiment will be explained referring to the block diagram of According to the constitution shown in The forward matching unit The backward matching unit In the configuration, the image signal of motion image is divided into a plurality of frames and then processed. In this time, each frame is stored in the frame memory An occlusion region must be detected adequately for the occlusion region to be interpolated adequately. The ground that can detect the occlusion region by the present embodiment adequately is explained referring to The uncovered occlusion appears on the rear of the round object between the times t and t+lΔt as explained referring to The stretch of the spring corresponds to the uncovered occlusion region as shown in With the above it is found that the occlusion region is detected and a region corresponding to the occlusion region can be subjected to the matching. It is found that the uncovered occlusion region can be detected by the expansion of the spring. This means what can compute surface reliability. If the surface reliability can be computed not only by the expansion of the spring but also by plural information, it is thought that the uncovered occlusion region can be detected at higher precision by the joint probability of these reliabilities. The following characteristics should be noted. 1. An occlusion occurs before and after movement of the object. The occlusion appearing before movement of the object is covered occlusion and the occlusion appearing thereafter is uncovered occlusion. 2. In dynamic matching, there are geometric characteristics such that the spring stretches on the rear of the object and shortens on the front thereof. According to the characteristics 1 and 2, the uncovered occlusion can be detected by expansion of the spring, and the covered occlusion can be detected by the contraction thereof. It is assumed that the matching is carried out with respect to the bidirection of the time axis to execute the forward matching and backward matching. In this time, one occlusion region is provided on the rear of the object by the forward matching, and the other occlusion region is provided on the front thereof by the backward matching. If these two occlusion regions are added, an ideal result as shown on the upper right of In contrast, the case that the occlusion disappeared due to contraction of the spring is detected at the same time is considered. An example of this case is explained referred to As shown in How to carry out matching is explained. A method for matching corresponding points from an source image to a reference image when the source image and reference image are input, that is, generating a relation between matching images is explained referring to a block circuit of A Continuous Image Model
X⊂R This is a continuous image model of a real number base. In the present embodiment, because a digital image is regarded as an source image, the following model obtained by sampling the above model is used. Sampling Image Model
where N⊂Λ
is used. The image matching is a problem to find which point of the reference image corresponds to a point on the source image In this time, the image matching problem is formulated as a problem to search for a motion vector d:X×T→X satisfying the following conditional expression. It should be noted that since a motion vector d is a real number, the right side of the equation uses notation of the continuous image model equation 77. Because the image matching between two images of the source image and reference image is thought about, the following equivalent problem is considered. where s s k The motion vector is simplified to be d:X→X. Since only one motion vector d may be decided with respect to the point x, the motion vector d is assumed to be a unique mapping. Since the image x+d(x) can be considered to be a mapping with respect to the point x, it is defined by the following equation 80. where it is a unique mapping of g:X→X. The above image matching problem results in a problem for searching for the mapping g satisfying the following equation 81. Image Matching Problem where a point determined by the mapping g is defined by the following equation 82. where yεX. Because x=Vn, the point x corresponds uniquely to a point n on the lattice space. Because the mapping g is a unique mapping, the point y corresponds uniquely to the point x, too. Therefore, the point y corresponds uniquely to the point n. This is expressed by Because y=g(x)=g(Vn) as described above, this is redefined by the following equation in order to make clearly understandable to correspond one-to-one to the point n. Then, the image matching problem (equation 3) result in a problem searching for yn satisfying the next expression. Image Matching Problem Measure for solving the problem is explained. Energy function is defined by the following equation to solve the equation 84 of the image matching problem. Energy Function The equation 84 of the image matching problem is formulated to the following optimization problem as the minimization problem of the energy function of the whole screen. Optimization Problem
where Eu is assumed to be an image energy, and Ek to be an elastic energy. This is used in Snakes [“M. Kass, A. Witkin and D. Terzopoulos, “Snakes: Active Contour Models”, International Journal of Computer Vision, 1, 4, pp. 321-331, 1988”], Active Net [“Sakaue, Yamamoto”, Model Active Net of a dynamic network and Application to Region extraction, Television studies magazine, 45, 10, pp. 1155-1163, 1991”] and CPF [Y. Shinagawa and T. L. Kunii, “Unconstrained Automatic Image Matching Using Multiresolutional Critical-Point Filters”, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 20, No. 9, 1998”]. Eu is explained as follows. The equation 84 is transformed into the following equation. It is usually difficult by noise components included in an image to search for a point to satisfy the equation 87 strictly. Accordingly, the following energy function is conceived.
A point that this energy function Eu becomes minimum is searched for. Ek is explained as follows. The image to be subjected to matching is assumed to be one obtained by projecting a three-dimensional space to a two-dimensional area. When an object on the three-dimensional space is assumed to be a rigid body, the surface of the rigid body is observed as an object on the two-dimensional image. If an object on the three-dimensional space is observed with the source image and reference image, the probability that the phase of the object observed on each image is kept is high. As shown in Therefore, the energy minimization equation 86 due to the equation 85 of energy function is to search for solution such that the equation 84 is satisfied as much as possible and the whole image becomes smooth. The optimality requirement of the following equation is found from the equation 86 in Snakes and Active Net. Optimality Requirement
A solution is obtained by solving this. This provides a condition of static balance to satisfy the energy function (equation 5). This method builds up a dynamic solution method by introducing kinetic energy T. A dynamic behavior of the point y is interested, so that the function yn(i) is defined by introducing a new time axis τεR to the point y. Because the new time axis is introduced, a differentiation concerning a time can be defined as follows.
The kinetic energy is defined as the following expression. Kinetic Energy
A Lagrangian is defined as follows. Lagrangian An equation of motion to satisfy a variation principle then is provided by the following equation 93. Euler Equation
When this is developed, the equation of motion of the following expression is provided. An Equation of Motion
However, the motion according to the equation of motion 94 is a conservative system, and a solution does not converge by an energy conservation law. Therefore, the friction energy of the following equation 95 is introduced to make the system a non-conservative system. Friction Energy
where μ>0 is a coefficient of dynamic friction. The Euler equation when the friction energy is introduced is expressed as follows. The Euler Equation Including the Friction Energy.
The equation of motion 97 is provided by solving this equation 96. The equation of motion including friction.
These static energy function and dynamic energy function are intuitively compared with each other as follows. In optimization according to the static energy function as shown in In contrast, in the dynamic optimization of the present embodiment a search is done in a dynamic motion. More specifically, the search is expressed as motion. Therefore, the first potential energy is converted into the kinetic energy as shown in It will be proved that the dynamic search, that is, a solution according to the equation of motion (equation 15) satisfies an optimality requirement condition for an optimization problem (equation 6). 1. The search is assumed to be completed. Then, the point y stops completely, that is, the following equation 98 is established. Then, the following equation is established by the equation of motion 97.
Thus, the following equation 100 is established.
Therefore, the optimality requirement equation 89 is satisfied. When the search is completed, the point is optimum. 2. The point y is assumed to be stationary. Then, the following equation 101 is established. Then, the following equation 102 is obtained by the equation of motion 97.
If the point y is an optimal point now, the following equation 103 is established by the optimality requirement equation 89, and the search is completed. If the point y is not an optimal point, the search is not completed. In other words, the search is not completed except for the optimal point. 3. The following equation is established by a property of friction energy.
In other words, the motion stops. Thus, the search is completed sometime. The motion according to the equation of motion (equation 15) satisfies a requirement condition of optimality for the optimization problem equation 86 by the conditions 1, 2 and 3 as mentioned above. The secondary optimality requirement is not satisfied partly. The reason is because the search may be finished on the top of potential hill. Therefore, if the condition that the search is not completed on the top of potential hill, that is, the following equation 105 is assumed, the secondary optimality requirement is proved, too.
This additional assumption can be achieved by requesting for the next condition for an algorithm. “if
in {dot over (y)}=0,˙=0, random perturbation is given.” As described above, the search according to the equation of motion 97 provides a solution for the optimization problem equation 86, and suggests to have a robust in comparison with a conventional static search. The structure of
will be conceived successively. Since this can be developed like the following equation 106, this equation will be discussed.
represents a force of a reverse direction with respect to a gradienta gradient of the point yn(T) of image energy. The following equation is given to make that it is a force clear.
Various methods are conceived to compute the gradient, but the following method is adopted. As shown in the equation 7, the gradient is obtained by local optimization directly. The image model Sc is a continuous image model, but in practice only the sampled image model Sp can be used. Thus, the local optimization is done based on the sampled image model, too. Because the sampling point which is at the nearest to yn(T) should be a local space center yc, it is obtained by the following equation 107.
When the adjacency space is defined by the following equation 108, a local search set Q can be defined by the following equation 109.
The gradient is subjected to the local optimization, and a vector in the gradient direction is obtained. The vector is normalized and multiplied by the size of the gradient. As a result, the following equation is provided. A Force Due to the Image Energy (Square Error Energy)
In view of handling easiness in mounting, the equation 112 of the force due to the image energy that the energy function is defined by the following equation 111 can be used. Force Due to the Image Energy (Absolute Value Difference Error Energy)
represents a force due to the gradient in a point y The following equation is given to make that it is a force clear. The elastic body is assumed to be configured by a manifold M. A point on the manifold M is assumed to be expressed by the following equation 113.
The manifold is assumed to have coordinate neighborhood. In this time, partial differentiation is defined by the equation 114.
The primary differential term of the elastic energy is expressed by the following equation 115.
This is energy to express the shortening of the elastic member. This is differentiated by y as shown by the following equation 116.
The manifold M is assumed to be realized by the spring model coupling lattice points, and the partial differentiation is approximated by the difference equation. If this is substituted for the equation 117 and arranged, 4-point connection spring model shown by the following equation 119 is provided. 4-Point Connection Spring Models
It is natural that an initial condition is expressed by the following equation 120, if y
When the above matters are summarized, the equation of motion is expressed by the following equation. Equation of Motion
Since the force Fu due to the image energy is not solved in analysis, the normal differential equation (equation 19) is not solved in analysis. Therefore, it is difficult to take the limit inτ→∞ of the system. For this reason, an enough time necessary for converging the system is considered, and the convergence state of the system is estimated by computing t=(0,T) interval by numeric analysis. If the initial value is determined, the normal differential equation provides a solution uniquely by a discrete variable method. Generally, it is said an initial value problem of the normal differential equation. There are many numerical solutions for solving this problem, for example, an Euler method, a Runge Kutta method, a Bulirsch-Soer method, a predictor-corrector method, a hidden Runge Kutta method as famous methods. The Runge Kutta method is most famous and has a high use frequent. However, the following equation 121 is hard to fit a complicated numerical solution since it has a dimension for the size of an image. Therefore, it is thought that the Euler method that realization is most simplified is applied. Since the Euler method is a numerical solution for a normal differential equation of one time, at first, the equation 19 is converted to the ordinary differential equation of one time. When transformation of variable is carried out by the following equation 122, a transformed equation of motion 123 is provided. Transformation of Variable
Transformed Equation of Motion
where ψ A scheme of the Euler method for the normal differential equation 124 is represented by the following equation 125. [Equation 122]] A Scheme of Euler Method This progresses a solution from t x(n) shows a step and h indicates a step size. When the scheme of the Euler method is applied to the equation 123, an updated equation of the following equation 126 is provided. Update Equation Due to Euler Method The update equation is solved repeatedly, so that a convergence state can be obtained. The interpolation frame generation step is explained. This step adopts a method similar to Digital Image Warping1 [“J. Nieweglowski, P. Haavisto, “Motion Compensated Video Sequence Interpolation Using Digital Image Warping”, IEEE Acoustics, Speech, and Signal Processing, vol. 5, p 205-208, 1994”, “J. Nieweglowski, T. George Campbell, P. Haavisto, “A Novel Video Coding Sheme Based on Temporal Prediction Using Digital Image Warping”, IEEE Transactions on, Volume: 39 Issue: 3, Aug. 1993”]. In the Digital Image Warping, a motion vector is obtained for each point of a rectangle, the rectangle is divided into two triangles, and affine transformation is computed on the basis of three vertexes for each triangle. A triangular image to be interpolated by the affine transformation is obtained. The affine transformation from a triangle having vertexes V={(u Affine Transformation
It can be computed using the affine transformation that the point (u,v) An interpolation frame is obtained by the affine transformation. The time position of the reference frame t is set to 0, the time position of the reference frame t+lΔt is set to 1, and the time position of the interpolation frame to be interpolated is assumed to be Δt The interpolation frame is generated as shown in Similarly, the point within the triangle Y corresponding to the point w can compute by the following equation 131. If a weight average of both frames is computed according to the time position, a pixel value of the interpolation frame can be computed by the following equation 132. Interpolation Frame Due to a Mapping g(•,t;lΔt) The right side of the equation 132 is written in a form of a continuous image model. Therefore, a pixel value is computed by being interpolated by a bi-linear method to compute the right side from the discrete image model. An interpolation frame based on the mapping g(•,t;lΔt) is generated by doing this processing for all triangles in a screen. The step of the surface reliability computation In practice, since the reliability in the region must be computed, the area of a triangle surrounded by the region expressed by, for example, the following equation as shown in The area of the original triangle is derived from the absolute value of outer product of a vector according to the following equation 133.
Similarly, the area after mapping is obtained by the following equation 134.
Therefore, the differential value of the area is assumed to be expressed by the following equation 135. Assuming that the probability of surface enhances as the differential value nears to 0, and the probability of surface lowers as the differential value increases. In this case, it is convenient that if the differential value is greater than a threshold, it is set to 0, and if the differential value is smaller than the threshold, it is set to 1. The step of the surface reliability computation The lattice point of the covered occlusion region (covered occlusion region) has a geometric characteristic to be shortened to 0 ideally. This is symmetrical with that the uncovered occlusion region (uncovered occlusion region) stretches. The probability of reliability given by the uncovered occlusion region is associated with the expansion of the spring, so that it will be natural that the probability of reliability given by the covered occlusion region is associated with the contraction of the spring. In other words, the expansion and contraction of the spring are dealt with equivalently till now. Hereinafter, the different confidence probabilities generate according to the expansion and contraction of the spring. This seems to be natural from the symmetry of a theory Using the difference of area used in the step of the surface reliability computation If the reliability of the mapping is positive, it is probability to seem to be reliable as the surface. This is modeled as probability of the image Sk Assuming that a noise is added to an observed image, the pixel values of the lattice point x and mapping g(x) are expressed by the following equation 136. where ε Displaced Pixel Difference (referred to as DPD hereinafter) is modeled as shown by the following equation 137.
If this DPD is smaller than the threshold, the mapping is probable. Accordingly, the probability is set to 1. If DPD is greater than the threshold, the probability is set to 1. A configuration is explained referring to Because the forward matching (
A subscript f is added to an output result in the forward matching, and a subscript b is added to the output result in the backward matching. Matching Step with Occlusion ( The matching g(Vn,t;lΔt),∀nεN is derived from t t+lΔt by the matching step (later described) (S11).
2. The occlusion matching g(Vn,t+lΔt;lΔt),∀nεN is derived from t+lΔt t+2lΔt by an occlusion matching step (the same as the matching step) (S12).
A surface reliability map is derived from g(Vn,t;lΔt),∀nεN by the step of the surface reliability computation (a) The area S (b) The surface probability inside the quadrilateral is obtained with respect to a threshold T
(c) A result is stored in the surface reliability map. (e) The above-mentioned processing is carried out for the whole screen**. 2. A mapping reliability map is derived from g(Vn,t;lΔt),∀nεN by the step of the surface reliability computation (a) The probability of the mapping is obtained with respect to a threshold T
(b) A result is stored in the mapping reliability map. (c) The above-mentioned processing is carried out for the whole screen ∀nεN. 3. A mapping reliability map is derived from g(Vn,t+lΔt;lΔt),∀nεN by a step of the surface reliability computation (a) The probability of the mapping is obtained with respect to the threshold T
(b) A result is stored to a mapping reliability map. (c) The above-mentioned processing is done for the whole screen ∀nεN. 4. A covered surface reliability map is derived from g(Vn,t;lΔt),∀nεN by the step of the surface reliability computation (S (a) The area S (b) The surface probability inside the quadrilateral is obtained with respect to a threshold T
(c) A result is stored in the covered surface reliability map. (d) The above-mentioned processing is done for the whole screen ∀nεN. 5. g 6. g 7. Surface reliability map Pr Covered surface reliability map Pr Mapping reliability map Pr Mapping reliability map Pr (a) The following triangles are obtained. (b) Affine transformation T(W,Y),T(W,X) is obtained. (c) [y [x It is stored in Pr It is stored in Pr (d) The above-mentioned processing is done for the whole screen ∀nεN. A combination step is explained. 8. combination is carries out to generate an interpolation frame s (a) The probabilities of the surface side and the occlusion side are obtained as probability of the product event (S (b) An alpha blend coefficient for composition is lost as follows.
(d) The above-mentioned processing is done for the whole screen ∀nεN. Each step is described concretely according to the block diagram shown in In a discrete variable method step, a position and a speed are updated according to an update equation 126 of the discrete variable method, and a computation is done from the initial value to a predetermined time. In this embodiment, a discrete variable method based on Euler method is used. Concretely, computation is performed according to a flow shown in In the image force computation step, a force Fu occurring by the potential energy due to the image correlation is computed. If a position and mapping n,ψ
Concretely, computation is carried out according to a flow chart shown in If a position and mapping n,ψ
Concretely, computation is carried out according to a flow shown in In the frictional force computation step, the force F When a speed ψ Concretely, computation is carried out according to a flow shown in In an interpolation frame generation step, an interpolation image of a 0≦Δt The present embodiment can provide a robust and high-quality interpolation frame and interpolate an appropriate image in an occlusion region as mentioned above. The present embodiment is explained referring to the block diagram of
where α indicates a gradient and β indicates a center. For example, α=30.0 and β=0.25 can be used. The following part is changed from the third embodiment. The Step of the Surface Reliability Computation (a) The area S (b) The surface probability of the quadrilateral inside is obtained by the next expression.
(c) A result is stored in the surface reliability map as follow: (d) The above processing is done about the whole screen ∀nεN. A mapping reliability map is obtained from g(Vn,t;lΔt),∀nεN by the step of surface reliability computation (a) The probability of mapping is obtained according to the following equation:
(b) A result is stored in the mapping reliability map as shown by the following equation. (c) The above processing is done about the whole screen ∀nεN. (d) A mapping reliability map is obtained from g(Vn,t+lΔt;lΔt),∀nεN by the step of surface reliability computation (e) The probability of mapping is obtained with respect to the threshold T
(f) The result is stored in the mapping reliability map as represented by the following equation. (g) The above processing is done about the whole screen ∀nεN. Step of Surface Reliability Computation (a) The area S (b) Surface probability of the quadrilateral inside is obtained with respect to the threshold T
(c) A result is stored in a covered surface reliability map. (e) The above-mentioned processing is done about the whole screen ∀nεN. The present embodiment can provide an interpolation frame of higher image quality because the more smooth probability map is provided. The present embodiment is explained referring to the block diagram of In the present embodiment, the more robust surface reliability is obtained by describing the probability of surface reliability as a probability distribution according to a Gaussian distribution. About the Step of Surface Reliability Computation The part that a spring expands is assumed to be a region having a high reliability as occlusion, in other words, the part which the spring does not change has a high reliability as the surface. Actually, an area of the triangle surrounded by, for example, three points y
Similarly, the area of the triangle after mapping is obtained as follows:
Therefore, a differential value between the areas is computed by the following equation. Assuming that the surface probability increases as the differential value draws closer to 0, and the surface probability decreases as the differential value increases. If the differential value is 0 in average and accords to the Gaussian distribution of dispersion σ
Because this is a probability density function, a distribution function is expressed by the following equation: Therefore, the probability that the differential value R(n) is between a and b is given by the following equation: If the reliability when the differential value R(n)=0 is provided will be 95%, the following equation may be concluded. Accordingly, the surface probability when a certain differential value is given is defined by the following equation 142: Surface Probability Because this is the reliability of the surface inside the triangle, this is assigned to the inside of the triangle and stored in a memory. Because when the triangle y Surface reliability computation This is modeled as plausibility of the image S Assuming that an observed image contains a noise, pixel values of the lattice point x and mapping g(x) are expressed by the following equations: where ε Displaced Pixel Difference (referred to as DPD hereinafter) is modeled as follows.
Assuming that the observed noise is 0 in average and accords to the Gaussian distribution of dispersion σ
Because this is a probability density function, the distribution function is described by the following equation. Therefore, a probability that the DPD r(g(x),x) exists between a and b is determined by the following equation. If in true mapping {tilde over (g)}(x) the differential value between corresponding pixels becomes zero, DPD in {tilde over (g)}(x), that is, the true DPD{tilde over (r)}({tilde over (g)}(x),x) is {tilde over (r)}({tilde over (g)}(x),x)=ε. In other words, the true DPD{tilde over (r)}({tilde over (g)}(x),x) is equal to a noise component. Therefore, {tilde over (r)}({tilde over (g)}(x),x) is to be 0 in average and accord to the Gaussian distribution of dispersion σ For example, when the reliability of 95% will be given to the true DPD{tilde over (r)}({tilde over (g)}(x),x), it may be expressed by the following equation. Therefore, the probability of the mapping g(x) when DPD is provided may be defined by the following equation 144. Probability of Mapping g(x) Surface reliability computation Covered occlusion (covered occlusion) is a transformed and compressed region. The quadrilateral is defined as follows: The transformed quadrilateral is defined as follows The difference between these areas is represented by the following equation. where S represents an operator for computing an area. If this differential value r(n) is a negative, the occlusion is compressed. If the whole screen is a single area, the distribution of differential value R(n) looks good according to the Gaussian distribution. If the screen comprises a plurality of regions, and the covered occlusion occurs, the differential value R(n) between the regions may be distributed as outlier. Therefore, the probability that it is the surface builds reliability to be high so that the differential value R(n) is almost zero. Probability density function of differential value R is modeled as follows.
Because this is a probability density function, a distribution function is expressed by the following equation: Therefore, the probability that the differential value R(n) is between b and a is represented by the following equation: If the reliability when the differential value R(n)=0 is obtained will be 95%, the following equation is preferable. Surface probability when the differential value is obtained is defined by the following equation 145. Surface Probability Real constitution seems to become not more than. One Surface Reliability Computation Step (a) An area S (b) (a sample average μ of R(n),∀nεN and a sample variance a σ (c) Surface probability of the quadrilateral inside is obtained.
(d) A result is stored in the surface reliability map. (e) The above processing is done about the whole screen ∀nεN. Surface Trust Computation (a) Probability of mapping is obtained.
(b) A result is stored in the mapping reliability map. (c) The above processing is done about the whole screen ∀nεN. A mapping reliability map is obtained from g(Vn,t+lΔt;lΔt),∀nεN by the surface reliability computation (a) Probability of representation is found.
(b) A result is stored in the mapping reliability map. (c) The above processing is done about the whole screen ∀nεN. Surface Reliability Computation (a) An area S (b) Surface probability of the quadrilateral inside is obtained with respect to the threshold T (c) A result is stored in the covered surface reliability map. (d) The above processing is done about the whole screen ∀nεN. As described above, according to the present embodiment, the probability of the surface reliability is described as a probability distribution according to the Gaussian distribution, whereby the more robust surface reliability can be obtained and the interpolation frame of higher image quality more can be provided. The present embodiment is explained referring to the block diagram of In the case of the image wherein the background remains stationary, a part that an occlusion region can occur is only a part which changes between frames. The part which changes between frames can be detected by detecting a difference between the frames. Therefore, the surface reliability can be computed from the difference between the frames. Surface reliability computation where ε Pixel Difference (referred to as PD hereinafter) is modeled as follows.
If this PD is smaller than a threshold, the probability can be set to 1, and if it is greater than the threshold, it can be set to 0. The modeling is not the above threshold modeling but may be the following method according to the Gaussian distribution model. Assuming that an observed noise is 0 in average, and accords to the Gaussian distribution of dispersion σ Likelihood.
Because this is a probability density function, the distribution function is expressed by the following equation.
Therefore, the probability that PDγ(x) is between a and b is expressed by the following equation. Assuming that a differential value between corresponding pixels becomes zero in a stationary region, the stationary PD is equal to a noise component. Therefore, the stationary PD is 0 in average and accords to the Gaussian distribution of dispersion σ Accordingly, the surface probability when a certain PD is provided is defined by the following equation 147. Surface Probability The dispersion σ Therefore, the surface reliability computation Surface Trust Computation (d) Probability of the stationary surface is computed.
(e) A result is stored in a stationary surface reliability map. (f) The above processing is done about the whole screen ∀nεN. If the background completely stands still, it may use as it is without occlusion matching. A combination step is improved for the fourth embodiment. Combination is performed to generate an interpolation frame s (a) Probabilities on the surface side and the occlusion side are obtained as a probability of product event. (b) An alpha blend coefficient for combination is determined by the following equation.
(d) The above processing is done about the whole screen ∀nεN. According to the present embodiment, there is provided a high-quality interpolation frame in the case that the background stands still as mentioned above. The embodiment may accord to the Gaussian distribution as in the sixth embodiment in the case that the whole screen is a single area, but needs not always accord to the Gaussian distribution in the case that it comprises a plurality of areas. For this reason, the precision may fall. Accordingly, in the present embodiment, a precision is maintained for a plurality of regions by flipping the region coming off from the Gaussian distribution by M estimation. Covered occlusion is assumed to be a transformed and compressed region. A quadrilateral is defined as Quad The difference between the areas of these quadrilateral is assumed to be R(n)=S(Quad S indicates an operator for computing the areas. If this differential value R(n) is negative, the region is compressed. If the whole screen is a single region, the distribution of differential value R(n) is seemed to be good according to the Gaussian distribution. If the screen is formed of a plurality of regions and the covered occlusion occurs, the possibility that the differential value R(n) between the regions is distributed as an outlier is high. Therefore, it is possible to built up the reliability that the probability that the region is the surface increases as the differential value R(n) nears zero. The probability density function of the differential value R is modeled as follows.
Because this is a probability density function, the distribution function is expressed by the following equation.
Therefore, the probability that the differential value R(n) exists between a and b is given by the following equation. If the reliability when the differential value R(n)=0 is provided will be 95%, the following equation may be established. Accordingly, the surface probability when the differential value is acquired is defined by the following equation 148. The surface probability is defined by the following equation. This surface probability is one obtained when the whole screen is a single region. When the probability which is not an outlier is assumed to be w:N→R:n w(n), the surface probability in the case of a plurality of regions can be defined by the following equation 149. This probability w can be associated with the weight function of the M estimation. The weight function of the M estimation nears 1 as the probability nears to an average, and nears zero as the probability falls. A modeling will be done within a frame of the M estimation. Because it is desired to discuss on the covered occlusion, this modeling is intended for only a compression direction. In other words, the expanded region should be ignored, the modeling is intended for a point of R(n)<1 (R(n)≧1 represents the expanded region). The surface on the covered occlusion is an average of the Gaussian distribution. In other words, a gap between the model and the differential value R(n) can be defined by the following equation.
According to the TuKey's Biweight method, the weight function can be expressed by the following equation.
where c>0 is a constant, and c=6.0 in the case of normal distribution. A sample average and a specimen deviation can be computed using the weight function by the following equation.
where the superscript expresses iteration. The M estimation is done by iterating a computation of the sample average & deviation and a computation of the weight function alternately once. When the probability is developed using a result of this M estimation, the modeling can be carried out as follows.
As described above, the present embodiment can provide an interpolation frame with a good precision even if the screen is formed of a plurality of regions. The present embodiment is explained referring to the block diagram of The outputs of the forward and backward matching units Assuming that the image wherein an object moves from the left to the right is subjected to matching as shown in The similar action occurs even if the matching is carried out in an opposite direction. Therefore, the regions which the forward and backward matching could not be carried out are detected as the expansion and shrinkage of the spring respectively. In other words, it is very likely that the region which could not be detected is contraction with being combined with the occlusion region. The matching error can be compensated by matching the occlusion regions together again. A surface reliability computation A step corresponding to the forward matching unit A region that the back side reliability is more than a threshold (0.5, for example) is detected as the back side. These regions are assumed to be the first back side and the second back side respectively. A frame obtained by masking the reference frame t with the second back side is assumed to be a mask reference frame t, and a frame obtained by masking the reference frame t+lΔt with the second side is assumed to be a mask reference frame t+lΔt. The matching is done using the mask reference frame t as an source image and the mask reference frame t+lΔt as a reference image to produce a forward matching image. The matching is done using the mask reference frame t+lΔt as a source image and the mask reference frame t as a reference image to produce a backward matching image. For example, the reliabilities of the forward matching image and backward matching image are obtained using the step of the surface reliability computation By doing combination as in the fourth embodiment together with the result of the surface reliability computation According to the present invention, there can be produced an image wherein even if the image is smoothed off, expression of discontinuity is not lost, and even if discontinuity is expressed, the smoothness is obtained. Additional advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described herein. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents. Referenced by
Classifications
Rotate |