CA1229157A - Method for determining the displacement of image sequences and arrangement and use thereof - Google Patents
Method for determining the displacement of image sequences and arrangement and use thereofInfo
- Publication number
- CA1229157A CA1229157A CA000475675A CA475675A CA1229157A CA 1229157 A CA1229157 A CA 1229157A CA 000475675 A CA000475675 A CA 000475675A CA 475675 A CA475675 A CA 475675A CA 1229157 A CA1229157 A CA 1229157A
- Authority
- CA
- Canada
- Prior art keywords
- output
- input
- image sequences
- values
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/577—Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
Abstract
ABSTRACT OF THE DISCLOSURE
In a method and apparatus for determining, in a system having a transmitting end and a receiving end, displacements of image features occurring between successively transmitted image sequences, by the operations of determining, at the receiving end, peak values of pixel correlation between successively transmitted image sequences; deriving representations of estimated displace-ment values from the peak values of pixel correlation; and produc-ing representations of optimized displacement values by a recursion operation which includes recursively deriving a correction value and adding the correction value to an existing estimated displace-ment value to form a new estimated value, the correction value is derived by performing a mathematical division operation in which the dividend is a selected function of pixel values in successive image sequences and the divisor is proportional to the average of a first value resulting from crosscorrelation of successively transmitted image sequences and a second value resulting from autocorrelation of the currently received image sequence.
In a method and apparatus for determining, in a system having a transmitting end and a receiving end, displacements of image features occurring between successively transmitted image sequences, by the operations of determining, at the receiving end, peak values of pixel correlation between successively transmitted image sequences; deriving representations of estimated displace-ment values from the peak values of pixel correlation; and produc-ing representations of optimized displacement values by a recursion operation which includes recursively deriving a correction value and adding the correction value to an existing estimated displace-ment value to form a new estimated value, the correction value is derived by performing a mathematical division operation in which the dividend is a selected function of pixel values in successive image sequences and the divisor is proportional to the average of a first value resulting from crosscorrelation of successively transmitted image sequences and a second value resulting from autocorrelation of the currently received image sequence.
Description
~9~7 27371-141 BACKGROUND OF THE INVENTION
The present invention relates to a method and circuit arrangement for determining displacements of scene elements between successively transmitted image sequences corresponding to a field or frame. Such scene elements are associated with moving objects appearing in the scene depicted by the image sequences.
Such a method is described by Hans C. Bergman in an article entitled "Analysis of Different Displacement Estimation Algorithms for Digital Television Signals," published in NATO AS Series, Vol. F2, Image Sequence Processing and Dynamic Scene Analysis, edited by T. S. Hung, published by Sprinter Verlag, Berlin, Heidelberg, 1983, at pages 215-234.
Various recursive and non recursive estimating methods have been published for determination of displacement in television signals under real-time conditions.
Non recursive methods are disclosed, for example, by Schalkoff in IRE Trans. on Pattern Analysis and Machine Intel-loginess," Vol. 4, No. 1, January, 19~2, at pages 2-10. They have the advantage that hardware implementation thereof is relatively easy but they operate with sufficient accuracy in their estimation only for displacements of up to 2 pixels per frame.
See, in this connection, Bergman, swooper, page 222. Since displacements of more than 10 pixels per frame occur in natural television image sequences, These methods find only limited use.
The present invention relates to a method and circuit arrangement for determining displacements of scene elements between successively transmitted image sequences corresponding to a field or frame. Such scene elements are associated with moving objects appearing in the scene depicted by the image sequences.
Such a method is described by Hans C. Bergman in an article entitled "Analysis of Different Displacement Estimation Algorithms for Digital Television Signals," published in NATO AS Series, Vol. F2, Image Sequence Processing and Dynamic Scene Analysis, edited by T. S. Hung, published by Sprinter Verlag, Berlin, Heidelberg, 1983, at pages 215-234.
Various recursive and non recursive estimating methods have been published for determination of displacement in television signals under real-time conditions.
Non recursive methods are disclosed, for example, by Schalkoff in IRE Trans. on Pattern Analysis and Machine Intel-loginess," Vol. 4, No. 1, January, 19~2, at pages 2-10. They have the advantage that hardware implementation thereof is relatively easy but they operate with sufficient accuracy in their estimation only for displacements of up to 2 pixels per frame.
See, in this connection, Bergman, swooper, page 222. Since displacements of more than 10 pixels per frame occur in natural television image sequences, These methods find only limited use.
- 2 -`
~2~9~S~ 27371-141 Recursive estimating methods are generally based on an optimization criterion, with a correction value being added to the respectively preceding estimated value so that the optimization value gradually approaches the optimum value. Examination of known recursive displacement estimation methods has shown that they con-verge at sufficient speed, i.e. already after a few recursion steps, for average displacements up to about 4 pixels per frame.
However, for large displacements up to 20 pixels per frame, five and more recursion steps are required to overcome the transient state.
However, with respect to real-time estimation, the number of recursion steps is limited to one or two steps so that these methods likewise find only limited use for the estimation ; of large displacements.
SUMMARY OF THE INVENTION
It is therefore an object of the present invention to determine displacement in successively transmitted image sequences so as to furnish an accurate estimated value already after a few recursion steps, i.e. to achieve a very high convert genre rate, with the prerequisite that the convergence range is at least as large as in the known methods.
~2~7 27371-141 The above and other objects are achieved, according to the invention, in connection with a method for determining, in a system having a -transmitting end and a receiving end, displace-mints of image features occurring between successively transmitted image sequences which method includes: determining, at the no-ceiling end, peak values of pixel correlation between successively transmitted image sequences, deriving representations of estimated displacement values from the peak values of pixel correlation; and producing representations of optimized displace-mint values by a recursion operation which includes recursively : deriving a correction value and adding the correction value to an existing estimated displacement value to form a new estimated value. According to the invention r the step of deriving a correction value comprises performing a mathematical division operation in which the dividend is a selected function of pixel values in successive image sequences and the divisor is proper-tonal to the average of a first value resulting from crosscor-relation of successively transmitted image sequences and a second value resulting from auto correlation of the currently received image sequence.
The objects of the invention are further achieved by apparatus for determining, in a system having a transmitting end and a receiving end, displacements of scene features occurring between successively transmitted image sequences, the apparatus comprising:
a first input point connected to receive a signal representing the image sequence currently being transmitted;
a second input point connected to receive a signal representing a previously transmitted image sequence;
first and second differentiating means each having an input connected to a respective one of the input points and each constituting means for producing an output signal representing the derivative of the signal at its input;
first subtraction means having two inputs each connected to a respective one of the first and second input points and an output for providing a difference signal having a value equal to the difference between the values of the signals at the first and second input points;
adder means having two inputs each connected to receive the output signal produced by a respective one of the different tinting means, and an output for providing a sum signal having a value equal to the sum of the values of the signals produced by the first and second differentiating means;
Jo ..
1~2~5~ 27371-141 first and second multiplying means each having first and second inputs and an accumulating output device for storing a representation of the accumulated products of the signal values supplied to the first and second inputs over a selected window w of neighboring pixels, the first input of the first multiplying means and the second input of the second multiplying means being connected to receive the output signal produced by the first differentiating means, the second input of the first multiplying means being connected to the output of the first subtraction means, 10 and the first input of the second multiplying means being connected to the output of the adder means;
dividing means having two inputs each connected to receive the representation stored in the accumulating output device of a respective one of the multiplying means, and an output providing a representation of the quotient of the representations received by the dividing means inputs;
second subtraction means having a subtrahend input con-netted to the output of the dividing means, a minuend input and an I: output providing a representation of the difference between the 20 representations at its minuend and subtrahend inputs; and memory means having an input connected to receive the representation presently provided at the output of the second subtraction means, storage means for storing the representation provided at the output of the second subtraction means for the next iteration, and an output connected to the minuend input of the second subtraction means for supplying the representation stored by the storage means;
whereby the output of the second subtraction means provides a recursively optimized representation of the displace mints of image features occurring between successively transmitted image sequences.
A particular advantage of the present invention is its better and faster convergence rate of the estimated value for the displacement compared to those methods~hich use either only cross-correlation or auto correlation functions for the divisor. Because of the fast convergence rate, the method is particularly suitable for real-time estimation.
The present invention will now be described in greater detail with reference to the drawings.
.
BRIEF DESCRIPTION OF TOE DRAWINGS
Figure 1 is a diagram showing one-dimensional attacker-relation and cross correlation functions of the luminance signals of two successive image sequences corresponding either to fields or frames.
. go Figure 2 is a diagram showing the first derivative of the correlation functions and the estimated displacement values.
Figure 3 is a block circuit diagram of a preferred embodiment of an arrangement for implementing the method of the present invention.
Figure 4 is a block circuit diagram of an embodiment of a circuit according to the invention for reducing the image flicker in standard television.
DESCRIPTION OF THE PREFERRED EMBODIMENTS
Some basics and prerequisites for better understanding of the relationships given below shall be explained first. A
moving object in a televised scene is being observed. The two-dimensional luminance signal of the image sequence just being transmitted is designated Ik(x,y) and the image sequence trays-milted immediately preceding in time is designated Ik_l(x,y).
Thus, x and y are the horizontal and vertical coordinates of each picture element of an image sequence and k is a parameter for the succession of image sequences. For example, each image sequence can be a television picture field. Correlation R of these two signals, in dependence on the displacements do and dye which are the localized signal offset between these two image sequences in the x and y directions, then provides l~:Z~LS~
Cook IkIk_l(dx,dy) O O Ik(X'Y) Ik_l(X dry dy)dX dye with a correlation peak at do, dye (1) Through use of the recursion formulas:
do = dx(i-l) + do (2 Dow Doyle) dye displacement can be determined in the ithiteration step, whereby the iterations can be carried out either at one pixel or from one to the next pixel spatially as well as temporarily.
Displacement in the horizontal direction, the x direction, can be expressed by means of the following equation:
RI I' (O'ER (d d ) RI If (d d ) do= k-l k-l k k-l x' y (i-l) k k-l x' y (i-l) ROY, (0,0) RI ION
k k _ + RIkI'k-l (dX,dy)(i-l) wherein I' represents the partial derivative a - I, and (0,0) signifies that for the respective correlation there is no relative displace-mint associated with the two signals being correlated because they relate to the same image sequence.
`'',' I lS7 In the denominator of this term, there appear only the sum of the first derivative of the auto correlation function for It at point (0,0) and the firs-t derivative of the cross correlation function of It and It 1 at point (do, dye), i.e. the points where the auto correlation function (AKF) and the cross correlation function (KKF ) have their peak values. Figure 1 shows these auto correlation and cross correlation functions for the displaced luminance signals with respect to a one-dimensional example. The peak value of AKF~ RI do lies at do = and the peak value of KKF, Rococo lox lies at do = duo. For stationary portions of a scene depicted by successive image sequences, the following then applies:
R . I (0,0) = I I I (') . (5) lk-lIk-l k k Moreover, the order of differentiation and integration can be exchanged for stationary processes of It and It 1 By way of conversions, the following results:
Jo Rococo 1 (do dye = adieux Rococo 1 (dX,dy) (6) IkIk 1 (dX,dy) = - adieux Rococo 1 (dX,dy) (7) kit l ~dx'dY~ I Cook Jo ye I
and thus for Equation I
2 * Ilk Sty a X Audi owe) err C
k k k-l Y (~) In this case, Equation is used to locally determine the maximum ox the crosscorrela~ion function of the disk : placed video signals It lay and Ik(x,y). Figure 2, in conjunction with Figure 1, shows these relationships again for one-dimensional functions.
In Figure 1, the auto correlation function (AKF) RI I (do) and the cross correlation function ~KKF) RI I (do) of the : video signals approximately coincide in their shape - 11 `
~Z'~9~5~
and are displaced with respect to one another in direction do by the distance duo. Let it be assumed that the starting value for the recursive estimation method is dx(i-l) = 0. The first estimated value dub is calculated according to Equation (9) from the quotient of the first derivative of KKF at point (0,0) and the second derive-lives of AKF and KKF, each weighted with the factor 1/2. By averaging both second derivatives in Equation I the estimated value approximates the actual displacement duo better than the known methods in which the divisor contains either only the second derivative of KKF or of AKF. In these cases, one obtains the estimated values dXa and dxc, respectively, which are shown in dot-dash lines in Figure 2.
lo In a comparison with known methods for determining displacement it was found for the example of a special KKF
that the novel displacement estimating method converges faster by a factor of 4 than the Newton method described by Bergman in "Displacement Estimation Based On The Charlie-lion Of Image Segments", published by the International Conference on Electronic Image Processing, July, 1982, University of York, Great Britain and faster by a factor of 2 than a method described by Cougher and Rex in an Jo ..
~22~
article published in IEEE Transact. on Information Theory, Vol.
IT-22, No. 5, September, 1976, pages 573-579. Experimental exam-inactions have further shown that the novel estimation method attains an accuracy sufficient for the interpolation of television image sequences already after two recursion steps.
The block circuit diagram according to Figure 3 shows an arrangement for implementing the method according to the present invention. realization of the method was based on the following relationship for displacement determination:
- -2 w [Ik-2(x-dx~Y-dy)(i-l)-yk(x~y)] a/ax It (zoo) 10 do = dX(i-l) We / X Icky (X dX~Y-dy) (isle Ajax It (zoo) ] a/ax It (x my) (10 ) where It and It 2 are the luminance of respective video fields.
Displacement dye for the vertical direction is determined correspondingly.
This relationship (10) is a modified form of Equation I
It should be noted that in Equation (9), it is not the field It lay immediately prior to field Ike y) that is utilized for forming the cross correlation with Ike y) but the corresponding field It 2(x,Y) transmitted immediately prior to It lay .,~ ., Z~3~57 27371~
Equation (10) is a recursive formula with which a new estimated value do is always formed from the estimated value dx(i-l) determined immediately before and a correction value do. In the first step, dX(i~ is determined with dX(i_l~ = O and in the sea step dX(i+l) is determined with do from the first step, etc.
In equation (10) the estimated values dX(i 1) and Dow 1) determined in preceding iteration are used to calculate It 2 (X-dx/ y-dy) (i-l) As can be seen in Figure 3, the difference between image sequences It and Ik_2(x-dx) (i-l) fed to signal inputs 1 and 2 is formed in subtracter Sly The two image sequences It and It 2(x-dx) (i 1) axe also conducted through respective different shutters Do and Do so that the first derivatives of the image sequences are available at their outputs. Multipliers Ml and My ; are additionally equipped with respective accumulation devices Age with which the multiplied signals are summed - in the thus-treated embodiment this summing is done with respect to an image window w (see Equation 10).
Window w is preferably chosen rectangular. For instance the window size can be 5 x 5 pixels. With a sampling frequency of 10 MHz for the image sequences the time for multiplication will take 1,125 us. That will mean, that in a time interval of 1,2 -1,3 us, i.e. for each Thea to Thea pixel, a displacement vector will be determined. Multipliers as well as differentiators realized in the form of subtracters for these purposes are well known, e.g. the 16 bit multiplier by TRW.
- aye -27371~
For multiplier Ml, the signal fed to the first multiplier input is the output signal of differentiator Do and the signal fed to the second multiplier input is the output signal of subtracter Sly Multiplier My receives the output signal of differentiator Do as well as the output signal of an adder I
which links the output signals of the two differentiators Do and Do.
The output signals of Ml and My are supplied to respective inputs of a divider Q. The result of the division is weighted with a factor of 2. The actual estimated value do is obtained by means of a subtracter So by linking the estimated value determined during the last recursion step, which is stored in a memory Spy and the correction value, i.e. the output signal of divider Q.
For storing the previously determined estimated value, memory Sup is provided with a connection to output 3 of subtracter So which simultaneously constitutes the output of the arrangement.
To take care that signals It and Ik_2(x-dx)(i-l) can be evaluated together, a signal delaying device (not shown) is connected ahead of the arrangement to delay the luminance signal It for the duration of two fields or one frame.
' ':, I So The method according to the present invention can ad van-tageously be used particularly or motion adaptive interpolation of video image sequences, e.g. for the reconstruction of eliminated image sequences (fields) in transmission channels employing data reduction. The principle involved is based on the reduction of the image repetition rate at the transmitting end by omitting video fields and motion adapting interpolation of the not trays-milted images in the receiver. A further advantageous use for the method according to the invention is the removal of flicker in standard television reception. The image repetition rate of the standard video signal on the monitor can be increased in that Jan additional image sequence is obtained by interpolation from two successively transmitted image sequences (fields), with the additional sequence being inserted at a monitor M between two conventionally displayed fields and the interpolated field being corrected in a motion adaptive manner under consideration of displacement.
the differentiators Do and Do are realized in form of subtracters, i.e. spatial derivatives are substituted by spatial differentials.
Jo ~2~9~7 27371-141 The block circuit diagram of Figure 4 shows an arrangement for such flicker removal By means of an evaluation device, or image memory, Al for a first image sequence (for example, the first field) and an evaluation device, or image memory, A for a second image sequence for example, the field preceding the first field), an intermediate image sequence is produced in an inter poultry It. This interpolator It has control input Sty to which is fed a motion adaptive control signal. The displacement, as obtained with the arrangement according to Figure 3, serves as the motion adaptive control signal.
Output I' of evaluation device Al corresponds to signal input 1 of Figure 3. The same applies for output 2' of evaluation device A. For the further components, such as adder, different shutters, multipliers, eta, the reference characters of Figure 3 are also used in Figure 4. Thus an interpolated intermediate image composed under consideration of displacement appears additionally on monitor M. Due to the transmission of the intermediate image, the observer is offered a higher image repetition rate which noticeably reduces image flicker, particularly for the playback of high luminance values. Visible distortions, e.g. along the edges of moving image portions, which would occur if displacement err not considered, are reduced. In order for the images to be displayed on monitor M in their correct time sequence, suitable delaying devices are required from case to case ahead of the inputs of Monitor M.
The method according to the present invention is portico-laxly suitable for use in connection with source coding methods for the purpose of data reduction, as for example in DPCM image transmission, and for use in connection with noise suppression by means of image-to-image filters.
The interpolator It is described in greater detail in our corresponding Canadian application Serial No. 475l674 filed March 4, 1985.
The input 2 of the circuit of Figure 4 corresponds to the input 2 of Figure 3. In Figure 4 only one step of iteration is assumed.
~2Z9~
It will be understood that the above description of the present invention is susceptible to various modifications, changes and adaptations, and the same are intended to be comprehended within the meaning and range of equivalents of the appended claims.
. "
~2~9~S~ 27371-141 Recursive estimating methods are generally based on an optimization criterion, with a correction value being added to the respectively preceding estimated value so that the optimization value gradually approaches the optimum value. Examination of known recursive displacement estimation methods has shown that they con-verge at sufficient speed, i.e. already after a few recursion steps, for average displacements up to about 4 pixels per frame.
However, for large displacements up to 20 pixels per frame, five and more recursion steps are required to overcome the transient state.
However, with respect to real-time estimation, the number of recursion steps is limited to one or two steps so that these methods likewise find only limited use for the estimation ; of large displacements.
SUMMARY OF THE INVENTION
It is therefore an object of the present invention to determine displacement in successively transmitted image sequences so as to furnish an accurate estimated value already after a few recursion steps, i.e. to achieve a very high convert genre rate, with the prerequisite that the convergence range is at least as large as in the known methods.
~2~7 27371-141 The above and other objects are achieved, according to the invention, in connection with a method for determining, in a system having a -transmitting end and a receiving end, displace-mints of image features occurring between successively transmitted image sequences which method includes: determining, at the no-ceiling end, peak values of pixel correlation between successively transmitted image sequences, deriving representations of estimated displacement values from the peak values of pixel correlation; and producing representations of optimized displace-mint values by a recursion operation which includes recursively : deriving a correction value and adding the correction value to an existing estimated displacement value to form a new estimated value. According to the invention r the step of deriving a correction value comprises performing a mathematical division operation in which the dividend is a selected function of pixel values in successive image sequences and the divisor is proper-tonal to the average of a first value resulting from crosscor-relation of successively transmitted image sequences and a second value resulting from auto correlation of the currently received image sequence.
The objects of the invention are further achieved by apparatus for determining, in a system having a transmitting end and a receiving end, displacements of scene features occurring between successively transmitted image sequences, the apparatus comprising:
a first input point connected to receive a signal representing the image sequence currently being transmitted;
a second input point connected to receive a signal representing a previously transmitted image sequence;
first and second differentiating means each having an input connected to a respective one of the input points and each constituting means for producing an output signal representing the derivative of the signal at its input;
first subtraction means having two inputs each connected to a respective one of the first and second input points and an output for providing a difference signal having a value equal to the difference between the values of the signals at the first and second input points;
adder means having two inputs each connected to receive the output signal produced by a respective one of the different tinting means, and an output for providing a sum signal having a value equal to the sum of the values of the signals produced by the first and second differentiating means;
Jo ..
1~2~5~ 27371-141 first and second multiplying means each having first and second inputs and an accumulating output device for storing a representation of the accumulated products of the signal values supplied to the first and second inputs over a selected window w of neighboring pixels, the first input of the first multiplying means and the second input of the second multiplying means being connected to receive the output signal produced by the first differentiating means, the second input of the first multiplying means being connected to the output of the first subtraction means, 10 and the first input of the second multiplying means being connected to the output of the adder means;
dividing means having two inputs each connected to receive the representation stored in the accumulating output device of a respective one of the multiplying means, and an output providing a representation of the quotient of the representations received by the dividing means inputs;
second subtraction means having a subtrahend input con-netted to the output of the dividing means, a minuend input and an I: output providing a representation of the difference between the 20 representations at its minuend and subtrahend inputs; and memory means having an input connected to receive the representation presently provided at the output of the second subtraction means, storage means for storing the representation provided at the output of the second subtraction means for the next iteration, and an output connected to the minuend input of the second subtraction means for supplying the representation stored by the storage means;
whereby the output of the second subtraction means provides a recursively optimized representation of the displace mints of image features occurring between successively transmitted image sequences.
A particular advantage of the present invention is its better and faster convergence rate of the estimated value for the displacement compared to those methods~hich use either only cross-correlation or auto correlation functions for the divisor. Because of the fast convergence rate, the method is particularly suitable for real-time estimation.
The present invention will now be described in greater detail with reference to the drawings.
.
BRIEF DESCRIPTION OF TOE DRAWINGS
Figure 1 is a diagram showing one-dimensional attacker-relation and cross correlation functions of the luminance signals of two successive image sequences corresponding either to fields or frames.
. go Figure 2 is a diagram showing the first derivative of the correlation functions and the estimated displacement values.
Figure 3 is a block circuit diagram of a preferred embodiment of an arrangement for implementing the method of the present invention.
Figure 4 is a block circuit diagram of an embodiment of a circuit according to the invention for reducing the image flicker in standard television.
DESCRIPTION OF THE PREFERRED EMBODIMENTS
Some basics and prerequisites for better understanding of the relationships given below shall be explained first. A
moving object in a televised scene is being observed. The two-dimensional luminance signal of the image sequence just being transmitted is designated Ik(x,y) and the image sequence trays-milted immediately preceding in time is designated Ik_l(x,y).
Thus, x and y are the horizontal and vertical coordinates of each picture element of an image sequence and k is a parameter for the succession of image sequences. For example, each image sequence can be a television picture field. Correlation R of these two signals, in dependence on the displacements do and dye which are the localized signal offset between these two image sequences in the x and y directions, then provides l~:Z~LS~
Cook IkIk_l(dx,dy) O O Ik(X'Y) Ik_l(X dry dy)dX dye with a correlation peak at do, dye (1) Through use of the recursion formulas:
do = dx(i-l) + do (2 Dow Doyle) dye displacement can be determined in the ithiteration step, whereby the iterations can be carried out either at one pixel or from one to the next pixel spatially as well as temporarily.
Displacement in the horizontal direction, the x direction, can be expressed by means of the following equation:
RI I' (O'ER (d d ) RI If (d d ) do= k-l k-l k k-l x' y (i-l) k k-l x' y (i-l) ROY, (0,0) RI ION
k k _ + RIkI'k-l (dX,dy)(i-l) wherein I' represents the partial derivative a - I, and (0,0) signifies that for the respective correlation there is no relative displace-mint associated with the two signals being correlated because they relate to the same image sequence.
`'',' I lS7 In the denominator of this term, there appear only the sum of the first derivative of the auto correlation function for It at point (0,0) and the firs-t derivative of the cross correlation function of It and It 1 at point (do, dye), i.e. the points where the auto correlation function (AKF) and the cross correlation function (KKF ) have their peak values. Figure 1 shows these auto correlation and cross correlation functions for the displaced luminance signals with respect to a one-dimensional example. The peak value of AKF~ RI do lies at do = and the peak value of KKF, Rococo lox lies at do = duo. For stationary portions of a scene depicted by successive image sequences, the following then applies:
R . I (0,0) = I I I (') . (5) lk-lIk-l k k Moreover, the order of differentiation and integration can be exchanged for stationary processes of It and It 1 By way of conversions, the following results:
Jo Rococo 1 (do dye = adieux Rococo 1 (dX,dy) (6) IkIk 1 (dX,dy) = - adieux Rococo 1 (dX,dy) (7) kit l ~dx'dY~ I Cook Jo ye I
and thus for Equation I
2 * Ilk Sty a X Audi owe) err C
k k k-l Y (~) In this case, Equation is used to locally determine the maximum ox the crosscorrela~ion function of the disk : placed video signals It lay and Ik(x,y). Figure 2, in conjunction with Figure 1, shows these relationships again for one-dimensional functions.
In Figure 1, the auto correlation function (AKF) RI I (do) and the cross correlation function ~KKF) RI I (do) of the : video signals approximately coincide in their shape - 11 `
~Z'~9~5~
and are displaced with respect to one another in direction do by the distance duo. Let it be assumed that the starting value for the recursive estimation method is dx(i-l) = 0. The first estimated value dub is calculated according to Equation (9) from the quotient of the first derivative of KKF at point (0,0) and the second derive-lives of AKF and KKF, each weighted with the factor 1/2. By averaging both second derivatives in Equation I the estimated value approximates the actual displacement duo better than the known methods in which the divisor contains either only the second derivative of KKF or of AKF. In these cases, one obtains the estimated values dXa and dxc, respectively, which are shown in dot-dash lines in Figure 2.
lo In a comparison with known methods for determining displacement it was found for the example of a special KKF
that the novel displacement estimating method converges faster by a factor of 4 than the Newton method described by Bergman in "Displacement Estimation Based On The Charlie-lion Of Image Segments", published by the International Conference on Electronic Image Processing, July, 1982, University of York, Great Britain and faster by a factor of 2 than a method described by Cougher and Rex in an Jo ..
~22~
article published in IEEE Transact. on Information Theory, Vol.
IT-22, No. 5, September, 1976, pages 573-579. Experimental exam-inactions have further shown that the novel estimation method attains an accuracy sufficient for the interpolation of television image sequences already after two recursion steps.
The block circuit diagram according to Figure 3 shows an arrangement for implementing the method according to the present invention. realization of the method was based on the following relationship for displacement determination:
- -2 w [Ik-2(x-dx~Y-dy)(i-l)-yk(x~y)] a/ax It (zoo) 10 do = dX(i-l) We / X Icky (X dX~Y-dy) (isle Ajax It (zoo) ] a/ax It (x my) (10 ) where It and It 2 are the luminance of respective video fields.
Displacement dye for the vertical direction is determined correspondingly.
This relationship (10) is a modified form of Equation I
It should be noted that in Equation (9), it is not the field It lay immediately prior to field Ike y) that is utilized for forming the cross correlation with Ike y) but the corresponding field It 2(x,Y) transmitted immediately prior to It lay .,~ ., Z~3~57 27371~
Equation (10) is a recursive formula with which a new estimated value do is always formed from the estimated value dx(i-l) determined immediately before and a correction value do. In the first step, dX(i~ is determined with dX(i_l~ = O and in the sea step dX(i+l) is determined with do from the first step, etc.
In equation (10) the estimated values dX(i 1) and Dow 1) determined in preceding iteration are used to calculate It 2 (X-dx/ y-dy) (i-l) As can be seen in Figure 3, the difference between image sequences It and Ik_2(x-dx) (i-l) fed to signal inputs 1 and 2 is formed in subtracter Sly The two image sequences It and It 2(x-dx) (i 1) axe also conducted through respective different shutters Do and Do so that the first derivatives of the image sequences are available at their outputs. Multipliers Ml and My ; are additionally equipped with respective accumulation devices Age with which the multiplied signals are summed - in the thus-treated embodiment this summing is done with respect to an image window w (see Equation 10).
Window w is preferably chosen rectangular. For instance the window size can be 5 x 5 pixels. With a sampling frequency of 10 MHz for the image sequences the time for multiplication will take 1,125 us. That will mean, that in a time interval of 1,2 -1,3 us, i.e. for each Thea to Thea pixel, a displacement vector will be determined. Multipliers as well as differentiators realized in the form of subtracters for these purposes are well known, e.g. the 16 bit multiplier by TRW.
- aye -27371~
For multiplier Ml, the signal fed to the first multiplier input is the output signal of differentiator Do and the signal fed to the second multiplier input is the output signal of subtracter Sly Multiplier My receives the output signal of differentiator Do as well as the output signal of an adder I
which links the output signals of the two differentiators Do and Do.
The output signals of Ml and My are supplied to respective inputs of a divider Q. The result of the division is weighted with a factor of 2. The actual estimated value do is obtained by means of a subtracter So by linking the estimated value determined during the last recursion step, which is stored in a memory Spy and the correction value, i.e. the output signal of divider Q.
For storing the previously determined estimated value, memory Sup is provided with a connection to output 3 of subtracter So which simultaneously constitutes the output of the arrangement.
To take care that signals It and Ik_2(x-dx)(i-l) can be evaluated together, a signal delaying device (not shown) is connected ahead of the arrangement to delay the luminance signal It for the duration of two fields or one frame.
' ':, I So The method according to the present invention can ad van-tageously be used particularly or motion adaptive interpolation of video image sequences, e.g. for the reconstruction of eliminated image sequences (fields) in transmission channels employing data reduction. The principle involved is based on the reduction of the image repetition rate at the transmitting end by omitting video fields and motion adapting interpolation of the not trays-milted images in the receiver. A further advantageous use for the method according to the invention is the removal of flicker in standard television reception. The image repetition rate of the standard video signal on the monitor can be increased in that Jan additional image sequence is obtained by interpolation from two successively transmitted image sequences (fields), with the additional sequence being inserted at a monitor M between two conventionally displayed fields and the interpolated field being corrected in a motion adaptive manner under consideration of displacement.
the differentiators Do and Do are realized in form of subtracters, i.e. spatial derivatives are substituted by spatial differentials.
Jo ~2~9~7 27371-141 The block circuit diagram of Figure 4 shows an arrangement for such flicker removal By means of an evaluation device, or image memory, Al for a first image sequence (for example, the first field) and an evaluation device, or image memory, A for a second image sequence for example, the field preceding the first field), an intermediate image sequence is produced in an inter poultry It. This interpolator It has control input Sty to which is fed a motion adaptive control signal. The displacement, as obtained with the arrangement according to Figure 3, serves as the motion adaptive control signal.
Output I' of evaluation device Al corresponds to signal input 1 of Figure 3. The same applies for output 2' of evaluation device A. For the further components, such as adder, different shutters, multipliers, eta, the reference characters of Figure 3 are also used in Figure 4. Thus an interpolated intermediate image composed under consideration of displacement appears additionally on monitor M. Due to the transmission of the intermediate image, the observer is offered a higher image repetition rate which noticeably reduces image flicker, particularly for the playback of high luminance values. Visible distortions, e.g. along the edges of moving image portions, which would occur if displacement err not considered, are reduced. In order for the images to be displayed on monitor M in their correct time sequence, suitable delaying devices are required from case to case ahead of the inputs of Monitor M.
The method according to the present invention is portico-laxly suitable for use in connection with source coding methods for the purpose of data reduction, as for example in DPCM image transmission, and for use in connection with noise suppression by means of image-to-image filters.
The interpolator It is described in greater detail in our corresponding Canadian application Serial No. 475l674 filed March 4, 1985.
The input 2 of the circuit of Figure 4 corresponds to the input 2 of Figure 3. In Figure 4 only one step of iteration is assumed.
~2Z9~
It will be understood that the above description of the present invention is susceptible to various modifications, changes and adaptations, and the same are intended to be comprehended within the meaning and range of equivalents of the appended claims.
. "
Claims (5)
PROPERTY OR PRIVILEGE IS CLAIMED ARE DEFINED AS FOLLOWS:
1. In a method for determining, in a system having a trans-mitting end and a receiving end, displacements of image features occurring between successively transmitted image sequences, which method includes: determining, at the receiving end, peak values of pixel correlation between successively transmitted image sequences, deriving representations of estimated displacement values from the peak values of pixel correlation, and producing representations of optimized displacement values by a recursion operation which includes recursively deriving a correction value and adding the correction value to an existing estimated displace-ment value to form a new estimated value, the improvement wherein said step of deriving a correction value comprises performing a mathematical division operation in which the dividend is a selected function of pixel values in successive image sequences and the divisor is proportional to the average of a first value resulting from crosscorrelation of successively transmitted image sequences and a second value resulting from autocorrelation of the currently received image sequence.
2. The method as defined in claim 1, carried out for the motion adaptive interpolation of intermediate image sequences for the reconstruction, in dependence of displace-ment, of image sequences eliminated during transmission.
3. The method as defined in claim 1, carried out for the removal of flicker in standard television transmissions, wherein additional image sequences are generated for playback from the transmitted image sequences by inter-polation and the interpolation takes place in dependence of displacement.
4. The method as defined in claim 1, carried out for the suppression of noise by means of image-to-image fil-tering.
5. Apparatus for determining, in a system having a transmitting end and a receiving end, displacements of scene features occurring between successively transmitted image sequences, said apparatus comprising:
a first input point connected to receive a signal representing the image sequence currently being transmitted;
a second input point connected to receive a signal representing a previously transmitted image sequence:
first and second differentiating means each having an input connected to a respective one of said input points and each constituting means for producing an output signal representing the derivative of the signal at its input;
first subtraction means having two inputs each connected to a respective one of said first and second input points and an output for providing a difference signal having a value equal to the difference between the values of the signals at said first and second input points;
adder means having two inputs each connected to receive the output signal produced by a respective one of said differen-tiating means, and an output for providing a sum signal having a value equal to the sum of the values of the signals produced by said first and second differentiating means;
first and second multiplying means each having first and second inputs and an accumulating output device for storing a representation of the accumulated products of the signal values supplied to said first and second inputs over a selected window w, said first input of said first multiplying means and said second input of said second multiplying means being connected to receive the output signal produced by said first differentiating means, said second input of said first multiplying means being connected to said output of said first subtraction means, and said first input of said second multiplying means being connected to said output of said adder means;
dividing means having two inputs each connected to receive the representation stored in said accumulating output device of a respective one of said multiplying means, and an output provid-in a representation of the quotient of the representations re-ceived by said dividing means inputs;
second subtraction means having a subtrahend input connected to said output of said dividing means, a minuend input and an output providing a representation of the difference between the representations at its minuend and subtrahend inputs; and memory means having an input connected to receive the representation presently provided at said output of said second subtraction means, storage means for storing the representation provided at said output of said second subtraction means for the next iteration, and an output connected to said minuend input of said second subtraction means for supplying the representation stored by said storage means;
whereby said output of said second subtraction means provides a recursively optimized representation of the displacements of image features occurring between succes-sively transmitted image sequences.
a first input point connected to receive a signal representing the image sequence currently being transmitted;
a second input point connected to receive a signal representing a previously transmitted image sequence:
first and second differentiating means each having an input connected to a respective one of said input points and each constituting means for producing an output signal representing the derivative of the signal at its input;
first subtraction means having two inputs each connected to a respective one of said first and second input points and an output for providing a difference signal having a value equal to the difference between the values of the signals at said first and second input points;
adder means having two inputs each connected to receive the output signal produced by a respective one of said differen-tiating means, and an output for providing a sum signal having a value equal to the sum of the values of the signals produced by said first and second differentiating means;
first and second multiplying means each having first and second inputs and an accumulating output device for storing a representation of the accumulated products of the signal values supplied to said first and second inputs over a selected window w, said first input of said first multiplying means and said second input of said second multiplying means being connected to receive the output signal produced by said first differentiating means, said second input of said first multiplying means being connected to said output of said first subtraction means, and said first input of said second multiplying means being connected to said output of said adder means;
dividing means having two inputs each connected to receive the representation stored in said accumulating output device of a respective one of said multiplying means, and an output provid-in a representation of the quotient of the representations re-ceived by said dividing means inputs;
second subtraction means having a subtrahend input connected to said output of said dividing means, a minuend input and an output providing a representation of the difference between the representations at its minuend and subtrahend inputs; and memory means having an input connected to receive the representation presently provided at said output of said second subtraction means, storage means for storing the representation provided at said output of said second subtraction means for the next iteration, and an output connected to said minuend input of said second subtraction means for supplying the representation stored by said storage means;
whereby said output of said second subtraction means provides a recursively optimized representation of the displacements of image features occurring between succes-sively transmitted image sequences.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| DEP3408016.3 | 1984-03-05 | ||
| DE3408016A DE3408016A1 (en) | 1984-03-05 | 1984-03-05 | METHOD FOR DETERMINING THE DISPLACEMENT OF IMAGE SEQUENCES AND ARRANGEMENT AND APPLICATIONS THEREFOR |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA1229157A true CA1229157A (en) | 1987-11-10 |
Family
ID=6229589
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA000475675A Expired CA1229157A (en) | 1984-03-05 | 1985-03-04 | Method for determining the displacement of image sequences and arrangement and use thereof |
Country Status (5)
| Country | Link |
|---|---|
| US (1) | US4630114A (en) |
| EP (1) | EP0154126B1 (en) |
| JP (1) | JP2509174B2 (en) |
| CA (1) | CA1229157A (en) |
| DE (2) | DE3408016A1 (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE3523424C1 (en) * | 1985-06-29 | 1986-10-30 | Deutsche Forschungs- und Versuchsanstalt für Luft- und Raumfahrt e.V., 5000 Köln | Method for the transmission and reproduction of video scenes, in particular aerial scenes, with a reduced frame rate with a relative movement of the recorded scene relative to the recording sensor |
| DE3704777C1 (en) * | 1987-02-16 | 1988-04-07 | Ant Nachrichtentech | Method of transmitting and playing back television picture sequences |
| US4893182A (en) * | 1988-03-18 | 1990-01-09 | Micronyx, Inc. | Video tracking and display system |
| US5101354A (en) * | 1988-04-18 | 1992-03-31 | Brunswick Bowling & Billards Corporation | Multi-lane bowling system with remote operator control |
| US5255185A (en) * | 1988-04-18 | 1993-10-19 | Brunswick Bowling & Billiards Corp. | Bowling center video display system |
| US4931868A (en) * | 1988-05-31 | 1990-06-05 | Grumman Aerospace Corporation | Method and apparatus for detecting innovations in a scene |
| US5122876A (en) * | 1988-12-26 | 1992-06-16 | Canon Kabushiki Kaisha | Image signal restoring apparatus |
| US4894716A (en) * | 1989-04-20 | 1990-01-16 | Burle Technologies, Inc. | T.V. motion detector with false alarm immunity |
| US5265172A (en) * | 1989-10-13 | 1993-11-23 | Texas Instruments Incorporated | Method and apparatus for producing optical flow using multi-spectral images |
| JP2827424B2 (en) * | 1990-03-31 | 1998-11-25 | ソニー株式会社 | Image stabilization device |
| IT1245060B (en) * | 1991-04-12 | 1994-09-13 | Firestone Int Dev Spa | VULCANIZATION UNIT FOR TIRES |
| US5257325A (en) * | 1991-12-11 | 1993-10-26 | International Business Machines Corporation | Electronic parallel raster dual image registration device |
| US5309237A (en) * | 1992-03-31 | 1994-05-03 | Siemens Corporate Research, Inc. | Apparatus and method of compensating image-sequences for motion |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US3632865A (en) * | 1969-12-23 | 1972-01-04 | Bell Telephone Labor Inc | Predictive video encoding using measured subject velocity |
| JPS5739597B2 (en) * | 1974-04-08 | 1982-08-21 | ||
| US3890462A (en) * | 1974-04-17 | 1975-06-17 | Bell Telephone Labor Inc | Speed and direction indicator for video systems |
| JPS5851471B2 (en) * | 1975-04-09 | 1983-11-16 | 日本電気株式会社 | TV synopsis |
| JPS54127623A (en) * | 1978-03-27 | 1979-10-03 | Nec Corp | S/n improving device for television video signal |
| US4193092A (en) * | 1978-06-21 | 1980-03-11 | Xerox Corporation | Image interpolation system |
| US4383272A (en) * | 1981-04-13 | 1983-05-10 | Bell Telephone Laboratories, Incorporated | Video signal interpolation using motion estimation |
| US4437161A (en) * | 1981-06-29 | 1984-03-13 | Siemens Gammasonics Inc. | Medical imaging apparatus |
-
1984
- 1984-03-05 DE DE3408016A patent/DE3408016A1/en not_active Withdrawn
-
1985
- 1985-01-18 EP EP85100478A patent/EP0154126B1/en not_active Expired - Lifetime
- 1985-01-18 DE DE8585100478T patent/DE3579012D1/en not_active Expired - Lifetime
- 1985-03-04 CA CA000475675A patent/CA1229157A/en not_active Expired
- 1985-03-05 JP JP60042044A patent/JP2509174B2/en not_active Expired - Fee Related
- 1985-03-05 US US06/708,457 patent/US4630114A/en not_active Expired - Lifetime
Also Published As
| Publication number | Publication date |
|---|---|
| DE3579012D1 (en) | 1990-09-13 |
| DE3408016A1 (en) | 1985-09-12 |
| US4630114A (en) | 1986-12-16 |
| JP2509174B2 (en) | 1996-06-19 |
| JPS60244171A (en) | 1985-12-04 |
| EP0154126A2 (en) | 1985-09-11 |
| EP0154126B1 (en) | 1990-08-08 |
| EP0154126A3 (en) | 1987-05-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US4651207A (en) | Motion adaptive interpolation of television image sequences | |
| US5111511A (en) | Image motion vector detecting apparatus | |
| CA1229157A (en) | Method for determining the displacement of image sequences and arrangement and use thereof | |
| US7145607B1 (en) | Spatio-temporal video noise reduction system | |
| US4890160A (en) | TV picture motion vector measurement by correlation of pictures | |
| De Haan et al. | True-motion estimation with 3-D recursive search block matching | |
| US6445833B1 (en) | Device and method for converting two-dimensional video into three-dimensional video | |
| US6215899B1 (en) | Motion and disparity estimation method, image synthesis method, and apparatus for implementing same methods | |
| CA2227390C (en) | Improved correlation processing for motion estimation | |
| US8358878B2 (en) | Method and apparatus for interpolating an image | |
| EP1230805B1 (en) | Video signal noise level estimator | |
| JPS60229594A (en) | Method and device for motion interpolation of motion picture signal | |
| JPS63313981A (en) | Digital television image motion vector processor | |
| JPH06500225A (en) | video image processing | |
| EP1865711A2 (en) | Noise suppression method, noise suppression method program, recording medium recording noise suppression method program, and noise suppression apparatus | |
| JP2000102041A (en) | Method for measuring image quality | |
| KR970002964B1 (en) | A filter of time interpolation for image signal | |
| JPH1117954A (en) | Video signal processing unit | |
| GB2202706A (en) | Video signal processing | |
| JPH07240858A (en) | Digital signal filter for image, image filter and method of filtering image for film | |
| Hang et al. | Motion estimation for image sequence compression | |
| GB2514557A (en) | Image processing | |
| JP3176652B2 (en) | Image distortion correction device, inter-image calculation device, image motion calculation device | |
| KR930002332B1 (en) | Apparatus for detection of image movement | |
| KR100230308B1 (en) | Apparatus for interpolation with perturbation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| MKEX | Expiry |