|Publication number||USRE39455 E1|
|Application number||US 09/691,857|
|Publication date||Jan 2, 2007|
|Filing date||Oct 18, 2000|
|Priority date||Dec 27, 1995|
|Also published as||EP0782343A2, EP0782343A3, US5825421|
|Publication number||09691857, 691857, US RE39455 E1, US RE39455E1, US-E1-RE39455, USRE39455 E1, USRE39455E1|
|Inventors||Thiow Keng Tan|
|Original Assignee||Matsushita Electric Industrial Co., Ltd.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (18), Non-Patent Citations (11), Referenced by (6), Classifications (30), Legal Events (1)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This application is a divisional reissue application of U.S. Pat. No. 5,825,421, issued Oct. 20, 1998. This application also has a related divisional reissue application Ser. No. 09/691,858, filed on Oct. 18, 2000.
1. Field of the Invention
This invention can be used in low bit rate video coding for tele-communicative applications. It improves the temporal frame rate of the decoder output as well as the overall picture quality.
2. Related art of the Invention
In a typical hybrid transform coding algorithm such as the ITU-T Recommendation H.261  and MPEG  motion compensation is used to reduce the amount of temporal redundancy in the sequence. In the H.261 coding scheme, the frames are coded using only forward prediction, hereafter referred to as P-frames. In the MPEG coding scheme, some frames are coded using bi-direction prediction, hereafter referred to as B-frames. B-frames improve the efficiency of the coding scheme. Now the  is ITU-T Recommendation H.261 (Formerly CCITT Recommendation H.261) Codes for audiovisual services at p×64 kbit/s Geneva, 1990 , and the  is ISO/IEC 11172-2 1993 , Information technology—Coding of moving pictures and associated audio for digital storage media at up to about 1,5 Mbit/s - Part 2: Video.
However, it introduces delay in the encoding and decoding, making it unsuitable for applications in the communicative services where delay is an important parameter.
In the PB-block only the motion vectors for the P-block is transmitted to the decoder. The forward and backward motion vectors for the B-block is derived from the P motion vectors. A linear motion model is used and the temporal reference of the B and P frame is used to scale the motion vector appropriately.
Currently the method used in the prior art assumes a linear motion model. However this assumption is not valid in a normal scene where the motion is typically not linear. This is especially true when the camera shakes and when objects are not moving at constant velocities.
A second problem involves the quantization and transmission of the residual of the prediction error in the B-block Currently the coefficients from the P-block and the B-block are interleaved in some scanning order which requires the B-block efficients to be transmitted even when they are all zero. This is not very efficient as it is quite often that there are no residual coefficients to transmit (all coefficients are zero).
In order to solve the first problem, the current invention employs a delta motion vector to compensate for the non-linear motion. Thus it becomes necessary for the encoder to perform an additional motion search to obtain the optimum delta motion vector that when added to the derived motion vectors would result in the best match in the prediction This delta motion vectors are transmitted to the decoder at the block level only when necessary. A flag is used to indicate to the decoder if there are delta motion vectors present for the B-block.
For the second problem, this invention also uses a flag to indicate if there are coefficients for the B-block to be decoded.
The operation of the Invention is described as follows.
In the current invention the problem is solved by adding a small delta motion vector to the derived motion vector to compensate for the difference between the derived and true motion vector. Therefore the equations in (1) and (2) are now replaced by equations (3) and (4), respectively.
The preferred embodiment of the current invention is described here.
The encoding functionality block diagram depicts an encoder using a motion estimation and compensation for reducing the temporal redundancy in the sequence to be coded. The input sequences is organized into a first frame and pairs of subsequent frames. The first frame, hereafter referred to as the I-frame, is coded independent of all other frames. The pairs of subsequent frames, hereafter referred to as PB-frame, consist of a B-frame followed by a P-frame. The P-frame is forward predicted based on the previously reconstructed I-frame or P-frame and the B-frame is bi-directionally predicted based on the previously reconstructed I-frame or P-frame and the information in the current P-frame.
The input frame image sequence, 1, is placed in the Frame Memory 2. If the frame is classified as an I-frame or a P-frame it is passed through line 14 to the Reference Memory 3, for use as the reference frame in the motion estimation of the next PB-frame to be predictively encoded. The signal is then passed through line 13 to the Block Sampling module 4, where it is partitioned into spatially non-overlapping blocks of pixel data for further processing.
If the frame is classified as an I-frame, the sampled blocks are passed through line 16 to the DCT module 7. If the frame is classified as a PB-frame, the sampled blocks are passed through line 17 to the Motion Estimation module 5. The Motion Estimation module 5 uses information from the Reference Frame Memory 3 and the current block 17 to obtain the motion vector for that provides the best match for the P-block The motion vector and the local reconstructed frame, 12, are passed through line 19 and 20, respectively, to the Motion Compensation module 6. The difference image is formed by subtracting the motion compensated decoded frame, 21, from the current P-block, 15. This signal is then passed through line 22 to the DCT module 7.
In the DCT module 7, each block is transformed into the DCT domain coefficients. The transform coefficients are passed through line 23 to the Quantization module 8, where they are quantized. The quantized coefficients are then passed through line 24 to the Run-length & Variable Length Coding module 9. Here the coefficients are entropy coded to form the Output Bit Stream, 25.
If the current block is an I-block or a P-block, the quantized coefficients are also passed through line 26 to the Inverse Quantization module 10. The output of the Inverse Quantization 10, is then passed through line 27 to the Inverse DCT module 11. If the current block is an I-block then the reconstructed block is placed, via line 28, in the Local Decoded Frame Memory 12. If the current block is a P-block then the output of the Inverse DCT 29 is added to the motion compensated output 21, to from the reconstructed block 30. The reconstructed block 30, is then placed in the Local Decoded Frame Memory 12, for the motion compensation of the subsequent frames.
After the P-block have been locally reconstructed, the information is passed again to the Motion Compensation Module 6, where the prediction of the B-block is formed.
The prediction difference block is then passed through line 22 to the DCT module 7. The DCT coefficients are then passed through line 23 to the Quantization module 8. The result of the Quantization module 8, is passed through line 24 to the Run-length & Variable Length Coding 9. In this module the presence of the delta motion vector and the quantized residual error in the Output Bitstream 25, is indicated a variable length code, NOB which is the acronym for No B-block This flag is generated in Run-length & Variable Length Coding module 9 based on whether there are residual error in the Quantization module 8 and delta motion vectors found in the Delta Motion Search module 54 is not zero. Table 1 provides the preferred embodiment of the variable length code for the NOB flag. The variable length code of the NOB flag is inserted in the Output Bitstream, 25, prior to the delta motion vector and quantized residual error codes.
(Variable length code for the NOB flag)
If the current frame is an I-frame then the output of Inverse DCT 34, is passed through line 39 and stored in the Frame Memory 42.
If the current frame is a PB-frame, the side information containing the motion vectors are passed through line 45 to the Motion compensation module 36. The motion Compensation module 36, uses this information and the information in the Local Decoded Memory, 35, to from the motion compensated signal, 44. This signal is then added to the output of the Inverse DCT module 34, to form the reconstruction of the P-block.
The Motion Compensation module 36, then uses the additional information obtained in the reconstructed P-block to obtain the bi-directional prediction for the B-block. The B-block is then reconstructed and placed in the Frame Memory, 42, together with the P-block.
By implementing this invention, the temporal frame rate of the decoded sequences can be effectively doubled at a fraction of the expected cost in bit rate. The delay is similar to that of the same sequence decoded at half the frame rate.
As described above in the present invention a new predictive coding is used to increase the temporal frame rate and coding efficiency without introducing excessive delay. Currently the motion vector for the blocks in the bi-directionally predicted frame is derived from the motion vector of the corresponding block in the forward predicted frame using a linear motion model. This however is not effective when the motion in the image sequence is not linear. According to this invention, the efficiency of this method can be further improved if a non-linear motion model is used. In this model a delta motion vector is added to or subtracted from the derived forward and backward motion vector, respectively. The encoder performs an additional search to determine if there is a need for the delta motion vector. The presence of this delta motion vector in the transmitted bitstream is signalled to the decoder which then takes the appropriate action to make use of the delta motion vector to derive the effective forward and backward motion vectors for the bi-directionally predicted block.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US5136378 *||Aug 15, 1991||Aug 4, 1992||Matsushita Electric Industrial Co., Ltd.||Moving picture coding apparatus|
|US5144426 *||Oct 12, 1990||Sep 1, 1992||Matsushita Electric Industrial Co., Ltd.||Motion compensated prediction interframe coding system|
|US5150432 *||Mar 22, 1991||Sep 22, 1992||Kabushiki Kaisha Toshiba||Apparatus for encoding/decoding video signals to improve quality of a specific region|
|US5155593 *||Sep 27, 1990||Oct 13, 1992||Sony Corporation||Video signal coding method|
|US5267334 *||Jan 21, 1993||Nov 30, 1993||Apple Computer, Inc.||Encoding/decoding moving images with forward and backward keyframes for forward and reverse display|
|US5293229 *||Mar 27, 1992||Mar 8, 1994||Matsushita Electric Corporation Of America||Apparatus and method for processing groups of fields in a video data compression system|
|US5315326 *||Apr 24, 1992||May 24, 1994||Victor Company Of Japan, Ltd.||Efficient coding/decoding apparatuses for processing digital image signal|
|US5361105 *||Mar 5, 1993||Nov 1, 1994||Matsushita Electric Corporation Of America||Noise reduction system using multi-frame motion estimation, outlier rejection and trajectory correction|
|US5386234 *||Nov 5, 1992||Jan 31, 1995||Sony Corporation||Interframe motion predicting method and picture signal coding/decoding apparatus|
|US5412428 *||Dec 9, 1993||May 2, 1995||Sony Corporation||Encoding method and decoding method of color signal component of picture signal having plurality resolutions|
|US5436665||Nov 19, 1993||Jul 25, 1995||Kabushiki Kaisha Toshiba||Motion picture coding apparatus|
|US5467136||Feb 17, 1994||Nov 14, 1995||Kabushiki Kaisha Toshiba||Video decoder for determining a motion vector from a scaled vector and a difference vector|
|US5481310 *||Apr 22, 1994||Jan 2, 1996||Sharp Kabushiki Kaisha||Image encoding apparatus|
|US5905534 *||Jul 12, 1994||May 18, 1999||Sony Corporation||Picture decoding and encoding method and apparatus for controlling processing speeds|
|US6104753 *||Feb 3, 1997||Aug 15, 2000||Lg Electronics Inc.||Device and method for decoding HDTV video|
|US6184935 *||Mar 11, 1998||Feb 6, 2001||Matsushita Electric Industrial, Co. Ltd.||Upsampling filter and half-pixel generator for an HDTV downconversion system|
|US6219383 *||Jun 8, 1998||Apr 17, 2001||Daewoo Electronics Co., Ltd.||Method and apparatus for selectively detecting motion vectors of a wavelet transformed video signal|
|EP0651574A1||Mar 24, 1994||May 3, 1995||Sony Corporation||Method and apparatus for coding/decoding motion vector, and method and apparatus for coding/decoding image signal|
|1||*||"Recommendation H.261-Video Codec for Audiovisual at px 64 kbit/s", International Telegraph and Telephone Consultative Committee, Study Group XV-Report R 37, Aug. 1990.|
|2||"Transmission of Non-Telephone Signals Information Technology-Generic Coding of Moving Pictures and Associated Audio Information: Video", ITU-T Telecommunication Standardization Sector of ITU, XX, XX, Jul. 1, 1995, pp. A-B, I-VIII, 1, XP000198491.|
|3||A. Nagata, "Moving Picture Coding System for Digital Storage Media Using Hybrid Coding", vol. 2, No. 2 Aug. 1, 1990, pp. 109-116, XP000243471.|
|4||European Search Report dated Nov. 8, 2000, application No. EP 96120920.|
|5||*||Han et al., Efficient encoding of dense motion fields of motion-compensated compression, Image Processing, 1999. ICIP 99. Proceedings 1999 International Conference vol. 1, 1999, pp. 84-88.|
|6||K. Rijkae, "H-263: Video Coding For Low-Bit-Rate Communication", IEEE Communications Magazine vol. 34, No. 12, Dec. 1, 1996, pp. 42-45, XP000636452.|
|7||K. Rijkse, "ITU Standardization of Very Low Bitrate Video Coding Algorithms", vol. 7, No. 4, pp. 553-565, XP004047099, Nov. 1, 1995.|
|8||Kozu et al., "A new Technique for Block-Based Motion Compensation", pp. V/217-20, vol. 5, XP002151093, Apr. 1, 1994.|
|9||*||Secretariat: Japan (JISC), "Coded Representation of Audio, Picture, Multimedia and Hypermedia Information," ISO/IEC JTC 1/SC 29 N 313, dated May 20, 1993.|
|10||W. Lynch, "Bidirectional Motion Estimation Based On P Frame Motion Vectors and Area Overlap", vol. Conf. 17, Mar. 23, 1992, pp. 445-448, XP000378964.|
|11||*||Yo, Y et al., Adaptive quantization without side information using scalar-vector quantization and trellis quantization, Signals, Systems and Computers, 1995. Conference Record of the Twenty-Nine Asilomar Conference on, vol. 2, 1996, pp. 1398-1402.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7848410 *||Dec 7, 2010||Institute For Information Industry||Video decoding methods and devices|
|US8879631||Nov 21, 2008||Nov 4, 2014||Dolby Laboratories Licensing Corporation||Temporally smoothing a motion estimate|
|US9232230 *||Mar 21, 2012||Jan 5, 2016||Vixs Systems, Inc.||Method and device to identify motion vector candidates using a scaled motion search|
|US20060083306 *||Dec 21, 2004||Apr 20, 2006||Wen-Hao Hsu||Video decoding methods and devices|
|US20100321583 *||Nov 21, 2008||Dec 23, 2010||Jerome Shields||Temporally Smoothing a Motion Estimate|
|US20130251024 *||Mar 21, 2012||Sep 26, 2013||Vixs Systems, Inc.||Method and device to identify motion vector candidates using a scaled motion search|
|U.S. Classification||375/240.15, 348/699|
|International Classification||H04N19/523, H04N19/102, H04N19/57, H04N19/61, H04N19/51, H04N19/423, H04N19/463, H04N19/625, H04N19/36, H04N19/91, H04N19/517, H04N19/70, H04N19/159, H04N19/46, H04N19/577, H04N19/93, H04N19/176, H04N19/85, H04N19/50, H04N19/513, H04N19/503, H03M7/36, G06T9/00, H04B1/66|
|Cooperative Classification||H04N19/51, H04N19/577|
|European Classification||H04N7/26M2, H04N7/46E|