|Publication number||USRE43062 E1|
|Application number||US 12/553,836|
|Publication date||Jan 3, 2012|
|Filing date||Sep 3, 2009|
|Priority date||Mar 31, 2003|
|Also published as||CN1826812A, CN1826812B, CN101854552A, CN101854553A, CN101854553B, DE602004012540D1, DE602004012540T2, DE602004027847D1, EP1611747A1, EP1611747A4, EP1611747B1, EP1791369A2, EP1791369A3, EP1791369B1, EP2209319A2, EP2209319A3, EP2209319B1, US7266147, US20040190606, WO2004088988A1|
|Publication number||12553836, 553836, US RE43062 E1, US RE43062E1, US-E1-RE43062, USRE43062 E1, USRE43062E1|
|Inventors||Sachin Govind Deshpande|
|Original Assignee||Sharp Kabushiki Kaisha|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (26), Non-Patent Citations (21), Classifications (10), Legal Events (1)|
|External Links: USPTO, USPTO Assignment, Espacenet|
The present invention relates to a hypothetical reference decoder.
A digital video system includes a transmitter and a receiver which assemble video comprising audio, images, and ancillary components for coordinated presentation to a user. The transmitter system includes subsystems to receive and compress the digital source data (the elementary or application data streams representing a program's audio, video, and ancillary data components); multiplex the data from the several elementary data streams into a single transport bit stream; and transmit the data to the receiver. At the receiver the transport bit stream is demultiplexed into its constituent elementary data streams. The elementary data streams are decoded and the audio and video data streams are delivered as synchronized program elements to the receiver's presentation subsystem for display as parts of a coordinated program.
In many video coding standards, a compliant bit stream to the decoder is decoded by a hypothetical decoder that is conceptually connected to the output of an encoder and consists of a decoder buffer, a decoder, and a display unit. This virtual decoder is known as the hypothetical reference decoder (HRD) in H.263 and the video buffering verifier (VBV) in MPEG-2. The encoder creates a bit stream so that the hypothetical decoder buffer does not overflow or underflow.
As a result, the quantity of data the receiver may be required to buffer might exceed its capacity (a condition of memory overflow) or throughput capabilities. Alternatively, the receiver may fail to receive all of the data in a data access unit in time for decoding and synchronized presentation with a specified instant in the audio or video data streams resulting in a loss of data and inconsistent performance (a condition of memory underflow).
In existing hypothetical reference decoders, the video bit stream is received at a given constant bit rate (usually the average rate in bits/sec of the stream) and is stored into the decoder buffer until the buffer fullness reaches a desired level. Such a desired level is denoted as the initial decoder buffer fullness and is directly proportional to the transmission or start-up (buffer) delay. At that point, the decoder instantaneously removes the bits for the first video frame of the sequence, decodes the bits, and displays the frame. The bits for the following frames are also removed, decoded, and displayed instantaneously at subsequent time intervals.
Traditional hypothetical decoders operate at a fixed bit rate, buffer size, and initial delay. However, in many of today's video applications (e.g., video streaming through the Internet or ATM networks) the available bandwidth varies according to the network path (e.g., how the user connects to the network: by modem, ISDN, DSL, cable, etc.) and also fluctuates in time according to network conditions (e.g., congestion, the number of users connected, etc.). In addition, the video bit streams are delivered to a variety of devices with different buffer capabilities (e.g., hand-sets, PDAs, PCs, Set-top-boxes, DVD-like players, etc.) and are created for scenarios with different delay requirements (e.g., low-delay streaming, progressive download, etc.). As a result, these applications require a more flexible hypothetical reference decoder that can decode a bit stream at different peak bit rates, and with different buffer sizes and start-up delays.
Jordi Ribas-Corbera and Philip A. Chou in a paper entitled, “A Generalized Hypothetical Reference Decoder For H.26L”, on Sep. 4, 2001, proposed a modified hypothetical reference decoder. The decoder operates according to N sets of rate and buffer parameters for a given bit stream. Each set characterizes what is known as a leaky bucket model and contains three values (R, B, F), where R is the transmission bit rate, B is the buffer size, and F is the initial decoder buffer fullness (F/R is the start-up or initial buffer delay). An encoder can create a video bit stream that is contained by some desired N leaky buckets, or can simply compute the N sets of parameters after the bit stream has been generated. The hypothetical reference decoder may interpolate among the leaky bucket parameters and can operate at any desired peak bit rate, buffer size, or delay. For example, given a peak transmission rate R′, the reference decoder may select the smallest buffer size and delay (according to the available leaky bucket data) that will be able to decode the bit stream without suffering from buffer underflow or overflow. Conversely, for a given buffer size B′, the hypothetical decoder may select and operate at the minimum required peak transmission rate.
There are benefits of using such a generalized hypothetical reference decoder. For example, a content provider can create a bit stream once, and a server can deliver it to multiple devices of different capabilities, using a variety of channels of different peak transmission rates. Or a server and a terminal can negotiate the best leaky bucket for the given networking conditions—e.g., the ones that will produce the lowest start-up (buffer) delay, or the one that will require the lowest peak transmission rate for the given buffer size of the device.
As described in Document VCEG-58 Sections 2.1-2.4, a leaky bucket is a model for the state (or fullness) of an encoder or decoder buffer as a function of time. The fullness of the encoder and the decoder buffer are complements of each other. A leaky bucket model is characterized by three parameters (R, B, F), where:
In a leaky bucket model, the bits enter the buffer at rate R until the level of fullness is F (i.e., for D seconds), and then b0 bits for the first frame are instantaneously removed. The bits keep entering the buffer at rate R and the decoder removes b1, b2, . . . , bn−1 bits for the following frames at some given time instants, typically (but not necessarily) every 1/M seconds, where M is the frame rate of the video.
Let Bi be the decoder buffer fullness immediately before removing bi bits at time ti. A generic leaky bucket model operates according to the following equations:
Bi+1=min (B, Bi−bi+R(ti+1−ti)), i=0, 1, 2, . . . (1)
Typically, ti+1−ti=1/M seconds, where M is the frame rate (normally in frames/sec) for the bit stream.
A leaky bucket model with parameters (R, B, F) contains a bit stream if there is no underflow of the decoder buffer. Because the encoder and decoder buffer fullness are complements of each other this is equivalent to no overflow of the encoder buffer. However, the encoder buffer (the leaky bucket) is allowed to become empty, or equivalently the decoder buffer may become full, at which point no further bits are transmitted from the encoder buffer to the decoder buffer. Thus, the decoder buffer stops receiving bits when it is full, which is why the min operator in equation (1) is included. A full decoder buffer simply means that the encoder buffer is empty.
The following observations may be made:
Assume that the system fixes F=aB for all leaky buckets, where a is some desired fraction of the initial buffer fullness. For each value of the peak bit rate R, the system can find the minimum buffer size Bmin that will contain the bit stream using equation (1). The plot of the curve of R-B values, is shown in
By observation, the curve of (Rmin, Bmin) pairs for any bit stream (such as the one in
MPEG Video Buffering Verifier (VBV)
The MPEG video buffering verifier (VBV) can operate in two modes: constant bit rate (CBR) and variable bit rate (VBR). MPEG-1 only supports the CBR mode, while MPEG-2 supports both modes.
The VBV operates in CBR mode when the bit stream is contained in a leaky bucket model of parameters (R, B, F) and:
R=Rmax=the average bit rate of the stream.
The VBV operates in VBR mode when the bit stream is constrained in a leaky bucket model of parameters (R, B, F) and:
R=Rmax=the peak or maximum rate. Rmax is higher than the average rate of the bit stream.
The decoder buffer fullness follows the following equations:
Bi+1=min (B, Bi−bi+Rmax/M), i=0, 1, 2, . . . (3)
The encoder ensures that Bi−bi is always greater than or equal to zero. That is, the encoder must ensure that the decoder buffer does not underflow. However, in this VBR case the encoder does not need to ensure that the decoder buffer does not overflow. If the decoder buffer becomes full, then it is assumed that the encoder buffer is empty and hence no further bits are transmitted from the encoder buffer to the decoder buffer.
The VBR mode is useful for devices that can read data up to the peak rate Rmax. For example, a DVD includes VBR clips where Rmax is about 10 Mbits/sec, which corresponds to the maximum reading speed of the disk drive, even though the average rate of the DVD video stream is only about 4 Mbits/sec.
Broadly speaking, the CBR mode can be considered a special case of VBR where Rmax happens to be the average rate of the clip.
H.263's Hypothetical Reference Decoder (HRD)
The hypothetic reference model for H.263 is similar to the CBR mode of MPEG's VBV previously discussed, except for the following:
Previously existing hypothetical reference decoders operate at only one point (R, B) of the curve in
A generalized hypothetical reference decoder (GHRD) can operate given the information of N leaky bucket models,
(R1, B1, F1), (R2, B2, F2), . . . , (RN, BN, RN), (4)
each of which contains the bit stream. Without loss of generality, let us assume that these leaky buckets are ordered from smallest to largest bit rate, i.e., Ri<Ri+1. Lets also assume that the encoder computes these leaky buckets models correctly and hence Bi<Bi+1.
The desired value of N can be selected by the encoder. If N=1, the GHRD is essentially equivalent to MPEG's VBV. The encoder can choose to: (a) pre-select the leaky bucket values and encode the bit stream with a rate control that makes sure that all of the leaky bucket constraints are met, (b) encode the bit stream and then use equation (1) to compute a set of leaky buckets containing the bit stream at N different values of R, or (c) do both. The first approach (a) can be applied to live or on-demand transmission, while (b) and (c) only apply to on-demand.
The number of leaky buckets N and the leaky bucket parameters (4) are inserted into the bit stream. In this way, the decoder can determine which leaky bucket it wishes to use, knowing the peak bit rate available to it and/or its physical buffer size. The leaky bucket models in (4) as well as all the linearly interpolated or extrapolated models are available for use.
The interpolated buffer size B between points k and k+1 follow the straight line:
Likewise, the initial decoder buffer fullness F can be linearly interpolated:
The resulting leaky bucket with parameters (R, B, F) contains the bit stream, because the minimum buffer size Bmin is convex in both R and F, that is, the minimum buffer size Bmin corresponding to any convex combination (R, F)=a(Rk, Fk)+(1−a)(Rk+1, Fk+1), 0<a<1, is less than or equal to B=aBk+(1−a)Bk+1.
It is observed that if R is larger than RN, the leaky bucket (R, BN, FN) will also contain the bit stream, and hence BN and FN are the buffer size and initial decoder buffer fullness recommended when R>=RN. If R is smaller than R., the upper bound B=B1+(R1−R)T can be caused (and once can set F=B), where T is the time length of the stream in seconds. These (R, B) values outside the range of the N points are also shown in
The Joint Video Team of ISO/IEC MPEG and ITU-T VCEG Working Draft Number 2, Revision 0 (WD-2) incorporated many of the concepts of the hypothetical reference decoder proposed by Jordi Ribas-Cobera, et al. of Microsoft Corporation, incorporated by reference herein. The WD-2 document is similar to the decoder proposed by Jordi Ribas-Cobera, et al. of Microsoft Corporation, though the syntax is somewhat modified. In addition, WD-2 describes an example algorithm to compute B, and F for a given rate R.
As previously described, the JVT standard (WD-2) allows the storing of (N>=1) leaky buckets, (R1, B1, F1), . . . , (RN, BN, FN) values which are contained in the bit stream. These values may be stored in the header. Using Fi as the initial buffer fullness and Bi as the buffer size, guarantees that the decoder buffer will not underflow when the input stream comes in at the rate Ri. This will be the case if the user desires to present the encoded video from start to end. In a typical video-on-demand application the user may want to seek to different portions of the video stream. The point that the user desires to seek to may be referred to as the access point. During the process of receiving video data and constructing video frames the amount of data in the buffer fluctuates. After consideration, the present inventor came to the realization that if the Fi value of the initial buffer fullness (when the channel rate is Ri) is used before starting to decode the video from the access point, then it is possible that the decoder will have an underflow. For example, at the access point or sometime thereafter the amount of bits necessary for video reconstruction may be greater than the bits currently in the buffer, resulting in underflow and inability to present video frames in a timely manner. It can likewise be shown that in a video stream the value of initial buffer fullness required to make sure there in no underflow at the decoder varies based on the point at which the user seeks to. This value is bounded by the Bi. Accordingly, the combination of B and F provided for the entire video sequence, if used for an intermediate point in the video will not likely be appropriate, resulting in an underflow, and thus freezing frames.
Based upon this previously unrealized underflow potential, the present inventor then came to the realization that if only a set of R, B, and F values are defined for an entire video segment, then the system should wait until the buffer B for the corresponding rate R is full or substantially full (or greater than 90% full) to start decoding frames when a user jumps to an access point. In this manner, the initial fullness of the buffer will be at a maximum and thus there is no potential of underflow during subsequent decoding starting from the access point. This may be achieved without any additional changes to the existing bit stream, thus not impacting existing systems. Accordingly, the decoder would use the value of initial buffering Bj for any point the user seeks to when the rate is Rj, as shown in
The initial buffer fullness (F) may likewise be characterized as a delay until the video sequence is presented (e.g., initial_cpb_removal_delay). The delay is temporal in nature being related to the time necessary to achieve initial buffer fullness (F). The delay and/or F may be associated with the entire video or the access points. It is likewise to be understood that delay may be substituted for F in all embodiments described herein (e.g., (R,B,delay)). One particular value for the delay may be calculated as delay=F/R, using a special time unit (units of 90 KHz Clock).
To reduce the potential delay the present inventor came to the realization that sets of (R, B, F) may be defined for a particular video stream at each access point. Referring to
The sets of R, B, F values for each access point may be located at any suitable location, such as for example, at the start of the video sequence together with sets of (R, B, F) values for the entire video stream or before each access point which avoids the need for an index; or stored in a manner external to the video stream itself which is especially suitable for a server/client environment.
This technique may be characterized by the following model:
(R1, B1, F1, M1, f11, t11, . . . , fM11, tM11) . . . , (RN, BN, FN, MN, f1N, t1N, . . . , fMNN, tMNN),
where fkj denotes the initial buffer fullness value at rate Rj at access point tkj (time stamp). The values of Mj may be provided as an input parameter or may be automatically selected.
For example, Mj may include the following options:
The system may, for a given Rj, use an initial buffer fullness equal to fjk if the user seeks an access point tkj. This occurs when the user selects to start at an access point, or otherwise the system adjusts the user's selection to one of the access points.
It is noted that in the case that a variable bit rate (in bit stream) is used the initial buffer fullness value (or delay) is preferably different than the buffer size, albeit it may be the same. In the case of variable bit rate in MPEG-2 VBV buffer is filled till it is full, i.e. F=B (value of B is represented by vbv_buffer_size).
If the system permits the user to jump to any frame of the video in the manner of an access point, then the decoding data set would need to be provided for each and every frame. While permissible, the resulting data set would be excessively large and consume a significant amount of the bitrate available for the data. A more reasonable approach would be to limit the user to specific access points within the video stream, such as every second, 10 seconds, 1 minute, etc. While an improvement, the resulting data set may still be somewhat extensive resulting in excessive data for limited bandwidth devices, such as mobile communication devices.
In the event that the user selects a position that is not one of the access points with an associated data set, then the initial buffer fullness may be equal to max(fkj, f(k+1)j) for a time between tkj and t(k+1)j, especially if the access points are properly selected. In this manner, the system is guaranteed of having a set of values that will be free from resulting in an underflow condition, or otherwise reduce the likelihood of an underflow condition, as explained below.
To select a set of values that will ensure no underflow condition (or otherwise reduce) when the above-referenced selection criteria is used, reference is made to
Based upon the selection criteria a set of 10 points for
In addition, if the bit rate and the buffer size remain the same while selecting a different access point, then merely the modified buffer fullness, F, needs to be provided or otherwise determined.
All the references cited herein are incorporated by reference.
The terms and expressions that have been employed in the foregoing specification are used as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding equivalents of the features shown and described or portions thereof, it being recognized that the scope of the invention is defined and limited only by the claims that follow.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US5159447||May 23, 1991||Oct 27, 1992||At&T Bell Laboratories||Buffer control for variable bit-rate channel|
|US5365552 *||Nov 16, 1992||Nov 15, 1994||Intel Corporation||Buffer fullness indicator|
|US5398072 *||Oct 25, 1993||Mar 14, 1995||Lsi Logic Corporation||Management of channel buffer in video decoders|
|US5481543||Jun 10, 1994||Jan 2, 1996||Sony Corporation||Rational input buffer arrangements for auxiliary information in video and audio signal processing systems|
|US5534944 *||Jul 21, 1995||Jul 9, 1996||Matsushita Electric Corporation Of America||Method of splicing MPEG encoded video|
|US5537408||Jun 5, 1995||Jul 16, 1996||International Business Machines Corporation||apparatus and method for segmentation and time synchronization of the transmission of multimedia data|
|US5619341||Feb 23, 1995||Apr 8, 1997||Motorola, Inc.||Method and apparatus for preventing overflow and underflow of an encoder buffer in a video compression system|
|US5629736||Nov 1, 1994||May 13, 1997||Lucent Technologies Inc.||Coded domain picture composition for multimedia communications systems|
|US5652749||Jul 25, 1996||Jul 29, 1997||International Business Machines Corporation||Apparatus and method for segmentation and time synchronization of the transmission of a multiple program multimedia data stream|
|US5831688||Jun 5, 1995||Nov 3, 1998||Mitsubishi Denki Kabushiki Kaisha||Image coded data re-encoding apparatus|
|US5877812 *||Nov 21, 1995||Mar 2, 1999||Imedia Corporation||Method and apparatus for increasing channel utilization for digital video transmission|
|US5995151||Sep 18, 1997||Nov 30, 1999||Tektronix, Inc.||Bit rate control mechanism for digital image and video data compression|
|US6272566||Nov 18, 1998||Aug 7, 2001||International Business Machines Corporation||System for maintaining proper buffering within video play list|
|US6389072||Dec 23, 1998||May 14, 2002||U.S. Philips Corp.||Motion analysis based buffer regulation scheme|
|US6542549||Oct 12, 1999||Apr 1, 2003||Matsushita Electric Industrial Co., Ltd.||Method and model for regulating the computational and memory requirements of a compressed bitstream in a video decoder|
|US6637031 *||Dec 4, 1998||Oct 21, 2003||Microsoft Corporation||Multimedia presentation latency minimization|
|US6909743||Oct 29, 1999||Jun 21, 2005||Sarnoff Corporation||Method for generating and processing transition streams|
|US6912251||Jul 2, 1999||Jun 28, 2005||Sarnoff Corporation||Frame-accurate seamless splicing of information streams|
|US20020037161||Sep 25, 2001||Mar 28, 2002||Takayuki Sugahara||MPEG picture data recording apparatus, MPEG picture data recording method, MPEG picture data recording medium, MPEG picture data generating apparatus, MPEG picture data reproducing apparatus, and MPEG picture data reproducing method|
|US20030053416||Sep 19, 2001||Mar 20, 2003||Microsoft Corporation||Generalized reference decoder for image or video processing|
|EP0930786A2||Jan 6, 1999||Jul 21, 1999||Sony Corporation||Compressed picture data editing apparatus and method|
|JP2000124958A||Title not available|
|JP2002112183A||Title not available|
|JP2003092752A||Title not available|
|JPH02272851A||Title not available|
|JPH07107429A||Title not available|
|1||"Annex C-Video Buffering Verifier, Information Technology-Generic coding of moving pictures and associated audio information: Video," ITU-T Recommendation H.262, Feb. 2000, pp. 1, 138-142, XP 002 248 658.|
|2||"Working Draft No. 2, Revision 3 (WD-2r3)," Joint Video Team (JVT) of ISO/IEC MPEG and ITU-T VCEG, Gary Sullivan, One Microsoft Way, Redmond, WA 98052 USA, Mar. 25, 2002, Dovument JVT-B118r3, 99 pages.|
|3||"Annex C—Video Buffering Verifier, Information Technology—Generic coding of moving pictures and associated audio information: Video," ITU-T Recommendation H.262, Feb. 2000, pp. 1, 138-142, XP 002 248 658.|
|4||Annex B-"Hypothetical Reference Decoder, Video Coding for Low Bit Rate Communication," ITU-T Recommendation H.263, Feb. 1998, pp. 1, 49-50, XP 002 248 657.|
|5||Annex B—"Hypothetical Reference Decoder, Video Coding for Low Bit Rate Communication," ITU-T Recommendation H.263, Feb. 1998, pp. 1, 49-50, XP 002 248 657.|
|6||Annex C, "Hypothetical Reference Decoder," Draft ISO/IEC 14496-10:2002(E), Draft ITU-T Rec. H.2649, (2002)E, pp. 160-188.|
|7||Annex C, "Video Buffering Verifier," H.262/MPEG-2, ITU-T Rec. H.262 (1995E), pp. 143-147.|
|8||Annex D, "Features Supported by the Algorithm," ITU-T Rec. H.262 (1995 E), pp. 148-150.|
|9||Hannuksela, M., "Simple Definition of GOP for Random Access," Nokia Corporation, JVT-B041, Jan. 23, 2002, pp. 1-6.|
|10||Hannuksela, Miska M., et al., Random Access and Time Information; JVT-B109, Mar. 2002, pp. 1-6.|
|11||Hiroshi, Fujiwara, "Point Illustrated Newest MPEG Textbook," ASCII Corporation, Aug. 1, 1994, p. 113-114.|
|12||Miska M. Hannuksela, Stephan Wenger, Thomas Stockhammer; Random Access and Time Information; JVT-B109, Mar. 1, 2002, pp. 1-6.|
|13||Misska M. Hannuksela; Simple Definition of GOP for Random Access; Nokia Corporation, JVT-BO41, Jan. 23, 2002, pp. 1-6.|
|14||Regunathan, Shankar L., et al., "Video Complexity Verifier for HRD", Microsoft, JVT-B050, Jan. 18, 2002, pp. 1-19.|
|15||Ribas-Corbera et al., "A Generalized Hypothetical Reference Decoder for H.26L," ITU Telecommunications Standardization Sector, Sep. 24-27, 2001.|
|16||Ribas-Corbera, Jordi et al., "A Flexible Decoder Buffer Model for JVT Video Coding," International Conference on Image Processing ICIP 2002, vol. 2, pp. II 493-II 496.|
|17||Ribas-Corbera, Jordi, "A Generalized Hypothetical Reference Decoder for H.264/AVC," IEEE Transactions on Circuits and Systems for Video Technology, IEEE Service Center, Piscataway, NJ, US, vol. 13, No. 7 Jul. 2003, pp. 674-687, XP 001 051 195.|
|18||Shankar L. Regunathan, Phil A. Chou, Jordi Ribas-Corbera; Video Complexity Verifier for HRD; Microsoft, JVT-BO50, Jan. 18, 2002, pp. 1-19.|
|19||Sullivan, Gary, "On Random Access and Bitstream Format for JVT Video," Microsoft Corporation, JVT-B063R1, pp. 1-16.|
|20||Sullivan, Gary, "Working Draft No. 2, Revision 3 (WD-2r3)," Joint Video Team (JVT) of ISO/IEC MPEG and ITU-T VCEG, Mar. 2002, Document JVT-B118r3.|
|21||Viscito, Eric, "H.26L Buffering Ad-Hoc Group Report," Globespan Virata, ITU Sector Member, JVT-B013, Jan. 23, 2002, pp. 1-3.|
|International Classification||H04N7/50, H04N7/26, H04B1/66|
|Cooperative Classification||H04N19/44, H04N19/61, H04N19/15|
|European Classification||H04N7/26D, H04N7/26A6E4G, H04N7/50|