|Publication number||USRE39955 E1|
|Application number||US 08/881,965|
|Publication date||Dec 25, 2007|
|Filing date||May 16, 1997|
|Priority date||Nov 30, 1993|
|Also published as||US5416520, US5485211|
|Publication number||08881965, 881965, US RE39955 E1, US RE39955E1, US-E1-RE39955, USRE39955 E1, USRE39955E1|
|Inventors||Andrew J. Kuzma|
|Original Assignee||Intel Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (22), Referenced by (1), Classifications (16), Legal Events (1)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This is a reissue of Ser. No. 08/159,665 filed Nov. 30, 1993, now U.S. Pat. No. 5,416,520.
1. Field of the Invention
The present invention relates to the transmission of information over a communications path. More particularly, the present invention relates to the communications of high bandwidths information over networks of varying types.
2. Art Background
Until recently, telecommunications and computing were considered to be entirely separate disciplines. Telecommunications was analog and done in real time whereas computing was digital and performed at a rate determined by the processing speed of a computer. Today, such technologies as speech processing, electronic mail and facsimile have blurred these lines. In the coming years, computing and telecommunications will become almost indistinguishable in a race to support a broad range of new multimedia (i.e., voice, video and data) applications. These applications are made possible by emerging digital-processing technologies, which include: compressed audio (both high fidelity audio and speech), high resolution still images, and compressed video. The emerging technologies will allow for collaboration at a distance, including video conferencing.
Of these technologies, video is particularly exciting in terms of its potential applications. But video is also the most demanding in terms of processing power and sheer volume of data to be processed. Uncompressed digital video requires somewhere between 50 and 200 Mb/s (megabits per second) to support the real-time transmission of standard television quality images. This makes impractical the widespread use of uncompressed digital video in telecommunications applications.
Fortunately, there is considerable redundancy in video data, both in terms of information theory and human perception. This redundancy allows for the compression of digital video sequences into lower transmission rates. For some time, researchers have been aware of a variety of techniques that can be used to compress video data sequences anywhere from 2:1 to 1000:1, depending on the quality required by the application. Until recently, however, it was not practical to incorporate these techniques into low cost video-based applications.
A number of standards have been recently developed for such activities as video conferencing, the transmission and storage of standard high quality still images, as well as standards for interactive video playback to provide interoperability between numerous communications points. The standards recognize a need for quality video compression to reduce the tremendous amount of data required for the transmission of video information.
Two important methods of data compression for video information are used widely throughout the various standards for video communication. These are the concepts of frame differencing and motion compensation. Frame differencing recognizes that a normal video sequence has little variation from one frame to the next. If, instead of coding each frame, only the differences between a frame and the previous frame are coded, then the amount of information needed to describe the new frame will be dramatically reduced. Motion compensation recognizes that much of the difference that does occur between successive frames can be characterized as a simple translation of motion, caused either by the moving of objects in the scene or by a pan of the field of view. Rather than form a simple difference between blocks in a current frame and the same block in the previous frame, the area around those blocks can be searched in the previous frame to find an offset block that more closely matches the block of the current frame. Once a best match has been identified, the difference between a reference block in the current frame and the best match in the previous frame are coded to produce a vector that describes the offset of the best match. This motion vector then can be used with the previous frame to produce the equivalent of what the current frame should be. These methods, and others are incorporated into systems which make possible the rapid transmission of real-time video information.
As the worlds of telecommunications and computers blend closely together, the telecommunications aspects of communications will have to contend with some of the constraints of the computer world. Particularly, video conferencing over existing computer networks will prove a challenge in that maintaining real time information communication over traffic-burdened existing network protocols may prove insurmountable.
Current video algorithms assume a nearly constant bandwidth availability for the encoding of video information. This is evidenced by the use of only a single output buffer for traditional video encoder output. It is common to use the output buffer fullness as a feedback parameter for encoding subsequent images; i.e., with higher or lower levels of quantization. A well-known effect resulting from using a single output buffer is called “bit-bang” where the output buffer is over depleted by the interface to the communications channel, causing the feedback loop to indicate that the buffer can handle lots of data, which in turn causes the video compression algorithm to under optimize the subsequent image coding. The user perceives the bit-bang as an uneven quality and frame rate.
To alleviate bit-bang, the typical approach has been to limit the amount of data pulled out from the encoder video output buffer to a fraction of the total size of the output buffer; 10% to 30% is typical. This approach keeps the feedback indicator rather small, and encoding more uniform. The underlying assumption of this approach is that the communications channel will usually not be changing rapidly. Exceptions are caused by connectivity interruptions, such as burst errors, which are handled strictly as exceptions to the call. In a local area network (LAN), or other collision-sensing multiple access channel, or in other networks with burst characteristics (such as noisy RF channels), this underlying assumption no longer holds. Over these sorts of communications channels, unanticipated transmission delays may result in bit-bang problems which are not so readily overcome by limiting the size of the feedback buffer. Thus, video jerkiness will result in real-time video communication over such channels. It would be advantageous, and is therefore an object of the present invention to provide a video transmission mechanism which can be accommodated on such potential bursty networks.
From the foregoing, it can be appreciated that there is a need for a mechanism of incorporating real-time video data communication over traditional network protocols to smooth video transmission. It is therefore an object of the present invention to provide a method and apparatus for the conveyance of video data over such networks as local area networks.
These and other objects of the present invention are provided by introducing feedback between the video CODEC and the intended communications channel such that the characteristics of the channel are used to drive multiple video output buffers. These multiple output buffers share an original temporal video reference, but have different subsequent temporal video images. The communications channel interface then picks the subsequent video image buffer that best matches the current conditions experienced by it. By using a predictor of the channel performance, the video algorithm can be tuned to provide video output buffers with the best guess of how the buffers should be configured. A number of subsequent histories of an image are buffered until the receiving channel indicates it is ready to receive the next. Then the appropriate output buffer having the corresponding temporal change in the video is used to supply the next frame change information to the receiving station.
The objects, features and advantages of the present invention will be apparent from the following detailed description in which:
A method and apparatus are described for the conveyance of real-time isochronous data over bursty networks. Although the present invention is described predominantly in terms of the transmission of video information, the concepts and method are broad enough to encompass the transmission of real-time audio and other data requiring isochronous data transfer. Throughout this detailed description, numerous details are specified such as bit rates and frame sizes, in order to provide a thorough understanding of the present invention. To one skilled in the art, however, it will be understood that the present invention may be practiced without such specific details. In other instances, well-known control structures and gate level circuits have not been shown in detail in order not to obscure unnecessarily the present invention. Particularly, some functions are described to be carried out by various logic circuits. Those of ordinary skill in the art, having been described the various functions will be able to implement the necessary logic circuits without undue experimentation.
From the capture buffer 22, the video information is processed by motion estimation circuitry 23. The motion estimation circuitry is used to generate motion vectors which describe the difference of a portion of a video image from the previously recorded image in terms of a translational offset. The motion estimation circuitry compares the currently decoded frame from the previous frame stored in the transmit reference image buffer 30 about which more will be described further herein. From the motion estimation circuitry, the outputs are the motion vectors and the motion compensated image 24. The motion compensated image 24 is then processed by the differential pulse code modulation (DPCM) circuitry 25 which generates digital information of the changes to the previously stored transmit reference image. Finally, a final stage of coding is done at transform coding block 26 which also performs quantization and run-length encoding. Run-length encoding is a technique for compressing data sequences that have large numbers of zeros and is well-known to those of ordinary skill in the art. This transform coder may perform a discrete cosine transform (DCT).
From the transform coding block, the coded sequence is propagated to the output buffer 27 which is used to maintain a constant bit rate for the output to the network. As was described, prior art methods used the output buffer fullness to regulate the degree of quantization that would be applied to the compressing and encoding circuitry because a constant bandwidth availability was assumed.
The transform coder block 26 also outputs the compressed image data to generate a new transfer reference image for storage in the transmit reference image buffer. The encoding logic provides the compressed image data to a decoding block 28 that has an inverse quantizer and inverse discrete cosine transform decoder which can be used to combine the decoded image data with the previously stored transfer reference image to yield a new transmit reference image which corresponds to the image that was most recently propagated on the network. It is this image data that would be used in calculating the changes in the image in sending the next frame of information. In other words, the transmit reference image, which is the same image that will be reconstructed at the other end by the video decoder, is used as the basis of subsequent encoding, including motion vectors and motion compensated image compression.
As was described in the previous section, the prior art feedback mechanism using the output buffer assumed a constant bit rate would be available for the transmission of information. This assumption no longer holds for video conferencing type devices which are on bursty networks such as CSMA LAN networks. The solution proposed by the present invention is to provide feedback between the video CODEC and the communications channel such that the characteristics of the channel are used to drive multiple video output buffers. These buffers share an original temporal video reference but will have different subsequent temporal video images. The communications channel interface then picks the subsequent video image buffer that best matches the current condition. By using a predictor of the channel performance, the video algorithm can be tuned to provide video output buffers with the best guess of how the buffers should be configured. Once a particular output buffer's image data is selected, the remaining buffers can be flushed to be refilled again based on a newly calculated transmit reference image. In the limit, the final action is to revert to an exception handler similar to current video CODECS, i.e., insert a key frame to restart the encoding of video data transmission.
The video encoder and camera circuitry described may be incorporated as part of a station that is on the network and are responsive to information received over the communications channel. When a given node again has the bus, the output buffer with the most current image may be signaled to transmit its information to the receiving node. Likewise, the channel information is used to then calculate the next transmit reference image for storage. The output buffers are then flushed and are again loaded in a time sequential manner until the data is again ready to be sent over the network. While four (4) output buffers are illustrated, this is purely for illustrative purposes in that as many buffers may be implemented as computing power and resources provide.
Prediction of Bandwidth per Unit
Time Relative to Current Transmit
Logical Output Buffer
about the same
a lot more
a lot less
For video coding, more bandwidth could be used to get sharper images and/or higher frame rate. The actual data contained in the logical output buffers can be significantly different, too. For example, in video coding, the new transmit reference might be calculated from different input images in time and/or spatial resolution. Logical output buffer 1 might represent the data from an image taken 1/15th of a second later than transmit reference 0, while logical output buffer 2 might represent the differential coding from an image half a second later than transmit reference 0. Such an approach would be good for video coding for channels where the bit rate allocated to video may undergo extreme fluctuations such as in the bursty networks described above.
While with reference to
In a more general description of the present invention, reference is now made to FIG. 7. Information about a real-time object 100 that is desired to be conveyed from a transmitting node to a receiving node on some sort of network is shown. This real-time object 100 may be a video image or it may be a sound depending on the particular implementation. A capture mechanism 110 detects the real-time object and encodes it into electronic information. The capture mechanism may be a camera for video information as described above or a microphone or stereo microphones for audio information. This information is then processed by differential encoder 115 which compares the newly captured real-time object to the previously stored recorded object in transmit reference buffer 120. The differentially encoded data is then propagated to a logical output buffer 125 which operates as those described above. When the network clears the output buffers for transmission, the particular output buffer having the best information conveys it over the network and that same information is used to calculate a new transmit reference to be stored in transmit reference buffer 120.
There has thus been described a method and apparatus of differential coding for use in bursty transmission networks which greatly improves the quality of transmitted compressed information. Although the present invention has been described in terms of preferred embodiments, it will be appreciated that various modifications and alterations might be made by those skilled in the art without departing from the spirit and scope of the invention. The invention should, therefore, be measured in terms of the claims which follow.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4320500 *||Feb 8, 1980||Mar 16, 1982||Cselt - Centro Studi E Laboratori Telecomunicazioni S.P.A.||Method of and system for routing in a packet-switched communication network|
|US4860244 *||Nov 7, 1983||Aug 22, 1989||Digital Equipment Corporation||Buffer system for input/output portion of digital data processing system|
|US5010401 *||Aug 11, 1988||Apr 23, 1991||Mitsubishi Denki Kabushiki Kaisha||Picture coding and decoding apparatus using vector quantization|
|US5077742 *||Dec 21, 1990||Dec 31, 1991||Ricoh Company, Ltd.||Error-corrected facsimile communication control system|
|US5117350 *||Dec 15, 1988||May 26, 1992||Flashpoint Computer Corporation||Memory address mechanism in a distributed memory architecture|
|US5121202 *||May 11, 1990||Jun 9, 1992||Nec Corporation||Adaptive interframe prediction coded video communications system|
|US5128931 *||Apr 24, 1990||Jul 7, 1992||Mitsubishi Denki Kabushiki Kaisha||Data exchange apparatus|
|US5140417 *||Jun 20, 1990||Aug 18, 1992||Matsushita Electric Co., Ltd.||Fast packet transmission system of video data|
|US5155484 *||Sep 13, 1991||Oct 13, 1992||Salient Software, Inc.||Fast data compressor with direct lookup table indexing into history buffer|
|US5164939 *||May 17, 1991||Nov 17, 1992||Kabushiki Kaisha Toshiba||Packet switching device|
|US5196933 *||Mar 19, 1991||Mar 23, 1993||Etat Francais, Ministere Des Ptt||Encoding and transmission method with at least two levels of quality of digital pictures belonging to a sequence of pictures, and corresponding devices|
|US5227875 *||Aug 20, 1991||Jul 13, 1993||Kabushiki Kaisha Toshiba||System for transmitting encoded image data with quick image expansion and contraction|
|US5276681 *||Jun 25, 1992||Jan 4, 1994||Starlight Networks||Process for fair and prioritized access to limited output buffers in a multi-port switch|
|US5343465 *||Jun 11, 1993||Aug 30, 1994||Bell Communications Research, Inc.||Method and system for real-time burstiness analysis of network traffic|
|US5396494 *||Mar 25, 1993||Mar 7, 1995||At&T Corp.||Method for operating an asynchronous packet bus for transmission of asynchronous and isochronous information|
|US5408465 *||Jun 21, 1993||Apr 18, 1995||Hewlett-Packard Company||Flexible scheme for admission control of multimedia streams on integrated networks|
|US5440344 *||Apr 21, 1993||Aug 8, 1995||Mitsubishi Denki Kabushiki Kaisha||Video encoder using adjacent pixel difference for quantizer control|
|US5497153 *||Jul 23, 1993||Mar 5, 1996||Samsung Electronics Co., Ltd.||System for variable-length-coding and variable-length-decoding digital data for compressing transmission data|
|US5502727 *||Apr 20, 1993||Mar 26, 1996||At&T Corp.||Image and audio communication system having graphical annotation capability|
|US5629736 *||Nov 1, 1994||May 13, 1997||Lucent Technologies Inc.||Coded domain picture composition for multimedia communications systems|
|US5640208 *||Apr 25, 1995||Jun 17, 1997||Sony Corporation||Video signal encoding in accordance with stored parameters|
|US5905821 *||Dec 8, 1994||May 18, 1999||Canon Kabushiki Kaisha||Compression/expansion circuit having transfer means and storage means with address management of the storage means|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US20100038044 *||Oct 27, 2009||Feb 18, 2010||Mark Alan Burazin||Method of Making Textured Tissue Sheets Having Highlighted Designs|
|International Classification||H04N21/236, H04N21/434, H04N7/50, H04N7/32, H04N7/12, G06T9/00|
|Cooperative Classification||H04N19/152, H04N19/50, H04N19/61, H04N21/434, H04N21/236|
|European Classification||H04N21/434, H04N21/236, H04N7/32B, H04N7/50|
|Dec 8, 1997||AS||Assignment|
Owner name: INTEL CORPORATION, CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:KUZMA, ANDREW J.;REEL/FRAME:008853/0472
Effective date: 19971118