|Publication number||US8218649 B2|
|Application number||US 13/099,313|
|Publication date||Jul 10, 2012|
|Filing date||May 2, 2011|
|Priority date||Jun 13, 2008|
|Also published as||US20090310684, US20110206134|
|Publication number||099313, 13099313, US 8218649 B2, US 8218649B2, US-B2-8218649, US8218649 B2, US8218649B2|
|Original Assignee||Kabushiki Kaisha Toshiba|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (18), Non-Patent Citations (1), Classifications (21), Legal Events (1)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This is a continuation of U.S. patent application Ser. No. 12/428,339 filed on Apr. 22, 2009, which is based upon and claims the benefit of priority from Japanese Patent Application No. 2008-155608, filed Jun. 13, 2008, the entire contents of both, which are incorporated herein by reference.
1. Field of the Invention
One embodiment of the invention relates to a decode control technology for performing a suitable decode on a video stream, which is applied to, for example, a personal computer including a TV function capable of recording and reproducing digital broadcast program data which is broadcast with a television broadcast signal.
2. Description of the Related Art
Recently, battery-drivable and easy-to-carry notebook type personal computers are in wide use. These types of computers have been subjected to reduction in thickness and weight, while simultaneously making improvements in functionality. As a result, such computers incorporating a TV function for showing digital television broadcast programs have appeared. Therefore, if a user carries such type of computer while traveling or away from home, a digital television broadcast program can be enjoyed.
Furthermore, with the improvement in the wireless communication environment, it has become easy to be connected to the Internet while a user is outside or traveling. It has become thus possible to receive digital television broadcast program data from a server connected via the Internet and watch the received program outside the home or office.
Digital broadcast program data is a video stream that is compression-encoded with, for example, a method compliant with an MPEG (Moving Picture Experts Group) 2 standard. During a transfer of such a compression-encoded video stream, a part thereof may be lost or an error may occur therein. In order to reproduce such a video stream in which an error occurs or a part thereof is missing, various recovery mechanisms have been proposed so far (e.g., Jpn. Pat. Appln. KOKAI publication No. 9-154135).
Generally, a computer is equipped with a graphics accelerator (hereinafter referred to as accelerator), which draws an image to be displayed by various application programs on behalf of a CPU. Furthermore, recently, accelerators including a function for executing a part of a process for decoding a compression-encoded video stream have begun to appear. Among those computers equipped with a TV function, quite a lot of them realize such TV function with software. In that case, a decoding process of a video stream needs to be executed by one module in the aforementioned software, which is operating on a CPU. A load on the CPU is expected to be greatly reduced by letting a part of the decoding process be carried out by the accelerator.
A video stream that is compression-encoded with a method compliant with the MPEG2 standard has a hierarchical structure comprising six layers: a sequence layer, a GOP (Group Of Picture) layer, a picture layer, a slice layer, a macroblock layer, and a block layer. The picture layer is a layer to be provided in pairs with respective images in the video. The slice layer located beneath the picture layer is an aggregate of partial image data (slice), which is produced by dividing each of the images in the horizontal direction. Also, the macroblock is a layer produced by dividing each of the images in a matrix form. That is to say, a slice is equal to macroblocks arranged in a row direction. A representative accelerator including a function for executing a part of the process for decoding a compression-encoded video stream is one that executes a decoding process by receiving data in slice units or macroblock units.
However, the above-mentioned accelerator, which performs a decoding process by receiving data in slice units or macroblock units, is unable to execute such a recovery process using peripheral data as is executed by an error processing apparatus disclosed in the above-mentioned Jpn. Pat. Appln. KOKAI publication No. 9-154135. Therefore, software, which tries to make the aforementioned accelerator execute the decoding process in slice units or macroblock units, is required to be provided with the following mechanism: the mechanism which supplements data for enabling the accelerator to successfully complete the decoding process and supplying the accelerator with the supplemented data, even when a part of the compression-encoded video stream is lost or an error occurs therein while being transferred.
A general architecture that implements the various feature of the invention will now be described with reference to the drawings. The drawings and the associated descriptions are provided to illustrate embodiments of the invention and not to limit the scope of the invention.
Various embodiments according to the invention will be described hereinafter with reference to the accompanying drawings. In general, according to one embodiment of the invention, an information processing apparatus includes a recovery module which causes a decoding module to decode supplementary slice data when it is determined that the value of the information indicating a position of processing target slice data in the image is inappropriate. The recovery module creates the supplementary slice data by editing slice data located upwardly adjacent to the processing target slice in such a manner that information indicating a position thereof in the image is edited into information for the processing target slice.
First of all, a configuration of an information processing apparatus according to one embodiment of the embodiment will be described with reference to
The computer 10 includes a television (TV) function for executing reproduction and recording of broadcast program data broadcast with a television broadcast signal. This TV function is realized with, for example, a TV application program which is preliminarily installed in the computer 10.
The display unit 12 is attached to the computer body 11 in such a manner that it is rotatable between a opened position where a top surface of the computer body 11 is exposed and a closed position where the computer body 11 is covered. The computer body 11 has a thin box shaped chassis on which a keyboard 13, a power button 14 for powering on/off the computer 10, an input operation panel 15, a touch pad 16, and speakers 18A and 18B are arranged.
The input operation panel 15 is an input device for inputting events corresponding to the pressed button, and includes a plurality of buttons for activating a plurality of functions. This group of buttons includes a group of operation buttons for controlling the TV function.
Also, a remote control unit interface portion 20 for performing communications with a remote control unit, which remote controls the TV function of the computer 10, is provided on a front face of the computer body 11. The remote control unit interface portion 20 comprises an infrared signal receiver, etc. Furthermore, an antenna terminal 19 is provided on, for example, a right side surface of the computer body 11.
Next, a system configuration of the computer 10 will be explained with reference to
The computer 10 includes, as shown in
The CPU 101 is a processor that controls operation of the computer 10. It executes an operating system (OS) 201 and various application programs such as a TV application program 202 operating under the OS 201, which are loaded from the HDD 109 into the main memory 103. The TV application program 202 is software for executing the TV function. This TV application program 202 executes: a live reproduction process for reproducing broadcast program data, which is received by the TV tuner 114 as an encoded video stream; a recording process for recording the received broadcast program data on the HDD 109; and a reproduction process for reproducing the broadcast program data/video data recorded on the HDD 109. Also, another possible way of utilizing the TV application program 202 is to reproduce broadcast program data obtained by way of the LAN controller 108 or the wireless LAN controller 111. Furthermore, the CPU 101 also executes a BIOS (Basic Input/Output System) stored in the BIOS-ROM 107. The BIOS is a program for hardware control.
The north bridge 102 is a bridge device for connecting a local bus of the CPU 101 and the south bridge 104. The north bridge 102 incorporates a memory controller for access controlling the main memory 103. Also, the north bridge 102 includes a function of executing communications with the GPU 105 via a serial bus of a PCI EXPRESS standard.
The GPU 105 is a display controller for controlling the LCD 17, which is used as a display monitor of the computer 10. A display signal generated by the GPU 105 is transmitted to the LCD 17. The GPU 105 includes an accelerator for drawing images to be displayed by various application programs on behalf of the CPU 101. The accelerator included in the GPU 105 includes a function to execute a part of a process of decoding an encoded video stream received by the TV tuner 114 on behalf of the TV application program 202. More specifically, the accelerator receives data included in the video stream in every predetermined unit from the TV application program 202, and executes the decoding process in the predetermined units. The computer 10 according to the invention is designed to recover data, even when a part of a video stream is lost or an error occurs therein while being transferred, in such a manner that the data in a predetermined unit to be supplied to the accelerator is effectively supplemented (in order to prevent the accelerator from abnormally completing the decoding process). This point will be described later.
The south bridge 104 controls devices on an LPC (Low Pin Count) bus and devices on a PCI (Peripheral Component Interconnect) bus. The south bridge 104 also incorporates an IDE (Integrated Drive Electronics) controller for controlling the HDD 109 and the DVD drive 110. Furthermore, the south bridge 114 includes a function of executing communications with the sound controller 106.
The sound controller 106 is a sound source device, and it outputs audio data as a reproduction target to the speakers 18A and 18B.
The wireless LAN controller 111 is a wireless communication device for executing wireless communications of, for example, the IEEE 802.11 standard. The IEEE 1394 controller 112 executes communications with an external device via a serial bus of the IEEE 1394 standard.
The EC/KBC 113 is a one-chip microcomputer on which an embedded controller for performing power management, and a keyboard controller for controlling the keyboard (KB) 13 and the touch pad 16 are integrated. The EC/KBC 113 includes a function of powering on/off the computer 10 in response to a user's operation of the power button 14. Furthermore, the EC/KBC 113 includes a function of executing communications with the remote controller interface section 20.
The TV tuner 114 is a receiver for receiving broadcast program data broadcast with a television broadcast signal, and is connected to the antenna terminal 19. The TV tuner 114 is realized as, for example, a digital TV tuner capable of receiving digital broadcast program data such as digital terrestrial broadcasting. Also, the TV tuner 114 includes a function of capturing video data input from an external device.
In the TV application program 202 which operates on the computer 10 having the above-described configuration and reproduces broadcast program data (video stream), a software decoder 2021, which conducts a decoding process of the broadcast program data, supplies an accelerator 1051 of the GPU 105 with data in the video stream in a predetermined units (stream units or macroblock units to be described later), as shown in
According to the MPEG2 standard, which is a major method for compression-encoding videos, an image is divided into macroblocks of 16×16 pixels, and the image is encoded in these macroblock units. Then, the macroblocks are arranged in row directions to form slices.
As shown in
On the other hand, the accelerator 1051 includes a function of performing a process of decoding the video stream in slice layer units or macroblock layer units. Therefore, the software decoder 2021 sequentially hands over data in a unit of the slice layer or the macroblock layer to the accelerator, for letting the accelerator carry out the decoding process.
Next, a basic principle of a recovery process, which is executed by the above-described computer 10, wherein the software decoder 2021 and the accelerator 1051 cooperate for decoding a video stream, when a part of the video stream is lost or an error occurs while being transferred, will be explained with reference to
First of all, assume that an error occurs in the 4th slice (“d1”) of a frame as shown in “A” of
Furthermore, the value “100” shown in “A” of
Next, a case where the 4th slice of a frame is lost, as shown in “A” of
The software decoder 2021, which sequentially hands over data in units of the macroblock layer to the accelerator 1051, checks, at the time of handing over, the data for the following errors: a Huffman coding table error in which variable length data (such as motion vector) deviates from a table when the variable length data is Huffman coded; and a skip macroblock error in which a skip macroblock value exceeds a standard. Here, it is assumed that an error is detected with respect to a hatched macroblock (“g1”) shown in “A” of
In this case, the software decoder 2021 abandons the macroblock (“g1”) and all of the immediately subsequent macroblocks in the same slice. Then, the software decoder 2021 performs a supplement with respect to the abandoned macroblocks by using data of the upwardly adjacent macroblocks (“g2”, “g3”, “g4”), as shown in “B” of
In other words, as shown in
The software decoder 2021 checks a value of a slice number of a slice to be handed over to the accelerator 1051 (block A1), and determines whether or not the value is appropriate (block A2). When the value is appropriate (YES of block A2), the software decoder 2021 supplies the accelerator 1051 with data of the slice (block A7).
On the other hand, when the value of the slice number is not appropriate (NO of block A2), the software decoder 2021 subsequently determines whether or not the value is suitable for any of subsequent slices (block A3). When the value the slice is suitable for none of the subsequent slices (YES of block A3), the software decoder 2021 edits data of the preceding slice which is upwardly adjacent to the slice into data for the slice, and supplies the edited data to the accelerator 1051 (block A8).
On the other hand, when the value of the slice is suitable for one of the subsequent slices (NO of block A3), the software decoder 2021 executes the following procedures: editing the data of the preceding slice upwardly adjacent to the slice into the data for the slice and supplying the edited data to the accelerator 1051 (block A4); determining whether or not the value of the slice is suitable for the next slice (blocks A5 and A6); and supplying, when the value is suitable for the next slice (YES of block A6), the data of the slice to the accelerator 1051 (block A7). When the value of the slice is not suitable for the next slice (NO of block A6), the software decoder 2021 repeats blocks from A4 onward, that is to say, repeats a supplement using the data of the immediately preceding slice.
After supplying the accelerator 1051 with the data of the slice, the software decoder 2021 determines whether or not the slice is the last slice in the frame (block A9). When it is determined that the slice is not the last slice (NO of block A9), the software decoder 2021 repeats processes from block A1 onward with respect to the next slice. On the other hand, when it is determined that the slice is the last slice (YES of block A9), the software decoder 2021 terminates processes with respect to the frame.
The software decoder 2021 checks a macroblock to be handed over to the accelerator 1051 for errors such as a Huffman coding table error or a skip macroblock error (block B1). When no error is detected (NO of block B2), the software decoder 2021 supplies the accelerator 1051 with data of the macroblock.
On the other hand, when an error is detected with respect to the macroblock (YES of block B2), the software decoder 2021 edits data of a macroblock, which is in a slice upwardly adjacent to the slice including the errored macroblock and located at a position corresponding to the errored macroblock, into data for the errored macroblock, and supplies the edited data to the accelerator 1051 (block B3).
When the above supplement is performed, the software decoder 2021 determines whether or not the errored macroblock is the last macroblock in the slice (block B4). When it is determined that the errored macroblock is not the last macroblock (NO of block B4), the software decoder 2021 repeats processes from block B4 onward with respect to subsequent macroblocks (block B5). In other words, the software decoder 2021 repeats a supplementing process using data of a macroblock in an immediately preceding slice which is located in a position corresponding to an errored macroblock.
After supplying the accelerator 1051 with the data of the macroblock, the software decoder 2021 determines whether or not the macroblock is the last macroblock in the frame (block B7). When it is determined that the macroblock is not the last macroblock (NO of block B7), the software decoder 2021 repeats the processes from block B1 onward with respect to the next macroblock. When it is determined that the macroblock is the last macroblock (YES of block B7), the software decoder 2021 terminates the processes with respect to the frame.
As described above, according to the computer 10, it is possible to realize a recovery process wherein the software decoder 2021 lets the accelerator 1051 carry out a part of the process of decoding a video stream for the software decoder 2021.
In the above-described example, a supplement using data of a macroblock located upwardly adjacent to an errored macroblock is performed with respect to the errored macroblock, as a recovery process wherein the software decoder 2021 lets the accelerator 1051 carry out a process of decoding a video stream in macroblock units for the software decoder 2021. It is obviously possible to combine this method with a recovery process wherein the software decoder 2021 lets the accelerator 1051 carry out a process of decoding a video stream in slice units for the software decoder 2021.
More specifically, when a slice number for one slice is determined to be not suitable, the software decoder 2021 immediately executes a supplement using data of a slice upwardly adjacent the slice. When the slice number is determined to be suitable, the software decoder 2021 subsequently checks each of the macroblocks for errors. When an error is detected in a macroblock, the software decoder 2021 generates data of a slice wherein the error macroblock and the immediately subsequent macroblocks are replaced with data of macroblocks located upwardly adjacent to the errored macroblock and the immediately subsequent macroblocks. The software decoder 2021 then supplies the accelerator 1051 with the data of the slice in which a partial replacement is performed.
The various modules of the systems described herein can be implemented as software applications, hardware and/or software modules, or components on one or more computers, such as servers. While the various modules are illustrated separately, they may share some or all of the same underlying logic or code.
While certain embodiments of the inventions have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US6480546 *||May 27, 1999||Nov 12, 2002||Lg Electronics Inc.||Error concealment method in a motion video decompression system|
|US6614843 *||Dec 10, 1999||Sep 2, 2003||Diva Systems Corporation||Stream indexing for delivery of interactive program guide|
|US6754278 *||Jul 14, 2000||Jun 22, 2004||Lg Electronics Inc.||Method for recovering moving picture by extending a damaged region in which an error occurs|
|US6842219 *||Jul 15, 2002||Jan 11, 2005||C & S Technology||Moving picture decoding processor for multimedia signal processing|
|US6987805 *||Sep 24, 1999||Jan 17, 2006||Lsi Logic Corporation||Macroblock level intrarefresh technique for encoded video|
|US7027515 *||Oct 15, 2002||Apr 11, 2006||Red Rock Semiconductor Ltd.||Sum-of-absolute-difference checking of macroblock borders for error detection in a corrupted MPEG-4 bitstream|
|US7254824 *||May 30, 2000||Aug 7, 2007||Sedna Patent Services, Llc||Encoding optimization techniques for encoding program grid section of server-centric interactive programming guide|
|US7606313 *||Jan 15, 2004||Oct 20, 2009||Ittiam Systems (P) Ltd.||System, method, and apparatus for error concealment in coded video signals|
|US20050111743 *||Nov 20, 2003||May 26, 2005||Hao-Song Kong||Error resilient encoding method for inter-frames of compressed videos|
|US20060256872 *||May 11, 2006||Nov 16, 2006||Matsushita Electric Industrial Co., Ltd.||Filtering characteristic error concealing apparatus|
|US20090002379 *||Jun 30, 2007||Jan 1, 2009||Microsoft Corporation||Video decoding implementations for a graphics processing unit|
|JP2005318255A||Title not available|
|JP2006262406A||Title not available|
|JP2006345046A||Title not available|
|JP2007043575A||Title not available|
|JPH06311052A||Title not available|
|JPH09154135A||Title not available|
|JPH09154142A||Title not available|
|1||Notice of Reasons for Rejection mailed by Japan Patent Office on Dec. 1, 2009 in the corresponding Japanese patent application No. 2008-155608.|
|International Classification||H04N19/70, H04N19/134, H04N19/91, H04N19/44, H04N19/176, H04N19/00, H04N19/132, H04N19/174, H04N19/895, H04B1/66|
|Cooperative Classification||H04N19/176, H04N19/895, H04N19/42, H04N19/61, H04N19/174|
|European Classification||H04N7/26A8B, H04N7/50, H04N7/68, H04N7/26L, H04N7/26A8L|