|Publication number||US20050062892 A1|
|Application number||US 10/391,465|
|Publication date||Mar 24, 2005|
|Filing date||Mar 17, 2003|
|Priority date||Jul 23, 1998|
|Also published as||US6587158|
|Publication number||10391465, 391465, US 2005/0062892 A1, US 2005/062892 A1, US 20050062892 A1, US 20050062892A1, US 2005062892 A1, US 2005062892A1, US-A1-20050062892, US-A1-2005062892, US2005/0062892A1, US2005/062892A1, US20050062892 A1, US20050062892A1, US2005062892 A1, US2005062892A1|
|Inventors||Dale Adams, Laurence Thompson, Jano Banks|
|Original Assignee||Dvdo, Inc.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (31), Referenced by (4), Classifications (28), Legal Events (2)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This application claims the benefits of co-pending U.S. Patent Provisional Application No. 60/093,815 filed on Jul. 23, 1998, and is related to U.S. patent application Ser. No. 09/167,527 filed on Oct. 6, 1998, both of which are incorporated herein by reference.
1. Field of the Invention
The present invention relates generally to video processing and, more particularly, to techniques for vertical processing of horizontally scanned video images.
2. Description of the Related Art
Many types of processing only utilize data from horizontally adjacent pixel locations. This naturally fits with the existing data ordering of the raster-scanned format and is relatively economical to implement. In addition, the row ordering of the video data naturally aligns with the current dynamic random access memory (DRAM) row/column chip organization which provides significantly reduced access times for in-row (or horizontal) accesses.
However, when video data must be processed in the vertical direction, particularly when multiple vertically adjacent pixel locations are simultaneously needed for processing, the raster-scanned format no longer provides the data needed in the correct or appropriate order.
Since the pixel data is presented in a horizontally ordered sequence, multiple lines of video data must be available in order to have simultaneous access to multiple vertically aligned pixels. This requires storing or buffering of those multiple lines in order to make the data available for processing. In the past, this has either not been done at all due to implementation cost reasons, or has been done by using multiple on-chip line memories to store a number of horizontal lines of pixel data. If nothing is done, the result is nonexistent or poor quality processing due to lack of vertical pixel data being available.
The main problem with using on-chip line memories for vertical processing of horizontal raster-scanned video is that they are extremely large, thus requiring a significant increase in die area (and therefore chip cost). This is particularly true if a large number of line memories are needed for high-quality processing. A single line memory for the ITU-R BT.601 standard digital video formats typically contains 720 16-bit pixels, or 11520 memory bits.
Because multiple line memories are needed for quality vertical processing, and since some video processing implementations require multiple serial processing stages, each with its own set of line memories, the required amount of on-chip memory can be very large in many different scenarios. For instance, with 6 line memories per processing stage, and with 3 serial processing stages the number of required memory bits would be over 200 kbits.
External memories have not been often utilized due to the fact that accessing vertically adjacent data results in the crossing of DRAM row (or page) boundaries, with the attendant severe reduction in available memory bandwidth. The implementation cost issue is compounded by the fact that high quality processing typically requires more data, i.e., better vertical processing requires a larger number of simultaneously available vertically aligned pixels.
On-chip memory requirements of this order significantly reduce the available implementation options (e.g., prototyping with field programmable gate arrays (FPGA) or most gate-arrays is not viable) and increase the chip die area and cost. While these expensive line memories for vertical video processing cannot be eliminated completely, any reduction of the memory requirements would be valuable.
In view of the foregoing, it is desirable to have a method that provides for high quality vertical processing of horizontally scanned video while minimizing cost and reducing the number of full line memories required.
The present invention fills these needs by providing an efficient and economical method and apparatus for high quality vertical video processing utilizing off-chip commodity memory and an alternative scan sequence for vertical processing. It should be appreciated that the present invention can be implemented in numerous ways, including as a process, an apparatus, a system, a device or a method. Several inventive embodiments of the present invention are described below.
In one embodiment of the present invention, a digital image processor is provided. The digital image processor includes an input buffer for storing raster-scanned data. A slice-buffer memory is coupled to the input buffer to store a portion of a vertical slice of said raster-scanned data. The vertical slice is processed by a vertical slice processor having an input coupled to the slice-buffer memory. The vertical slice processor reassembles the vertical slices into processed raster-scanned data in an output buffer that is coupled to the output of the vertical slice processor. The digital image processor preferably utilizes multiple sequential processing stages and processes the raster-scanned data along the horizontal axis of the vertical slices.
In another embodiment of the present invention, a method of processing image data is provided. The method includes buffering a block of raster-scanned data in an input buffer. Vertical slices of the raster-scanned data are sequentially retrieved and processed, forming processed vertical slices. The processed vertical slices are then stored in an output buffer to form a processed block of raster-scanned data. The vertical slices are preferably comprised of a slice core and at least one pair of wings, which overlap the slice core of horizontally adjacent vertical slices. The width of the processed block of raster-scanned data is preferably equal to the width of the slice core.
An advantage of the present invention is that the on-chip-memory requirements for high quality vertical processing are significantly reduced. By dividing the rectangular video field or frame into smaller portions, the memory requirements of the system can be reduced by an order of magnitude. Therefore, the image processing chip is not limited by the constraints of having only a small number of on-chip line memories. In addition, by not having to use the line memories, costs are dramatically reduced.
Other aspects and advantages of the invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention.
The present invention will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements.
An invention for a method and apparatus for reducing on-chip memory in vertical video processing is disclosed. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be understood, however, to one skilled in the art, that the present invention may be practiced without some or all of these specific details. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present invention.
However, difficulties do arise from utilizing this “slice” scan organization. First, it is often the case that processing must simultaneously be done in both the horizontal and vertical directions. This results in a problem on the left most and right most slice boundaries where horizontal pixel data outside the slice may not be available. Second, the conventional raster-scan sequencing has been changed, resulting in a potential incompatibility with common video sources and display/storage devices. Both of these problems will be addressed in the following sections as solved by the present invention.
It should be noted that the situation at the top edge 42 and bottom edge 44 of the slice core 30, where data above the top-most pixel and data below the bottom-most pixel is not available, is identical to that with the conventional raster-scanned format. This can be solved in a number of ways, such as substituting zero data for the nonexistent upper/lower pixel data. Therefore, the top and bottom edges 42 and 44 of the slice core 30 will not cause problems with unavailable data. In contrast, processing matrix 38 is on the left edge 32 of the slice core 30, and is missing horizontally adjacent data. Two columns of pixel data 40 are missing because they are outside the left edge 32 of the slice core 30.
To resolve this situation, data for these columns are provided from the slice immediately to the left of the slice being processed.
The method 60 utilizes a first processing stage 70 and a second processing stage 74. The first processing stage 70 utilizes and then removes the outer wings 62 and 68 leaving an output slice 72 consisting of the slice core 30 and the inner wings 64 and 66. The second processing stage 74 utilizes and then removes the inner wings 64 and 66. Therefore, the wings 48 and 50 are effectively removed in the processing and the output of the processing block is a slice 76 with the width equal to the original slice core 30.
One effect of the wings 48 and 50, is to increase the on-chip slice-line memory requirements by the width of the wings 48 and 50. However, the wing width is typically small relative to the overall slice width. The actual slice and wing width is implementation dependent and will depend on processing requirements and available external memory bandwidth.
A preferred embodiment of the present invention utilizes three vertical video processing blocks. The first processing stage 70 requires a pair of outer wings 62 and 68 having a width of 2 pixels; the second processing stage 74 requires a pair of inner wings 64 and 66 with a width of 4 pixels; and the third processing stage 77 requires no wings as the specific processing algorithm used does not require data horizontal to the vertical data being processed. The slice core width chosen was 36 pixels, resulting in an initial input slice width of 48 pixels. (Core+left-inner-wing+right-inner-wing+left-outer-wing+right-outer-wing=36+4+4+2+2=48.)
Unfortunately, the data inputs and outputs of the vertical processing blocks are not in the raster-scan video format, which is standard to virtually all video input sources and video output display and storage devices. The present invention includes a standardized input/output format conversion, which is accomplished via the use of a memory external to the video processing device. A commodity DRAM memory device is used for reasons of cost and availability.
Depending on the type of video processing to be done, a field or frame size buffer(s) serves other necessary purposes other than conversion between full field/frame raster-scan and slice-scan formats. For instance, the deinterlacing process typically requires one (sometimes several) field buffers to store multiple fields of video data for temporal processing. Buffers are also needed in frame rate conversion, where the output frame rate is different than the input rate; in this case multiple output field or frame buffers may be required for the frame rate conversion process.
A similar process is used for output. The second output buffer 86 receives processed data in slice-scan format from the vertical video processing section 90, while the first output buffer 84 (filled in the previous field/frame period) is used to output data in the conventional raster-scan format to the output data formatter 92. The output data stream may actually provide data to additional video processing stages that process data in the horizontal direction only (e.g. horizontal scaling and color space conversion).
During the period in which data is written to the first field memory buffer 102, data is read in vertical slice scan sequence from the second, third and fourth field memory buffers 106, 108 and 110, all of which are in the external memory source. The field buffers 106, 108 and 110 feed the vertical video processing section 112 that is located inside the video processor 96. The data is processed in the vertical video processing section 112, which removes the wings.
Data is written from the vertical video processing section 112 in a slice-scan format back to a first frame buffer area 114 in the external memory source 98. Data is read from a second frame buffer area 116 in the external memory source 98 in a conventional raster-scan sequence for input to a horizontal processing block 118 located in the video processor 96. The output of the horizontal processing block 118 is in raster-scan format and is the output of the video processor 96.
In one preferred embodiment of the present invention, video input data is provided as interlaced fields of data in a 720×240 pixel field format. Each video field is conceptually broken into 20 slices of width 36 pixels, each having left and right wings of 6 pixels each (outer wings of 2 pixels each and inner wings of 4 pixels each). The wings are added at the appropriate points in the video input data stream, and the resulting data stream is written in raster-scan sequence into a first field buffer in an external SDRAM.
Three fields of data are read from the SDRAM simultaneously. The data for these fields is sourced by second, third and fourth field buffers and is read in vertical slices of 48 pixel wide (slice core and wings) by 240 rows. The data is processed by a first vertical processing stage that provides slice-scan format data at twice the input rate of a single field to a second stage. Slice data input to the second stage is formatted in slices of 44 pixels wide by 480 rows (due to the rate-doubling action of the first stage). The second vertical processing stage processes the data and provides 36 pixel wide slice-scan format data at the same rate as the input to that stage to a third vertical processing stage.
The third stage is a vertical scaler and performs no horizontal processing, and so does not require wings on the slice format data. Data is output from the third processing stage in a 36 pixel wide slice-scan format to a first frame buffer area in the SDRAM. The number of rows in each slice is dependent on the specific vertical scaling ratio chosen. Data is input to a horizontal-only processing stage in conventional raster-scan format of 720×480*M pixels, where M is the vertical scaling factor in the third vertical processing stage. This data is processed by the horizontal processor (which includes a horizontal scaler) and is output in a conventional raster-scan format at a resolution of 720*N×480*N, where N is the horizontal scaling factor.
Overall, this implementation results in a greater than 10× reduction in on-chip memory requirements due to the slice-scan architecture. This expense saved with the reduction in on-chip memory requirements more than offsets the additional required external memory, and provides a variety of prototyping and production options.
It will therefore be appreciated that the present invention provides a method and apparatus of reducing on-chip memory requirements by processing a digital image along a vertical axis by sequencing the image in vertical slices. The invention has been described herein in terms of several preferred embodiments. Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention. Furthermore, certain terminology has been used for the purposes of descriptive clarity, and not to limit the present invention. The embodiments and preferred features described above should be considered exemplary, with the invention being defined by the appended claims.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4002827 *||May 15, 1975||Jan 11, 1977||General Electric Company||Polar coordinate format to a cartesian coordinate format scan converter|
|US4689675 *||May 23, 1985||Aug 25, 1987||Advanced Systems Development, Inc.||Digital scan converter and method therefor|
|US4959715 *||Jun 16, 1989||Sep 25, 1990||North American Philips Corporation||Method and apparatus for converting an interlaced television transmission to a sequential display using maximum likelihood estimation|
|US5170259 *||Sep 30, 1991||Dec 8, 1992||Victor Company Of Japan, Ltd.||Motion compensated predictive coding/decoding system of picture signal|
|US5347314 *||Sep 30, 1992||Sep 13, 1994||Yves C. Faroudja||Video scan converter|
|US5444493 *||Jun 22, 1993||Aug 22, 1995||Deutsche Thomson-Brandt Gmbh||Method and apparatus for providing intra-field interpolation of video signals with adaptive weighting based on gradients of temporally adjacent fields|
|US5543858 *||Jun 3, 1994||Aug 6, 1996||U. S. Philips Corporation||Method of and apparatus for reducing noise in a motion detector signal|
|US5600731 *||May 9, 1991||Feb 4, 1997||Eastman Kodak Company||Method for temporally adaptive filtering of frames of a noisy image sequence using motion estimation|
|US5784115 *||Dec 31, 1996||Jul 21, 1998||Xerox Corporation||System and method for motion compensated de-interlacing of video frames|
|US5850487 *||Apr 24, 1995||Dec 15, 1998||Fuji Photo Film Co., Ltd.||Digital image processing apparatus|
|US5854636 *||Sep 16, 1997||Dec 29, 1998||Hitachi, Ltd.||Semiconductor IC with a plurality of processing circuits which receive parallel data via a parallel data transfer circuit|
|US5856930 *||Mar 28, 1994||Jan 5, 1999||Sony Corporation||Disc-shaped recording medium, disc recording apparatus and disc reproducing apparatus|
|US6014182 *||Oct 10, 1997||Jan 11, 2000||Faroudja Laboratories, Inc.||Film source video detection|
|US6034733 *||Jul 29, 1998||Mar 7, 2000||S3 Incorporated||Timing and control for deinterlacing and enhancement of non-deterministically arriving interlaced video data|
|US6055018 *||Nov 4, 1997||Apr 25, 2000||Ati Technologies, Inc.||System and method for reconstructing noninterlaced captured content for display on a progressive screen|
|US6104755 *||Sep 12, 1997||Aug 15, 2000||Texas Instruments Incorporated||Motion detection using field-difference measurements|
|US6166772 *||Apr 1, 1997||Dec 26, 2000||Compaq Computer Corporation||Method and apparatus for display of interlaced images on non-interlaced display|
|US6222589 *||Aug 8, 1996||Apr 24, 2001||Yves C. Faroudja||Displaying video on high-resolution computer-type monitors substantially without motion discontinuities|
|US6266092 *||May 12, 1998||Jul 24, 2001||Genesis Microchip Inc.||Method and apparatus for video line multiplication with enhanced sharpness|
|US6269484 *||Jun 24, 1997||Jul 31, 2001||Ati Technologies||Method and apparatus for de-interlacing interlaced content using motion vectors in compressed video streams|
|US6295041 *||Mar 5, 1997||Sep 25, 2001||Ati Technologies, Inc.||Increasing the number of colors output by an active liquid crystal display|
|US6298144 *||May 20, 1998||Oct 2, 2001||The United States Of America As Represented By The National Security Agency||Device for and method of detecting motion in an image|
|US6489998 *||Aug 11, 1999||Dec 3, 2002||Dvdo, Inc.||Method and apparatus for deinterlacing digital video images|
|US6504577 *||Jul 6, 2000||Jan 7, 2003||Compaq Information Technologies Group, L.P.||Method and apparatus for display of interlaced images on non-interlaced display|
|US6545719 *||Mar 31, 2000||Apr 8, 2003||Matsushita Electric Industrial Co., Ltd.||Apparatus and method for concealing interpolation artifacts in a video interlaced to progressive scan converter|
|US6577345 *||Jul 26, 2000||Jun 10, 2003||Lg Electronics Inc.||Deinterlacing method and apparatus based on motion-compensated interpolation and edge-directional interpolation|
|US6847405 *||Sep 14, 2001||Jan 25, 2005||Sony Corporation||Motion-adaptive de-interlacing method and system for digital televisions|
|US6867814 *||Apr 18, 2001||Mar 15, 2005||Silicon Image, Inc.||Method, system and article of manufacture for identifying the source type and quality level of a video sequence|
|US6909469 *||Aug 28, 2001||Jun 21, 2005||Silicon Image, Inc.||Interlace motion artifact detection using vertical frequency detection and analysis|
|US20010016009 *||Dec 30, 1997||Aug 23, 2001||Robert N. Hurst||Reduced cost decoder using bitstream editing for image cropping|
|US20020109790 *||Dec 22, 2000||Aug 15, 2002||Mackinnon Andrew Stuart||Method and apparatus for detecting motion and absence of motion between odd and even video fields|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US6944821 *||Dec 7, 1999||Sep 13, 2005||International Business Machines Corporation||Copy/paste mechanism and paste buffer that includes source information for copied data|
|US7853734 *||Mar 11, 2003||Dec 14, 2010||Broadcom Corporation||Video bus for a video decoding system|
|US20040078504 *||Mar 11, 2003||Apr 22, 2004||Patrick Law||Video bus for a video decoding system|
|US20110109794 *||May 12, 2011||Paul Wiercienski||Caching structure and apparatus for use in block based video|
|U.S. Classification||348/716, 348/441, 348/E05.133, 386/E09.013, 348/E05.062, 386/E05.07, 348/E07.015, 348/E05.108|
|International Classification||H04N5/44, H04N5/66, H04N5/85, H04N5/775, H04N5/14, H04N7/01, H04N9/804|
|Cooperative Classification||H04N5/85, G09G2310/0229, H04N5/14, H04N5/775, H04N5/4401, H04N5/66, H04N7/012, H04N7/0112, H04N9/8042|
|European Classification||H04N7/01G3, H04N9/804B, H04N5/775, H04N7/01F|
|May 3, 2004||AS||Assignment|
Owner name: SILICON IMAGE, INC., CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:FREELAND, HOWARD, SECRETARY OF DVDO, INC.;REEL/FRAME:015275/0046
Effective date: 20031121
|Apr 11, 2007||AS||Assignment|
Owner name: DVDO, INCORPORATED, CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ADAMS, DALE R.;THOMPSON, LAURENCE A.;BANKS, JANO D.;REEL/FRAME:019148/0282
Effective date: 19990722