WO2004044843A2

WO2004044843A2 - Systems and methods for image enhancement in multiple dimensions

Info

Publication number: WO2004044843A2
Application number: PCT/US2003/035577
Authority: WO
Inventors: Richard G. Hier; Terrence J. Coleman
Original assignee: Digivision, Inc.
Priority date: 2002-11-06
Filing date: 2003-11-06
Publication date: 2004-05-27
Also published as: JP2006506021A; EP1590764A4; US20040234154A1; US7668390B2; JP2010213284A; CA2505260A1; US20040227773A1; AU2003287577A1; KR20050084992A; EP1590764A2; JP5000737B2; US7092582B2; WO2004044843A3; JP4503441B2; AU2003287577A8

Abstract

A multi-dimensional data enhancement system uses large kernel filtering, decimation, and interpolation, in multi-dimensions to enhance the multi-dimensional data in real-time (2). The multi-dimensional data enhancement system is capable of performing large kernel processing in real-time because the required processing overhead is significantly reduced (7). The reduction in processing overhead is achieved through the use of substantial low pass filtering and decimation that significantly reduces the amount of data that needs to be processed in order to generate an un-sharp mask comprising only very low spatial frequencies (9), that can be used to further process the data in a more natural way than is normally achieved with conventional edge sharpening.

Description

S P E C I F I C A T I O N

SYSTEMS AND METHODS FOR IMAGE ENHANCEMENT IN MULTIPLE

DIMENSIONS

RELATED APPLICATION INFORMATION [001] This application claims priority under 35 USC §119 to U.S. Provisional

Patent Application 60/424,472, entitled "Multi-Dimensional Filtering, Decimation, and Processing of data Sets and Images," filed November 11, 2002, and to U.S. Provisional Patent Application 60/424,473, entitled "Large Kernel Processing of Pixels Near Edges of Data Frames With Additional Artificial Border Data by Processing the Border Data at a Higher Rate," filed November 11, 2002, both of which are incorporated herein in their entirety as if set forth in full.

BACKGROUND

1. Field of the Invention

[002] The present invention relates to multi-dimensional data processing, and more particularly, to the enhancement of image data.

2. Background Information

[003] Imaging systems play a varied and important role in many different applications. For example, medical imaging applications, such as endoscopy, fluoroscopy, X-ray, arthroscopy, and microsurgery applications are helping to save lives and improve health. Industrial applications, such as parts inspection systems that can detect microscopic errors on an assembly line, are leading to increased yields and efficiencies. A wide variety of military and law enforcement applications are taking advantage of imaging technology for target acquisition, surveillance, night vision, etc. Even consumer applications are taking advantage of advanced video imaging technology to produce heightened entertainment experience, such as the improved picture quality provided by High Definition Television (HDTV).

[004] While there have been many advancements in video imaging technology, conventional video imaging systems can still suffer from deficiencies that impact the quality and usefulness of the video imagery produced. For example, video images generated with uncontrolled illumination often contain important, but subtle, low- contrast details that can be obscured from the viewer's perception by large dynamic range variations in the image. Any loss, or difficulty, in perceiving such low-contrast details can be detrimental in situations that require rapid responses to, or quick decisions based on, the images being presented.

[005] A number of techniques have been applied to enhance video imagery.

These techniques include image filtering applied in real-time. Conventional real-time filtering techniques can today be implemented as digital convolution over kernels comprising a relatively small number of image pixels, e.g., 3x3 pixel kernels, or 7x7 pixel kernels. These techniques can use high-pass filtering to emphasize details that are small relative to the size of the kernel being used. The improvement that can be achieved using such small kernels, however, is often limited. Studies have shown that significantly larger kernels are far more effective at achieving meaningful video enhancement. Unfortunately, the processing overhead required to perform large kernel convolution, in real-time, using conventional techniques, is prohibitive at the present state of digital signal processing technology. SUMMARY

[006] A multi-dimensional data enhancement system uses large kernel convolution techniques, in multi-dimensions, to improve image data in real-time. The multi-dimensional data enhancement system is capable of performing large kernel processing in real-time because the required processing overhead is significantly reduced. The reduction in processing overhead is achieved through the use of multidimensional filtering, decimation, and processing that reduces the amount of data that needs to be handled in certain stages of the operation, but still provides the same beneficial image enhancement.

[007] In another aspect of the invention, the enhancement system can reduce the effect of pixels in surrounding image frames and the effect of blanking data on the processing of a pixel near the edge of an image frame by inserting fictional blanking data into the blanking areas.

[008] These and other features, aspects, and embodiments of the invention are described below in the section entitled "Detailed Description of the Preferred Embodiments."

BRIEF DESCRIPTION OF THE DRAWINGS

[009] Features, aspects, and embodiments of the inventions are described in conjunction with the attached drawings, in which:

[010] Figure 1 is a diagram illustrating an exemplary display of image data;

[011] Figure 2 is a diagram illustrating filtering, and decimation of image data in accordance with one embodiment of the invention;

[012] Figure 3 is a diagram illustrating interpolation of the filtered and decimated image data of figure 2 in accordance with one embodiment of the invention; [013] Figure 4 is a diagram illustrating a 3 -dimensional image that can be the subject of filtering, decimation, and interpolation similar to that of figures 2 and 3;

[014] Figure 5 is a diagram illustrating an example circuit for filtering and decimating image data in one dimension in accordance with one embodiment of the systems and methods described herein;

[015] Figure 6 is a diagram illustrating an example circuit for filtering and decimating the image data of figure 5 in a second dimension in accordance with one embodiment of the systems and methods described herein;

[016] Figure 7 is a diagram illustrating an example circuit for interpolating the image data of figure 6 in accordance with one embodiment of the systems and methods described herein;

[017] Figure 8 is a diagram illustrating an example circuit for interpolating the image data of figure 7 in accordance with one embodiment of the systems and methods described herein;

[018] Figure 9 is a diagram illustrating the application of a relatively small kernel filter to input data in order to generate an unsharp mask that can be used to generate an enhanced version of the input data in accordance with one embodiment of the systems and methods described herein;

[019] Figure 10 is a diagram illustrating the application of a larger kernel filter to input data in order to generate an even smoother unsharp mask that can be used to generate an enhanced version of the input data in accordance with one embodiment of the systems and methods described herein;

[020] Figure 11 A is a diagram illustrating a curve representing input image data; [021] Figure 1 IB is a diagram illustrating a curve representing an un-sharp mask generated from the image data of figure 11 A;

[022] Figure 11C is a diagram illustrating a curve representing the amplified difference between the curve of figures 11A and the curve of figure 1 IB;

[023] Figure 1 ID is a diagram illustrating a curve representing another un-sharp mask generates from the image data of figure 11 A;

[024] Figure 1 IE is a diagram illustrating a curve representing the amplified difference between the curve of figures 11A and the curve of figure 1 ID;

[025] Figure 1 IF is a diagram illustrating a curve representing the amplified difference between the curve of figures 1 IB and the curve of figure 1 ID;

[026] Figure 12 is a diagram illustrating an exemplary cable system comprising a video enhancement device configured in accordance with one embodiment of the invention;

[027] Figure 13 is a diagram illustrating an exemplary NTSC video image display;

[028] Figure 14 is a diagram illustrating an exemplary HDTV video image display; and

[029] Figure 15 is a diagram illustrating an example circuit for creating fictional blanking area in accordance with the systems and methods described herein.

DETAILED DESCRIPTION

[030] The systems and methods described below are generally described in relation to a two dimensional video image system; however, it will be understood that the systems and methods described are not limited to applications involving video image systems nor to image processing systems comprising only two dimensions. For example, the filtering techniques described herein can also be used in data storage and data compression schemes.

[031] Figure 1 is a diagram illustrating an exemplary video image display 100.

The image comprises a plurality of pixels arranged in rows 102 and columns 104. In most conventional, e.g., rasterized, video imaging systems, the pixels are scanned in both the horizontal and vertical dimensions.

[032] It is often difficult to provide meaningful enhancement of video images in a conventional system. Real-time correction, or enhancement techniques do exist, but many such conventional techniques have used global approaches. In other words, all points of a given intensity on an input image must be mapped to the same corresponding output intensity. When applied correctly, such techniques can selectively help expand subtle details in some image areas; however, such approaches also often result in undesirable side effects, such as saturation of other broad bright or dark areas resulting in lose of detail in these areas. In order to provide enhancement and avoid some of the previously encountered drawbacks, relatively small kernel processing techniques have been used for non-global types of enhancement through a convolution process that generates a filtered output image from an input image, where each pixel in the output image results from considering the input image pixel values in an area surrounding the corresponding pixel, where the area is defined by the kernel size. [033] In convolution processing, the values of pixels in an area around, and including, a pixel of interest are each multiplied by a coefficient, which is the corresponding element in the so-called "convolution kernel" and then these products are added to generate a filtered output value for the pixel of interest. The result of the convolutional processing is a low pass filtered version, or an "un-sharp mask", of the input data. The values of the coefficients assigned to the elements of kernel can be configured so as to set the cut-off for the low pass filter operation being performed. In order to enhance the original image data, the un-sharp mask can be subtracted from the original input data, which will produce a high pass version of the input data. The high pass version of the input data can then be amplified, or otherwise enhanced, and then can be recombined with the original data. The result can be an enhancement, or sharpening of the original image.

[034] Often, however, it is easier to define coefficients that accomplish both the low pas filtering and subtraction from the original data in one step. Thus, simply passing the data through a single filtering step can generate enhanced data. But because the filter becomes more complex, it is only really practical to apply such one step filtering techniques to a limited kernel size, i.e., a 3x3 pixel kernel or a 7x7 pixel kernel. [035] Small kernel buffering techniques require more processing overhead than global techniques. Thus, the buffering, and processing overhead requirements have limited conventional kernel based approaches to relatively small kernel sizes. As mentioned above, however, small kernel operation produce limited enhancement. Larger kernel operation can produce greater enhancement, but as explained above, the overhead required by large kernel sizes has traditionally proved prohibitive. As explained below, however, the systems and methods described herein can allow for large kernel sizes to be used to enhance video images in real-time, without the excessive overhead that plagues conventional systems. Moreover, the enhancement can be provided in multi-dimensional space, i.e., space with (N) dimensions, where N= 2, 3, 4, ...n.

[036] Briefly, the systems and methods described herein take advantage of the fact that most image data resides at low frequencies. Therefore, the data can be low passed filtered to reduce the amount of data being handled in each dimension. Then, this data can be decimated to further reduce the amount of data being handled. The data can then be interpolated and subtracted from the original data. The resulting image data can then be used, or combined with the original data to create an enhanced image that can be displayed.

[037] Thus, as illustrated in figure 2, in a multi-dimensional data space, a low pass filter or a plurality of separable low pass filters can be implemented to decimate, or subsample, each of the N-dimensions in a successive order. The order is typically based on the scanning sequence associated with the data in the multi-dimensional data space. Examples of multi-dimensional data space can, for example, include two-dimensional video images in x and > dimensions. For a typical two-dimensional image that is scanned first in the horizontal direction and then in the vertical direction, decimation, or subsampling, can be performed in the same order as that of the scan, i.e., first in the horizontal direction and then in the vertical direction. The image data can be digital, analog, or mixed signal. For a typical two-dimensional digital video image, the pixel intensities can be represented as numbers in a two-dimensional data space. [038] In order for decimation, or sub-sampling, operations described herein to achieve the desired results of greatly easing the requirements of computing speed and data storage in subsequent dimensions, the low pass filtering of data within each dimension is preferably substantial, such that much of the high-frequency information is intentionally suppressed. In large-kernel low pass filtering operations on N-dimensional data sets, for example, all data except those at the very low frequencies are suppressed by using a low pass filter with a very low cut-off frequency. After the large-kernel low pass filtering and decimation operations, the data can then be interpolated and combined with the original data or the high frequency data using various types of algorithms to produce enhanced results. [039] Ordinarily, for correct alignment, the data containing the high spatial frequencies should ordinarily be delayed to match the filter delay inherent in the low pass filtering operation; however, in cases in which a sequence of similar images exist, the low frequency data can be delayed by one field time minus the filter delay, and an "unsharp mask" of the data, which is low spatial frequency data from one frame, can be used as information for an approximate correction term to be applied to the next frame. This is because the accuracy required is not stringent in most cases due to the fact that the low pass filtered data by nature contains little or no detailed information which would otherwise require precise alignment.

[040] Because of the very low frequency filtering in large kernel operations, the decimation or subsampling of data in each dimension can greatly reduce the need for computing speed and data storage in subsequent dimensions in the multi-dimensional data space. For example, for a two-dimensional video image, low pass filtering and decimation can be performed together for each dimension, first horizontally and then vertically. The decimation, or subsampling, in the horizontal direction can make realtime processing efficient by reducing the number of operations necessary to perform filtering in the subsequent dimension. Furthermore, the decimation, or sub-sampling in the vertical direction can greatly reduce the amount of storage space required for the low frequency data.

[041 ] In general, the advantages of reduced requirements for data storage and computing power are more pronounced in data processing operations in a data space with a greater number of dimensions. For example, if low pass filtering results in data compression by a factor of 10 in each dimension, the processing of data in an N- dimensional data space will result in a reduction of the required data storage and processing power by a factor of 10^. [042] For practical applications, because of the reduced requirements for processing power and data storage space, it is possible to combine different circuits for various functions, including, e.g., low pass filters, decimation or subsampling processors, and/or data storage memory, into a single device, such as an application specific integrated circuit (ASIC). Further, processor circuits for interpolation and other processing functions including, for example, various types of algorithms for enhancements, can also be integrated on the same ASIC.

[043] In embodiments in which low frequency data is delayed while high frequency data is not, there is no need for any memory to store the high frequency data, such that the high frequency data can remain pristine with a high bandwidth. The high frequency data can even, depending on the implementation, remain in analog form and need not be sampled at all.

[044] Returning to Figure 2, a diagram is presented illustrating an example of filtering and decimation of a two-dimensional image that is scanned first horizontally and then vertically. The two-dimensional image can, for example, be a raster-scanned image in a traditional NTSC, or HDTV, system. It is assumed that the two-dimensional video image in Figure 2 is scanned horizontally pixel by pixel from left to right and vertically line by line from top to bottom. In Figure 2, an initial image 202 is scanned horizontally with pixel values represented by a curve 204 that is transmitted to a first low pass filter (LPF) and decimator 206, which generates a filtered and decimated output signal represented by a curve 208. The video image that has been filtered and decimated by the LPF and decimator 206 in the horizontal dimension is represented as an image 210, with a plurality of vertical columns of pixels that are spaced relatively far apart from one another. The pixel values of one of the columns of the image 210 can be represented by a signal curve 212, which is transmitted to a second LPF and decimator 214. The output signal of the second LPF and decimator 214 can be represented by a curve 216, which forms an output video image 218 after the processes of low pass filtering and decimation in both horizontal and vertical dimensions.

[045] Figure 3 is a diagram illustrating an example interpolation of low pass filtered and decimated video signals for constructing an un-sharp mask in accordance with the systems and methods described herein. In Figure 3, the video image 218, which has been low pass filtered and decimated as illustrated in Figure 2, with a signal curve 216, passes through a first interpolator with an optional low pass filter 322, column by column, to construct a signal curve 324. Signal curve 324 forms an intermediary video image 326, which has a plurality of columns of pixels spaced relatively far apart from one another. In the horizontal direction, signal curve 328 as shown in Figure 3, is transmitted to a second interpolator with an optional low pass filter 330 to construct a signal curve 332 in the horizontal direction. The video signals which have been interpolated both vertically and horizontally form an output un-sharp mask 334. [046] The two-dimensional low pass filtering, decimation, and interpolation processes, as illustrated in Figures 2 and 3, can also be extended to data sets of three or more dimensions, such as a three-dimensional representation of an apple 436 as shown in Figure 4. The pixels representing apple representation 436 can be scanned successively in x, y and z directions. The low pass filtering and decimation processes can then be performed with separable low pass filters and successive dimensional decimators in the same order as that of pixel scanning. The three-dimensional object, which in this case is apple 436, can be scanned in three dimensions with magnetic resonance imaging (MRI), for example. A three-dimensional data set is formed by sampling the density at each point in the three-dimensional space. [047] The filtering, decimation, and interpolation in subsequent dimensions of the ^-dimension can be performed in parallel or in serial, depending on the embodiment. Parallel filtering operation have the added benefit that data does not need to be queued up, or buffered, as with serial operations. But serial operation typically require less resources, at least in hardware implementations.

[048] Thus, using the systems and methods described in relation to figures 2 and

3, large kernel filtering operations can be performed, providing greater enhancement than was previously possible, do to the fact that the data is decimated, which reduces the amount of data that needs to be stored and processed in subsequent operations. Unlike current conventional small kernel techniques, which perform low pass filtering and subtraction in one step, the systems and methods described herein perform the low pass filtering independently of the other steps, which allows the data to then be decimated enabling large kernel operations. Low pass filtering allows decimation, because you do not need as many data points to represent the low pass filtered, or low spatial frequency data. As a result, once the data is low pass filtered in the first dimension, the data can be decimated and subsequent operations in other dimensions can be performed on the reduced data set, which reduces storage and processing burdens. [049] Once the low pass filtered data is decimated stored and then interpolated, producing un-shaφ mask 334, it can be recombined with the original data in such a manner as to produce enhanced data. For example, un-sharp mask 334 can be subtracted from the original data. As mentioned above, an un-sharp mask 334 produced form one frame can actually be used to enhance the next frame, since un-sharp mask 334 comprises strictly low spatial frequency data. In other words, since un-sharp mask 334 generally does not vary much from frame to frame, an un-sharp mask produced form a previous frame can be used to enhance the image data in the next frame. Using this technique, the original data does not need to be slowed down, or buffered, while unsharp mask 334 is generated. Thus, the large kernel enhancement described herein can be performed in real time. Conversely, the input data can be slowed down to allow time for the generation of un-sharp mask 334, but this is often not preferable, because it is not in real time of the input data is slowed down.

[050] As mentioned above, the filtering, decimation, and interpolation process described herein can also be used for data compression. For example, depending on the embodiment, the low frequency and high frequency data can be separated and transmitted separately, with the low frequency data being subsampled to save data storage space in a compression scheme, for example, and then recombine with the high frequency data to recover the original data set or image.

[051] Example filtering circuits configured to implement the systems and methods described herein are described in detail in the following paragraphs. Thus, figures 5-8 illustrate embodiments of low pass filtering with decimation and separable interpolation using polyphase Finite Impulse Response (FIR) filters for the filtering of two-dimensional data sets, for example, two-dimensional video images. Figures 5 and 6 form a block diagram illustrating low pass filtering and decimation in two dimensions, whereas Figures 7 and 8 form a block diagram illustrating separable interpolation in two dimensions using polyphase FIR filters. In Figure 5, input data representing full- bandwidth video images can be passed to a series of multiple-pixel delay buffer beginning with buffer 802a and a string of multipliers beginning with multiplier 504a. As shown, a set of selectable filter coefficients 806a can be provided for selection by a coefficient multiplexer 508a. The coefficient selected by coefficient multiplexer 508a can then be transmitted to multiplier 504a for multiplication with the full-bandwidth input data. The result of the multiplication can then be passed from the output of multiplier 504a to an adder and accumulator 512.

[052] The full-bandwidth data delayed by multiple-pixel delay buffer 502a, can then be passed to a second multiplier 504b. A second set of filter coefficients 506b can then be provided for selection by a second coefficient multiplexer 508b. The selected coefficient can be passed to second multiplier 504b for multiplication with the full- bandwidth input data, which has been delayed by multiple-pixel delay buffer 502a. A plurality of such multiple-pixel delay buffers 502a, 502b (not shown), 502c (not shown), . . . 502n as well as a plurality of multipliers 504a, 504b, 504c (not shown), . . . 504n, and a plurality of selectable coefficient multiplexers 508a, 508b, 508c (not shown), . . . 508n can be connected to form a polyphase FIR filter in accordance with the systems and methods described herein.

[053] It should be noted that the selection of coefficients 506a, 506b, . . ., 506n, can be controlled by a processor (not shown) interfaced with the polyphase FIR filter of Figure 5. In certain embodiments, the coefficients are simply loaded by the processor initially and then the processor is not involved. The number of coefficients is equal to the number of delays 502a-502n. As each new pixel is processed, a new coefficient 506a is selected. For example, the first of coefficients 506a can be selected when a first pixel is being processed, the second of coefficients 506a can be selected when the next pixel is being processed, and so on until the process loops back and the first of coefficients 506a is selected again.

[054] Thus, for example, if the decimation in figure 2 is 4-to-l, i.e., for every four pixels in data set 202, there is one pixel in data set 210, then there would be four delay buffers 502a-502d and four coefficients 506a. Similarly, there will be four coefficients 506b-506n. [055] The results of multiplication by multipliers 504a, 504b, 504c (not shown),

. . . 504n are passed to adder and accumulator 512, which can be configured to generate horizontally decimated video data for decimation in a second dimension, e.g., by another polyphase FIR filter, such as the one illustrated in Figure 6. The full-bandwidth input data can be sent to successive multiple-pixel delay buffers 602a, 602b, . . . 602n and multipliers 604a, 604b, . . . 604n at the full-pixel data rate, i.e., the rate at which the pixels of a video image are scanned. The data transmitted from adder and accumulator 612 can, however, be at a horizontally decimated data rate, which can be a much lower data rate than the full-pixel data rate. As a result, the overhead required for subsequent operations can be reduced.

[056] Figure 6 is a block diagram illustrating a polyphase FIR filter configured to decimate data sets in a second dimension after the data sets, or video images, have been decimated in a first dimension, e.g., by the polyphase FIR filter of Figure 5, in accordance with one embodiment of the systems and methods described herein. For example, two-dimensional video images scanned first in the horizontal direction and then in the vertical direction, can be first decimated horizontally by the polyphase FIR filter of Figure 5 and then decimated vertically by the polyphase FIR filter of Figure 6. [057] Referring to Figure 6, the output data from adder and accumulator 512 can be passed to a multiple-line horizontally decimated delay buffer 622a and a multiplier 624a. Selectable filter coefficients 626a can then be provided for selection by a coefficient multiplexer 628a. The selected coefficient can then be multiplied with the horizontally decimated data by first multiplier 624a to generate an output that is transmitted to an adder an accumulator 632.

[058] The horizontally decimated input data which has passed through the first multiple-line horizontally decimated delay buffer 622a can then be passed to a second multiplier 624b. A second set of selectable filter coefficients 626b can be provided for selection by a second coefficient multiplexer 628b. Second multiplier 624b can multiply the coefficient selected by multiplexer 628b with the data that has been delayed by first multiple-line horizontally decimated delay buffer 622a. The result of multiplication by second multiplier 624b can then be passed from the output of second multiplier 624b to adder and accumulator 632. After a series of successive delays of horizontally decimated data, the last multiple-line horizontally decimated delay buffer 622n can be configured to transmit the delayed data to a last multiplier 624n. A set of selectable filter coefficients 626n can be provided for selection by a last coefficient multiplexer 628n, which can transmit the selected coefficient to multiplier 624n for multiplication with the data delayed by last delay buffer 622n. The result of the multiplication can then be transmitted from multiplier 624n to adder and accumulator 632. [059] In the embodiment shown in figure 6, the horizontally decimated input data are transmitted through a successive chain of multiple-line horizontally decimated delay buffers 622a, 622b (not shown), 622c (not shown), . . . 622n for multiplication with respective filter coefficients selected by respective multiplexers. The results of multiplication by multipliers 624a, 624b, 624c (not shown), . . . 624n can be passed to adder and accumulator 632, which can be configured to generate the result of horizontal decimation. The results generated by adder and accumulator 632 can be stored in a two- dimensionally decimated frame storage and delay buffer 634 for further processing. The data received from the polyphase FIR filter of Figure 5 can be passed through multiple- line horizontally decimated delay buffers 622a, 622b, . . . 622n and the multipliers 624a, 624b, . . . 624n at the horizontally decimated data rate, whereas the data transmitted from adder and acςumulator 632 to two-dimensionally decimated frame storage and delay buffer 634 can be transmitted at a two-dimensionally decimated data rate, which can be a much lower data rate than the horizontally decimated data rate. [060] Figure 7 is a block diagram illustrating a polyphase FIR filter for interpolation in the vertical dimension of data which have been decimated in both the horizontal and vertical dimensions by the polyphase FIR filters of figures 5 and 6. In the embodiment of figure 7, two-dimensionally decimated frame storage and delay buffer 634 transmits the decimated data to a first horizontally decimated recirculating line delay 742a for vertically decimated lines. Horizontally decimated recirculating line delay 742a outputs temporally delayed data to a first multiplier 744a and to a second horizontally decimated recirculating line delay 742b for the vertically decimated lines. [061] A first set of selectable filter coefficients 746a can be provided for selection by a first coefficient multiplexer 748a, which can be configured to select a first coefficient for multiplication with the delayed data received from horizontally decimated recirculating line delay 742a. First multiplier 744a can be configured to multiply the first coefficient with the temporally delayed input data to produce a result which is transmitted from the output of the first multiplier 744a to an adder 752. [062] Similarly, a second set of selectable filter coefficients 746b can be provided for selection by a second coefficient multiplexer 748b, which can be configured to select a coefficient from the second set of coefficients for multiplication with twice delayed input data by a second multiplier 744b. Multiplier 744b multiplies the selected coefficient with the decimated input data that has been passed through the first two horizontally decimated recirculating line delays 742a and 742b for vertically decimated lines, to produce a result at the output of the second multiplier 744b. [063] Thus, the two-dimensionally decimated data can pass through a plurality of horizontally decimated recirculating line delays 742a, 742b, . . . until it reaches the last horizontally decimated recirculating line delay 742n. At which point, a set of selectable filter coefficients 746n can be provided for selection by a coefficient multiplexer 748n, which can be configure to select a coefficient from the set of selectable coefficients 742n for multiplication by the multiplier 744n. Multiplier 744n can be configured to multiply the coefficient with two-dimensionally decimated data that has passed through the series of horizontally decimated recirculating line delays 742a, 742b, . . . 742n, to generate a result at the output of the multiplier 744n. [064] Adder 752, which is connected to the outputs of multipliers 744a, 744b, . .

. 744n, respectively, can be configured to then calculate the resulting video data that has been reconstructed by interpolation in the vertical dimension. [065] In the embodiment of figure 7, the two-dimensionally decimated video data is transmitted at a two-dimensionally decimated data rate from the frame storage and delay buffer 634 to the series of horizontally decimated recirculating line delays 742a, 742b, . . . 742n for the vertically decimated lines. In contrast, the video data that has been vertically interpolated by multipliers 744a, 744b, . . . 744n and adder 752 can be transmitted at a horizontally decimated data rate that is a higher data rate than the two- dimensionally decimated data rate. The vertically interpolated video data generated at the output of adder 1052 can then be transmitted to the polyphase FIR filter of figure 8 for horizontal interpolation.

[066] Figure 8 is a block diagram illustrating a polyphase FIR filter configured for horizontal interpolation of the horizontally decimated video data generated by the FIR filter of figure 10 to produce output video data at full-bandwidth, but with reduced spectral content, in accordance with one example embodiment of the systems and methods described herein. As shown in the embodiment of figure 8, the horizontally decimated data received from the output of the polyphase FIR filter of figure 7 can be transmitted to a first latch 862a for horizontally decimated pixels, which in turn can be configured to transmit the temporally delayed data to a second latch 862b and to a first multiplier 864a.

[067] A set of selectable filter coefficients 866a can then be provided for selection by a coefficient multiplexer 868a. Coefficient multiplexer 868a can be configured to output the selected coefficient to a multiplier 864a that can be configured to multiply the coefficient with the temporally delayed video data that has passed through latch 862a, to produce a result at the output of multiplier 864a. Again the coefficients can be loaded by a processor (not shown) and then selected as each pixel is processed.

[068] Similarly, a second set of selectable filter coefficients 866b can be provided for selection by a second coefficient multiplexer 868b, which can be configured to select a coefficient form the second set of coefficients for multiplication by a second multiplier 864b. Multiplier 864b can be configure to multiply the coefficient with the video data that has passed through the first two latches 862a and 862b to generate a result at the output of the second multiplier 864b.

[069] The input video data that has passed through the series of latches 862a,

862b, . . . 82n for the horizontally decimated pixels can then be transmitted to a final coefficient multiplier 864n. At which point, a set of selectable filter coefficients 866n can be provided for selection by a coefficient multiplexer 868n, which can be configured to select a coefficient from the set of coefficients 866n for multiplication with the temporally delayed video data that has passed through the series of latches 862a, 862b, . . . 862n. Multiplier 864n generates a result at the output of multiplier 864n. [070] An adder 872 can be interfaced with the outputs of respective multipliers

864a, 864b, . . . 864n to produce vertically and horizontally interpolated output data with a reduced bandwidth at the output of adder 872.

[071] Thus, in the embodiment shown in figure 8, the vertically interpolated input data can be transmitted to the series of latches 862a, 862b, . . . 862n for the horizontally decimated pixels at the horizontally decimated data rate, whereas the output data resulting from interpolation by multipliers 864a, 864b, . . . 864n and adder 872 can be transmitted at a full-pixel data rate, which can be the same rate at which the full- bandwidth input data is transmitted to the polyphase FIR filter for horizontal decimation illustrated in figure 5. The resulting output from the polyphase FIR filter of figure 8 can, however, have be at full bandwidth, but comprise less spectral content, because it is a low pass filtered version of the input data. The output data from the polyphase FIR filter of figure 8 can, therefore, be used as low pass filter data for video enhancement algorithms to produce various enhancement effects.

[072] Although embodiments have been described with respect to specific examples of two-dimensional separable low pass filtering, decimation, and interpolation using polyphase FIR filters, the systems and methods described herein should not be seen as limited to such specific implementations. For example, three-dimensional video images and other types of multi-dimensional data sets with two or more dimensions can also be processed according to the systems and methods described herein. Furthermore, other types of filters such as IIR filters can also be used for low pass filtering, decimation, and interpolation operations as required by a specific implementation. [073] The output of the filtering, decimation, and interpolation systems illustrated in figures 2 and 3 can be referred to as an unsharp mask of the input data. Different kernel sizes, i.e., numbers of coefficients, will result in different unsharp masks. Thus, a plurality for unshaφ masks can be predefined for a given system, e.g., an HDTV system, and the user can be allowed to select the enhancement in different spatial frequency bands by selecting which of the plurality of unshaφ masks should be applied to the input data for the system.

[074] For example, Figure 9 illustrates the application of a relatively smaller kernel to the input data of curve 902. It should be kept in mind that figure 9 still depicts a large kernel operation. Thus, when the large kernel processing described above is applied to eh input data represented by curve 902, un-shaφ mask 904 is generated. A scan be seen, the high frequency spectral content of curve 902 is suppressed in curve 904, due to the low pass filtering operation. Un-shaφ mask 904 can then be subtracted form input data 902, which will leave the high frequency content of input data 902. The high frequency version can then be amplified to produce a curve 906. The high frequency data of curve 906 can then be recombined with input curve 902 to produce an enhanced version of the input data.

[075] If an even larger kernel is used, however, then even more high frequency data will be suppressed in the resulting un-shaφ mask as illustrated in figure 10. Thus, from the same input curve 902, and even smoother un-shaφ mask 1002 can be generated using a larger kernel than that used to produce un-shaφ mask 904. A high frequency version 1004 can again be generated through subtraction and amplification, but curve 1004 will have even more high frequency spectral content. High frequency version 1004 can then be combined with input data of curve 902 to generated an enhanced version of the input data.

[076] Using different kernel sizes, a plurality of unshaφ masks can be applied and combined to produce different frequency bands in a manner similar to the filtering and combining of frequency bands by graphic equalizers in typical audio applications. Thus, a video equalizer can be created using a plurality of unshaφ masks such as those illustrated in figures 9 and 10 in a manner analogous to an audio graphic equalizer. As described below an N-dimensional bandpass function, and other unshaφ masks with other cut-off frequencies and corresponding kernel sizes can be combined to produce other passbands, each of which can be further manipulated for various effects and puφoses. In a video graphic equalizer configured in accordance with eth systems and methods described herein, a contiguous, nearly non-overlapping set of bands can be produced to manipulate the gain of each band independently. [077] Figures 11 A-1 IF illustrate an example of using two different unshaφ masks with different cutoff frequencies, such as those depicted in relation to figures 9 and 10, to produce enhanced output signals in accordance with one embodiment of the systems and methods described herein. In the examples of figures 11 A-1 IF, input curve 902 is a simple one-dimensional data set; however, it will be easily understood that the same techniques can be applied to an M-dimensional system. Figure 11 A, therefore, illustrates an input data curve 902. Figure 1 IB illustrates an unshaφ mask 904 that results from applying a relatively small kernel filter with a relatively high cutoff frequency. Figure 11C illustrates a curve 906 that is the result of subtracting unshaφ mask 904 from input data curve 902. The result of subtraction can then be amplified or rescaled as explained above.

[078] Figure 1 ID illustrates an output curve 1002 representing an unshaφ mask resulting from the application of a relatively large kernel filter, i.e., a low pass filter with a relatively low cutoff frequency, to input data curve 902. It should be apparent, as explained above, that un-shaφ mask 1002, is generally "smoother" than curve 904. Figure 1 IE illustrates a curve 1004 representing amplified pixel values resulting from the subtraction of un-shaφ mask 1002 from input data curve 902. [079] Figure 1 IF illustrates a curve 1202, which is the result of subtracting the un-shaφ mask 904 from unshaφ mask 1002.

[080] Curves 904 and 906 can be regarded as a pair of low and high frequency bands of a video graphic equalizer, respectively. Similarly, curves 1002 and 1004 can also be regarded as another pair of low and high frequency bands of a video graphic equalizer, respectively, with a cutoff frequency different from that of curves 902 and 904. Curve 1202 can then be regarded as a mid band, e.g., of a graphic equalizer. By using two different unshaφ masks with different cutoff frequencies, a three-band video graphic equalizer can, therefore, be formed with a low band, e.g., curve 1002, a mid band, e.g., 1202, and a high band, e.g., curve 906. These bands can be relatively contiguous and non-overlapping.

[081] A two-band graphic equalizer can also be formed with a low band, e.g., curve 904 or 1002, and a high band, e.g., curve 906 or 1004. [082] As mentioned above, the filtering, decimation, and re-inteφolation systems and methods described above can provide enhanced video in a wide variety of applications including, medical, industrial, military and law enforcement, and consumer entertainment applications. Moreover, since the filters, decimators, and inteφolators described above reduce the processing and storage requirements, the filtering, decimation, and inteφolation circuits can be included in small form factor chip sets or even a single Application Specific Integrated Circuit (ASIC), which helps to enable an even wider variety of applications.

[083] For example figure 12 illustrates a typical cable system 1200 configured to deliver cable television program to a consumer's cable set top box 1202. The cable television programming can, for example, comprise NTSC or HDTN cable television signals. The cable network 1204 can comprise a head-end 1204 that is configured to receive television programming and deliver it to a particular consumers set top box 1202. Set top box 1202 can be configured to then delver the television programming for viewing via the consumer's television, or display, 1210.

[084] In system 1200, however, a video enhancement device 1208 can be included to enhance the cable television programming being delivered in accordance with the systems and methods described above. In other words, video enhancement device 1208 can be configured to perform the filtering, decimation, and inteφolation steps described above. Thus, the cable television programming delivered to television 1210 for viewing can be significantly enhanced, even if the television programming comprises HDTN signals. Moreover, a user can be allowed to select the level of enhancement desired, e.g., using a multiple band graphic equalizer configured from a plurality of unshaφ masks as described above.

[085] Further, other video generation devices 1212 can be interfaced with video enhancement device 1208. Exemplary video generation devices 1212 can include, for example, a DND player, a digital video camera, or a NCR. Thus, the video signals displayed via television 1210 can be enhanced regardless of the source, simply by routing them through video enhancement device 1208. Alternatively, signals from video generation devices 1210 can be routed through set top box 1202 to video enhancement device 1208.

[086] Moreover, because the circuits comprising video enhancement device

1208 can be made very small using today's circuit fabrication techniques, video enhancement device 1208 can actually be included within one of the other components comprising system 1200. For example, video enhancement device 1208 can be included in head-end 1204 and delivered to television 1210 via set top box 1202. Alternatively, video enhancement device 1202 can be included in set top box 1202 or television 1210. Video enhancement device 1208 can even be included in a video generation devices 1210.

[087] Thus, the ability to miniaturize the circuits comprising video enhancement device 1208 provides flexibility in the enhancement and design of various consumer electronics and entertainment devices. The same flexibility can also be afforded to more specialized implementations, such medical imaging, military target acquisition, and/or military or law enforcement surveillance systems.

[088] Various filtering techniques can be used to implement the system sand methods described above. These techniques can comprise analog and/or digital filtering techniques. Digital filtering techniques can be preferred, especially form the viewpoint of aiding integration with other circuits, e.g., into one or more ASICs. [089]

[090] It will be understood that using higher order filters can perform better than a first order filter. This is especially true when large changes in the spectral content of the data need to be detected. For example, in video imaging systems, it can be important to detect when there is a relatively large changes, or motion, in the data. If relatively heavy filtering is being applied when there is a lot of motion, then blurring, or artifacts, can result from the filtering operation. Thus, it can be preferable to detect when there is a lot of motion and then turn down the amount of filtering that is being applied. [091]

[092] With a first order filter a low pass version of the data will be available, but the low pass version is not very useful for detecting motion. A high pass version of the data can be obtained by subtraction the low pass version form the original data. The high pass version can be used to detect movement; however, this technique is susceptible to false indications due to noise. In other words, noise can masquerade as motion and limit the systems ability to adequately detect when in fact there is motion. Conventional higher order digital filters are often built using delay taps, but all that is produced from these is delayed versions of the data, which is not necessarily useful, e.g., for detecting motion.

[093] In order, for example, to better detect motion, the systems and methods described herein can make use of a higher order temporal digital filter that is configured to generate a high pass, band pass, and low pass version of the data. The band pass version of the data can then be used, for example, to detect motion. The band pass version can be used to more accurately detect motion, because changes in the data that are persistent over a few frames are more likely to be motion as oppose dot a noise spike. Thus, a higher order temporal filter, such as a digital state variable filter can be used to generate a high pass, band pass, and low pas version of the data. Te band pass version can then be use, for example to detect motion. The amount of filtering can then be modulated based on the amount of motion detected.

[094] Accordingly, not only can the systems and methods described herein provided better enhancement, it can also reduce artifacts, or blurring when there is significant motion in the data. It should also be noted that the band pass version of the data can also be used for other beneficial puφoses. For example, the band pass information can be combined with the high pass, or lo pass data to detect other useful information about the data.

[095] In image processing operations, such as those described above, filtering operations with a large kernel size is applied to an image such that, when processing a given pixel, the algorithm uses the pixel values of the surrounding pixels within the boundary defined by the size of the kernel. In general, when processing pixels near the edge of a data set, such as the edge of a television image frame, the kernel would extend past the edge and some assumption must be applied to the pixel values in the area outside the data set. This is because the pixel values outside the data set contribute to and will most likely contaminate or corrupt the processing of a pixel near the edge of, but still within, the active data area of a data set such as a television image frame. [096] In rasterized images such as a conventional television system, the raster format naturally provides "space" for at least some amount of this extra border area, also called blanking intervals, for system processing in real time. Mechanisms for filtering such images in real time can typically keep the filtering process going during these blanking intervals without much additional burden on the processing power; however, the existence of blanking intervals between active image areas of rasterized images does not generally mean that suitable pixel values are provided for use within the blanking intervals, thereby resulting in artifacts near the edges of the active areas. [097] Furthermore, in HDTN systems for example, the vertical blanking area between adjacent image frames is considerably smaller in proportion to the active image area than in a conventional television system, and therefore is more likely to be smaller than the size of the kernel in large kernel processing operations. Thus, for a given pixel near the edge of a given frame, a large kernel size means that a significant portion of the kernel will be outside the frame and can be even more likely to include pixels in effectively adjacent frames, for example on the opposite side of the current frame, on the bottom of the previous frame, or on top of the next frame. Being from a different area of the image, these pixels will likely contain data that is unrelated to the pixel being processed.

[098] The systems and methods described herein can account for, and reduce the effect of blanking areas on the large kernel processing of pixel data. When pixels comprising a blanking area are processed using the systems and methods described herein, additional effective blanking data can be added to the blanking area such that the additional "fictional blanking area" will occupy the area within the kernel size surrounding the pixel being processed, instead of image data in the adjacent frame being in that area. This operation can be aided, for example, by the availability of additional processing power achieved by multi-dimensional filtering, decimation, or sub-sampling, and data processing operations as described above.

[099] The addition of fictional blanking area to the small existing vertical or horizontal blanking area between adjacent frames in, for example, an HDTN system can be achieved by speeding up the processing of data during a blanking interval. For example, in an HDTN system, the vertical blanking area is of special concern. The data is coming at a certain rate and is normally processed at a rate dictated by the rate the data is being provided. But during a blanking interval, there is no real data present, so the system can be allowed to speed up the processing rate, i.e., act like data is coming much faster. Thus, the blanking area can be made to appear larger, thus creating fictional blanking data.

[0100] With the addition of, for example, such fictional scan lines, the pixels near the top or bottom edge of a given frame of, e.g., an HDTN image can be processed without contamination by data values of the pixels near the opposite edge of the adjacent frame, which would otherwise undesirably affect the large-kernel filtering operations on the pixels near the edge of the frame currently being processed. The process of filling in intelligent pixel values in the blanking area, including the naturally existing blanking area and the artificially added fictional scan lines, is dependent on the requirements of a particular implementation and can be optimized for various types of signals and blanking area sizes. It is generally sufficient for this intelligently assumed blanking data to be slowly varying in nature, comprising only low spatial frequency content, and simply maintaining consistency with the average values of the data in the nearby area of the active frame.

[0101] In one embodiment, a feedback loop can be connected during blanking time to the horizontal and vertical low pass filters for two-dimensional video processing, such that after each iteration, the blanking area including the actual blanking area and the fictional scan lines is filled up progressively with more appropriately assumed data. After multiple iterations, the blanking area is filled up with artificial data resulting from the feedback loop in the circuit, such that the transition from the edge of a given frame to the blanking area, including the artificially added fictional blanking data, and from the blanking area to the edge of the immediately adjacent frame will be a smooth transition. The artificial data assigned to the blanking area will then be used as inputs in the filtering or processing of the pixel near the edge of the frame.

[0102] The amount of fictional blanking data added is dependent on the specific implementation. Generally, however, it is sufficient to simply add enough fictional blanking data such that overlap of adjacent frames is sure to be avoided. [0103] Figure 13 illustrates an example of a traditional NTSC display 1300. The current frame 1318 is shown along with another copy of the current frame 1330. Thus, replicated columns of frames are depicted illustrating effectively horizontal adjacent frames resulting from the video raster scanning process. The previous frame 1326 is shown on the top of the figure and the next frame 1340 is shown at the bottom of the figure. Various pixels 1310, 1312, 1314, and 1316, within a current fame 1318 are highlighted along with the associated processing kernel sizes 1302, 1304, 1306 and 1308, respectively. For pixel 1310, for example, associated kernel 1302 is completely inside frame 1318. Thus, the pixel values from the adjacent frames or non-related areas of the current frame would not likely cause significant contamination of the filtering, decimation, and/or other types of processing operations performed on pixel 1310. Pixel 1312, which is at or near the upper horizontal edge of frame 1318, however, has an associated kernel size 1304 that includes a portion of vertical blanking area 1322 and a small image area within adjacent frame 1326. Thus, the processing of pixel 1312 can be affected by the values applied to the portion of blanking area 1322 that falls within kernel size 1304. The pixel values from adjacent frame 1326 can cause even further issues with the processing of pixel 1312, because these pixel values are usually unrelated to, and can be quite different from, pixel 1312, thereby potentially causing undesirably noticeable artifacts. Considering that disparate motion can exist between different areas of frames, the lack of relationship between the pixels in the effectively adjacent frame and the pixel that is being processed can potentially cause even more noticeable artifacts. [0104] For pixel 1314 at or near the right vertical edge of frame 1318, the size of associated kernel 1306 is not large enough, in the example of figure 13, to overlap any image area of the second copy of the current frame 1330. Therefore, pixel 1314 can be processed without substantial contamination from non-related data values in effectively adjacent frame 1330; however, even without contamination by pixel values of frame 1330, suitable values should still be assigned to the portion of blanking area 1334 within kernel size 1306 in order to avoid artifacts relating to the blanking area. [0105] For a pixel 1316 at or near a corner of frame 1318, corresponding kernel

1308 can include a significant portions of blanking area 1334 and an area near the corner of vertically adjacent frame 1340. As mentioned, the overlapping area in adjacent frame 1340 can contribute directly to the processed value of pixel 1316. In addition, the pixel values of other image areas outside kernel area 1308 in adjacent frame 1340 can also indirectly impact the processing of pixel 1316, if they affect the values being filled into blanking area 1334. [0106] Thus, in one embodiment, values are assigned to pixels comprising the blanking area, some of which will then fall within the area of a kernel, such as kernel area 1304 or kernel area 1306. The values assigned should provide a slow, smooth data transition across the blanking area between the data in adjacent frames. It should be noted that, depending on the particular implementation, the values in the blanking area near the edge of one frame can still be affected slightly by the pixel values in effectively adjacent frames. The values in the blanking area closer to the effectively adjacent frames can be affected by the pixel values in those effectively adjacent frames to a greater degree, but in producing a filtered output value for, e.g., pixels 1312, 1314, or 1316, the outer kernel coefficients are typically assigned smaller values, and therefore the assumed blanking values farther away from the pixel, e.g., those closer to the effectively adjacent frames, do not cause a great amount of impact on the filtering result. [0107] As mentioned, in order to limit the effects of pixels in effectively adjacent frames 1326, 1330, and 1340 on the processing of pixels 1312, 1314, and 1316, fictional blanking data can be added to blanking area 1322 and 1334, e.g., by speeding up the processing during the blanking periods. Values can then be assigned to the blanking areas, as described, including the fictional blanking data. Adding fictional blanking data can keep kernel areas 1304, 1306, and 1308 from overlapping pixels in effectively adjacent frames 1326, 1330, and 1340, or at least reduce the extent to which there is overlap. The reduction in overlap can prevent pixels in adjacent frames 1326, 1330, and 1340 from having any significant effect on the processing of, e.g., pixels 1312, 1314, and 1316.

[0108] As mentioned, due to the smaller blanking areas present in HDTV signals, pixels in adjacent frames are even more of a concern. Figure 14 illustrates an example of an HDTV display. The current frame 1462 is shown along with another copy of the current frame 1468. The previous frame 1400 is shown at the top of figure 14 and the next frame 1478 is shown at the bottom. Representative pixels 1446, 1448, 1450, and

1452 are shown with associated kernel sizes 1454, 1456, 1458, and 1460. The processing of pixel 1446 is not likely to be significantly affected by the pixel values of i adjacent frames, or non-related areas of current frame 1462, because kernel 1454 is completely within frame 1462. For pixel 1448 at or near the right vertical edge of frame

1462, however, kernel 1456 includes a portion of blanking area 1466 between frame

1462 and the other copy of current frame 1468. But because horizontal blanking area

1456 between HDTV image frames is relatively large compared to the typical kernel size for filtering operations, the processing of pixel 1448 is not likely to be substantially contaminated by the pixel values in frame 1468. Suitable artificial blanking values should, however, still be formulated and filled into blanking area 1456, as described above, in order to avoid artifacts relating to the blanking area. Further, since in some schemes the artificial values in blanking area 1456 can still be affected by the pixel values in frame 1468 at least to some extent, the processing of pixel 1448 can still be indirectly affected by the pixel values in frame 1468.

[0109] For pixel 1450 at or near the upper horizontal edge of frame 1462, a significant portion of vertically adjacent frame 1474 can reside within kernel 1458, because vertical blanking area 1442 between frames 1462 and 1474 is small compared to the typical kernel size for filtering and decimation operations in HDTN systems. Thus, the pixel values in frame 1474 can significantly contaminate the processing of pixel

1450, thereby producing substantially noticeable artifacts. In addition to the contamination of artificial values filled into the blanking area, the processing of pixel

1450 can be significantly affected by the actual pixel values in adjacent frame 1474, especially considering that these pixel values are most likely unrelated to, and can be quite different from, pixel 1450.

[0110] Similarly, kernel 1460 associated with pixel 1452 at or near the corner of frame 1462 can comprise a large area overlapping the corner portion of vertically adjacent frame 1478. The pixel values in the corner portion of vertically adjacent frame 1478 can, therefore, cause significant contamination to the processing of pixel 1452, thereby producing significant artifacts. Thus, the artificial values assigned to blanking areas 1466 and 1444 can contaminate the processing of pixel 1452, and in addition, the pixel values in adjacent frame 1478 can significantly exacerbate the contamination of the processing of the pixel 1452.

[0111] The addition of fictional scan lines, especially to the vertical blanking areas between vertically adjacent image frames in an HDTN system, can eliminate, or at least significantly reduce, the amount of overlap between, e.g., kernels 1454, 1456, and 1458 and adjacent frames 1474, 1444, and 1468, thereby avoiding, or at least mitigating, the contamination of data values of pixels 1446, 1448, and 1450. [0112] Figure 15 is a block diagram illustrating a circuit 1500 configured to insert fictional blanking data and associated values to blanking areas including the fictional blanking data in accordance with one embodiment of the systems and methods described herein. Full-bandwidth input data, e.g., the active video image of frame, can be received at a first input 1500 of a video data multiplexer 1502, which can also be configured to receive a control signal 1504 that indicates when blanking occurs between adjacent frames. The original blanking data can be received on the same path as the input data, e.g. on input 1500, because the original blanking data is part of the original video data, but this data can be ignored by operation of the multiplexer. [0113] Multiplexer 1502 can be configured to pass the input data to a N- dimensional low pass filter 1508 configured to low pass filter the input data, e.g., as described above. The output of N-dimensional low pass filter 1508 can then be fed back to a feedback input 1512 of multiplexer 1502. Multiplexer 1502 can, therefore, be configured to multiplex, under the control of control signal 1504, the input data and the filtered output of N-dimensional low pass filter 1510 so that, during non-blanking time, the filter is responding to the actual image data, while during blanking time the filter responds to its previously estimated value, which can for example be initially set to a default zero value, and thereby effectively takes on some additional temporal characteristics. Thus, the output of N-dimensional low pass filter 1508 can be used to develop an estimate of the value that should be assigned to a blanking area. [0114] Control signal 1504 can also be supplied to N-dimensional low pass filter

1508 to control the processing rate, i.e., effectively speed up the processing during blanking time, to thereby add the fictional blanking data. It should be noted that, for example, the decimation of data described above can be useful for freeing up processing resources that can then be used to perform more cycles, i.e., speed up the pixel clock. The initial value of the fictional blanking data can also be initially set to a zero value and then filled in using estimates based on the filtered output of N-dimensional low pass filter 1508.

[0115] Thus, the blanking area can be progressively filled with data values after a number of iterations of low pass filtering through N-dimensional low pass filter 1508, until a smooth transitions is formed between the actual data values of the pixels near the edges of adjacent frames and the artificial data values filled into the blanking areas, including the fictional data added to the blanking area. [0116] In certain embodiments, the N-dimensional low pass filter can, for example, be a two-dimensional low pass filter comprising separable horizontal and vertical low pass filters according to the systems and methods described herein. But as mentioned, the systems and methods described herein can be applied in N-dimensions, regardless of how the filter is implemented, e.g., whether it is separable or not. [0117] Thus, circuit 1500 can be included in the same device, e.g. video enhancement device 1208, or even ASIC, as the circuits described above. Alternatively, some or all of the circuits can be included in different devices and/or ASICs. Further, by implementing the systems and method described herein significant enhancement in video imagery for a variety of systems can be achieved.

[0118] While certain embodiments of the inventions have been described above, it will be understood that the embodiments described are by way of example only. Accordingly, the inventions should not be limited based on the described embodiments. Rather, the scope of the inventions described herein should only be limited in light of the claims that follow when taken in conjunction with the above description and accompanying drawings.

Claims

What is Claimed:

1. A method for enhancing multi-dimensional image data, comprising: scanning image data in a plurality of dimensions; and for each of the plurality of dimensions performing a sub-sampling operation that reduces the amount of image data in each of the plurality of dimensions.

2. The method of claim 1, wherein sub-sampling comprises low pass filtering the image data in each of the plurality of dimensions using a relatively large kernel size.

3. The method of claim 2, wherein sub-sampling further comprises decimating the low pass filtered image data in each of the plurality of dimensions.

4. The method of claim 2, wherein the low pass filtering for at least some of the plurality of dimensions is performed serially.

5. The method of claim 2, wherein the low pass filtering for at least some of the plurality of dimensions is performed in parallel.

6. The method of claim 2, wherein the low pass filtering of the image data in the plurality of dimensions is performed in the same order as the scanning of the image data in the plurality of dimensions.

7. The method of claim 2, wherein the low pass filtering comprises substantial low pass filtering using very low cut-off frequencies.

8. The method of claim 1, further comprising, for each of the plurality of dimensions, inteφolating the sub-sampled image data.

9. The method of claim 8, wherein inteφolating the sub- sampled image data comprises low pass filtering the sub-sampled image data.

10. The method of claim 8, further comprising processing the inteφolated image data.

11. The method of claim 10, wherein processing the inteφolated image data comprises subtracting the inteφolated image data from the original image data to generate a high frequency version of the original image data.

12. The method of claim 11 , wherein processing the inteφolated image data further comprises amplifying the high frequency version of the original data.

13. The method of claim 12, wherein processing the inteφolated image data further comprises combining the amplified high frequency version of the original data with the original data to generated an enhanced version of the original data.

14. The method of claim of claim 10, wherein processing the inteφolated image data further comprises separating high frequency image data from low frequency image data, wherein the sub-sampling occurs in relation to the low frequency image data, and wherein processing the inteφolated image data comprises combining the inteφolated low frequency image data with associated high frequency image data.

15. A method for enhancing multi-dimensional image data, comprising: scanning image data in a plurality of dimensions; and for each of the plurality of dimensions, low pass filtering the image data using a relatively large kernel size, and decimating the image data.

16. The method of claim 15, wherein the low pass filtering for at least some of the plurality of dimensions is performed serially.

17. The method of claim 15, wherein the low pass filtering for at least some of the plurality of dimensions is performed in parallel.

18. The method of claim 15, wherein the low pass filtering of the image data is performed in the same order as the scanning of the image data for each of the plurality of dimensions.

19. The method of claim 15, wherein the low pass filtering comprises substantial low pass filtering using very low cut-off frequencies.

20. The method of claim 15, further comprising, for each of the plurality of dimensions, inteφolating the low pass filtered and decimated image data.

21. The method of claim 20, further comprising processing the inteφolated image data.

22. The method of claim 21 , wherein processing the inteφolated image data comprises subtracting the inteφolated image data from the original image data to generate a high frequency version of the original image data.

23. The method of claim 22, wherein processing the inteφolated image data further comprises amplifying the high frequency version of the original data.

24. The method of claim 23, wherein processing the inteφolated image data further comprises combining the amplified high frequency version of the original data with the original data to generated an enhanced version of the original data.

• 25. A method for enhancing multi-dimensional image data, comprising: low pass filtering the image data in each of a plurality of dimensions using a large kernel size; decimating the low pass filtered image data in each of the plurality of dimensions inteφolating the decimated image data to generate an un-shaφ mask of the image data; and using the un-shaφ mask of the image data to generate an enhanced version of the image data.

26. The method of claim 25, wherein using the un-shaφ mask of the image data to generate an enhanced version of the image data comprises subtracting the un-shaφ mask of the image data from the image data to produce a high frequency version of the image data.

27. The method of claim 26, wherein using the un-shaφ mask of the image data to generate an enhanced version of the image data further comprises amplifying the high frequency version of the image data.

28. The method of claim 26, wherein using the un-shaφ mask of the image data to generate an enhanced version of the image data further comprises combining the high frequency version of the image data with the image data.

29. A multi-dimensional image data enhancement system, comprising: an output device configured to output the image data by scanning image data in a plurality of dimensions; and an image data processor configured, for each of the plurality of dimensions, to perform a sub-sampling operation that reduces the amount of image data in each of the plurality of dimensions.

30. The multi-dimensional image data enhancement system of claim 29, wherein the image data processor comprises a large kernel low pass filter, and wherein sub-sampling comprises low pass filtering the image data in each of the plurality of dimensions using the large kernel low pass filter.

31. The multi-dimensional image data enhancement system of claim 30, wherein the image data processor further comprises a decimator, and wherein sub-sampling further comprises decimating the low pass filtered image data in each of the plurality of dimensions using the decimator.

32. The multi-dimensional image data enhancement system of claim 30, wherein the large kernel low pass filtering for at least some of the plurality of dimensions is performed serially.

33. The multi-dimensional image data enhancement system of claim 30, wherein the large kernel low pass filtering for at least some of the plurality of dimensions is performed in parallel.

34. The multi-dimensional image data enhancement system of claim 30, wherein the large kernel low pass filter is configured to low pass filter the image data in the same order that the image data is scanned by the image processor.

35. The multi-dimensional image data enhancement system of claim 29, wherein the image data processor further comprises an inteφolator configured, for each of the plurality of dimensions, to inteφolate the sub-sampled image data.

36. The multi-dimensional image data enhancement system of claim 35, wherein the image data processor is further configured to process the inteφolated image data.

37. The multi-dimensional image data enhancement system of claim 36, wherein processing the inteφolated image data comprises subtracting the inteφolated image data from the original image data to generate a high frequency version of the original image data.

38. The multi-dimensional image data enhancement system of claim 37, wherein processing the inteφolated image data further comprises amplifying the high frequency version of the original data.

39. The multi-dimensional image data enhancement system of claim 38, wherein processing the inteφolated image data further comprises combining the amplified high frequency version of the original data with the original data to generated an enhanced version of the original data.

40. The multi-dimensional image data enhancement system of claim of claim 36, wherein processing the inteφolated image data further comprises separating high frequency image data from low frequency image data, wherein the sub-sampling occurs in relation to the low frequency image data, and wherein processing the inteφolated image data comprises combining the inteφolated low frequency image data with associated high frequency image data.

41. A multi-dimensional image data enhancement system, comprising: an output device configured to output the image data by scanning image data in a plurality of dimensions; and for each of the plurality of dimensions, a large kernel low pass filter configured to low pass filter the image data, and a decimator configured to decimate the image data.

42. The multi-dimensional image data enhancement system of claim 41, wherein the large kernel low pass filtering for at least some of the plurality of dimensions is performed serially.

43. The multi-dimensional image data enhancement system of claim 41, wherein the large kernel low pass filtering for at least some of the plurality of dimensions is performed in parallel.

44. The multi-dimensional image data enhancement system of claim 41, wherein the large kernel low pass filtering of the image data is performed in the same order as the scanning of the image data for each of the plurality of dimensions.

45. The multi-dimensional image data enhancement system of claim 41, further comprising, for each of the plurality of dimensions, an inteφolator configured to inteφolate the low pass filtered and decimated image data.

46. The multi-dimensional image data enhancement system of claim 45, further comprising an image data processor configured to process the inteφolated image data.

47. The multi-dimensional image data enhancement system of claim 46, wherein processing the inteφolated image data comprises subtracting the inteφolated image data from the original image data to generate a high frequency version of the original image data.

48. The multi-dimensional image data enhancement system of claim 47, wherein processing the inteφolated image data further comprises amplifying the high frequency version of the original data.

49. The multi-dimensional image data enhancement system of claim 48, wherein processing the inteφolated image data further comprises combining the amplified high frequency version of the original data with the original data to generated an enhanced version of the original data.

50. A method for enhancing multi-dimensional image data, comprising: adding sufficient fictional blanking data to a blanking area such that pixel data for pixels in effectively adjacent frames will not unduly influence processing of pixel data for a pixel near the edge of a current frame; and processing pixel data for a pixel near the edge of the current frame.

51. The method of claim 50, wherein adding fictional blanking data comprises processing data at a higher data rate during blanking.

52. The method of claim 51 , wherein processing the pixel data comprises low pass filtering and decimating the pixel data and using the resulting processing overhead to effectively process data at a higher data rate during blanking.

53. The method of claim 51, wherein processing the pixel data during vertical blanking time comprises vertical low pass filtering the pixel data and horizontally decimating the pixel data to produce vertical blanking lines that comprise horizontally decimated data, and thereby increasing the number of vertical lines that can be processed during vertical blanking time.

54. The method of claim 50, wherein processing the pixel comprises performing a convolutional filtering operation.

55. The method of claim 54, wherein the convolutional processing is performed in a plurality of dimensions.

56. The method of claim 50, wherein processing the pixel comprises performing low pass filtering in a plurality of dimensions.

57. The method of claim 50, further comprising assigning artificial data to the blanking area including the fictional blanking area, such that the artificial data is used when processing the pixel data for the pixel near the edge of the cuπent frame.

58. The method of claim 57, further comprising iteratively generating artificial data for the blanking area including the fictional blanking area.

59. A multi-dimensional image data enhancement device, comprising: a convolutional low pass filter configured to convolutionally filter image data in a plurality of dimensions; a feedback loop coupled to an output of the convolutional low pass filter, the feedback loop configured to receive filtered data from the output of the convolutional low pass filter; and a multiplexer, coupled to the feedback loop and an input of the convolutional low pass filter, the multiplexer configured to multiplex image data and filtered data under the control of a control signal, the control signal configured to indicate when the image data comprises blanking data and to control the operational rate of the convolutional low pass filter.

60. The multi-dimensional image data enhancement device of claim 59, wherein the control signal is configured to cause the operational rate of the convolutional filter to speed up in order to add fictional blanking data to the blanking data.

61. The multi-dimensional image data enhancement device of claim 59, wherein convolutional filtering the image data further comprises low pass filtering and decimating the image data and using the resulting processing overhead to effectively increase the operational rate of the convolutional filter.

62. The multi-dimensional image data enhancement device of claim 59, wherein convolutional filtering the pixel data during vertical blanking time comprises vertical low pass filtering the pixel data and horizontally decimating the pixel data to produce vertical blanking lines that comprise horizontally decimated data, and thereby increasing the number of vertical lines that can be processed during vertical blanking time.

63. The multi-dimensional image data enhancement device of claim 61, wherein the convolutional filtering is performed in a plurality of dimensions.

64. The multi-dimensional image data enhancement device of claim 59, wherein the convolutional filtering comprises performing low pass filtering in a plurality of dimensions.

65. The multi-dimensional image data enhancement device of claim 59, wherein the feedback loop is configured to provide artificial data for the blanking data including the fictional blanking data, such that the artificial data is used when processing image data near the edge of a cuπent frame.

66. The multi-dimensional image data enhancement device of claim 65, wherein the feedback loop is further configured to iteratively provide artificial data for the blanking area including the fictional blanking area.

67. The multi-dimensional image data enhancement device of claim 66, wherein iteratively providing artificial data comprises feeding back the convolutionally filtered data and multiplexing the convolutionally filtered data with newly received image data during blanking intervals under the control of the control signal.