CA2148433A1 - Electronic image resolution enhancement by frequency-domain extrapolation - Google Patents
Electronic image resolution enhancement by frequency-domain extrapolationInfo
- Publication number
- CA2148433A1 CA2148433A1 CA002148433A CA2148433A CA2148433A1 CA 2148433 A1 CA2148433 A1 CA 2148433A1 CA 002148433 A CA002148433 A CA 002148433A CA 2148433 A CA2148433 A CA 2148433A CA 2148433 A1 CA2148433 A1 CA 2148433A1
- Authority
- CA
- Canada
- Prior art keywords
- pixel array
- pixels
- original
- array
- final
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4023—Decimation- or insertion-based scaling, e.g. pixel or line decimation
Abstract
The resolution of an original image, represented as a pixel array (49, 66), is increased by applying a spectral transformation to the pixel array, producing a spectral array (52, 66) representing the image, and then applying an inverting spectral transformation to produce a larger pixel array (56, 70). The inverting spectral transformation is of a kind designed to convert high frequency spectral components not present in the original image or identified by the spectral array.
These high-frequency terms of the inverting spectral transformation are either ignored or supplied with zero values. As a result, the final, larger pixel array has the same spectral distribution as the original pixel array, but a larger number of pixels.
These high-frequency terms of the inverting spectral transformation are either ignored or supplied with zero values. As a result, the final, larger pixel array has the same spectral distribution as the original pixel array, but a larger number of pixels.
Description
~ 2118~33 AITORNI~Y DOCKE~ NO: SONY/~
ELECTRONIC IMAGE RESOLUTION ENHANCEMENT BY
FREOUENCY-DOMAIN EXTRAPOLATION
Field of the Invention The present invention relates to image enhancement to increase the number of pixels in a pixellated image.
Background of the Invention It is known to apply computers and other electronic computing circuitry to problems of image generation and enhancement. Image enhancement has been used in scientific applications, producing higher definition images providing improved insight, and also in consumer products such as enhanced definition television sets.
Typically, images are electronically stored in the form of pixels each having separate brightness and color characteristics. These pixels are displayed in an array to produce an image.
21~33 -Often it is desirable to increase the number of pixels in an image, e.g. to display the image on a high resolution display or to zoom into a region of the image. A common ter-hn;que for increasing the number of pixels in an image is known as interpolation. Under this technique, new pixels are inserted between existing pixels. The brightness and color characteristics of the new pixels are computed from the brightness and color characteristics of the original pixels which neighbor the new pixel, e.g. by averaging two or more nearest neighbors.
A difficulty with this t~hn;que is that it tends to reduce the sharpness of edges in the image. Edges of the image are characterized by large magnitude transitions in brightness or color between neighboring pixels. However, when a new pixel is inserted between neighboring pixels, the severity of the brightness or color transition is necessarily reduced.
Sophisticated interpolation techniques derive color and brightness values for new pixels using a weighted average of a large number of neighboring pixels, and can to some extent improve the sharpness of the resulting image. However, such ~ 214~3~
sophisticated t~chniques can be computationally complex and therefore too slow for some applications, such as real-time video.
Thus there remains a need for a high quality image ~h~nc~ment t~hni que which does not significantly degrade the sharpness of the image and can also be computed in a reasonable time frame.
Summary of the Invention In accordance with the present invention, the spacial resolution of an original image, represented as a pixel array, is increased by applying a spectral transformation to the pixel array, producing a spectral array representing the spectral components of the image, and then applying an inverting spectral transformation to the spectral array to produce a larger pixel array. The resulting, larger pixel array has the same spectral distribution as the original pixel array, but spread over a larger number of pixels.
As noted above, conventional interpolation can be dissatisfactory because sharp edges are often obscured when new pixels are inserted between existing pixels. In contrast, the present invention produces a final pixel array having the same spectral distribution as the original pixel array, ~4~3 and therefore produces a final image having the same intensity of sharp edges as the original image.
To compensate for reductions in brightness caused by spreading spectral energy across a greater number of pixels, each entry of the original pixel array, the final pixel array, or the transformed spectral array may be increased in value by an amount related to the increase in resolution.
As noted above, elaborate interpolation techniques using large numbers of original pixels to compute each new pixel can improve sharp edges, but at a severe cost in computational complexity. While the present invention does involve a relatively complex computation, spectral computations are used in many different environments, and as a result, near-optimal computing t~chn;ques, and custom-designed computing circuitry, are available to increase the speed of computation. As a result, the present invention can achieve a similar level of image quality as elaborate interpolation t~r-hniques in significantly less computation time.
Principles of the present invention may be applied to video monitors, cameras, recorders or printers, for example to provide electronic zooming, or to modify a low-definition image to permit output ~ 21484~3 on a higher definition device. In some video formats, images are represented as scan lines, in which case the scan lines are converted to form rows of pixels of a pixel array prior to conversion, and vice-versa after conversion.
Specific spectral transforms which can be used include the discrete cosine transform, the discrete sine transform, the fast Fourier transform, the Hadamard transform, the Harr transform, the Walsh transform, the Slant transform. Other transforms, and combinations of transforms, may also be used. The discrete cosine transform has the advantage that custom integrated circuits for performing this transform are commercially available and can be used to speed the transformation.
Large images may be parsed into several blocks, and the above methodology applied to the individual blocks. ~hen the processed blocks are recombined, the resulting image quality can rival that achieved by processing the entire image at once. Some artifacts of the blocks may appear, but these can be reduced by selecting overlapping blocks from the original image; after processing the overlapping blocks are combined to form the final image by discarding overlapping pixels.
r A~ 2 1 ~ 8 ~ 3 ~
Principles of the present invention may be applied in one dimension, thereby increasing the number of pixels in the original image in one direction, or in two dimensions, thereby increasing the number of pixels in two directions. Specific applications include:
Using a two-dimensional transform to electronically zoom into a portion of an image.
Using a one-dimensional transform to display an image in a different resolution. One specific application is displaying an image on a monitor designed for a different aspect ratio, by selecting a central portion of the original aspect ratio image, and then increasing the number of pixels in the selected portion to allow display on the monitor in the new aspect ratio.
Producing a still image from a video frame containing two fields of interleaved scan lines, by selecting one of the fields of scan lines and using this field of scan lines with a one-dimensional transform to produce a greater number of scan lines.
Principles of the present invention may also be applied to applications requiring reduction in the number of pixels, in which case spectral information is discarded after transformation of the ~ 2i~8~3~
original pixel array and before inverse transformation.
The above and other applications and advantages of the present invention shall be made apparent from the accompanying drawings and the description thereof.
Brief Description of the Prawing The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with a general description of the invention given above, and the detailed description of the embodiments given below, serve to explain the principles of the invention.
Fig. 1 is a functional block diagram of a system for increasing the resolution of an image by increasing the number of pixels in the image;
Fig. 2 is a functional block diagram of one embodiment of the computing circuitry 12 of Fig. 1;
Fig. 3 is an illustration of a pixellated image including new pixels added by the system of Fig. 1;
Fig. 4 is flow chart of the process used by the system of Fig. 1 to add pixels to an image;
214843~
Figs. 5A, 5B, and 5C are diagrammatic illustrations of arrays formed during the æteps of the process of Fig. 4 for different types of images;
Fig. 6 is a flow chart of the software program which is on implementation of the process of Fig. 4; and Figs. 7A, 7B, 7C and 7D are diagrammatic illustrations of arrays formed during the steps of the software program for different types of images.
Detailed Descri~tion of S~ecific Embodiments Fig. 1 illustrates a system 10 in accordance with principles of the present invention for enhancing the spacial resolution of electronic images. The system includes computing circuitry 12 which performs the enhancement of the image and stores the enh~nced final image in a memory 14 for display on monitor 16, recording in recorder 17 or printing on printer 18. The image is retrieved by computing circuitry 12 from any suitable source, e.g. from disk storage 20 via line 22 or from a camera 15 via line 24.
The following figures and description detail procedures for enhancing images. The image processing procedures described below can be carried out in various ways, e.g. by software running on a 2148~33 microprocessor embedded in a special purpose device such as video camera 15, video monitor 16, video recorder 17 or video printer 18, or a stand-alone image processing unit which may be connected to any or all of these sourcss. Alternatively, some or all of the procedures described below can be performed by a special purpose computing circuitry, for example, custom integrated circuits, rather than general purpose computers using software. For clarity, in the following description, the general term "computing circuitry" will be used to collectively refer to any electronic configuration which can perform a computation, including by one or more microprocessors running software, by special purpose circuitry, or by a combination of both.
Furthermore, principles of the present invention may be used to process still or moving images from various different sources including, e.g., data files, television broadcasts, video tapes and cameras.
Referring to Fig. 2, in one particular embodiment, computing circuitry 12 comprises a microprocessor 26 coupled to a special purpose circuit 28. Microprocessor 26 is programmed to retrieve images obtained on lines 22 and 24 and to 21~8~33 display, print, or record images by transmission to memory 14 via line 13. Mathematical operations, described below, for processing the images are performed by special purpose circuit 28.
Microprocessor 26 relays unprocessed image information to special purpose circuit 28 over line 27 and obtains processed image information from special purpose circuit 28 over line 27 and forwards the processed image information over line 13 to memory 14.
As noted below, in one embodiment of the present invention, image processing is performed through the use of the discrete cosine transform.
In this embodiment, special purpose circuit 28 may be a commercially available fast cosine transform processor, such as the processor sold under part no.
TMC-2311, by TRW LSI Products, Inc. of P.O. Box 2472, LaJolla, CA 92038. Other custom processors are available for performing discrete cosine transforms, or other suitable transforms.
Referring to Fig. 3, in accordance with the principles of the present invention, computing circuitry 12 increases the number of pixels in an image by generating new pixels 30 for insertion between existing pixels 32 in the image. The new ~i 214843~
pixels 30 may be inserted between vertical columns of existing pixels 32, between horizontal rows of existing pixels, or both, as is illustrated in Fig.
3. By inserting additional pixels between horizontal rows of original pixels, the image resolution is enhanced in the vertical direction.
By inserting additional pixels between columns of original pixels the resolution of the image is enhanced in the horizontal direction.
Referring to Fig. 4, in general, in accordance with the principles of the present invention, pixels are inserted between existing pixels by a spectral transformation. After obtaining 35 an original image of pixels, the original image is spectrally transformed 37 to form a spectral array whose entries identify band limited spectral components in the spatial variation of the luminance or color values of pixels in the original image. This spectral array is then used (after brightness adjustment and other steps, noted below) as the basis for an inverse spectral transformation 39. To perform the inverse spectral transformation, the spectral array is expanded in size to include zero valued entries corresponding to high frequency spectral components not identified or present in the 4 ~ ~
spectral array prior to expansion. The expanded æpectral array is then inversely transformed to produce a resulting pixel array. Through the inverse transformation, a pixel array having a larger number of pixels than the original image is generated. This final array is then printed or displayed 41.
The procedure described above will generate a final array for printing or displaying which has a spectral distribution similar to that of original image prior to expansion. However, the final array will have a larger number of pixels and therefore, can be displayed at a higher resolution.
The procedure illustrated in Fig. 4 includes a step 43 during which the brightness of the image is adjusted. This step is n~ce~y because the expansion of the spectral array reduces the overall brightness of the image by spreading the spectral energy over a larger number of pixels. To compensate for this reduction in brightness, at step 43, the entries of the spectral array are increased in value by a multiplier proportional to the square root of the increase in the number of pixels. By adjusting the brightness in this manner, the resulting image printed or displayed in step 41 will ^ 214~3~
have the same overall brightness as the original image obtained in step 35.
The general procedure described in Fig. 4 can be applied to various types of image processing as illustrated in Figs. 5A through 5C.
Referring to Fig. 5A, one application of the general procedure of Fig. 4 is to perform an electronic zoom into a region 45 of a larger image 47. In this case, a region 45 to be zoomed is extracted from the larger image 47 and used as the original pixel array 49. This original pixel array is subjected to a two-dimensional spectral transformation to form a transformed spectral array 52. This transformed spectral array is then expanded to form an expanded spectral array 54 by inserting zero valued entries corresponding to higher frequency spectral components not identified by the transformed spectral array 52. Then, as a final step, the expanded spectral array 54 is sub~ected to a two-dimensional inverse spectral transformation, forming a final pixel array 56 which is an expanded version of the region 45 of the original image 47.
Referring to Fig. 5B, in another application, the procedure of Fig. 4 is used to 2 1 9~ 3 3 change the aspect ratio of an image for display in a different aspect ratio. For example, an NTSC
television frame, which has a 4:3 aspect ratio, may be converted to an HDTV frame, which has, for example, a 16:9 aspect ratio. As shown in Fig. 5B, a 4:3 frame 60 may be converted to a 16:9 frame by extracting a central group 62 of raster scan lines from the 4:3 image 60. The central raster scan lines 62 are used to produce a display on an HDTV
monitor. HDTV monitors are designed for a greater number of scan lines and therefore, to display the central scan lines 62 on the HDTV monitor, it is necessary to fabricate additional scan lines.
To create these additional scan lines, the extracted raster scan lines 62 are used to produce an original pixel array 64 by converting each scan line into a row of pixels. Next, the original pixel array is subjected to a one-dimensional spectral transform in the vertical direction to produce a transformed spectral array 66. Then, this transformed spectral array is expanded to produce an expanded spectral array 68 by inserting zero valued entries corresponding to high frequency spectral components not identified by the transformed spectral array 66. Finally, the expanded spectral 21~43~
array is sub;ected to an inverse one-dimensional spectral transform to produce a final pixel array 70, and rows of pixels of the final pixel array 70 are used to generate HDTV scan lines.
In a third application, shown in Fig. 5C, the method described in Fig. 4 is used to generate a still image from a NTSC frame of scan lines. An NTSC frame of scan lines includes two fields of scan lines which are displayed interleaved with each other on an NTSC monitor. Thus, as shown in Fig. 5C
a NTSC frame 72 includes a first field of scan lines 74, represented by solid lines in Fig. 5C, and a second field of scan lines 76, represented by dashed lines in Fig. 5C, which is interleaved with the first set of scan lines 74.
A difficulty with producing still images from NTSC frames is that the subject of the NTSC
frame often moves between the recording of the first field of scan lines 74 and the second field of scan lines 76. As a result, the information contained in the first and second fields of scan lines does not properly align if the two fields are displayed simultaneously to produce a frame. (The misalignment is not visible to the eye when NTSC
frames are displayed at normal speed such as during ~ 214~43~
a television program; however, the misalignment becomes visible when a frame is displayed as a still image.) To eliminate the misalignment, one approach is to produce the still image from only one of the two NTSC fields 74, 76 which form the frame 72. This approach produces an image having reduced clarity, but can be acceptable for certain applications. It is in this situation, where a single field is used to produce a still image from a frame, where procedure of Fig. 4 can be particularly useful. For example, it may be necessary to display the still image on a monitor which uses a large number of scan lines. In such a situation, it is necessary to fabricate additional scan lines from the f ield of scan lines taken from the original frame 72. To do this, the field 74 of scan lines from the NTSC frame is used to produce an original pixel array 78 by converting each scan line of the field into a row of pixels. Then this original pixel array is subjected to a one-dimensional spectral transform in the vertical direction producing a transformed spectral array 80. Then the transformed spectral array 80 is expanded to form an expanded spectral array 82 by inserting zero valued j~ 2148~3~
entries corresponding to high frequency spectral components not present in the original pixel array.
Then the expanded spectral array 82 is subjected to an inverse one-dimensional transform to produce a final pixel array 84 from which æcan lines can be recreated.
As seen by the above examples, the combination of a spectral transformation, expansion and inverse spectral transformation can be used to extrapolate additional pixels from an image in either one dimension or two dimensions by using respectively a one-dimensional or two-dimensional spectral transform. Any spectral transform can be used, for example, a fast Fourier transform, a discrete cosine transform, a discrete sine transform, a Hadamard transform, a Harr transform, a Walsh transform, a Slant transform, or a combination of two or more of the above, or any other transform which uses orthogonal eigenfunctions to produce an output related to spectral components of an input.
Note, however, that some transforms (e.g., the Hadamard) have eigenfunctions comprised of long strings of binary coefficients (to reduce computational complexity); such transforms are best used with large images: if these transforms are used ~ 3 3 with images having smaller dimensions than the length of the eigenfunctions, the eigenfunctions will lose orthogonality and the transform will not properly separate spectral components.
In one embodiment, the discrete cosine transform is used to perform spectral transformations. The discrete cosine transform was chosen because there are several commercially available custom circuits which perform the discrete cosine transform, and thus in a commercial embodiment the discrete cosine transform would be relatively inexpensive to implement with custom circuitry.
Further information about the discrete cosine transform and other transforms can be found in Fundamental Imaqe Transforms by Anil Jain, published by Prentice-Hall, Englewood Cliffs, New Jersey, copyright 1989, ISDN 0-13-336165-9, incorporated by reference herein in its entirety.
As is generally illustrated in Figs. 5A
through 5C, the discrete cosine transform produces a spectral array in which low frequency spectral components (which typically contain the greatest spectral energy) are collected at a number of entries at one corner or end of the spectral array.
~48~33 Thus, as shown in Fig. 5A, when a two-dimensional discrete cosine transform is applied to an original pixel array, the resulting spectral array has entries corresponding to low frequency components in the upper left hand corner and entries corresponding to succeedingly higher frequency components in locations succ~;ngly spaced away from the upper left hand corner. Thus, to expand a spectral array produced by the two-dimensional discrete cosine transform, zeros are added to the spectral array to the right, below, and to the lower right of the array, in positions which correspond to higher frequencies not identified in the spectral array. As a result, when an inverse discrete cosine transform is applied to the expanded array, the result is a final pixel array having the same spectral pattern as the original pixel array, but having a larger number of pixels.
Similarly, as shown in Figs. 5B and 5C, when a one-dimensional discrete cosine transform is applied in the vertical direction to an original pixel array, the resulting spectral array has entries along its top row corresponding to low frequency spectral components in the original pixel array and entries in each succee~;~g row ~ 214~43~
corresponding to succeedingly higher frequency spectral components. Thus, to form an expanded spectral array zero entries are added below the last row of the spectral array.
It will be appreciated that other types of transforms have other orientations of lower- and higher- frequency spectral entries, and will therefore need to be padded out differently than described above. Also, it will be appreciated that, as an alternative to forming an array padded with zero entries, the inverse transform calculations could be rewritten to disregard any terms involving high frequency spectral components not present in the original image or in the transformed spectral array. This modification would eliminate the step of expanding the spectral array to a larger array padded out with zero entries, and would reduce the number of calculations required.
Fig. 6 illustrates a particular embodiment of the procedure of Fig. 4 implemented by the software. This embodiment of the general procedure of Fig. 4 is particularly adapted for use with relatively large images. The large image is not processed as a whole, rather the image is processed as a sequence of smaller blocks.
21~8~3 The procedure of Fig. 6 starts by reading 90 an original file containing an image, for example, stored in a red, green, blue (RGB) format, into three buffers containing respectively red, green and blue intensity information for the image.
Subsequently, the procedure of Fig. 6 parses 92 the image into smaller blocks, processing each block in accordance with the subsequent steps to produce a final image having a greater number of pixels.
lo Each block is processed by reading 94 the block into temporary buffers, one buffer for red, green and blue values, respectively. Then the procedure performs 96 the spectral transformation, e.g. a discrete cosine transform, on the pixels in each of the red, green and blue buffers. Next, the procedure transfers 98 the resulting spectral information in the red, green and blue buffers to three larger temporary buffers which have been padded with zero values in the manner generally shown in Figs. 5A, 5B and 5C. At the same time, each of the non-zero spectral values is increased 98 by a factor proportional to the square root of the increase in the number of pixels to thereby maintain the brightness of the resulting image. The larger 2~ ~8~3~
temporary buffers are then inversely transformed 100 and the resulting pixels are written 102 from the larger temporary buffers into three final image buffers.
The steps described in the previous paragraph are repeated for each block in the image until the three final image buffers contain a complete image processed to include a greater number of pixels. At this point, the procedure of Fig. 6 lo writes 104 the red, green and blue values from the final image buffers into a file. Alternatively, the resulting image could be printed or displayed on a monitor.
Figs. 7A-7D illustrate two different methodologies for selecting blocks, overlapping and non-overlapping. Figs. 7A and 7B show the procedure of Fig. 5A performed in a block-by-block fashion; in Fig. 7A, non-overlapping blocks are used, whereas in Fig. 7B, overlapping blocks are used. Figs. 7C and 7D show the procedure of Fig. 5B performed in a block-by-block manner; in Fig. 7C non-overlapping blocks are used, whereas in Fig. 7D overlapping blocks are used.
2 1 ~
Referring to Figs. 7A and 7C, to perform a block-by-block transformation using non-overlapping blocks, the original image is parsed into non-overlapping blocks 110 which serve as original pixel arrays. Then, following steps 94 through 102 of Fig. 6, each of these non-overlapping blocks is subjected to a two-~i~^n~ional spectral transform (Fig. 7A) or a one-dimensional spectral transform (Fig. 7C) to form a spectral array 112.
Subsequently, each spectral array 112 is ~Yp~n~e~ to form an expanded array 114 including zero entries.
Finally, the expanded spectral array 114 is subjected to a two-dimensional inverse spectral transform (Fig. 7A) or a one-dimensional inverse spectral transform (Fig. 7C) to form a non-overlapping block 116 of the final image.
Thus, using non-overlapping blocks from the original image, non-overlapping blocks of the final image are produced and can be collected to build a complete final image.
Referring to Figs. 7B and 7D, to perform a transformation of an image using overlapping blocks, first overlapping blocks 120 are selected from the ~14843~
original image. The blocks should be selected so that up to one-half of each block overlaps neighboring blocks. The overlapping blocks 120 are then processed following steps 94 through 102 of Fig. 6. Thus, blocks 120 are subjected to a two-dimensional spectral transform (Fig. 7B) or a one-dimensional spectral transform (Fig. 7D) to form spectral arrays 122. Next, these spectral arrays are expanded to form expanded spectral arrays 124.
Finally, the expanded spectral arrays 124 are subjected to an inverse two-dimensional spectral transform (Fig. 7B) or an inverse one-dimensional spectral transform (Fig. 7D) to form blocks which may be assembled to create the final image.
The final blocks will partially overlap, so portions of the final blocks must be discarded when assembling them to produce the final image. To do this, the size of each final overlapping block i8 reduced by discarding peripheral pixels 125 to form a central portion 126 (indicated by dotted lines) having the same size as the zoomed non-overlapping blocks 116 illustrated in Figs. 7A and 7C. The central portions 126 of the overlapping blocks are then assembled to produce the final image.
21~8~33 Some ringing and other artifacts may be created when the processing is performed in blocks rather than on the image as a whole. However, these effects are most notable at the periphery of the blocks, and therefore can be substantially reduced by applying the overlapped block methodology described above, which eliminates the periphery of each individual block before assembling the block into the final image.
While the present invention has been illustrated by a description of various embodiments and while these embodiments have been described in considerable detail, it is not the intention of the applicants to restrict or in any way limit the scope of the appended claims to such detail. Additional advantages and modifications will readily appear to those skilled in the art. For example, a three dimensional spectral transformation may be used to generate additional pixels in three dimensions.
Higher-dimensional transforms may also be used.
Furthermore, principles of the present invention may be applied to fields other than image processing.
The invention in its broader aspects is therefore not limited to the specific details, representative apparatus and method, and illustrative example shown -- 2~ 4~3~
and described. Accordingly, departures may be made from such details without departing from the spirit or scope of applicant's general inventive concept.
ELECTRONIC IMAGE RESOLUTION ENHANCEMENT BY
FREOUENCY-DOMAIN EXTRAPOLATION
Field of the Invention The present invention relates to image enhancement to increase the number of pixels in a pixellated image.
Background of the Invention It is known to apply computers and other electronic computing circuitry to problems of image generation and enhancement. Image enhancement has been used in scientific applications, producing higher definition images providing improved insight, and also in consumer products such as enhanced definition television sets.
Typically, images are electronically stored in the form of pixels each having separate brightness and color characteristics. These pixels are displayed in an array to produce an image.
21~33 -Often it is desirable to increase the number of pixels in an image, e.g. to display the image on a high resolution display or to zoom into a region of the image. A common ter-hn;que for increasing the number of pixels in an image is known as interpolation. Under this technique, new pixels are inserted between existing pixels. The brightness and color characteristics of the new pixels are computed from the brightness and color characteristics of the original pixels which neighbor the new pixel, e.g. by averaging two or more nearest neighbors.
A difficulty with this t~hn;que is that it tends to reduce the sharpness of edges in the image. Edges of the image are characterized by large magnitude transitions in brightness or color between neighboring pixels. However, when a new pixel is inserted between neighboring pixels, the severity of the brightness or color transition is necessarily reduced.
Sophisticated interpolation techniques derive color and brightness values for new pixels using a weighted average of a large number of neighboring pixels, and can to some extent improve the sharpness of the resulting image. However, such ~ 214~3~
sophisticated t~chniques can be computationally complex and therefore too slow for some applications, such as real-time video.
Thus there remains a need for a high quality image ~h~nc~ment t~hni que which does not significantly degrade the sharpness of the image and can also be computed in a reasonable time frame.
Summary of the Invention In accordance with the present invention, the spacial resolution of an original image, represented as a pixel array, is increased by applying a spectral transformation to the pixel array, producing a spectral array representing the spectral components of the image, and then applying an inverting spectral transformation to the spectral array to produce a larger pixel array. The resulting, larger pixel array has the same spectral distribution as the original pixel array, but spread over a larger number of pixels.
As noted above, conventional interpolation can be dissatisfactory because sharp edges are often obscured when new pixels are inserted between existing pixels. In contrast, the present invention produces a final pixel array having the same spectral distribution as the original pixel array, ~4~3 and therefore produces a final image having the same intensity of sharp edges as the original image.
To compensate for reductions in brightness caused by spreading spectral energy across a greater number of pixels, each entry of the original pixel array, the final pixel array, or the transformed spectral array may be increased in value by an amount related to the increase in resolution.
As noted above, elaborate interpolation techniques using large numbers of original pixels to compute each new pixel can improve sharp edges, but at a severe cost in computational complexity. While the present invention does involve a relatively complex computation, spectral computations are used in many different environments, and as a result, near-optimal computing t~chn;ques, and custom-designed computing circuitry, are available to increase the speed of computation. As a result, the present invention can achieve a similar level of image quality as elaborate interpolation t~r-hniques in significantly less computation time.
Principles of the present invention may be applied to video monitors, cameras, recorders or printers, for example to provide electronic zooming, or to modify a low-definition image to permit output ~ 21484~3 on a higher definition device. In some video formats, images are represented as scan lines, in which case the scan lines are converted to form rows of pixels of a pixel array prior to conversion, and vice-versa after conversion.
Specific spectral transforms which can be used include the discrete cosine transform, the discrete sine transform, the fast Fourier transform, the Hadamard transform, the Harr transform, the Walsh transform, the Slant transform. Other transforms, and combinations of transforms, may also be used. The discrete cosine transform has the advantage that custom integrated circuits for performing this transform are commercially available and can be used to speed the transformation.
Large images may be parsed into several blocks, and the above methodology applied to the individual blocks. ~hen the processed blocks are recombined, the resulting image quality can rival that achieved by processing the entire image at once. Some artifacts of the blocks may appear, but these can be reduced by selecting overlapping blocks from the original image; after processing the overlapping blocks are combined to form the final image by discarding overlapping pixels.
r A~ 2 1 ~ 8 ~ 3 ~
Principles of the present invention may be applied in one dimension, thereby increasing the number of pixels in the original image in one direction, or in two dimensions, thereby increasing the number of pixels in two directions. Specific applications include:
Using a two-dimensional transform to electronically zoom into a portion of an image.
Using a one-dimensional transform to display an image in a different resolution. One specific application is displaying an image on a monitor designed for a different aspect ratio, by selecting a central portion of the original aspect ratio image, and then increasing the number of pixels in the selected portion to allow display on the monitor in the new aspect ratio.
Producing a still image from a video frame containing two fields of interleaved scan lines, by selecting one of the fields of scan lines and using this field of scan lines with a one-dimensional transform to produce a greater number of scan lines.
Principles of the present invention may also be applied to applications requiring reduction in the number of pixels, in which case spectral information is discarded after transformation of the ~ 2i~8~3~
original pixel array and before inverse transformation.
The above and other applications and advantages of the present invention shall be made apparent from the accompanying drawings and the description thereof.
Brief Description of the Prawing The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with a general description of the invention given above, and the detailed description of the embodiments given below, serve to explain the principles of the invention.
Fig. 1 is a functional block diagram of a system for increasing the resolution of an image by increasing the number of pixels in the image;
Fig. 2 is a functional block diagram of one embodiment of the computing circuitry 12 of Fig. 1;
Fig. 3 is an illustration of a pixellated image including new pixels added by the system of Fig. 1;
Fig. 4 is flow chart of the process used by the system of Fig. 1 to add pixels to an image;
214843~
Figs. 5A, 5B, and 5C are diagrammatic illustrations of arrays formed during the æteps of the process of Fig. 4 for different types of images;
Fig. 6 is a flow chart of the software program which is on implementation of the process of Fig. 4; and Figs. 7A, 7B, 7C and 7D are diagrammatic illustrations of arrays formed during the steps of the software program for different types of images.
Detailed Descri~tion of S~ecific Embodiments Fig. 1 illustrates a system 10 in accordance with principles of the present invention for enhancing the spacial resolution of electronic images. The system includes computing circuitry 12 which performs the enhancement of the image and stores the enh~nced final image in a memory 14 for display on monitor 16, recording in recorder 17 or printing on printer 18. The image is retrieved by computing circuitry 12 from any suitable source, e.g. from disk storage 20 via line 22 or from a camera 15 via line 24.
The following figures and description detail procedures for enhancing images. The image processing procedures described below can be carried out in various ways, e.g. by software running on a 2148~33 microprocessor embedded in a special purpose device such as video camera 15, video monitor 16, video recorder 17 or video printer 18, or a stand-alone image processing unit which may be connected to any or all of these sourcss. Alternatively, some or all of the procedures described below can be performed by a special purpose computing circuitry, for example, custom integrated circuits, rather than general purpose computers using software. For clarity, in the following description, the general term "computing circuitry" will be used to collectively refer to any electronic configuration which can perform a computation, including by one or more microprocessors running software, by special purpose circuitry, or by a combination of both.
Furthermore, principles of the present invention may be used to process still or moving images from various different sources including, e.g., data files, television broadcasts, video tapes and cameras.
Referring to Fig. 2, in one particular embodiment, computing circuitry 12 comprises a microprocessor 26 coupled to a special purpose circuit 28. Microprocessor 26 is programmed to retrieve images obtained on lines 22 and 24 and to 21~8~33 display, print, or record images by transmission to memory 14 via line 13. Mathematical operations, described below, for processing the images are performed by special purpose circuit 28.
Microprocessor 26 relays unprocessed image information to special purpose circuit 28 over line 27 and obtains processed image information from special purpose circuit 28 over line 27 and forwards the processed image information over line 13 to memory 14.
As noted below, in one embodiment of the present invention, image processing is performed through the use of the discrete cosine transform.
In this embodiment, special purpose circuit 28 may be a commercially available fast cosine transform processor, such as the processor sold under part no.
TMC-2311, by TRW LSI Products, Inc. of P.O. Box 2472, LaJolla, CA 92038. Other custom processors are available for performing discrete cosine transforms, or other suitable transforms.
Referring to Fig. 3, in accordance with the principles of the present invention, computing circuitry 12 increases the number of pixels in an image by generating new pixels 30 for insertion between existing pixels 32 in the image. The new ~i 214843~
pixels 30 may be inserted between vertical columns of existing pixels 32, between horizontal rows of existing pixels, or both, as is illustrated in Fig.
3. By inserting additional pixels between horizontal rows of original pixels, the image resolution is enhanced in the vertical direction.
By inserting additional pixels between columns of original pixels the resolution of the image is enhanced in the horizontal direction.
Referring to Fig. 4, in general, in accordance with the principles of the present invention, pixels are inserted between existing pixels by a spectral transformation. After obtaining 35 an original image of pixels, the original image is spectrally transformed 37 to form a spectral array whose entries identify band limited spectral components in the spatial variation of the luminance or color values of pixels in the original image. This spectral array is then used (after brightness adjustment and other steps, noted below) as the basis for an inverse spectral transformation 39. To perform the inverse spectral transformation, the spectral array is expanded in size to include zero valued entries corresponding to high frequency spectral components not identified or present in the 4 ~ ~
spectral array prior to expansion. The expanded æpectral array is then inversely transformed to produce a resulting pixel array. Through the inverse transformation, a pixel array having a larger number of pixels than the original image is generated. This final array is then printed or displayed 41.
The procedure described above will generate a final array for printing or displaying which has a spectral distribution similar to that of original image prior to expansion. However, the final array will have a larger number of pixels and therefore, can be displayed at a higher resolution.
The procedure illustrated in Fig. 4 includes a step 43 during which the brightness of the image is adjusted. This step is n~ce~y because the expansion of the spectral array reduces the overall brightness of the image by spreading the spectral energy over a larger number of pixels. To compensate for this reduction in brightness, at step 43, the entries of the spectral array are increased in value by a multiplier proportional to the square root of the increase in the number of pixels. By adjusting the brightness in this manner, the resulting image printed or displayed in step 41 will ^ 214~3~
have the same overall brightness as the original image obtained in step 35.
The general procedure described in Fig. 4 can be applied to various types of image processing as illustrated in Figs. 5A through 5C.
Referring to Fig. 5A, one application of the general procedure of Fig. 4 is to perform an electronic zoom into a region 45 of a larger image 47. In this case, a region 45 to be zoomed is extracted from the larger image 47 and used as the original pixel array 49. This original pixel array is subjected to a two-dimensional spectral transformation to form a transformed spectral array 52. This transformed spectral array is then expanded to form an expanded spectral array 54 by inserting zero valued entries corresponding to higher frequency spectral components not identified by the transformed spectral array 52. Then, as a final step, the expanded spectral array 54 is sub~ected to a two-dimensional inverse spectral transformation, forming a final pixel array 56 which is an expanded version of the region 45 of the original image 47.
Referring to Fig. 5B, in another application, the procedure of Fig. 4 is used to 2 1 9~ 3 3 change the aspect ratio of an image for display in a different aspect ratio. For example, an NTSC
television frame, which has a 4:3 aspect ratio, may be converted to an HDTV frame, which has, for example, a 16:9 aspect ratio. As shown in Fig. 5B, a 4:3 frame 60 may be converted to a 16:9 frame by extracting a central group 62 of raster scan lines from the 4:3 image 60. The central raster scan lines 62 are used to produce a display on an HDTV
monitor. HDTV monitors are designed for a greater number of scan lines and therefore, to display the central scan lines 62 on the HDTV monitor, it is necessary to fabricate additional scan lines.
To create these additional scan lines, the extracted raster scan lines 62 are used to produce an original pixel array 64 by converting each scan line into a row of pixels. Next, the original pixel array is subjected to a one-dimensional spectral transform in the vertical direction to produce a transformed spectral array 66. Then, this transformed spectral array is expanded to produce an expanded spectral array 68 by inserting zero valued entries corresponding to high frequency spectral components not identified by the transformed spectral array 66. Finally, the expanded spectral 21~43~
array is sub;ected to an inverse one-dimensional spectral transform to produce a final pixel array 70, and rows of pixels of the final pixel array 70 are used to generate HDTV scan lines.
In a third application, shown in Fig. 5C, the method described in Fig. 4 is used to generate a still image from a NTSC frame of scan lines. An NTSC frame of scan lines includes two fields of scan lines which are displayed interleaved with each other on an NTSC monitor. Thus, as shown in Fig. 5C
a NTSC frame 72 includes a first field of scan lines 74, represented by solid lines in Fig. 5C, and a second field of scan lines 76, represented by dashed lines in Fig. 5C, which is interleaved with the first set of scan lines 74.
A difficulty with producing still images from NTSC frames is that the subject of the NTSC
frame often moves between the recording of the first field of scan lines 74 and the second field of scan lines 76. As a result, the information contained in the first and second fields of scan lines does not properly align if the two fields are displayed simultaneously to produce a frame. (The misalignment is not visible to the eye when NTSC
frames are displayed at normal speed such as during ~ 214~43~
a television program; however, the misalignment becomes visible when a frame is displayed as a still image.) To eliminate the misalignment, one approach is to produce the still image from only one of the two NTSC fields 74, 76 which form the frame 72. This approach produces an image having reduced clarity, but can be acceptable for certain applications. It is in this situation, where a single field is used to produce a still image from a frame, where procedure of Fig. 4 can be particularly useful. For example, it may be necessary to display the still image on a monitor which uses a large number of scan lines. In such a situation, it is necessary to fabricate additional scan lines from the f ield of scan lines taken from the original frame 72. To do this, the field 74 of scan lines from the NTSC frame is used to produce an original pixel array 78 by converting each scan line of the field into a row of pixels. Then this original pixel array is subjected to a one-dimensional spectral transform in the vertical direction producing a transformed spectral array 80. Then the transformed spectral array 80 is expanded to form an expanded spectral array 82 by inserting zero valued j~ 2148~3~
entries corresponding to high frequency spectral components not present in the original pixel array.
Then the expanded spectral array 82 is subjected to an inverse one-dimensional transform to produce a final pixel array 84 from which æcan lines can be recreated.
As seen by the above examples, the combination of a spectral transformation, expansion and inverse spectral transformation can be used to extrapolate additional pixels from an image in either one dimension or two dimensions by using respectively a one-dimensional or two-dimensional spectral transform. Any spectral transform can be used, for example, a fast Fourier transform, a discrete cosine transform, a discrete sine transform, a Hadamard transform, a Harr transform, a Walsh transform, a Slant transform, or a combination of two or more of the above, or any other transform which uses orthogonal eigenfunctions to produce an output related to spectral components of an input.
Note, however, that some transforms (e.g., the Hadamard) have eigenfunctions comprised of long strings of binary coefficients (to reduce computational complexity); such transforms are best used with large images: if these transforms are used ~ 3 3 with images having smaller dimensions than the length of the eigenfunctions, the eigenfunctions will lose orthogonality and the transform will not properly separate spectral components.
In one embodiment, the discrete cosine transform is used to perform spectral transformations. The discrete cosine transform was chosen because there are several commercially available custom circuits which perform the discrete cosine transform, and thus in a commercial embodiment the discrete cosine transform would be relatively inexpensive to implement with custom circuitry.
Further information about the discrete cosine transform and other transforms can be found in Fundamental Imaqe Transforms by Anil Jain, published by Prentice-Hall, Englewood Cliffs, New Jersey, copyright 1989, ISDN 0-13-336165-9, incorporated by reference herein in its entirety.
As is generally illustrated in Figs. 5A
through 5C, the discrete cosine transform produces a spectral array in which low frequency spectral components (which typically contain the greatest spectral energy) are collected at a number of entries at one corner or end of the spectral array.
~48~33 Thus, as shown in Fig. 5A, when a two-dimensional discrete cosine transform is applied to an original pixel array, the resulting spectral array has entries corresponding to low frequency components in the upper left hand corner and entries corresponding to succeedingly higher frequency components in locations succ~;ngly spaced away from the upper left hand corner. Thus, to expand a spectral array produced by the two-dimensional discrete cosine transform, zeros are added to the spectral array to the right, below, and to the lower right of the array, in positions which correspond to higher frequencies not identified in the spectral array. As a result, when an inverse discrete cosine transform is applied to the expanded array, the result is a final pixel array having the same spectral pattern as the original pixel array, but having a larger number of pixels.
Similarly, as shown in Figs. 5B and 5C, when a one-dimensional discrete cosine transform is applied in the vertical direction to an original pixel array, the resulting spectral array has entries along its top row corresponding to low frequency spectral components in the original pixel array and entries in each succee~;~g row ~ 214~43~
corresponding to succeedingly higher frequency spectral components. Thus, to form an expanded spectral array zero entries are added below the last row of the spectral array.
It will be appreciated that other types of transforms have other orientations of lower- and higher- frequency spectral entries, and will therefore need to be padded out differently than described above. Also, it will be appreciated that, as an alternative to forming an array padded with zero entries, the inverse transform calculations could be rewritten to disregard any terms involving high frequency spectral components not present in the original image or in the transformed spectral array. This modification would eliminate the step of expanding the spectral array to a larger array padded out with zero entries, and would reduce the number of calculations required.
Fig. 6 illustrates a particular embodiment of the procedure of Fig. 4 implemented by the software. This embodiment of the general procedure of Fig. 4 is particularly adapted for use with relatively large images. The large image is not processed as a whole, rather the image is processed as a sequence of smaller blocks.
21~8~3 The procedure of Fig. 6 starts by reading 90 an original file containing an image, for example, stored in a red, green, blue (RGB) format, into three buffers containing respectively red, green and blue intensity information for the image.
Subsequently, the procedure of Fig. 6 parses 92 the image into smaller blocks, processing each block in accordance with the subsequent steps to produce a final image having a greater number of pixels.
lo Each block is processed by reading 94 the block into temporary buffers, one buffer for red, green and blue values, respectively. Then the procedure performs 96 the spectral transformation, e.g. a discrete cosine transform, on the pixels in each of the red, green and blue buffers. Next, the procedure transfers 98 the resulting spectral information in the red, green and blue buffers to three larger temporary buffers which have been padded with zero values in the manner generally shown in Figs. 5A, 5B and 5C. At the same time, each of the non-zero spectral values is increased 98 by a factor proportional to the square root of the increase in the number of pixels to thereby maintain the brightness of the resulting image. The larger 2~ ~8~3~
temporary buffers are then inversely transformed 100 and the resulting pixels are written 102 from the larger temporary buffers into three final image buffers.
The steps described in the previous paragraph are repeated for each block in the image until the three final image buffers contain a complete image processed to include a greater number of pixels. At this point, the procedure of Fig. 6 lo writes 104 the red, green and blue values from the final image buffers into a file. Alternatively, the resulting image could be printed or displayed on a monitor.
Figs. 7A-7D illustrate two different methodologies for selecting blocks, overlapping and non-overlapping. Figs. 7A and 7B show the procedure of Fig. 5A performed in a block-by-block fashion; in Fig. 7A, non-overlapping blocks are used, whereas in Fig. 7B, overlapping blocks are used. Figs. 7C and 7D show the procedure of Fig. 5B performed in a block-by-block manner; in Fig. 7C non-overlapping blocks are used, whereas in Fig. 7D overlapping blocks are used.
2 1 ~
Referring to Figs. 7A and 7C, to perform a block-by-block transformation using non-overlapping blocks, the original image is parsed into non-overlapping blocks 110 which serve as original pixel arrays. Then, following steps 94 through 102 of Fig. 6, each of these non-overlapping blocks is subjected to a two-~i~^n~ional spectral transform (Fig. 7A) or a one-dimensional spectral transform (Fig. 7C) to form a spectral array 112.
Subsequently, each spectral array 112 is ~Yp~n~e~ to form an expanded array 114 including zero entries.
Finally, the expanded spectral array 114 is subjected to a two-dimensional inverse spectral transform (Fig. 7A) or a one-dimensional inverse spectral transform (Fig. 7C) to form a non-overlapping block 116 of the final image.
Thus, using non-overlapping blocks from the original image, non-overlapping blocks of the final image are produced and can be collected to build a complete final image.
Referring to Figs. 7B and 7D, to perform a transformation of an image using overlapping blocks, first overlapping blocks 120 are selected from the ~14843~
original image. The blocks should be selected so that up to one-half of each block overlaps neighboring blocks. The overlapping blocks 120 are then processed following steps 94 through 102 of Fig. 6. Thus, blocks 120 are subjected to a two-dimensional spectral transform (Fig. 7B) or a one-dimensional spectral transform (Fig. 7D) to form spectral arrays 122. Next, these spectral arrays are expanded to form expanded spectral arrays 124.
Finally, the expanded spectral arrays 124 are subjected to an inverse two-dimensional spectral transform (Fig. 7B) or an inverse one-dimensional spectral transform (Fig. 7D) to form blocks which may be assembled to create the final image.
The final blocks will partially overlap, so portions of the final blocks must be discarded when assembling them to produce the final image. To do this, the size of each final overlapping block i8 reduced by discarding peripheral pixels 125 to form a central portion 126 (indicated by dotted lines) having the same size as the zoomed non-overlapping blocks 116 illustrated in Figs. 7A and 7C. The central portions 126 of the overlapping blocks are then assembled to produce the final image.
21~8~33 Some ringing and other artifacts may be created when the processing is performed in blocks rather than on the image as a whole. However, these effects are most notable at the periphery of the blocks, and therefore can be substantially reduced by applying the overlapped block methodology described above, which eliminates the periphery of each individual block before assembling the block into the final image.
While the present invention has been illustrated by a description of various embodiments and while these embodiments have been described in considerable detail, it is not the intention of the applicants to restrict or in any way limit the scope of the appended claims to such detail. Additional advantages and modifications will readily appear to those skilled in the art. For example, a three dimensional spectral transformation may be used to generate additional pixels in three dimensions.
Higher-dimensional transforms may also be used.
Furthermore, principles of the present invention may be applied to fields other than image processing.
The invention in its broader aspects is therefore not limited to the specific details, representative apparatus and method, and illustrative example shown -- 2~ 4~3~
and described. Accordingly, departures may be made from such details without departing from the spirit or scope of applicant's general inventive concept.
Claims (28)
1. A method for increasing the spacial resolution of an image comprising an original pixel array, each pixel characterized by luminance or color values, by transforming said original pixel array to a final pixel array having a greater number of pixels, comprising transforming said original pixel array into a spectral array, each entry of said spectral array identifying spectral components of spacial variation of luminance or color values of pixels of said original pixel array, and inverse transforming said transformed spectral array to generate said final pixel array having a greater number of pixels, the luminance or color values of pixels of said final pixel array having a distribution of spectral components of spacial variation similar to said original pixel array.
2. The method of claim 1 wherein inverse transforming said transformed spectral array comprises forming an expanded version of said transformed spectral array by padding said transformed spectral array with zero valued high order spectral components, and applying, to said expanded version, an inverting transformational formula including terms responsive to high order spectral components, said padded zero valued high order spectral components causing said high order terms to be disregarded.
3. The method of claim 1 adapted for use on a large original image, comprising selecting original sub-images each containing a subset of a smaller number of pixels of said large original image, applying the steps of claim 1 to each sub-image to generate final sub-images, and combining final sub-images to produce a large final image.
4. The method of claim 3 wherein selecting original sub-images includes selecting at least two original sub-images which contain overlapping pixels of said large original image, and combining final sub-images includes discarding overlapping pixels.
5. The method of claim 1 adapted to transform an original aspect ratio image to a different aspect ratio, further comprising selecting an original pixel array from a central portion of said original aspect ratio image, applying the steps of claim 1 to transform said original pixel array to a final pixel array having a greater number of pixels, and displaying said final pixel array in said different aspect ratio.
6. The method of claim 1 adapted to produce a video image from a video frame including two fields of interleaved raster scan lines, selecting a first field of interleaved raster scan lines, converting each raster scan line of said first field into a row of pixels, to form an original pixel array, applying the steps of claim 1 to transform said original pixel array to a final pixel array having a greater number of pixels, and converting each row of pixels of said final pixel array into a raster scan line to form said video image.
7. The method of claim 1 adapted to compensate for reductions in image brightness resulting from the increased number of pixels in said final pixel array, further comprising multiplying each entry in said original pixel array, said final pixel array, or said transformed spectral array, by a factor equal to the square root of the ratio of the number of pixels in said final pixel array to the number of pixels in said original pixel array.
8. The method of claim 1 wherein said transforming and inverse transforming steps are performed in one dimension.
9. The method of claim 8 wherein said transforming and inverse transforming steps are performed in two dimensions.
10. The method of claim 1 wherein said transforming and inverse transforming steps are performed in accordance with one or a combination of two or more of the group consisting of:
a discrete cosine transform;
a discrete sine transform;
a fast Fourier transform;
a Hadamard transform;
a Harr transform;
a Walsh transform;
a Slant transform.
a discrete cosine transform;
a discrete sine transform;
a fast Fourier transform;
a Hadamard transform;
a Harr transform;
a Walsh transform;
a Slant transform.
11. A method for reducing the spacial resolution of an image comprising an original pixel array, each pixel characterized by luminance or color values, by transforming said original pixel array to a final pixel array having fewer pixels, comprising:
transforming said original pixel array into a spectral array, each entry of said spectral array identifying spectral components in spacial variation of luminance or color values of pixels of said original pixel array, and inverse transforming said transformed spectral array to generate said final pixel array having fewer pixels, the luminance or color values of pixels of said final pixel array having a distribution of spectral components of spacial variation similar to low order spectral components of said original pixel array, but not including high order spectral components present in said original pixel array.
transforming said original pixel array into a spectral array, each entry of said spectral array identifying spectral components in spacial variation of luminance or color values of pixels of said original pixel array, and inverse transforming said transformed spectral array to generate said final pixel array having fewer pixels, the luminance or color values of pixels of said final pixel array having a distribution of spectral components of spacial variation similar to low order spectral components of said original pixel array, but not including high order spectral components present in said original pixel array.
12. Apparatus for increasing the spacial resolution of an image comprising an original pixel array, each pixel characterized by luminance or color values, by transforming said original pixel array to a final pixel array having a greater number of pixels, comprising:
computing circuitry transforming said original pixel array into a spectral array, each entry of said spectral array identifying spectral components in spacial variation of luminance or color values of pixels of said original pixel array, and computing circuitry inverse transforming said transformed spectral array to generate said final pixel array having a greater number of pixels, the luminance or color values of pixels of said final pixel array having a distribution of spectral components of spacial variation similar to said original pixel array.
computing circuitry transforming said original pixel array into a spectral array, each entry of said spectral array identifying spectral components in spacial variation of luminance or color values of pixels of said original pixel array, and computing circuitry inverse transforming said transformed spectral array to generate said final pixel array having a greater number of pixels, the luminance or color values of pixels of said final pixel array having a distribution of spectral components of spacial variation similar to said original pixel array.
13. The apparatus of claim 12 wherein said computing circuitry inverse transforming said transformed spectral array comprises circuitry forming an expanded version of said transformed spectral array by padding said transformed spectral array with zero valued high order spectral components, and circuitry applying, to said expanded version, an inverting transformational formula including terms responsive to high order spectral components, said padded zero valued high order spectral components causing said have high order terms to be disregarded.
14. The apparatus of claim 12 adapted for use on a large original image, comprising computing circuitry selecting original sub-images containing a subset of a smaller number of pixels of said large original image, and supplying said sub-images to the apparatus of claim 12 to generate final sub-images, and computing circuitry combining final sub-images to produce a large final image.
15. The apparatus of claim 14 wherein said computing circuitry selecting original sub-images includes circuitry selecting at least two original sub-images which contain overlapping pixels of said large original image, and said computing circuitry combining final sub-images includes circuitry discarding overlapping pixels.
16. The apparatus of claim 12 adapted to transform an original aspect ratio image to a different aspect ratio, further comprising computing circuitry selecting an original pixel array from a central portion of said original aspect ratio image, and supplying said original pixel array to the apparatus of claim 12 to generate a final pixel array having a greater number of pixels, and a display displaying said final pixel array in said different aspect ratio.
17. The apparatus of claim 12 adapted to produce a video image from a video frame including two fields of interleaved raster scan lines, computing circuitry selecting a first field of interleaved raster scan lines, computing circuitry converting each raster scan line of said first field into a row of pixels, to form an original pixel array, and supplying said original pixel array to the apparatus of claim 12 to generate a final pixel array having a greater number of pixels, and computing circuitry converting each row of pixels of said final pixel array into a raster scan line to form said video image.
18. The apparatus of claim 12 adapted to compensate for reductions in image brightness resulting from an increase in the number of pixels in said final pixel array, further comprising computing circuitry multiplying each entry in said original pixel array, said final pixel array, or said transformed spectral array, by a factor equal to the square root of the ratio of the number of pixels in said final pixel array to the number of pixels in said original pixel array.
19. The apparatus of claim 12 wherein said computing circuitry is configured to transform and inverse transform in one dimension.
20. The apparatus of claim 19 wherein said computing circuitry is configured to transform and inverse transform in two dimensions.
21. The apparatus of claim 12 wherein said computing circuitry transforms and inverse transforms in accordance with one or a combination of two or more of the group consisting of:
a discrete cosine transform;
a discrete sine transform;
a fast Fourier transform;
a Hadamard transform;
a Harr transform;
a Walsh transform;
a Slant transform.
a discrete cosine transform;
a discrete sine transform;
a fast Fourier transform;
a Hadamard transform;
a Harr transform;
a Walsh transform;
a Slant transform.
22. The apparatus of claim 12 wherein said computing circuitry is housed at least in part in a custom integrated circuit.
23. A video printer including circuitry for increasing the spacial resolution of an original image prior to printing said image, said original image comprising an original pixel array, each pixel characterized by luminance or color values, comprising:
computing circuitry transforming said original pixel array into a spectral array, each entry of said spectral array identifying spectral components in spacial variation of luminance or color values of pixels of said original pixel array, computing circuitry inverse transforming said transformed spectral array to generate a final pixel array having a greater number of pixels, the luminance or color values of pixels of said final pixel array having a distribution of spectral components of spacial variation similar to said original pixel array, and printing circuitry responsive to said final pixel array and generating a printed output having luminance or color characteristics identified by said final pixel array.
computing circuitry transforming said original pixel array into a spectral array, each entry of said spectral array identifying spectral components in spacial variation of luminance or color values of pixels of said original pixel array, computing circuitry inverse transforming said transformed spectral array to generate a final pixel array having a greater number of pixels, the luminance or color values of pixels of said final pixel array having a distribution of spectral components of spacial variation similar to said original pixel array, and printing circuitry responsive to said final pixel array and generating a printed output having luminance or color characteristics identified by said final pixel array.
24. A video monitor including circuitry for increasing the spacial resolution of an original image prior to displaying said image, said original image comprising an original pixel array, each pixel characterized by luminance or color values, comprising:
computing circuitry transforming said original pixel array into a spectral array, each entry of said spectral array identifying spectral components in spacial variation of luminance or color values of pixels of said original pixel array, computing circuitry inverse transforming said transformed spectral array to generate a final pixel array having a greater number of pixels, the luminance or color values of pixels of said final pixel array having a distribution of spectral components of spacial variation similar to said original pixel array, and display circuitry responsive to said final pixel array and generating a visual display having luminance or color characteristics identified by said final pixel array.
computing circuitry transforming said original pixel array into a spectral array, each entry of said spectral array identifying spectral components in spacial variation of luminance or color values of pixels of said original pixel array, computing circuitry inverse transforming said transformed spectral array to generate a final pixel array having a greater number of pixels, the luminance or color values of pixels of said final pixel array having a distribution of spectral components of spacial variation similar to said original pixel array, and display circuitry responsive to said final pixel array and generating a visual display having luminance or color characteristics identified by said final pixel array.
25. The video monitor of claim 24 adapted to display a television image comprising sets of scan lines, further comprising conversion circuitry converting scan lines of said television image into rows of pixels of said original pixel array, and conversion circuitry converting rows of pixels of said final pixel array into final scan lines, and wherein said display circuitry displays said final scan lines by writing said final scan lines across a display screen.
26. A video recorder including circuitry for increasing the spacial resolution of an original image prior to recording said image, said original image comprising an original pixel array, each pixel characterized by luminance or color values, comprising:
computing circuitry transforming said original pixel array into a spectral array, each entry of said spectral array identifying spectral components in spacial variation of luminance or color values of pixels of said original pixel array, computing circuitry inverse transforming said transformed spectral array to generate a final pixel array having a greater number of pixels, the luminance or color values of pixels of said final pixel array having a distribution of spectral components of spacial variation similar to said original pixel array, and recording circuitry responsive to said final pixel array and recording the final image identified by said final pixel array.
computing circuitry transforming said original pixel array into a spectral array, each entry of said spectral array identifying spectral components in spacial variation of luminance or color values of pixels of said original pixel array, computing circuitry inverse transforming said transformed spectral array to generate a final pixel array having a greater number of pixels, the luminance or color values of pixels of said final pixel array having a distribution of spectral components of spacial variation similar to said original pixel array, and recording circuitry responsive to said final pixel array and recording the final image identified by said final pixel array.
27. The video recorder of claim 26 adapted to record a television image comprising sets of scan lines, further comprising conversion circuitry converting a set of scan lines of said television image into rows of pixels of said original pixel array, and conversion circuitry converting rows of pixels of said final pixel array into a final set of scan lines, and wherein said recording circuitry records said final scan lines by writing data corresponding to said final scan lines across a recording medium.
28. A video camera including circuitry for increasing the spacial resolution of an original image received from a optical sensor, said original image comprising an original pixel array, each pixel characterized by luminance or color values, comprising:
computing circuitry transforming said original pixel array into a spectral array, each entry of said spectral array identifying spectral components in spacial variation of luminance or color values of pixels of said original pixel array, and computing circuitry inverse transforming said transformed spectral array to generate a final pixel array having a greater number of pixels, the luminance or color values of pixels of said final pixel array having a distribution of spectral components of spacial variation similar to said original pixel array.
computing circuitry transforming said original pixel array into a spectral array, each entry of said spectral array identifying spectral components in spacial variation of luminance or color values of pixels of said original pixel array, and computing circuitry inverse transforming said transformed spectral array to generate a final pixel array having a greater number of pixels, the luminance or color values of pixels of said final pixel array having a distribution of spectral components of spacial variation similar to said original pixel array.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US08/273,869 | 1994-07-12 | ||
| US08/273,869 US5666163A (en) | 1994-07-12 | 1994-07-12 | Electronic image resolution enhancement by frequency-domain extrapolation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA2148433A1 true CA2148433A1 (en) | 1996-01-13 |
Family
ID=23045762
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002148433A Abandoned CA2148433A1 (en) | 1994-07-12 | 1995-05-02 | Electronic image resolution enhancement by frequency-domain extrapolation |
Country Status (6)
| Country | Link |
|---|---|
| US (2) | US5666163A (en) |
| EP (1) | EP0696780A3 (en) |
| JP (1) | JPH08101917A (en) |
| KR (1) | KR960006587A (en) |
| CA (1) | CA2148433A1 (en) |
| TW (1) | TW296529B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3651511B2 (en) * | 1996-03-22 | 2005-05-25 | 富士写真フイルム株式会社 | Digital image recording apparatus and method |
| WO1998035449A1 (en) * | 1997-02-10 | 1998-08-13 | Sony Corporation | Method and equipment for processing data |
| AU766680B2 (en) | 1997-11-05 | 2003-10-23 | Sony Corporation | Method for converting digital signal and apparatus for converting digital signal |
| JP3343070B2 (en) * | 1998-01-30 | 2002-11-11 | 富士写真フイルム株式会社 | Image output system |
| JP3748489B2 (en) * | 1998-03-11 | 2006-02-22 | キヤノン株式会社 | Image processing apparatus, image processing method, and computer-readable storage medium |
| US20050123210A1 (en) * | 2003-12-05 | 2005-06-09 | Bhattacharjya Anoop K. | Print processing of compressed noisy images |
| KR100580094B1 (en) * | 2004-03-27 | 2006-05-16 | 주식회사 팬택앤큐리텔 | Apparatus and method for processing pixel array data of portable unit having funcion of digital camera |
| EP1880551A4 (en) * | 2005-02-24 | 2011-12-07 | Lg Electronics Inc | Method for up-sampling/down-sampling data of a video block |
| US8743963B2 (en) * | 2007-08-13 | 2014-06-03 | Ntt Docomo, Inc. | Image/video quality enhancement and super-resolution using sparse transformations |
| US8731062B2 (en) * | 2008-02-05 | 2014-05-20 | Ntt Docomo, Inc. | Noise and/or flicker reduction in video sequences using spatial and temporal processing |
| US20110097009A1 (en) * | 2008-06-20 | 2011-04-28 | Nxp B.V. | Digital image restoration |
| CN109598676A (en) * | 2018-11-15 | 2019-04-09 | 华南理工大学 | A kind of single image super-resolution method based on Hadamard transform |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4442454A (en) * | 1982-11-15 | 1984-04-10 | Eastman Kodak Company | Image processing method using a block overlap transformation procedure |
| US5028995A (en) * | 1987-10-28 | 1991-07-02 | Hitachi, Ltd. | Picture signal processor, picture signal coder and picture signal interpolator |
| JPH03160575A (en) * | 1989-11-20 | 1991-07-10 | Toshiba Corp | Picture display device |
| US5036281A (en) * | 1990-03-16 | 1991-07-30 | The Regents Of The University Of California | Resizing MRI images using fourier transformation |
| US5136377A (en) * | 1990-12-11 | 1992-08-04 | At&T Bell Laboratories | Adaptive non-linear quantizer |
| JPH04246992A (en) * | 1991-01-31 | 1992-09-02 | Sony Corp | Image conversion device |
| US5168375A (en) * | 1991-09-18 | 1992-12-01 | Polaroid Corporation | Image reconstruction by use of discrete cosine and related transforms |
| US5298997A (en) * | 1991-10-01 | 1994-03-29 | Ricoh Company, Ltd. | Line data-block data converting circuit |
| JP3278881B2 (en) * | 1991-12-13 | 2002-04-30 | ソニー株式会社 | Image signal generator |
| GB2262854B (en) * | 1991-12-24 | 1995-05-24 | Sony Broadcast & Communication | Image processing apparatus |
| JPH05261982A (en) * | 1992-03-23 | 1993-10-12 | Seiko Epson Corp | Printing device |
| JPH05316357A (en) * | 1992-05-14 | 1993-11-26 | Nippon Telegr & Teleph Corp <Ntt> | Resolution converting method |
-
1994
- 1994-07-12 US US08/273,869 patent/US5666163A/en not_active Expired - Lifetime
-
1995
- 1995-04-28 KR KR19950010889A patent/KR960006587A/ko not_active Application Discontinuation
- 1995-05-02 CA CA002148433A patent/CA2148433A1/en not_active Abandoned
- 1995-05-10 TW TW084104628A patent/TW296529B/zh not_active IP Right Cessation
- 1995-05-30 EP EP95303664A patent/EP0696780A3/en not_active Withdrawn
- 1995-06-09 JP JP7143550A patent/JPH08101917A/en active Pending
-
1997
- 1997-05-29 US US08/864,956 patent/US6141054A/en not_active Expired - Lifetime
Also Published As
| Publication number | Publication date |
|---|---|
| US6141054A (en) | 2000-10-31 |
| EP0696780A3 (en) | 1996-05-08 |
| TW296529B (en) | 1997-01-21 |
| EP0696780A2 (en) | 1996-02-14 |
| JPH08101917A (en) | 1996-04-16 |
| KR960006587A (en) | 1996-02-23 |
| US5666163A (en) | 1997-09-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2312983C (en) | Continuous gradation compression and expansion of an image or acoustics data based on a polynomial approximation | |
| JP5113971B2 (en) | Processing technique and apparatus for superimposing images for projection | |
| US6456745B1 (en) | Method and apparatus for re-sizing and zooming images by operating directly on their digital transforms | |
| US6078307A (en) | Method for increasing luminance resolution of color panel display systems | |
| JP4570336B2 (en) | A method for constructing an extended color gamut digital image from a limited color gamut digital image. | |
| US8275218B2 (en) | Forward and backward image resizing method | |
| US5666163A (en) | Electronic image resolution enhancement by frequency-domain extrapolation | |
| US20020109788A1 (en) | Method and system for motion image digital processing | |
| JP5451313B2 (en) | Image processing apparatus, image processing method, and program | |
| US4853794A (en) | Method and image processing system for reconstruction of an image | |
| US7548656B2 (en) | Method and apparatus for processing image signals by applying a multi-resolution conversion processing for reducing the image size and applying a dyadic wavelet transform | |
| US20120087582A1 (en) | Method and system for resizing an image | |
| US7245781B2 (en) | Applying a tone scale function to a digital image | |
| JPH1032763A (en) | Image pickup method of using offset spatial optical modulator array | |
| JP2522357B2 (en) | Image enlargement method | |
| JP2001008027A (en) | Image magnification device | |
| US6882752B1 (en) | Image compression and expansion apparatus and pixel number increasing apparatus | |
| JPS63239567A (en) | Image processing method and apparatus | |
| JP3600750B2 (en) | Image compression device and image decompression device | |
| JP3920122B2 (en) | Image processing method and apparatus, image processing system, and program | |
| JPH0345085A (en) | Sharpness emphasis method for television picture | |
| JPH0273475A (en) | Method and device for processing blurred picture | |
| JPH01222588A (en) | Picture correcting device | |
| JPH05260521A (en) | Reproduction processing method for compressed moving image data | |
| JPH0822084B2 (en) | Video image processor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| FZDE | Discontinued |