|Publication number||US5524162 A|
|Application number||US 07/733,479|
|Publication date||Jun 4, 1996|
|Filing date||Jul 22, 1991|
|Priority date||Jul 22, 1991|
|Publication number||07733479, 733479, US 5524162 A, US 5524162A, US-A-5524162, US5524162 A, US5524162A|
|Inventors||Raphael L. Levien|
|Original Assignee||Levien; Raphael L.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (7), Referenced by (43), Classifications (4), Legal Events (3)|
|External Links: USPTO, USPTO Assignment, Espacenet|
The present invention relates to the field of image processing, and particularly the present invention relates to a method and apparatus for sharpening an image.
An image may be represented in a memory as an array of stored pixels. Techniques for image sharpening are well known. The sharpness of the image is a measure of the high frequency content of a line scanned video signal representing a video scan line through the image. Sharper images have generally larger changes in intensity from one pixel to another, and therefore tend to have higher frequency content. Conversely, less sharp or blurry images have generally smaller changes in intensity from one pixel to another, and therefore tend to have less higher frequency content.
As is known to those skilled in the art, an image may be sharpened by creating a blurred (unsharp) image from the original image, and then subtracting the unsharp image from the original image to create a difference image. The difference image is then scaled (multiplied) by a sharpening factor, SF, a selected parameter of the sharpening process. The original image is combined with the scaled difference image to form a sharpened. image. Thus,
The problem in the prior art is that the two factors which control sharpening, i.e. the sharpening factor SF, and the amount of blurring are typically adjusted by an operator. More blurring will result in a more pronounced sharpening effect. Less blurring works better on images which are already somewhat sharp and therefore need less sharpening.
A more subtle problem exists in some images where the sharpness of the image varies within the same photograph. Uneven sharpness can result from motion blur where a moving object (or a moving background) has a difference in sharpness. Also, photographs taken with limited depth of field have areas with different sharpness. In some photographs the effect of finite depth of field is very pronounced, but it exists to some extent in almost every photograph. Too much sharpening produces a grainy effect in some areas (such as gray sky), while not enough sharpening leaves some areas (such as faces) blurry.
The present invention is embodied in a method and apparatus for adaptively sharpening an image. In accordance with the present invention, a method and apparatus is provided for sharpening an image by dividing the image area into smaller sub-areas each containing a sub-area image, and adaptively sharpening each sub-area image individually. That is, the sharpening factor SF, and the blur area would be set differently for individual sub-area images of the original photograph. The sharpening effect is thus improved over the entire image area since each sub-area image of the original image is sharpened in accordance with the amount of sharpening needed. For example, faces could receive more sharpening while areas of gray sky would receive almost no sharpening.
In accordance with a further aspect of the present invention, the image sharpening parameters for each sub-area image are automatically set by processing the original sub-area image. Thus, sharpening each of the sub-area images of the original image does not require individual operator adjustment.
Specifically, to automatically sharpen a stored image, the maximum frequency (FMAX) of the image energy content of each sub-area image is first determined. The fast Fourier transform (FFT), a well known technique for analyzing the frequency content of signals, is suitable for determining FMAX. Values for SF and filter coefficients for blur are derived from FMAX. It is desired to increase the energy at frequencies between 0.5 FMAX and FMAX of the original image signal in order to increase the sharpness of the image.
The sharpening apparatus suitable for use with the present invention includes a finite impulse response (FIR) digital filter to provide a blurred sub-area image, and means for subtracting the blurred sub-area image from the original sub-area image to provide a difference sub-area image, means for scaling the difference sub-area image by the sharpening factor, SF, and combining the scaled sub-area difference image with the original sub-area image to produce a sharpened sub-area image. The coefficients for the FIR digital filter are determined from the FFT calculation of FMAX, as is the sharpening factor SF. Coefficient values and SF are selected to increase the relative energies of the sharpened sub-area image at the frequencies 0.5 FMAX to FMAX of the original sub-area image.
FIG. 1 shows energy distribution curves in the frequency domain for various types of images.
FIG. 2 is a block diagram of an adaptive image sharpening system in accordance with the present invention.
FIG. 3 is a graphical representation of the relationship of blur factor and SF to FMAX which is used in conjunction with the present invention.
FIG. 4 shows the manner in which an original image area is scanned to divide the image area into a plurality of sub-area images.
FIG. 5A is a frequency response of a blurring filter used in accordance with the present invention.
FIG. 5B is the resulting frequency response of the sharpening filter used in accordance with the present invention.
FIG. 6 shows the frequency response of the blurring and sharpening filters from FIGS. 5A and 5B plotted on the same axes.
FIG. 7 is a block diagram of an adaptive image sharpening transfer function in accordance with the present invention.
FIG. 8 is an image sharpening apparatus for use in conjunction with the present invention.
FIG. 9 is an illustration of a non-linear curve for use with the image sharpening apparatus of FIG. 8.
FIG. 10 is a Gaussian distribution curve representing the filter coefficients of the FIR digital filter embodied in the image sharpening apparatus of FIG. 8.
The energy distribution of typical sharp and blurry images is shown in FIG. 1. Specifically, a very sharp image will tend to have energy in all frequency bands 10, while a very blurry image will have most of its energy in the very low frequencies and almost no energy in the higher frequency bands 14. A moderately sharp image will have energy in the lower to middle frequency bands, but will lack energy in the highest frequency bands.
A block diagram of an adaptive image sharpening system is shown in FIG. 2. The original image to be sharpened is stored in a memory 16. Under the control of an image scan controller 18, the original image is processed through a sharpening transfer function 20 to provide a sharpened image for storage 22.
As illustrated in FIG. 4, the present adaptive sharpening system processes an image area 25 as a plurality of sub-areas 26. An original image of 2048 by 1536 pixels is scanned along each horizontal line 24. The horizontal image scan signal is divided into groups of 128 pixels and processed to adaptively sharpen each individual group of 128 pixels. Similarly, the original image is scanned along each vertical line 30. The vertical image scan signal is divided into groups of 128 pixels and processed to adaptively sharpen each individual group of 128 pixels. The sub-area 26 of the original image area is thus adaptively sharpened according to the amount of sharpening needed for the image in sub-area 26. In such manner, each sub-area of the original image is individually and adaptively sharpened. While sub-areas of 128 by 128 pixels are shown, any other sub-area size such a 32 by 32 pixels or other non rectangular sub-area sizes may be used. Sub-area image size may be selected to optimize the adaptive sharpening process.
FIG. 5A illustrates the frequency response curve 27 for an unsharp filter, and FIG. 5B illustrates the frequency response curve 29 for the resulting sharpening operation. FIG. 6 shows the curves of FIGS. 5A and 5B plotted on the same axes. The effect of the sharpening factor, SF and the position of the center cutoff frequency f(0) of the digital filter is shown.
The adaptive image sharpening transfer function in FIG. 7 comprises a fast Fourier transform analyzer 38 to determine the maximum frequency content FMAX, of the image signal on input terminal 34. The FFT could be calculated over the whole 128 by 128 pixel block (which is a fairly major computational overhead for the FFT) or could be for a smaller square in the center of the block.
The output of the FFT 38 is coupled to means 40 for determining the digital filter coefficients which control the amount of blur, as well as generate SF. FIG. 3 shows the relationship between the determination of FMAX and the setting of the sharpening parameters, SF and blur. (It is noted that a different technique may be used to calculate the FMAX for the blur, and the FMAX for the SF). In any event, for high values of FMAX, less blur is used. At lower values of FMAX the amount of blurring is increased up to some maximum value at very low FMAX, for very blurry images. The blur curve illustrated in FIG. 3 is inversely proportional to the square of FMAX. Also shown in FIG. 3, SF increases linearly from low values of FMAX up to a maximum constant value for moderately sharp to very sharp sub-area images. Thus, sub-area images which are already fairly sharp will not be over-sharpened, while blurry sub-area images will receive relatively more sharpening. Sub-area images containing no detail, however, will not be sharpened.
Returning to FIG. 7, the filter coefficient generator 40 provides sharpening parameters, SF and a set of filter coefficients, which will determine blur for each block of 128 horizontal pixels, to the convolution filter 42. (An embodiment of the convolution filter 42 is discussed below in conjunction with FIG. 8). A good function to use in the digital FIR filter of the convolution filter 42 is the familiar Gaussian bell curve shown in FIG. 10. It should be noted that the width of curve 10 is proportional to blur. The convolution filter 42 also receives a delayed version of the original signal from delay 36. The delay 36 is needed to allow for the FFT calculation time and subsequent determination of SF and the FIR filter coefficients. The delay 36 may be implemented by reading the image from a disk storage twice, or by using RAM memory to temporarily store the sub-area image. The output of the convolution filer 42 is a final sharpened image on output terminal 44.
In operation, an original image, or sub-area image is input at terminal 34. The FFT analyzer 38 and filter coefficient generator 40 compute the sharpening parameters, SF and FIR filter coefficients. The original image, delayed by delay element 36 is also input to the convolution filter 42 which uses the computed sharpening parameters to process the image into a final sharpened image on output terminal 44.
FIG. 8 is a block diagram illustrating image sharpening. The input image signal on terminal 34 is delayed in delay elements 46A, 46B, 46C, and 46D. A digital FIR filter includes multipliers 48A, 48B, 48C and 48D, the output of which is summed in adders 50A, 50B and 50C. There is one delay element and one multiplier for each filter coefficient, F(-n) to F(+n), with F(0) being at the center position of the FIR filter. The number of adders (50A, 50B, and 50C) is one less than the number of filter coefficients. It should be noted that, although this technique suggests a blurring in one dimension only, a similar technique is used to blur in both dimensions. For example, the blur operation could consist of two convolutions, one to blur in the horizontal direction, one in the vertical direction The output of the FIR filter which is the output of adder 50C is the blurred (unsharp) image. The blurred image is then subtracted from the original image in adder 52. The difference image at the output adder 52 may be used directly in main sharpening formula (equation 1), or processed by a non-linear function f(x), in function generator 54 (equation 2). The non-linear function f(x) is illustrated in FIG. 9.
The modified difference image at the output of function generator 54 is scaled by the sharpening factor SF in multiplier 58. The final sharpened image is produced at the output of adder 56 which combines the original input image with the modified scaled difference image from multiplier 58 to produce the sharpened output image at terminal 44. The foregoing sharpening process, which is repeated for each sub-area of the original image to produce a sharpened final image can be expressed by the following formula (equation 2).
unsharp is the blurred image produced by the digital FIR filtering in FIG. 8, with coefficients determined from the FFT of the original image,
SF is the sharpening factor determined from the FFT of the original image, and
f(x) is an optional non-linear function shown in FIG. 9. A linear function may also be used.
For color photographs, individual color planes may be individually adaptively sharpened, or alternatively, a single composite luminance plane for the color image may be adaptively sharpened.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4053934 *||May 12, 1975||Oct 11, 1977||Kornreich Philipp G||Measuring the quality of images|
|US4319268 *||Jun 30, 1980||Mar 9, 1982||Dainippon Screen Seizo Kabushiki Kaisha||Reproduction picture sharpness emphasizing method|
|US4703349 *||Mar 19, 1985||Oct 27, 1987||Picturetel Corporation||Method and apparatus for multi-dimensional signal processing using a Short-Space Fourier transform|
|US4747052 *||Oct 28, 1985||May 24, 1988||Philips Medical Systems, Inc.||Radiation image processing|
|US4794531 *||Apr 22, 1987||Dec 27, 1988||Hitachi, Ltd||Unsharp masking for image enhancement|
|US5001573 *||Nov 2, 1989||Mar 19, 1991||Dainippon Screen Mfg. Co., Ltd.||Method of and apparatus for performing detail enhancement|
|US5038388 *||May 15, 1989||Aug 6, 1991||Polaroid Corporation||Method for adaptively sharpening electronic images|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US5835637 *||Mar 20, 1995||Nov 10, 1998||Eastman Kodak Company||Method and apparatus for sharpening an image by scaling spatial residual components during image reconstruction|
|US5867606 *||Aug 12, 1997||Feb 2, 1999||Hewlett-Packard Company||Apparatus and method for determining the appropriate amount of sharpening for an image|
|US5883971 *||Oct 23, 1996||Mar 16, 1999||International Business Machines Corporation||System and method for determining if a fingerprint image contains an image portion representing a smudged fingerprint impression|
|US5926578 *||Oct 22, 1996||Jul 20, 1999||Dainippon Screen Mfg. Co., Ltd.||Image processor having a partial image preprocessor|
|US5995640 *||Oct 23, 1996||Nov 30, 1999||International Business Machines Corporation||System and method for determining if a fingerprint image contains an image portion representing a dry fingerprint impression|
|US6005963 *||Oct 23, 1996||Dec 21, 1999||International Business Machines Corporation||System and method for determining if a fingerprint image contains an image portion representing a partial fingerprint impression|
|US6072895 *||Mar 25, 1997||Jun 6, 2000||International Business Machines Corporation||System and method using minutiae pruning for fingerprint image processing|
|US6330372||Sep 13, 1999||Dec 11, 2001||Intel Corporation||Compression edge adaptive video and image sharpening and scaling|
|US6351571 *||Dec 14, 1998||Feb 26, 2002||Eastman Kodak Company||Uniform convolution algorithm for frequency decomposition of image regions|
|US6421468||Jan 6, 1999||Jul 16, 2002||Seiko Epson Corporation||Method and apparatus for sharpening an image by scaling elements of a frequency-domain representation|
|US6526179||Apr 14, 1999||Feb 25, 2003||Intel Corporation||Compression sensitive sharpening|
|US6628329||Aug 26, 1998||Sep 30, 2003||Eastman Kodak Company||Correction of position dependent blur in a digital image|
|US6788824 *||Sep 29, 2000||Sep 7, 2004||Adobe Systems Incorporated||Creating image-sharpening profiles|
|US7054036 *||Jan 25, 2002||May 30, 2006||Kabushiki Kaisha Toshiba||Image processing method and image forming apparatus|
|US7103228||Feb 8, 2002||Sep 5, 2006||Imaging Solutions Ag||Local change of an image sharpness of photographic images with masks|
|US7164806||Aug 13, 2004||Jan 16, 2007||Adobe Systems Incorporated||Creating image-sharpening profiles|
|US7692715 *||Apr 6, 2010||Ricoh Company, Ltd.||Digital camera|
|US7826678 *||Oct 23, 2006||Nov 2, 2010||Primax Electronics Ltd.||Adaptive image sharpening method|
|US8294781 *||Oct 20, 2009||Oct 23, 2012||Apple Inc.||System and method for sharpening image data|
|US8483504 *||Nov 19, 2008||Jul 9, 2013||Samsung Electronics Co., Ltd.||Digital auto-focusing apparatus and method|
|US8593552||Sep 13, 2011||Nov 26, 2013||Omnivision Technologies, Inc.||Image based systems for detecting information on moving objects|
|US8717456||Nov 1, 2011||May 6, 2014||Omnivision Technologies, Inc.||Optical imaging systems and methods utilizing nonlinear and/or spatially varying image processing|
|US8736699 *||Aug 4, 2008||May 27, 2014||Omnivision Technologies, Inc.||Multi-region imaging systems|
|US20020171855 *||Mar 29, 2002||Nov 21, 2002||Edge Christopher J.||Automated sharpening of images for soft proofing|
|US20030142355 *||Jan 25, 2002||Jul 31, 2003||Toshiba Tec Kabushiki Kaisha||Image processing method and image forming apparatus|
|US20040081367 *||Oct 16, 2003||Apr 29, 2004||Shunichiro Nonaka||Image data correction apparatus and method|
|US20040239795 *||Mar 10, 2004||Dec 2, 2004||Tatsutoshi Kitajima||Digital camera|
|US20050018921 *||Aug 13, 2004||Jan 27, 2005||Adobe Systems Incorporated, A Delaware Corporation||Creating image-sharpening profiles|
|US20050220358 *||Jun 28, 2004||Oct 6, 2005||Laurent Blonde||Method of generating blur|
|US20060245665 *||Apr 29, 2005||Nov 2, 2006||Je-Ho Lee||Method to detect previous sharpening of an image to preclude oversharpening|
|US20080002904 *||Oct 23, 2006||Jan 3, 2008||Szu-Hao Lyu||Adaptive image sharpening method|
|US20090136148 *||Nov 19, 2008||May 28, 2009||Samsung Electronics Co., Ltd.||Digital auto-focusing apparatus and method|
|US20100045825 *||Sep 15, 2006||Feb 25, 2010||Kyocera Corporation||Image Apparatus and Image Processing Method|
|US20110085050 *||Aug 4, 2008||Apr 14, 2011||Omnivision Cdm Optics, Inc.||Multi-Region Imaging Systems|
|US20110090370 *||Apr 21, 2011||Apple Inc.||System and method for sharpening image data|
|EP0770973A1 *||Oct 24, 1996||May 2, 1997||Dainippon Screen Mfg. Co., Ltd.||Image processor|
|EP0898244A2 *||Jul 31, 1998||Feb 24, 1999||Hewlett-Packard Company||Image enhancement|
|EP1231773A1 *||Feb 9, 2001||Aug 14, 2002||GRETAG IMAGING Trading AG||Local sharpening of photographic images by utilizing masks|
|EP1530154A2 *||Nov 3, 2004||May 11, 2005||Noritsu Koki Co., Ltd.||Image processing method and apparatus for image sharpening|
|WO2001020551A1 *||Sep 13, 2000||Mar 22, 2001||Intel Corporation||Compression edge adaptive video and image sharpening and scaling|
|WO2003049035A2 *||Nov 14, 2002||Jun 12, 2003||Koninklijke Philips Electronics N.V.||Method and apparatus for automatic face blurring|
|WO2003049035A3 *||Nov 14, 2002||Jun 17, 2004||Koninkl Philips Electronics Nv||Method and apparatus for automatic face blurring|
|WO2006046182A1 *||Oct 21, 2005||May 4, 2006||Koninklijke Philips Electronics N.V.||Enhancement of blurred image portions|
|Nov 22, 1999||FPAY||Fee payment|
Year of fee payment: 4
|Jun 4, 2004||LAPS||Lapse for failure to pay maintenance fees|
|Aug 3, 2004||FP||Expired due to failure to pay maintenance fee|
Effective date: 20040604