US H2003 H1 Abstract Systems and methods are provided for enhancing an image by removing defects or blemishes from the image. A user selects a region of one or more pixels to be altered or modified. Interpolation techniques are used to determine interpolated values for pixels in the selected region based on the values of pixels surrounding the selected region. Thereafter, a smoothing function determines new values for each pixel in the selected region based on the values of pixels adjacent to the pixel being smoothed. The smoothing function kernel is applied iteratively to the pixels in the selected region until the image is smoothed to a desired degree.
Claims(29) 1. A method of calculating new values for pixels in an image desired to be altered, comprising the steps of:
a) providing an image, said image including:
a region of one or more first pixels desired to be altered, and
a perimeter surrounding said region and comprising second pixels having known values;
b) calculating a pixel value for each of said first pixels using linear interpolation based on at least a portion of said second pixels;
c) applying a smoothing function to said first pixels, wherein said function recalculates said pixel values for each of said first pixels based on the value of at least the first and second pixels adjacent the pixel being recalculated, said recalculation using the previously calculated pixel values for said first adjacent pixels; and
d) repeating step c) if a stop condition is not satisfied, wherein said recalculation uses the previously recalculated pixel values for said first adjacent pixels.
2. The method of claim
1, further comprising the step of determining a sum of the changes in each of said first pixel values during each iteration of step c), wherein said stop condition is satisfied when said sum is smaller than a predetermined value.3. The method of claim
1, further comprising the step of determining the average change in value of said first pixels during an iteration of step c), wherein said stop condition is satisfied when said average is smaller than a predetermined value.4. The method of claim
1, further comprising the step of determining, for each iteration of step c), a maximum value corresponding to the change in value of one of said first pixels having the maximum change during step c), wherein said stop condition is satisfied when said maximum value is smaller than a predetermined value.5. The method of claim
1, further comprising the step of entering an iteration number N, wherein said stop condition is satisfied if step c) has been repeated N times.6. The method of claim
1, wherein said image is a digital image, and wherein said step of providing an image includes the step of selecting said first pixels with a computer generated brush.7. The method of claim
1, wherein said image is a two-dimensional image, and wherein said calculating step b) uses four-way linear interpolation.8. The method of claim
1, wherein said image is an N-dimensional image, wherein N is an integer greater than two.9. The method of claim
1, wherein said smoothing function is a minimum curvature solution algorithm.10. A method of enhancing a computer generated image, comprising the steps of:
a) acquiring a digital image;
b) selecting a portion of said image, said portion including a plurality of first pixels, said portion surrounded by second pixels having known values;
c) calculating a pixel value for each of said first pixels using linear interpolation based on the known values of at least a portion of said second pixels; and
d) iteratively applying a minimum curvature algorithm to said first pixels until a stop condition is satisfied, wherein said algorithm recalculates said pixel values for each of said first pixels based on the value of at least the first and second pixels adjacent the pixel being recalculated, said recalculation using previously calculated pixel values for any of said first adjacent pixels.
11. The method of claim
10, further comprising the step of displaying said image using said recalculated values of said first pixels, after said stop condition is satisfied.12. The method of claim
10, further comprising the step of determining a sum of the changes in each of said first pixel values during each iteration of step d), wherein said stop condition is satisfied when said sum is smaller than a predetermined value.13. The method of claim
10, further comprising the step of determining the average change in value of said first pixels during each iteration of step d), wherein said stop condition is satisfied when said average is smaller than a predetermined value.14. The method of claim
10, further comprising the step of determining, for each iteration of step d), a maximum value corresponding to the change in value of one of said first pixels having the maximum change during step d), wherein said stop condition is satisfied when said maximum value is smaller than a predetermined value.15. The method of claim
10, further comprising the step of entering an iteration number N, wherein said stop condition is satisfied if step d) has been repeated N times.16. The method of claim
10, wherein said step of selecting includes the step of selecting said first pixels with a computer generated brush.17. The method of claim
10, wherein said digital image is a two-dimensional digital image, and wherein said calculating step b) uses four-way linear interpolation.18. The method of claim
10, wherein said digital image is an N-dimensional digital image, wherein N is an integer greater than two.19. The method of claim
10, wherein said smoothing function is a minimum curvature solution algorithm.20. An image processing system, comprising:
a) means for providing an image;
b) means for selecting a portion of said image, said portion including a plurality of first pixels, said portion surrounded by second pixels having known values;
c) a processor, wherein said processor calculates an initial pixel value for each of said first pixels using linear interpolation based on the known values of at least a portion of said second pixels, and wherein the processor thereafter iteratively applies a smoothing function to each of said first pixels until a stop condition has been satisfied, said function recalculating each of said first pixel values using the pixel values of at least the pixels adjacent the pixel being recalculated; and
d) means for displaying said image using the recalculated values of said first pixels after said stop condition has been satisfied.
21. The system of claim
20, wherein said means for displaying includes one of a monitor and a printer.22. The system of claim
20, wherein said means for selecting includes a computer generated brush.23. The system of claim
20, wherein said providing means includes means for providing a two-dimensional digital image, and wherein said processor calculates said initial pixel values using four-way linear interpolation.24. The system of claim
20, wherein said providing means includes means for providing an N-dimensional digital image, wherein N is an integer greater than two.25. The system of claim
20, wherein said smoothing function is a minimum curvature solution algorithm.26. The system of claim
20, wherein said processor further determines a sum of the changes in each of said first pixel values during each iteration of said smoothing function, and wherein said stop condition is satisfied when said sum is smaller than a predetermined value.27. The system of claim
20, wherein said processor further determines the average change in value of said first pixels during each iteration of said smoothing function, wherein said stop condition is satisfied when said average is smaller than a predetermined value.28. The system of claim
20, wherein said processor further determines, for each iteration of said smoothing function, a maximum value corresponding to the change in value of one of said first pixels having the maximum change during each iteration, wherein said stop condition is satisfied when said maximum value is smaller than a predetermined value.29. The system of claim
20, further comprising means for entering an iteration number N, wherein said stop condition is satisfied when said smoothing function has been applied N times.Description A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the xerographic reproduction by anyone of the patent document or the patent disclosure in exactly the form it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. The present invention relates generally to a technique for retouching, or removing defects or blemishes from, an image, and more specifically to a method and apparatus for recalculating values of pixels selected or identified using a computer generated brush. When using an image processing system, it is often desirable to alter one or more pixels in an image. For example, it is desirable to correct images containing defective or missing pixels caused by imperfections in the optical equipment used to acquire the image, such as scratches, smudges or other blemishes on camera lenses, photocopier platens or on the surface of the contact glass of a scanner. It is also desirable to manipulate images by replacing selected portions and filling in those portions with a continuation of the surrounding image. For instance, one may wish to alter a picture by removing a portion of the image, such as a person or writing on a wall, while maintaining continuity with the surrounding image. Some techniques for replacing, or correcting pixels in an image typically employ low pass filtering. However, low pass filtering often tends to blur the image. Other techniques use extrapolation based on the nearest neighbors. That is, the value of bad or missing pixels will be extrapolated based on the value of neighboring pixels having correct or valid values. These techniques can also result in a blurred image, mostly because they do not allow for smoothing of the image when there is a large variation in color in the neighboring pixels. Extrapolation techniques also can result in visual discontinuities or other visual artifacts at the edge of the filled-in region. Thus, it is clear that what is needed in the art is an improved technique for retouching, or removing defects or blemishes from, an image. The present invention provides systems and methods for enhancing an image by removing defects or blemishes from the image. The techniques of the present invention use interpolation to determine interpolated values for pixels in a selected region based on the values of pixels surrounding the selected region. Thereafter, a smoothing function determines new values for each pixel in the region based on the values of pixels adjacent the pixels being “smoothed”. The selected region comprises one or more pixels. A user selects a portion of an image comprising one or more pixels using a computer-generated eraser brush or air brush, for example. Alternatively, a region of one or more pixels is selected by designating pixels having a certain value, or no value (i.e., missing pixels). The selected portion is erased (or the values of the pixels in the region are ignored, for example), and the erased portion is filled in so as to “blend in” with the surrounding image. That is, a new value is determined for each pixel within the selected region. Specifically, the to-be-replaced pixels in the selected region are filled in using interpolation and a smoothing function. Initially, straight interpolation is used to fill in the to-be-replaced pixels based on the average of the pixel values of the surrounding area. For example, in a one-dimensional image, linear interpolation is used; in a two-dimensional image, four-way linear interpolation is used. A smoothing function such as a minimum curvature solution algorithm is then iteratively applied to each of the to-be-replaced pixels (now with interpolated values). The smoothing function calculates new values for the to-be-replaced pixels based on each pixel's nearest neighbors. For each iteration, the smoothing function uses the previously calculated values for each nearest neighbor pixel that lies within the selected region of to-be-replaced pixels. According to an aspect of the invention, a method is provided for calculating new values of pixels in an image desired to be altered, comprising the steps of: providing an image including a region of first pixels desired to be altered, and a perimeter surrounding the region and comprising second pixels having known values; calculating a pixel value for each of the first pixels using linear interpolation based on at least a portion of the second pixels; applying a smoothing function to the first pixels, wherein the function recalculates the pixel values for each of the first pixels based on the value of at least the first and second pixels adjacent the pixel being recalculated, wherein the recalculation is done using the previously calculated pixel values for the first adjacent pixels; and reapplying the smoothing function if a stop condition is not satisfied. According to another aspect of the present invention, a method is provided for enhancing a computer generated image, comprising the steps of: acquiring a digital image; selecting a portion of the image including a plurality of first pixels surrounded by second pixels having known values; calculating a pixel value for each of the first pixels using linear interpolation based on the known values of at least a portion of the second pixels; and iteratively applying a minimum curvature algorithm to the first pixels until a stop condition is satisfied, wherein the algorithm recalculates the pixel values for each of the first pixels based on the value of at least the first and second pixels adjacent to the pixel being recalculated, the recalculation using previously calculated pixel values for any of the first adjacent pixels. According to yet another aspect of the present invention, an image processing system is provided, comprising: means for providing an image; means for selecting a portion of the image including a plurality of first pixels surrounded by second pixels having known values; a processor, wherein the processor calculates an initial pixel value for each of the first pixels using linear interpolation based on the known values of at least a portion of the second pixels, and wherein the processor thereafter iteratively applies a smoothing function to each of the first pixels until a stop condition has been satisfied, the function recalculating each of the first pixel values using the pixel values of at least the pixels adjacent the pixel being recalculated; and means for displaying the image using the recalculated values of the first pixels after the stop condition has been satisfied. Reference to the remaining portions of the specification, including the drawings and claims, will realize other features and advantages of the present invention. Further features and advantages of the present invention, as well as the structure and operation of various embodiments of the present invention, are described in detail below with respect to the accompanying drawings. In the drawings, like reference numbers indicate identical or functionally similar elements. FIG. 1 depicts a flowchart which illustrates a preferred methodology according to the present invention; FIGS. 2 FIG. 3 depicts an exemplary image processing system; FIG. 4 illustrates an irregular patch of pixels for which interpolated values are to be determined according to the present invention; and FIG. 5 illustrates an exemplary adjustment Kernel according to the present invention. FIG. 1 is a flowchart which illustrates a preferred methodology according to the present invention. A simplified description of the present invention will be made with reference to FIGS. 2 FIG. 2 At step At step An exemplary image processing system is depicted in FIG. A preferred embodiment of the present invention will now be described in more detail with reference to a two dimensional image. Referring back to FIG. 1, at step In the preferred embodiment, at step According to one embodiment, at step In steps Prior to step The keystone procedure “erase”, collectively steps Within each color plane, each pixel (picture element) has a horizontal (x) and vertical (y) location, and has an intensity, normally either a value in the range 0 to 255 (a byte) or in the range 0 to 65535 (a pair of bytes or a word). A rectangle full of byte-valued or word-valued pixel intensities is stored in memory According to one embodiment, at step At step Consider one of the pixels in the selected region (i.e., pixels to be erased and replaced), for example, the pixel P and location (X,Y)=(3,4). Each pixel in the selected region has four clearly defined relatives, the Left (L), Right (R), Top (T) and Bottom (B) pixels, which are the closest pixels to the left, right, top and bottom of P from among the pixels which are NOT to be erased and replaced. Each cell represents a pixel in this example. In the particular case of pixel P: its Left pixel, L(P) is the pixel at (X,Y)=(1,4); its Right pixel, R(P) is the pixel at (X,Y)=(8,4); its Top pixel, T(P) is the pixel at (X,Y)=(3,1); its Bottom pixel, B(P) is the pixel at (X,Y)=(3,6); Each of these four boundary pixels has a value, which is denoted by V. Hence the four values are V(L(P)), V(R(P)), V(T(P)) and V(B(P)). Each of these four boundary pixels has a distance from P. This distance is denoted by D(L(P)), etc. The distance represents the number of cell boundaries that have to be crossed to get from P to the point in question. In this specific example: D(L(P))=2; D(R(P))=5; D(T(P))=3; D(B(P))=2. After pixel P is erased, its value is replaced with the following new value W(P): where K is the number that causes the four coefficients: to sum to one, so that, specifically, K is the value: Specifically, the new value, W(P) at pixel P is a weighted average of its four neighbor pixels, L(P), R(P), T(P) and B(P) in the unerased part of the image. Each neighbor pixel should exert an influence proportional to its closeness to P, or inversely proportional to its distance from P. One way to get this effect is to use coefficients in the sum which are reciprocals of the distance from P to the neighbor sum. E.g., one such coefficient is the reciprocal 1/[D(L(P))]. This is the coefficient (except for a scale factor, K) on the value V holding at the left point L(P). Finally, the scale factor K is introduced to ensure that the value W(P) as defined in equation (1) is a weighted average. What is required is to ensure that the sum of all four coefficients sum to 1, as stated in equation (2). The necessary solved value for K is as in equation (3). The following is a worked example corresponding to the case shown in FIG. 3 above, in which: where K is the number that causes the four coefficients:
to sum to one, so that, specifically, K is the value: From (3A): Substituting for K from (3B) into (1A) yields: According to the preferred embodiment, each selected pixel P is replaced by a weighted average of its four not-to-be-erased neighbors L(P), R(P), T(P) and B(P) as shown above with reference to FIG. In an alternate embodiment, the specific weighted sum of equations (1), (2) and (3) is constructed by making two successive passes over the image. For example, in one embodiment, represented as computer code, the first pass is a row-priority pass with embedded loops of the form of interp1_horz( ): for (int y=0; y<dim_.y( ); ++y, vY+=inc( ), dY+=inc( ), fY+=inc( )) for (int x=1; x<dim_.x( ); ++x, ++v, ++d, ++f) in which the embedded loop steps across horizontal lines of the image rectangle. The second pass is a column-priority pass with embedded loops of the form of interp2_vert( ): for (int x=0; x<dim_.x( ); ++x, ++vX, ++dX, ++fX) for (int y=1; y<dim_.y( ); ++y, v+=inc( ), d+=inc( ), f+=inc( )) in which the embedded loop steps down vertical lines of the image rectangle. The first pass in interp Within the interp procedures (see Appendix A), a pixel is inside the to-be-replaced part (the selected region) if *f=1. Procedure interp1_horz first looks for runs of to-be-replaced pixels. Having found a horizontal run of such pixels, the procedure computes and stores partial sums of the weighted value W(P) (as used in (1) above) and of the quantity 1/K, with K corresponding to K in (3) above. According to this embodiment, the relevant chunk of code in interp1_horz includes two key lines:
Using the terminology introduced earlier, equation (4) is equivalent to: from which it is clear that *pv is a portion of the sum W(P), without the factor K, and equation (5) is equivalent to: from which it is clear that *pd is a portion of the sum that constitutes the denominator of the factor K in equation (3). The values *pv and *pd are stored for every pixel which is within the to-be-replaced part of the image rectangle. Procedure interp2_vert then makes passes over successive vertical columns of the image rectangle. According to this embodiment, the relevant chunk of code in interp2_vert includes two key lines:
The quantity “*pd” of equation (6) is the quantity expanded in equation (5) and translated into the earlier notation in equation (5A). The quantity “1.0/frac1+1.0/frac2” in equation (6) is the same as the earlier notation's “1/D(T(P))+1/D(B(P))”. Hence equation (6) is, in the earlier notation: The latter is identical to 1/K in the earlier notation. The quantity “*pv” on the right side of equation (7) is the quantity expanded in equation (4) and translated into the earlier notation in (4A). The quantity “v1/frac1+v2/frac2” in equation (7) is the same as the earlier notation's “V(T(P))/D(T(P))+V(B(P))/D(B(P))”. Hence equation (7) is, in the earlier notation: where K is the number that causes the four coefficients: to sum to one, so that, specifically, K is the value: Equations (1C)-(3C) are identical to equations (1) to (3), respectively, so that equation (7) is precisely the same quantity defined in equations (1) to (3) above. Once interpolated values for the pixels in the selected region have been determined, at step An exemplary adjustment Kernel according to the present invention is shown in FIG. The center C of kernel #define V(x,y)( *(p+(x)+inc( )*(y))) double v=8./20.*(V(+1,0)+V(−1,0)+V(0,−1)+V(0,+1)) −2./20.*(V(+1,+1)+V(−1,+1)+V(+1,−1)+V(−1,−1)) −1./20.*(V(+2,0)+V(0,+2)+V(−2,0)+V(0,−2)); #undef V Hence, it is clear that when a pixel adjacent to the perimeter of the selected region is being operated on by the adjustment kernel According to one embodiment, at step At step The selected region specified above need not be a single contiguous region. It may include a plurality of disconnected subregions. Some of the disconnected subregions can include islands of one or more unselected pixels. In such cases, the selected region includes all the pixels that have been selected for replacement by interpolation from the unselected pixels, regardless of their contiguity relationships to one another. While the above is a complete description of the preferred embodiments of the invention, various alternatives, modifications, and equivalents may be used. Therefore, the above description should not be taken as limiting the scope of the invention which is defined by the appended claims.
Patent Citations
Non-Patent Citations
Referenced by
Classifications
Legal Events
Rotate |