Search Images Maps Play YouTube News Gmail Drive More »
Sign in
Screen reader users: click this link for accessible mode. Accessible mode has the same essential features but works better with your reader.

Patents

  1. Advanced Patent Search
Publication numberUS5621868 A
Publication typeGrant
Application numberUS 08/228,286
Publication dateApr 15, 1997
Filing dateApr 15, 1994
Priority dateApr 15, 1994
Fee statusPaid
Also published asCA2146740A1, CA2146740C, DE69514950D1, DE69514950T2, EP0678831A2, EP0678831A3, EP0678831B1, US5844565
Publication number08228286, 228286, US 5621868 A, US 5621868A, US-A-5621868, US5621868 A, US5621868A
InventorsMasao Mizutani, Chuen-Chien Lee
Original AssigneeSony Corporation, Sony Electronics, Inc.
Export CitationBiBTeX, EndNote, RefMan
External Links: USPTO, USPTO Assignment, Espacenet
Generating imitation custom artwork by simulating brush strokes and enhancing edges
US 5621868 A
Abstract
The precision of an initial image is reduced through image automatic processing in a manner simulating techniques used in the production of a custom art work. The resulting image, although generated without human intervention, has a pleasing, customized appearance which can be desirable to consumers seeking decorative art work. One way of reducing the precision of the initial image is modifying portions of the image to have more uniform luminance, color saturation, or color hue, producing a resulting image appearing to contain brush strokes. Alternatively, the precision of the initial image is reduced by locating contrast edges in the image, and modifying the luminance along these edges, producing a resulting image in which objects and features are outlined and have an enhanced definition from the background.
Images(7)
Previous page
Next page
Claims(32)
What is claimed is:
1. A method for modifying an initial image to produce a resulting image having the appearance of custom art work, comprising
storing a pixellated representation of said initial image in an electronic memory,
inspecting pixels of said pixellated representation of said initial image to derive, without human input, modifications which reduce the precision of said initial image in a manner simulating techniques used by humans to produce custom art work, by selecting locations in said initial image, determining contrast of said initial image at each location in at least two directions, determining a direction of maximum contrast at each location, and selecting elongated portions of said initial image at each location, the elongated portion at each respective location being oriented with its elongated dimension perpendicular to the direction of maximum contrast at said respective location,
automatically applying said modifications to said pixels of said pixellated representation of said initial image to generate a pixellated representation of said resulting image by blurring said initial image to form a background within which said portions are modified, and modifying each selected portion to approximate one or more of uniform luminance, uniform color saturation and uniform color hue as compared to said initial image, whereby said resulting image has the appearance of brush strokes, and
producing a visual output of said resulting image.
2. The method of claim 1 wherein producing a visual output of said resulting image comprises one or more of
printing a copy of said pixellated representation of said resulting image and
generating a display of said pixellated representation of said resulting image on an electronic display device.
3. The method of claim 1 wherein modifying a selected portion comprises
selecting a luminance and color value corresponding to one or more pixels within said selected portion of said initial image,
comparing said selected luminance value to an average luminance value derived from a plurality of pixels of said initial image,
reducing said selected luminance value if said selected luminance value is less than said average luminance value, otherwise increasing said selected luminance value,
deriving pixel luminance and color values from said selected color value and said reduced or increased selected luminance value, and
modifying pixels of said selected portion to approach said derived pixel luminance and color values.
4. The method of claim 3 wherein deriving pixel luminance and color values comprises augmenting said selected color value and said reduced or increased selected luminance value with pseudo-random noise to produce variation in luminance or color of pixels of said selected portion.
5. The method of claim 1 wherein selecting portions of said initial image comprises selecting elongated portions having parabolic curved elongated edges.
6. The method of claim 1 wherein selecting an elongated portion of said initial image at a respective location in said initial image further comprises selecting an elongated portion having a size related to contrast of said initial image in said direction of maximum contrast at said respective location.
7. The method of claim 1 wherein determining contrast at locations in said initial image comprises determining contrast from a blurred version of said initial image, resulting in increased uniformity of orientation of said elongated portions.
8. The method of claim 1 wherein
selecting locations for elongated portions of said initial image comprises selecting a first set of rows of locations extending through said image, and then selecting a second set of rows of locations interleaved with said first set of rows of blocks and extending through said image, and
modifying said elongated portions comprises modifying portions along said first set of rows prior to modifying portions along said second set of rows, whereby modifications of said elongated portions are variably overlapped within said image.
9. The method of claim 1 wherein
selecting elongated portions of said initial image comprises selecting large portions and smaller portions of said image, and
modifying said elongated portions comprises modifying said large portions prior to modifying said smaller portions.
10. The method of claim 9 wherein
selecting locations for large elongated portions of said image comprises selecting large portions along a first set of rows extending through said image, and selecting locations for large elongated portions along a second set of rows interleaved with said first set of rows and extending through said image, and
modifying said elongated portions comprises modifying portions along said first set of rows prior to modifying portions along said second set of rows,
whereby modifications of said large elongated portions are variably overlapped within said image.
11. The method of claim 8 wherein selecting locations for smaller elongated portions of said image comprises selecting locations for smaller portions at locations in said image determined to have high contrast.
12. A method for modifying an initial image to produce a resulting image having the appearance of custom art work by identifying contrast edges in the initial image, comprising
storing a pixellated representation of said initial image in an electronic memory,
generating at each pixel of said initial image a differential measure of luminance contrast by computing a luminance variation between pixels neighboring said pixel, and computing a logarithm of said luminance variation, and
modifying luminance of said initial image by combining the logarithm of each said differential measure with luminance of a corresponding pixel in said initial image, to generate luminance values for said resulting image, whereby objects in said resulting image have enhanced definition from background, and
producing a visual output of said resulting image.
13. The method of claim 12 wherein modifying luminance of said initial image comprises increasing luminance of said image in relatively high luminance areas of said image, and decreasing luminance of said image in relatively low luminance areas of said image.
14. The method of claim 12 wherein combining each said differential measure with a corresponding pixel in said initial image comprises multiplying each said differential measure with luminance of said corresponding pixel in said initial image.
15. The method of claim 12 wherein combining each said differential measure with a corresponding pixel in said initial image comprises
comparing said differential measure to a threshold value, and only if the comparison has a predetermined result, modifying said differential measure by subtracting it from a predetermined maximum value, otherwise not, and
multiplying each thus modified differential measure with luminance of said corresponding pixel in said initial image.
16. The method of claim 15 wherein said threshold value is derived from an average luminance value taken from a plurality of pixels of said initial image.
17. Apparatus for modifying an initial image to produce a resulting image having the appearance of custom art work, comprising
an electronic memory storing a pixellated representation of said initial image,
computing circuitry inspecting pixels of said pixellated representation of said initial image to derive, without human input, modifications which reduce the precision of said initial image in a manner simulating techniques used by humans to produce custom art work, by selecting locations in said initial image, determining contrast of said initial image at each location in at least two directions, determining a direction of maximum contrast at each location, and selecting elongated portions of said initial image at each locations, the elongated portion at each respective location being oriented with its elongated dimension perpendicular to the direction of maximum contrast at said respective location, and automatically applying said modifications to said pixels of said pixellated representation of said initial image to generate a pixellated representation of said resulting image, by blurring said initial image to form a background within which said portions are modified, and modifying each selected portion to approximate one or more of more uniform luminance, uniform color saturation and uniform color hue as compared to said initial image, whereby said resulting image has the appearance of brush strokes, and
an output port for connection to an output device capable of producing a visual output of said resulting image.
18. The apparatus of claim 17 wherein said output port is configured for connection to a printer.
19. The apparatus of claim 17 wherein said output port is configured for connection to an electronic display monitor.
20. The apparatus of claim 17 wherein elongated portions selected by said computing circuitry have a size related to contrast of said initial image in said direction of maximum contrast.
21. The apparatus of claim 17 wherein said computing circuit determines contrast of said initial image from a blurred version of said initial image, resulting in increased uniformity of orientation of said portions.
22. The apparatus of claim 17 wherein
said computing circuit selects portions of said initial image by selecting a first set of rows of blocks extending through said image, and then selecting a second set of rows of blocks interleaved with said first set of rows of blocks and extending through said image, and
said computing circuit modifies portions along said first set of rows prior to modifying portions along said second set of rows,
whereby modifications of said portions are variably overlapped within said image.
23. The apparatus of claim 17 wherein
said computing circuit selects large portions and smaller portions of said image, and
said computing circuit modifies said large portions prior to modifying said smaller portions.
24. The apparatus of claim 23 wherein said computing circuit selects smaller portions of said image in regions of said image having high contrast.
25. The apparatus of claim 17 wherein said computing circuitry modifies each selected portion by the steps of
selecting a luminance and color value corresponding to one or more pixels within said selected portion,
comparing said selected luminance value to an average luminance value derived from a plurality of pixels of said initial image,
reducing said selected luminance value if said selected luminance value is less than said average luminance value, otherwise increasing said selected luminance value,
deriving pixel luminance and color values from said selected color value and said reduced or increased selected luminance value, and
modifying pixels of said selected portion to approach said derived pixel luminance and color values.
26. The apparatus of claim 25 wherein said step of deriving pixel luminance and color values comprises augmenting said selected color value and said reduced or increased selected luminance value with pseudo-random noise to produce variation in luminance or color of pixels of said selected portion.
27. The apparatus of claim 17 wherein said computing circuitry selects elongated portions of said initial image having parabolic curved elongated edges.
28. Apparatus for modifying an initial image to produce a resulting image having the appearance of custom art work by identifying contrast edges in the initial image, comprising
an electronic memory storing a pixellated representation of said initial image,
computing circuitry inspecting pixels of said pixellated representation of said initial image to derive, without human input, modifications which reduce the precision of said initial image in a manner simulating techniques used by humans to produce custom art work, by generating at each pixel of said initial image a differential measure of luminance contrast by computing a luminance variation between pixels neighboring said pixel, and computing a logarithm of said luminance variation, and modifying luminance of said initial image by combining the logarithm of each said differential measure with luminance of a corresponding pixel in said initial image, to generate luminance values for said resulting image, whereby objects in said resulting image have enhanced definition from background, and
an output port for connection to an output device capable of producing a visual output of said resulting image.
29. The apparatus of claim 28 wherein said computing circuitry modifies luminance of said initial image by increasing luminance of said image in relatively high luminance areas of said image, and decreasing luminance of said image in relatively low luminance areas of said image.
30. The apparatus of claim 28 wherein said computing circuitry combines each said differential measure with a corresponding pixel in said initial image by multiplying each said differential measure with luminance of said corresponding pixel in said initial image.
31. The apparatus of claim 28 wherein said computing circuitry combines each said differential measure with a corresponding pixel in said initial image by the steps of
comparing said differential measure to a threshold value, and only if the comparison has a predetermined result, modifying said differential measure by subtracting it from a predetermined maximum value, otherwise not, and
multiplying each thus modified differential measure with luminance of said corresponding pixel in said initial image.
32. The apparatus of claim 31 wherein said computing circuitry derives said threshold value from an average luminance value taken from a plurality of pixels of said initial image.
Description
REFERENCE TO MICROFICHE APPENDIX

A microfiche appendix including 43 frames, is attached to this application. This appendix is subject to copyright protection. The copyright owner has no objection to copying of the appendix in the form in which it appears in the files of the United States Patent Office, but reserves all other rights under copyright law.

FIELD OF THE INVENTION

The present invention relates to the automatic generation, for display or printing, of images which resemble custom artwork.

BACKGROUND OF THE INVENTION

It is known to apply computers to problems of image generation and enhancement. Image enhancement techniques have been applied to scientific applications, producing higher definition images providing improved insight. Computer animation has been used to produce high-definition images, resembling photographs, for movies and other entertainment media. As computers and software have become more powerful, computer generated or enhanced images have become closer to the photographic ideal.

Some software packages also provide special effects which deform images. For example, some packages can modify the color or luminance of an image to produce what appears to be a photographic negative of the image.

SUMMARY OF THE INVENTION

Some consumers seeking decorative artwork prefer custom art work such as oil and watercolor paintings or lithography to more precise art forms such as photography. Custom art work includes an expressive aspect often lacking from photography; typically the colors and important elements of custom art work are emphasized and exaggerated as compared to a photograph, which can result in a pleasing overall impact. Furthermore, the customized nature of custom art work is often immediately apparent from the coarseness of presentation, e.g., from the brush strokes and object outlines which can be seen upon close inspection; this customized appearance can also enhance a consumer's perception and attraction to a work.

The drive in automated computer image processing has been toward either (1) enhancing the precision of an images to approach a photographic ideal, or (2) modifying an image to produce a manifestly computer-generated result. The present invention seeks a different goal. In accordance with principles of the present invention, the precision of an initial image is reduced through automatic image processing in a manner simulating techniques used in the production of a custom art work, producing an image with a pleasing, customized appearance which can be desirable to consumers seeking decorative art work. Although the resulting image appears to be the product of human labor, the process does not require human intervention.

In one aspect of the present invention, the precision of the initial image is reduced by modifying portions of the image to have more uniform luminance, color saturation, or color hue, producing a resulting image appearing to contain brush strokes similar to those in oil paintings.

Specifically, for each portion simulating a brush stroke, luminance and color values are selected from one pixel in the initial image and used as the basis for all of the pixels in the portion. To enhance the overall contrast of the image, the luminance of the selected value is reduced if the value is less than a computed average, otherwise the luminance of the selected value is increased. The pixels of the portion are then generated from the computed luminance and color values. A small random variation or dithering is also added to produce a more realistic result.

The portions are elongated, and in one embodiment have parabolic curved elongated edges, and thus have a more realistic brush-stroke-like appearance. These elongated portions are arranged in one of four orientations (vertical, horizontal, or diagonal); the orientation is determined by measuring the direction of maximum contrast in a blurred version of the initial image, and orienting the portion perpendicular to this direction, simulating typical painting techniques. For further improved simulation, the size of the portion is modified in response to the level of contrast. The portions thus created are overlaid on a background formed by blurring the initial image.

To further improve the simulation, smaller portions simulating small brush strokes are created over the initial set of large portions in regions of high contrast. These smaller portions simulate detailed brush strokes usually created near object edges in an original oil painting.

In another aspect, the precision of the initial image is reduced by locating contrast edges in the image, and modifying the luminance along these edges, producing a resulting image in which objects and features are outlined and therefore have an enhanced definition from the background, such as in watercolor paintings.

Specifically, a differential measurement of contrast is generated at each pixel of the initial image; then, after processing these contrast measurements with a logarithmic function, the contrast measurements are combined with the initial image, enhancing the edges in the initial image. For maximum enhancement, edges in bright regions are brightened, whereas edges in dark regions are darkened.

The above and other aspects and advantages of the present invention shall be made apparent from the accompanying drawings and the description thereof.

BRIEF DESCRIPTION OF THE DRAWING

The file of this patent contains at least one photographic drawing. Copies of this patent with photographic drawing(s) will be provided by the Patent and Trademark Office upon request and payment of the necessary fee.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with a general description of the invention given above, and the detailed description of the embodiments given below, serve to explain the principles of the invention.

FIGS. 1A and 1B are perspective views of a video printer, video camera, and monitor which can utilize principles of the present invention, and FIG. 1C is a functional block diagram of computing circuitry cooperating with the printer, camera, or monitor;

FIG. 2A is an initial image prior to processing, FIG. 2B is an image produced by generating simulated brush strokes in the image of FIG. 2A in accordance with principles of the invention, and FIG. 2C is an image produced by detecting and enhancing edges in the image of FIG. 2A in accordance with principles of the invention;

FIG. 3 is an overall flow chart of a computer program for adding simulated brush strokes to an image;

FIG. 4 is a flow chart of a procedure for reading a stored file for an image;

FIG. 5 is an illustration of the relationships of Saturation (S) and Hue (H) color values to U and V color values;

FIG. 6 is a flow chart of a procedure for converting U and V color values to S and H values;

FIG. 7 is a flow chart of a procedure for painting brush strokes onto an image;

FIG. 8 is a flow chart of a subprocedure for deriving brush stroke sizes and orientations;

FIG. 9 is a flow chart of a subprocedure for filling out a brush stroke on the image;

FIGS. 10A, 10B, 10C and 10D are illustrations of brush strokes formed by the subprocedures of FIGS. 8 and 9;

FIG. 11 is a flow chart of a procedure for converting S and H color values to U and V values;

FIG. 12 is a flow chart of a computer program for enhancing edges of an image;

FIG. 13 is a flow chart of a procedure for finding edges in an image by computing differential luminance values;

FIGS. 14A and 14B are illustrations of the manner in which differential luminance values are computed by the procedure of FIG. 13.

FIG. 15 is a flow chart of a procedure for modifying the output of the procedure of FIG. 13 to create a more realistic distribution of differential values;

FIGS. 16A and 16B are histograms of differential values respectively before and after modification by the procedure of FIG. 15;

FIG. 17 is a flow chart of a procedure for combining differential values with an initial image; and

FIG. 18 is a flow chart of a procedure for smoothing a combined image to produce a result.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

FIG. 1A illustrates a video printer 10. The printer 10 may be coupled to a video camera 12 via shielded cables 14. Printer 10 includes a control panel 16 having buttons 18 with which an operator can control video printer 10 to select a desired image on a video tape in camera 12. When a desired image has been located, the operator presses button 20 to generate a photograph 22 having a still print of the selected image. Video printer 10 may be used, for example, by consumers to produce still photographs from videotapes of vacations, family gatherings, etc. Thus, video printer 10 allows video camera 12 to serve the dual roles of a video and still camera.

FIG. 1B illustrates a monitor 23 which can be similarly coupled to video camera 12 via cable 14, to display video from camera 12. Alternatively, any other video source, such as a television broadcast, a desktop VCR, or a laser disc player, can provide video to monitor 23.

Referring to FIG. 1C, in accordance with principles of the present invention, computing circuitry 25, e.g. special purpose computing circuitry and/or one or more microprocessors running software, processes images before the images are printed by printer 10 or displayed on monitor 23.

As described in detail below, the processed image has characteristics simulating the appearance of original artwork. Thus, when printer 10 is used with computing circuitry 25, printer 10 generates still images simulating custom artwork; when monitor 23 is used with computing circuitry 25, monitor 23 generates either still images simulating custom artwork, or a series of sequentially-displayed images, which simulate custom animation.

Computing circuitry 25 may be mounted in a separate unit 25 as illustrated in FIG. 1C, or may be incorporated into the existing circuitry of printer 10, monitor 23 or camera 12.

Referring to FIG. 2A, there is shown an unprocessed image of the type which could be produced by printer 10 or monitor 23. Indicated on the image are two edges of contrast 24, 26 and areas 28 of varying contrasts.

Referring to FIG. 2B, in accordance with principles of the present invention, the image of FIG. 2A may be modified to simulate techniques used in producing oil paintings. As described in greater detail in the following figures, to perform such a modification, portions of the initial image are modified to have more uniform luminance color saturation and/or color hue, producing visible portions of uniform luminance or hue in the resulting image that resemble a brush strokes of oil paintings. Such simulated brush strokes 29 can be seen in areas 28 in FIG. 2B. Furthermore, as described in more detail below, the overall contrast of the image is increased, as is apparent by comparison between FIGS. 2B and 2A.

Referring to FIG. 2C, in accordance with another aspect of the present invention, the initial image shown in FIG. 2A may be modified by locating contrast edges in the image, such as edges 24 and 26, and modifying the luminance of the image along those edges, producing a resulting image in which contrast lines such as lines 24 and 26 are significantly more pronounced, simulating the appearance of watercolor original artwork.

The following figures describe procedures for producing the effects described above. The microfiche appendix being deposited in the Patent Office herewith details a computer program written for a general purpose microprocessor in standard C language which carries out the procedures described in the following figures, and was used to generate, on a computer display, the images illustrated in FIGS. 2B and 2C.

Although the attached appendix details a program for a general purpose computer, the image processing procedures described below can be carried out in various other ways, e.g. by software running on a microprocessor embedded in a special purpose device, such as video printer 10, monitor 23, camera 12, or a stand-alone image processing unit. Alternatively, these procedures can be performed by special purpose computing circuitry in an expansion slot of a general purpose computer or embedded in a special purpose device. For clarity, in the following description, the general term "computing circuitry" will be used to collectively refer to any electronic configuration which can perform a computation, including by one or more microprocessors running software, by special purpose computing circuitry, or by a combination of both.

Referring to FIG. 3, to produce simulated brush strokes on an initial image, the computing circuitry first reads 30 a stored input file containing a pixellated representation of the image to be processed. Next, the image is converted 32 into a representation in which it may be more easily processed (as discussed with reference to FIG. 5 below). Then, the computing circuitry steps through the pixels of the image, producing or "painting" 34 simulated brush strokes onto the image. Once the image has been fully painted, the computing circuitry converts 36 the color representation back to its initial format, and writes 38 an output file containing the processed image.

Referring to FIG. 4, to read an input file 30, the computing circuitry first reads 31 luminance (Y) values from input file and stores these values in a working buffer. Then the computing circuitry creates 33 a working copy of the image by subtracting a predetermined offset from each Y value and limiting the result to an allowable range. Thereafter, the computing circuitry reads 35 U color values, and reads 37 V color values from the input file and stores these values in working buffers.

Referring to FIG. 5, the relationship between U and V color values and saturation (S) and hue (H) color values are based on trigonometry. If the U and V values represent the length of the two legs of a right triangle, the length of the hypotenuse of this triangle represents the color saturation value S, and the angle adjacent to the U leg represents the color hue value H.

Thus, referring to FIG. 6, to convert 32 color values to a usable representation at each pixel in the working image, the computing circuitry computes 39 the saturation value using the Pythagorean theorem, and computes the hue value from the inverse tangent of the ratio of the V value and the U value.

Referring to FIG. 7, to paint 34 brush strokes onto the image in the working buffer, the computing circuitry first blurs 41 the initial image by replacing each pixel's luminance value with the average of the luminance values of, e.g., 100 pixels in a square 10 by 10 pixel surrounding the pixel. This blurring process fills in gaps between brush strokes to be added by the procedures described below, to produce a more realistic result.

After the image in the working buffer has been blurred, the computing circuitry computes 42 the average luminance of the central one-third of the image. This value is used in the subsequent processing to enhance the contrast of the image, resulting in a more realistic result.

After the proceeding preliminary processing, the computing circuitry begins to generate simulated brush strokes. To do so, the computing circuitry first creates a working buffer to containing luminance, saturation and hue values for each pixel of the resulting image. The computing circuitry then fills this buffer with the blurred luminance, saturation and hue values from the initial image produced in step 41 described above. Because the buffer is initialized with a blurred, background version of the initial image, any gaps between simulated brush strokes are not noticeable in the resulting image.

To produce brush strokes, the computing circuitry scans three times through the image in blocks of, e.g., 7 pixels on a side. During the scans, the computing circuitry produces simulated brush strokes in visited blocks. During the first two scans 44, the computing circuitry produces large brush strokes in every visited block; during a third scan 55, the computing circuitry produces small brush strokes in visited blocks which meet certain criteria, as elaborated below.

The large brush strokes may be larger than the size of the block, and thus may overlap with brush strokes of neighboring blocks; when large brush strokes overlap, the more recent brush stroke erases the overlapping portions of the less recent brush stroke. To enhance the realism of the result, large brush stroke scanning 44 is interleaved; the first scan is performed along regularly-spaced rows of blocks crossing the image, and the second scan is performed along regularly-spaced rows which are interleaved with the rows of the first scan. By performing two interleaved scans, the resulting image includes a realistic mix of complete brush strokes (from the second scan) and partially erased brush strokes (from the first scan).

At each block during the three scans, the computing circuitry first inspects 45 the image in the working buffer and derives a brush stroke size and a brush stroke orientation. Brush stroke sizes and orientations are generated from the blurred image rather than the original image. This is done because the blurred image is relatively uncontaminated by noise and therefore will produce a smoother and more regular pattern of brush stroke sizes and orientations, which results in a more realistic effect.

To create a brush stroke, the computing circuitry selects 46 the luminance, saturation and hue values for the pixel at the center of the new brush stroke, and computes a new luminance and saturation value by dithering 48 (i.e., adding a randomly generated positive or negative increment to) the luminance and saturation values from the pixel at the center of the brush stroke.

Next, to increase the overall contrast of the image, the computing circuitry compares 50 the computed luminance value to that from the central one-third of the image computed in step 42. If the computed luminance is less than that of the central one-third of the image, the computed luminance is decreased; however, if the computed luminance is greater than that of the central one-third of the image, the computed luminance is increased.

Finally, the brush stroke is filled out 52 with the computed luminance, saturation and hue values and the computing circuitry proceeds to the next step across the image and repeats at step 45.

As noted above, during the first two scans, large brush strokes are generated in every visited block; thus the four steps 46, 48, 50 and 52 and performed in every visited block. However, during the third scan, small brush strokes are created in only a few of the visited blocks.

In an original oil painting, object edges are typically defined by a relatively large number of small brush strokes, a natural result of the need for precision when painting such edges. However, the body of an object is typically defined by a smaller number of large brush strokes, because less precision is needed in these areas. This methodology is simulated in accordance with the invention by producing small brush strokes only along contrast lines in the image.

Specifically, during the scanning 55 which generates small brush strokes, the computing circuitry determines 57 at each block whether the image has high contrast at the location of the block, and only generates a small stroke if the image has high contrast. As elaborated below with reference to FIG. 8, inspection step 45 measures the contrast of the image at the current block and produces a small brush stroke size if the image has high contrast. Therefore, at step 57 the computing circuitry determines whether the brush stroke is small, e.g. less than seven pixels long, and if so generates the brush stroke through steps 46, 48, 50 and 52. If the brush stroke is larger than seven pixels, there are no significant lines of contrast in the current block, and the computing circuitry proceeds to the next block without generating a small brush stroke.

Referring to FIG. 8, at step 45, the computing circuitry inspects the image to derive brush stroke sizes and orientations. This is done by computing 47 the contrast of the initial image along four directions. These four directions can be, for example, vertical, horizontal and diagonal from lower left to upper right and diagonal from upper left to lower right. To compute the contrast in each of these directions, the computing circuitry computes the difference in luminance values of appropriate groups of pixels neighboring the pixel at the center of the brush stroke. For example, to compute the contrast in a horizontal direction, the computing circuitry computes the difference between luminance values at one or more pixels directly to the right of the central pixel and luminance values at one or more pixels directly to the left of the central pixel. The absolute value of the difference represents the contrast of the image at the brush stroke in the horizontal direction. Similar computations are performed along vertical and diagonal directions to generate four contrast values. The computing circuitry then determines which of these four values is the maximum value. The brush stroke is then oriented in a direction perpendicular to the direction of the maximum contrast value. By orienting brush strokes perpendicular to the direction of the maximum contrast, brush strokes in the processed image will be generally aligned along rather than across contrast lines in the image. This enhances the simulation of oil painting techniques, because oil paintings typically include brush strokes aligned along the direction of object outlines.

After computing the brush stroke direction, the brush stroke length is computed 49 with reference to the maximum contrast level determined in the proceeding step. Where the maximum contrast level is large, the length of the brush stroke is shortened. This also simulates oil painting techniques in that oil paintings typically include short brush strokes at detailed edges surrounding objects. The width of the brush stroke is computed 51 with reference to the length, so that the brush stroke is slightly longer than it is wide. Exemplary brush strokes are illustrated in FIGS. 10A, 10B, 10C and 10D, and are discussed in further detail below.

Referring to FIG. 9, to fill out a brush stroke with computed luminance saturation and hue values 52 the computing circuitry repeatedly performs 54 a series of steps along the length of the brush stroke. The brush strokes produced may be rectangular or leaf-shaped.

Exemplary rectangular brush strokes are shown in FIGS. 10A and 10B. FIG. 10A shows a horizontally oriented brush stroke having a horizontal length L and a vertical width W. FIG. 10B shows a diagonally oriented brush stroke of the same dimensions. (The horizontal and vertical dimensions in FIG. 10B are reduced by a factor of the square root of two because the stroke is at a 45 degree angle.)

FIG. 10C shows a leaf-shaped brush stroke in a horizontal orientation and FIG. 10D shows a leaf-shaped brush stroke in a diagonal orientation. The leaf-shaped brush strokes are characterized by curved sides along their long dimensions. The curves are generated by a quadratic equation, and thus the curves are parabolic curves. The leaf-shaped brush stroke shown in FIG. 10C has a horizontal length L equal to the length L of the brush stroke shown in FIG. 10A. The maximum width W of the leaf-shaped brush stroke shown in FIG. 10C is equal to the width of the brush stroke shown in FIG. 10A. Similar relationships can be found between the diagonally oriented leaf-shaped brush stroke shown in FIG. 10D and the diagonally oriented rectangular brush stroke shown in FIG. 10B. (Again, the horizontal and vertical dimensions in FIG. 10D are reduced by a factor of the square root of two because the brush stroke in FIG. 10D is at a 45 degree angle.)

As noted above with reference to FIG. 9, brush strokes are produced by a series of steps executed along the length of the brush stroke. During each step, the computing circuitry follows 58 a vertical, horizontal, or diagonal line of pixels, such as lines 59 illustrated in FIG. 10A-10D, across the width of the brush stroke, replacing each pixel along the line with the computed luminance, saturation and hue values. The computing circuitry starts at a pixel at a randomly selected position near to the edge of the brush stroke. For increased realism, the computing circuitry adds slight randomness to the appearance of the image by randomly varying the luminance value Y as a line 59 is followed across the stroke. To do this, the luminance value is varied based upon the current location across the width of the brush stroke.

After filling out a line 59 across the width of the brush stroke, the procedure of FIG. 9 proceeds to the subsequent line across the width of the brush stroke, and so on until the brush stroke has been filled out completely.

Leaf-shaped brush strokes are created by an additional step 56 performed during the above-described procedure. In this step the computing circuitry computes 56 a brush stroke width based upon the current location along the length of the brush stroke. The equation used to compute the width value is, as noted above, a quadratic equation, and generates a maximum width value at the center of the brush stroke. After the brush stroke width has been computed, the computing circuitry fills out a line 59 across the brush stroke having the computed width, in the manner described above.

Referring to FIG. 11, after the image has been processed to include brush strokes, the color representation is converted 36 back to U and V color values. To do this, for each pixel in the working buffer, the computing circuitry computes 60 the saturation S multiplied by the cosine of the hue H to generate the U value, and computes the saturation S multiplied by the sine of the hue H to produce the V value.

Referring to FIG. 12, in another aspect the invention features procedures for converting an image to have the appearance of watercolor painting. This procedure is shown generally in FIG. 12 and more specifically in the subsequent figures.

Referring to FIG. 12, to perform an edge detection conversion to generate the appearance of watercolor painting, the computing circuitry first reads 70 the luminance values from an input file into two buffers. (Color values are not processed in this procedure, although color processing, e.g., reducing or increasing color saturation, could be added to the procedure described herein to produce more realistic watercolor images.) The computing circuitry then computes 72 the average luminance value of the central one-third of the image. This value is used in subsequent processing described below. Next, the computing circuitry performs 74 a filtering operation on the first of the two buffers to locate contrast regions in the image. Various edge detecting filtering operations can be used at step 74, such as Laplacian filtering, Forsen filtering, and/or Sobel filtering. Sobel filtering has been found to create the strongest edges, and is used in the software in the attached appendix, and described below. After filtering, the computing circuitry processes 76 the luminance values in the first buffer by performing a logarithmic function on the values in the first buffer. After the above processing, the computing circuitry combines 78 the first buffer with the second, unmodified buffer. During this operation, the computing circuitry uses the contents of the first buffer to create dark edges in dark areas of the original image in the second buffer, and vice-versa. The resulting image is smoothed 80 in a manner discussed below, producing a final image which is written 82 to an output file.

To Sobel filter 74 an image in the first buffer, the computing circuitry performs 75 a sequence of steps shown in FIG. 13 for each pixel in the first buffer. First, the computing circuitry computes 77 a weighted differential in the horizontal direction. Then, the computing circuitry computes 79 a weighted differential in the vertical direction. The absolute values of these differentials are then summed 81 and the result is limited 83 to the range of permissible luminance values. Finally, the luminance values are reversed 85 from black to white and white to black, so that areas having high contrast are identified by relatively small luminance values in the first buffer, and vice-versa. To reverse white and black for the Sobel-filter result of a pixel, the minimum (black) and maximum (white) allowable luminance values are added together, and the result of the Sobel-filtering for the pixel is subtracted from the sum. If the Sobel-filter result for the pixel had a large (near-white) value, the result will be a proportionally small (near-black) value, and vice-versa.

To compute the weighted differentials steps 77 and 79 of FIG. 13, the computing circuitry multiplies neighboring pixels of the pixel being processed by weighted factors shown diagrammatically in FIG. 14A and FIG. 14B. To compute a weighted differential in the horizontal direction, and thereby locate vertically-oriented lines of contrast, the computing circuitry multiplies the three pixels to the left, upper left and to the lower left, respectively, by the values -2, -1 and -1; then, the computing circuitry multiplies the three pixels to the immediate right, upper right and lower right of the pixel being processed, respectively, by the values 2, 1 and 1. The results of these six multiplications are added together to produce a differential value.

Similarly, as shown in FIG. 14B, to compute a differential in the vertical direction, and thereby locate horizontally-oriented lines of contrast, the pixels immediately above, above and to the left, and above and to the right of the pixel being processed are respectively multiplied by -2, -1 and -1; and the pixels immediately below, below and to the left and below and to the right of the pixel being processed are multiplied respectively by 2, 1 and 1. The results of these multiplications are added together to generate a differential value in the vertical direction.

As can be seen from the proceeding discussion, the horizontal and vertical differential values represent the extent of horizontal and vertical contrast at the pixel for which the values are computed. Thus, the Sobel filtering procedure shown in FIG. 13 generates values representative of the contrast in the image at each respective pixel.

Referring to FIG. 15, the procedure of FIG. 12 equalizes 76 the results of the Sobel filtering procedure discussed above. To accomplish this, for each pixel in the first buffer 90 the computing circuitry reverses 92 black and white values in the manner discussed above, replaces 94 each luminance value in the first buffer with the logarithm of the luminance value, and then reverses 96 black and white values.

The effect of the logarithmic computation described above is illustrated in FIGS. 16A and 16B. As shown in FIG. 16A, a histogram of all of the Sobel-filter results in the first buffer after step 74 includes a large number of pixels with near zero values and a large number of pixels with values at the maximum permissible value. However, in the mid-range of values, there is a relatively low number of pixels. However, as seen in FIG. 16B, by replacing each value with the logarithm of the value, the number of pixels in the mid-range of luminance values is substantially increased. Thus, the logarithmic histogram equalization increases, in a non-linear manner, small Sobel-filter result values to larger values. Therefore, effect of this operation on the image is to increase the emphasis on edges in images.

It should be noted that, for the logarithmic equalization to operate properly, areas of high contrast must be represented by large luminance values, and other areas represented by small luminance values. However, in step 85 (FIG. 13) black and white are reversed so that contrast edges are represented by small luminance values. Therefore, black and white must be re-reversed in step 92 before the logarithmic computation.

These two reversals of black and white in steps 85 and 92 happen in immediate succession in the software in the attached appendix, and may therefore initially appear to be unnecessary. However, the two reversals are included so that the histogram equalization procedure, i.e., all of the processing illustrated in FIG. 15, can be disabled at any given time without modifying any other procedures. This might be done to, for example, examine the difference in results with and without the logarithmic operation. In another application, steps 85 and 92 could be eliminated.

Referring to FIG. 17, to combine 78 the processed pixels from the first buffer and the unprocessed pixels from the second buffer, the circuitry or software performs 100 a number of steps for each pixel in the image. First, the computing circuitry determines 102 whether the unprocessed pixel in the second buffer has a luminance greater than the average luminance of the central one-third of the image (as previously computed) and whether the processed pixel in the first buffer is less than midscale of the possible luminance values. If both are the case, the computing circuitry reverses 104 black and white values in the first buffer in the manner discussed above. This causes edges in relatively light-colored areas to have bright outlines, while causing edges in relatively dark-colored areas to have dark outlines. Then, the computing circuitry produces a final luminance value for the pixel by multiplying 106 the corresponding pixels from the first and second buffer with each other and with a gain value. The resulting luminance value is limited 108 to the allowable range for luminance values and is stored as the final luminance value for the pixel.

Referring to FIG. 18, to smooth 80 the image produced by the proceeding steps, the circuitry or software performs 110 a loop processing each pixel by averaging 112 the pixel with a neighboring pixel, e.g., the pixel vertically neighboring the pixel. The result of this operation is to slightly blur the resulting image and produce a more realistic watercolor effect.

While the present invention has been illustrated by a description of various embodiments and while these embodiments have been described in considerable detail, it is not the intention of the applicants to restrict or in any way limit the scope of the appended claims to such detail. Additional advantages and modifications will readily appear to those skilled in the art. For example, the procedures described above use the average luminance value from the central third of the image to approximate the average luminance value across the image. It has been found that this technique can result in erroneous results if the central third of the image is particularly bright or dark. Thus, in one alternative embodiment multiple different areas of the image may be used to compute the average luminance value. The invention in its broader aspects is therefore not limited to the specific details, representative apparatus and method, and illustrative example shown and described. Accordingly, departures may be made from such details without departing from the spirit or scope of applicant's general inventive concept.

Patent Citations
Cited PatentFiling datePublication dateApplicantTitle
US4782388 *Oct 24, 1986Nov 1, 1988The Grass Valley Group, Inc.Method and apparatus for providing video mosaic effects
US4907072 *Oct 25, 1988Mar 6, 1990Yamaha CorporationMosaic picture generation circuit
US4956872 *Oct 29, 1987Sep 11, 1990Canon Kabushiki KaishaImage processing apparatus capable of random mosaic and/or oil-painting-like processing
US5063448 *Jul 31, 1989Nov 5, 1991Imageware Research And Development Inc.Apparatus and method for transforming a digitized signal of an image
US5109282 *Jun 20, 1990Apr 28, 1992Eye Research Institute Of Retina FoundationHalftone imaging method and apparatus utilizing pyramidol error convergence
US5245432 *Nov 4, 1991Sep 14, 1993Imageware Research And Development Inc.Apparatus and method for transforming a digitized signal of an image to incorporate an airbrush effect
US5325200 *Nov 4, 1991Jun 28, 1994Imageware Research And Development Inc.Apparatus and method for transforming a digitized signal of an image into a reflective surface
US5363209 *Nov 5, 1993Nov 8, 1994Xerox CorporationImage-dependent sharpness enhancement
Non-Patent Citations
Reference
1"How Image Editors Use Filters to Alter Reality", Imaging Magazine, v2, n8, p44.
2Adobe Corporation, "Gallery Effects" Brochure, apparently Feb. 1992.
3 *Adobe Corporation, Gallery Effects Brochure, apparently Feb. 1992.
4Adobe Systems Incorporated, "Gallery Effects" Brochure, Copyright 1995.
5 *Adobe Systems Incorporated, Gallery Effects Brochure, Copyright 1995.
6 *B.J. Dawson, Changing Perceptions of Reality: Image editing lets you interactively improve, rearrange, modify and adjust images any way you want, BYTE, p. 293, vol. 14, No. 13, Dec. 1989.
7Fractal Design Corp., "Painter", X2, Sketcher, and ColorStudio brochure, copyright 1993.
8 *Fractal Design Corp., Painter , X2, Sketcher, and ColorStudio brochure, copyright 1993.
9Fractal Design Corporation, "Painter" User Guide, pp. 202-203, Aug. 1991.
10 *Fractal Design Corporation, Painter User Guide, pp. 202 203, Aug. 1991.
11 *How Image Editors Use Filters to Alter Reality , Imaging Magazine, v2, n8, p44.
12 *M. Hudnall et al., Test Lab (Paint Software Review), Compute, v15, n2, p8 (13), Feb. 1993.
13PC Connection, "Altamira Composer", Photoshop, Picture Publisher, Photomorph, Digital Morph catalog pp. 31-34, date unknown.
14 *PC Connection, Altamira Composer , Photoshop, Picture Publisher, Photomorph, Digital Morph catalog pp. 31 34, date unknown.
15 *Summary of Fractal Design Corporation s products release dates.
16Summary of Fractal Design Corporation's products release dates.
17 *Z Soft, PhotoFinish brochure, copyright 1993.
18Z-Soft, "PhotoFinish" brochure, copyright 1993.
Referenced by
Citing PatentFiling datePublication dateApplicantTitle
US5784301 *Jan 26, 1996Jul 21, 1998Sony CorporationMethod and apparatus for producing paper fiber structure data, and method and apparatus for drawing bled figure
US5940081 *Jan 25, 1996Aug 17, 1999Sony CorporationMethod and apparatus for forming a font and the font produced method and apparatus for drawing a blurred figure
US5966134 *Jun 28, 1996Oct 12, 1999SoftimageSimulating cel animation and shading
US6008862 *Sep 10, 1997Dec 28, 1999U.S. Philips CorporationPerceived color transient improvement
US6011536 *May 22, 1998Jan 4, 2000New York UniversityMethod and system for generating an image having a hand-painted appearance
US6021417 *Oct 31, 1997Feb 1, 2000Foto Fantasy, Inc.Method of stimulating the creation of an artist's drawing or painting, and device for accomplishing same
US6226015 *Feb 25, 1998May 1, 2001Intel CorporationMethod of automatically producing sketches and cartoon images from movies
US6385628 *Mar 1, 1999May 7, 2002Foto Fantasy, Inc.Method for simulating the creation if an artist's drawing or painting of a caricature, and device for accomplishing same
US6714207 *Jun 5, 2001Mar 30, 2004Adobe Systems IncorporatedRecoloring art work
US6791573 *Jun 4, 2002Sep 14, 2004Adobe Systems IncorporatedSmart erasure brush
US6850274Jul 10, 1998Feb 1, 2005Silverbrook Research Pty LtdImage texture mapping camera
US6885744Dec 20, 2001Apr 26, 2005Rockwell Electronic Commerce Technologies, LlcMethod of providing background and video patterns
US6894694 *Jul 10, 1998May 17, 2005Silverbrook Research Pty LtdProducing automatic “painting” effects in images
US7092011Dec 8, 2003Aug 15, 2006Silverbrook Research Pty LtdCamera for printing on media provided on print roll
US7187404Dec 8, 2003Mar 6, 2007Silverbrook Research Pty LtdCamera for printing manipulated images on media
US7283162Dec 19, 2002Oct 16, 2007Silverbrook Research Pty LtdUtilization of color transformation effects in photographs
US7289156Jan 3, 2005Oct 30, 2007Silverbrook Research Pty LtdOptical data card reader
US7295211Feb 24, 2005Nov 13, 2007Silverbrook Research Pty LtdMethod of automatic image processing
US7377706Mar 16, 2005May 27, 2008Silverbrook Research Pty LtdPrint media roll and ink supply cartridge
US7400763Jul 30, 2004Jul 15, 2008Adobe Systems IncorporatedSmart erasure brush
US7456979 *Mar 5, 1996Nov 25, 2008Sony CorporationApparatus having means for printing video signals of video camera attached thereto
US7460152Dec 8, 2003Dec 2, 2008Silverbrook Research Pty LtdCamera for printing manipulated images
US7469071Feb 10, 2007Dec 23, 2008Fotonation Vision LimitedImage blurring
US7483050Dec 8, 2003Jan 27, 2009Silverbrook Research Pty LtdCamera and controlling processing system
US7489413Jul 6, 2004Feb 10, 2009Sony CorporationApparatus having means for printing video signals of video camera attached thereto
US7606417Aug 30, 2005Oct 20, 2009Fotonation Vision LimitedForeground/background segmentation in digital images with differential exposure calculations
US7627168Jun 18, 2008Dec 1, 2009Adobe Systems IncorporatedSmart erasure brush
US7680342May 30, 2006Mar 16, 2010Fotonation Vision LimitedIndoor/outdoor classification in digital images
US7692696Dec 27, 2005Apr 6, 2010Fotonation Vision LimitedDigital image acquisition system with portrait mode
US7760235Jun 22, 2008Jul 20, 2010Silverbrook Research Pty LtdImage manipulation method for camera
US7773113Jun 22, 2008Aug 10, 2010Silverbrook Research Pty LtdCard based image manipulation method for camera
US7804521Jun 19, 2008Sep 28, 2010Silverbrook Research Pty LtdImage manipulation camera
US7821538Jun 22, 2008Oct 26, 2010Silverbrook Research Pty LtdCamera printer operation method with media pull sensing
US7868922Aug 21, 2006Jan 11, 2011Tessera Technologies Ireland LimitedForeground/background segmentation in digital images
US7912280 *Jun 28, 2007Mar 22, 2011Canon Kabushiki KaishaImage processing apparatus, image processing method, and image processing program
US7912285Sep 13, 2010Mar 22, 2011Tessera Technologies Ireland LimitedForeground/background segmentation in digital images with differential exposure calculations
US7924313Mar 17, 2008Apr 12, 2011Silverbrook Research Pty LtdCamera device incorporating a print roll validation apparatus
US7944473Jun 23, 2008May 17, 2011Silverbrook Research Pty LtdCard based image manipulation method with card skew correction
US7948655Jun 26, 2007May 24, 2011Canon Kabushiki KaishaImage processing apparatus, image processing method, image processing program, and storage medium
US7953287Oct 17, 2008May 31, 2011Tessera Technologies Ireland LimitedImage blurring
US7957597Sep 13, 2010Jun 7, 2011Tessera Technologies Ireland LimitedForeground/background segmentation in digital images
US7984965Nov 3, 2008Jul 26, 2011Silverbrook Research Pty LtdPrint head unit with printhead and transport rollers
US8139849Feb 1, 2011Mar 20, 2012Canon Kabushiki KaishaImage processing apparatus, image processing method, and image processing program
US8175155Jun 25, 2007May 8, 2012Canon Kabushiki KaishaImage processing apparatus, image processing method, image processing program, and storage medium
US8175385Mar 4, 2011May 8, 2012DigitalOptics Corporation Europe LimitedForeground/background segmentation in digital images with differential exposure calculations
US8212897Mar 15, 2010Jul 3, 2012DigitalOptics Corporation Europe LimitedDigital image acquisition system with portrait mode
US8363908May 3, 2007Jan 29, 2013DigitalOptics Corporation Europe LimitedForeground / background separation in digital images
US8760718 *Apr 12, 2012Jun 24, 2014Casio Computer Co., Ltd.Image printer for changing a tone of an original image based on associated identification information and printing the image data having the changed tone, and corresponding image printing method and recording medium
US20120262741 *Apr 12, 2012Oct 18, 2012Casio Computer Co., Ltd.Image printer, image printing method and recording medium storing image printing program
USH2003May 29, 1998Nov 6, 2001Island Graphics CorporationImage enhancing brush using minimum curvature solution
Classifications
U.S. Classification345/595, 382/274
International ClassificationG06T5/00, G06T11/00, G06T1/00, G06T5/20
Cooperative ClassificationG06T11/001, G06T15/02, G06T11/00
European ClassificationG06T11/00C, G06T11/00, G06T15/02
Legal Events
DateCodeEventDescription
Sep 30, 2008FPAYFee payment
Year of fee payment: 12
Oct 15, 2004FPAYFee payment
Year of fee payment: 8
Oct 13, 2000FPAYFee payment
Year of fee payment: 4
Jul 8, 1996ASAssignment
Owner name: SONY CORPORATION, JAPAN
Owner name: SONY ELECTRONICS, INC., NEW JERSEY
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SONY ELECTRONICS, INC.;REEL/FRAME:008021/0755
Effective date: 19960628
Apr 15, 1994ASAssignment
Owner name: SONY ELECTRONICS, INC., NEW JERSEY
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MIZUTANI, MASAO;LEE, CHUEN-CHIEN;REEL/FRAME:007031/0894
Effective date: 19940408