Search Images Maps Play YouTube News Gmail Drive More »
Sign in
Screen reader users: click this link for accessible mode. Accessible mode has the same essential features but works better with your reader.

Patents

  1. Advanced Patent Search
Publication numberUS20020061063 A1
Publication typeApplication
Application numberUS 10/000,741
Publication dateMay 23, 2002
Filing dateNov 30, 2001
Priority dateApr 17, 1998
Also published asEP0951177A2, EP0951177A3, US6356588
Publication number000741, 10000741, US 2002/0061063 A1, US 2002/061063 A1, US 20020061063 A1, US 20020061063A1, US 2002061063 A1, US 2002061063A1, US-A1-20020061063, US-A1-2002061063, US2002/0061063A1, US2002/061063A1, US20020061063 A1, US20020061063A1, US2002061063 A1, US2002061063A1
InventorsAnthony Otto
Original AssigneeOtto Anthony H.
Export CitationBiBTeX, EndNote, RefMan
External Links: USPTO, USPTO Assignment, Espacenet
Method for digital compression of color images
US 20020061063 A1
Abstract
The method for digital compression of color images digitally compresses color identification photographs into 512 to 2,048 bytes for storage of rectangular images in smart cards and databases. The method operates in the YCrCb color space and includes a novel color space conversion method for converting digital image data from a 15- or 24-bit RGB (Red-Green-Blue) color space to the YCrCb (Luminance-Chrominance) color space. The compression method uses a lookup table for color space conversion. A unique chromakey method is used to replace the background with a solid color to increase the visual quality of the compressed image, and to automate and simplify the chromakey process.
Images(27)
Previous page
Next page
Claims(37)
What is claimed is:
1. A method for digital compression of an image such as an identification photograph for storage of the image, said image containing image data consisting of a plurality of scan lines of pixels with scalar values, comprising the steps of:
filtering the image data by evaluating the scalar values of individual pixels in the image with respect to neighboring pixels; and
statistically encoding the image data by dividing the image into an array of blocks of pixels, and encoding each block of pixels into a fixed number of bits that represent the pixels in the block.
2. The method of claim 1, wherein said step of statistically encoding the image data comprises determining a first sample moment of each block as the arithmetic mean of the pixels in the block, determining a second sample moment of the pixels in the block, and determining a selection map of those pixels in the block having scalar values greater or less than a quantizer set to the first sample moment, along with a count of the pixels having scalar values less than the quantizer.
3. The method of claim 1, further comprising the step of replacing background in the image being compressed with a scalar value, in order to reduce noise in the image, and to increase the visual quality of the compressed image.
4. The method of claim 2, wherein said step of statistically encoding the image data further comprises determining the sample variance and the standard deviation of the pixels in the block based upon the first and second sample moments.
5. The method of claim 1, wherein said step of statistically encoding the image data comprises classifying each said block, quantifying each said block, and compressing each said block by codebook compression using minimum redundancy, variable-length bit codes.
6. The method of claim 5, wherein said step of classifying each said block comprises classifying each said block according to a plurality of categories.
7. The method of claim 1, wherein said step of filtering comprises evaluating each said individual pixel as a target pixel and a plurality of pixels in close proximity to the target pixel to determine an output value for the target pixel.
8. The method of claim 7, wherein said step of filtering comprises evaluating a sequence of five pixels, including two pixels on either side of the target pixel and the target pixel itself, for each said target pixel.
9. The method of claim 8, wherein said step of filtering comprises determining an average of the data for a window of the pixels immediately surrounding the target pixel for those pixels surrounding the target pixel that are within a specified range of values, according to the following protocol: if all five pixels are within the specified range, the output target pixel is determined to be the average of the four pixels in a raster line, two on each side of the target pixel; if the two pixels on either side are within a specified range and both sides themselves are within the range, the filtered output target pixel data is determined to be the average of the two pixels on each side of the target pixel; if the two pixels on either side of the target pixel and the target pixel itself are within a specified range, and the other two pixels on the other side are not within the specified range, the output target pixel is determined to be the average of the two neighboring pixels closest in value to the target pixel values and that fall within the specified range; if the five pixels are all increasing or decreasing, or are within a specified range, the output target pixel is determined to be the average of two pixels on whichever side of the target pixel is closest in value to the target pixel; and if the five pixels in the window do not fit into any of the prior cases, the output target pixel is unchanged.
10. The method of claim 3, wherein said step of replacing background in the image being compressed with a scalar value comprises calibrating an initial chromakey value and background scalar value.
11. The method of claim 10, wherein said step of calibrating comprises capturing at least one calibration image of the background prior to capturing an image without a subject of interest in place, consisting substantially of background pixels, and determining the average and standard deviation of said at least one calibration image to set at least an initial chromakey scalar value and range.
12. The method of claim 10, wherein said step of calibrating comprises capturing an image with a subject of interest in place, and beginning in the upper-left and upper-right corners of said at least one calibration image, collecting pixels down and towards the center of said image until an edge or image boundary is encountered, and determining the average and standard deviation of those pixels to set at least an initial chomakey value and range.
13. The method of claim 10, wherein said step of calibrating comprises manually specifying an initial chromakey value and range without respect to the properties of an individual image being captured prior to image capture.
14. A method for digital compression of a color image such as a color identification photograph for storage of the color image, said color image containing color image data consisting of a plurality of scan lines of pixels with color values, comprising the steps of:
filtering the color image data by evaluating the color values of individual pixels in the color image with respect to neighboring pixels; and
statistically encoding the color image data by dividing the color image into an array of blocks of pixels, and encoding each block of pixels into a fixed number of bits that represent the pixels in the block.
15. The method of claim 14, wherein said step of statistically encoding the color image data comprises determining a first sample moment of each block as the arithmetic mean of the pixels in the block, determining a second sample moment of the pixels in the block, and determining a selection map of those pixels in the block having color values darker or lighter than a quantizer set to the first sample moment, along with a count of the lighter pixels.
16. The method of claim 14, further comprising the step of replacing background in the color image being compressed with a solid color, in order to reduce noise in the image, and to increase the visual quality of the compressed image.
17. The method of claim 15, wherein said step of statistically encoding the color image data further comprises determining the sample variance and the standard deviation of the pixels in the block based upon the first and second sample moments.
18. The method of claim 14, wherein said step of statistically encoding the color image data comprises classifying each said block, quantifying each said block, and compressing each said block by codebook compression using minimum redundancy, variable-length bit codes.
19. The method of claim 18, wherein said step of classifying each said block comprises classifying each said block according to a plurality of categories.
20. The method of claim 19, wherein said step of classifying each said block comprises classifying each of said the blocks in one of four categories: null blocks exhibiting little or no change from the higher level or previous frame, uniform blocks having a standard deviation less than a predetermined threshold, uniform chroma blocks having a significant luminance component to the standard deviation, but little chrominance deviation, and pattern blocks having significant data in both luminance and chrominance standard deviations
21. The method of claim 20, further comprising the step of determining the number of bits to be preserved for each component of the block after each said block is classified.
22. The method of claim 21, further comprising the step of reducing the number of bits for the Y and Cr/Cb components independently for each classification.
23. The method of claim 21, further comprising the step of matching the texture map of the block with one of a plurality of common pattern maps for uniform chroma and pattern classified blocks.
24. The method of claim 21, further comprising the step of preserving all components of each said block for pattern blocks.
25. The method of claim 21, further comprising the step of preserving the mean luminance and chrominance, standard deviation luminance, and a selection map for uniform chroma blocks.
26. The method of claim 21, further comprising the step of preserving the mean values for all three color components.
27. The method of claim 14, wherein said step of filtering comprises evaluating each said individual pixel as a target pixel and a plurality of pixels in close proximity to the target pixel to determine an output value for the target pixel.
28. The method of claim 27, wherein said step of filtering comprises evaluating a sequence of five pixels, including two pixels on either side of the target pixel and the target pixel itself, for each said target pixel.
29. The method of claim 28, wherein said step of filtering comprises determining an average of the data for a window of the pixels immediately surrounding the target pixel for those pixels surrounding the target pixel that are within a specified range of values, according to the following protocol: if all five pixels are within the specified range, the output target pixel is determined to be the average of the four pixels in a raster line, two on each side of the target pixel; if the two pixels on either side are within a specified range and both sides themselves are within the range, the filtered output target pixel data is determined to be the average of the two pixels on each side of the target pixel; if the two pixels on either side of the target pixel and the target pixel itself are within a specified range, and the other two pixels on the other side are not within the specified range, the output target pixel is determined to be the average of the two neighboring pixels closest in value to the target pixel values and that fall within the specified range; if the five pixels are all increasing or decreasing, or are within a specified range, the output target pixel is determined to be the average of two pixels on whichever side of the target pixel is closest in value to the target pixel; and if the five pixels in the window do not fit into any of the prior cases, the output target pixel is unchanged.
30. The method of claim 16, wherein said step of replacing background in the color image being compressed with a solid color comprises calibrating an initial chromakey value and color of the background color.
31. The method of claim 30, wherein said step of calibrating comprises capturing at least one calibration image of the background prior to capturing an image without a subject of interest in place, consisting substantially of background pixels, and determining the average and standard deviation of said at least one calibration image to set at least an initial chromakey color and range.
32. The method of claim 30, wherein said step of calibrating comprises capturing an image with a subject of interest in place, and beginning in the upper-left and upper-right corners of said at least one calibration image, collecting pixels down and towards the center of said image until an edge or image boundary is encountered, and determining the average and standard deviation of those pixels to set at least an initial chomakey value and range.
33. The method of claim 30, wherein said step of calibrating comprises manually specifying an initial chromakey value and range without respect to the properties of an individual image being captured prior to image capture.
34. The method of claim 14, further comprising the step of converting digital color image data to the YCrCb (Luminance-Chrominance) color space.
35. The method of claim 34, wherein said step of converting digital color image data to the YCrCb (Luminance-Chrominance) color space comprises converting the color image data from the RGB (Red-Green-Blue) color space.
36. The method of claim 34, wherein said step of converting digital color image data to the YCrCb (Luminance-Chrominance) color space comprises utilizing lookup tables of selected color values for color space conversion.
37. The method of claim 36, wherein said step of converting digital color image data to the YCrCb (Luminance-Chrominance) color space comprises utilizing nine 256-entry one-byte lookup tables containing the contribution that each R, G and B make towards the Y, Cr and Cb components.
Description
    BACKGROUND OF THE INVENTION
  • [0001]
    2. Field of the Invention
  • [0002]
    The invention relates generally to information signal processing, and more particularly relates to digital compression of color images for storage of the images in smart cards and databases.
  • [0003]
    2. Description of Related Art
  • [0004]
    One technique that has been used in digital encoding of image information is known as run length encoding. In this technique, the scan lines of a video image are encoded as a value or set of values of the color content of a series of pixels along with the length of the sequence of pixels having that value, set of values, or range of values. The values may be a measure of the amplitude of the video image signal, or other properties, such as luminance or chrominance. Statistical encoding of frequent color values can also be used to reduce the number of bits required to digitally encode the color image data.
  • [0005]
    One basic encoding process is based upon the block truncation coding (BTC) algorithm published by Mitchell and Delp of Purdue University in 1979. The basic BTC algorithm breaks an image into 44 blocks of pixels and calculates the first and second sample moments. Based upon an initial quantizer, set to the first sample moment (the arithmetic mean), a selection map of those pixels lighter or darker than the quantizer is determined, along with a count of the lighter pixels. From the first and second sample moments, the sample variance, and therefore, the standard deviation, can be calculated. The mean, standard deviation, and selection map are preserved for each block. However, the original BTC method is limited to a grayscale, so that it would be desirable to adapt the BTC method to extend the BTC method to include YCrCb full-color. It would also be desirable to adapt the BTC method to handle delta values, allowing multi-level, or hierarchical, encoding and allowing encoding of differences between frames or from a specified background color.
  • [0006]
    The range of color values for any given pixel in a color image can be described, for example, as RGB color space, illustrated in FIG. 1, that can be represented by a conventional three-dimensional Cartesian coordinate system, with each axis representing 0 to 100% of the red, green and blue values for the color value of the pixel. A grayscale line can be described as running diagonally from black at 0% of each component to white at 100% of each. Since human vision can only discriminate a limited number of shades of color values, by selecting representative color values in such a color space, a limited number of color values can be used to approximate the actual color values of an image such that the human eye can not differentiate between the actual color values and the selected color values.
  • [0007]
    As is illustrated in FIGS. 2 and 3, human vision can be characterized by the Hue-Value-Saturation (HVS) color system. Hue is defined as the particular color in the visible spectrum ranging from red through green and blue to violet. Value is defined as the brightness level, ignoring color. Saturation is defined as the intensity of the particular color or the absence of other shades in the mixture. The HVS system can be represented by a generally cylindrical coordinate system with a polar base consisting of hue as the angle, and saturation as the radius. The value or brightness component is represented as the altitude above the base. The actual visible colors do not occupy the entire cylinder, but are approximately two cones, base to base with their vertices at 0% up to 100% on the value scale. The base is tilted in this example because the maximum saturation for blue occurs at a much lower brightness than the maximum saturation of green.
  • [0008]
    Referring to FIG. 2, in order to represent digitized NTSC/PAL video in a Cartesian coordinate system, the YCrCb color space is used. Because the method of the invention operates in the YCrCb color space, the method provides for a novel color space conversion method from 15- or 24-bit RGB. Eight-bit grayscale images are also supported. Referring also to FIG. 3, the chrominance components, Cr and Cb, are two axes that correspond to the polar hue and saturation components in the HVS system. The Y, or luminance, component corresponds to the brightness axis in the HVS graph. This description does not account for the slight differences between YIQ and YUV for NTSC- and PAL-based encoding, which does not form a part of the invention. The following equations can be used to convert from RGB to the YCrCb color space:
  • Y=0.299R+0.587G+0.114B
  • Cr=0.713(0.701R−0.587G+0.114B)
  • Cb=0.564(−0.299R−0.587G+0.866B)
  • [0009]
    Typical implementations of digitally representing color values in this fashion use floating point arithmetic (11 multiplications and 9 additions/subtractions per pixel) or 16-bit integer arithmetic (9 multiplications, 9 additions/subtractions and 3 division per pixel). Both of these methods are quite wasteful of computing power, particularly on smaller microcontrollers. There is thus a need for a method for representing color values of digitized images that takes advantage of the limitations of human vision in discriminating color in color images in order to reduce the software and hardware requirements, particularly for storage of such color images in smart cards and databases. Smart cards are commonly approximately the same shape and size of a common credit card, and typically contain a programmable microchip, having a memory such as a read only memory, or a read/write memory. Information stored in the memory of the card can be detected by a card interface device such as a card reader or connector.
  • [0010]
    Unfortunately, noise can seriously interfere with the efficiency of any image compression process, lossy or lossless, because a compression engine must use more unnecessary data to encode noise as if it were actual subject material. Since lossy compression tends to amplify noise creating more noticeable artifacts upon decompression, lossy compression processes therefore typically attempt to remove some of the noise prior to compressing the data. Such preprocessing filters must be used very carefully, because too little filtering will not have the desired result of improved compression performance, and too much filtering will make the decompressed image cartoon-like.
  • [0011]
    Another method used for removing unwanted noise from color image data is chromakeying, which is a process of replacing a uniform color background (usually blue) from behind a subject. A common application of this process is a television weather reporter who appears to be standing in front of a map. In actuality, the reporter is standing in front of a blue wall while a computer generated map is replacing all of the blue pixels in the image being broadcast.
  • [0012]
    While preprocessing filters can remove noise from the area surrounding a subject of interest in an image, subtle changes in lighting or shading can remain in the original image which can be eliminated by chromakeying. There is thus a need to provide a chromakey method in compression of color images for storage on smart cards and databases, in order to replace the background with a solid color to increase the visual quality of the compressed image. It would also be desirable to automate and simplify the chromakey process, to simplify the chromakey process for the operator. The present invention meets these and other needs.
  • SUMMARY OF THE INVENTION
  • [0013]
    The present invention provides for an improved method for digitally compressing color identification photographs into 512 to 2,048 bytes for storage in inexpensive contact-less smart cards or in databases. The compression method of the invention accepts rectangular images in 16 pixel increments ranging from 48 to 256 pixels on a side. The typical size is 9696 pixels. The method of the invention is designed for very low computational power implementations such as 8-bit microcontrollers, possibly with an ASIC accelerator.
  • [0014]
    Briefly, and in general terms, the present invention accordingly provides for a method for digital compression of a color image containing color image data consisting of a plurality of scan lines of pixels with color values, such as for a color identification photograph, for storage of the color image. The color image data is filtered by evaluating the color values of individual pixels in the color image with respect to neighboring pixels, and the color image data is statistically encoded by dividing the color image into an array of blocks of pixels, and encoding each block of pixels into a fixed number of bits that represent the pixels in the block.
  • [0015]
    In a presently preferred aspect of the method, the step of statistically encoding the color image data comprises determining a first sample moment of each block as the arithmetic mean of the pixels in the block, determining a second sample moment of the pixels in the block, and determining a selection map of those pixels in the block having color values darker or lighter than a quantizer set to the first sample moment, along with a count of the lighter pixels. Statistically encoding the color image data entails determining the sample variance and the standard deviation of the pixels in the block based upon the first and second sample moments.
  • [0016]
    In another aspect of the invention, each block is classified, quantized, and compressed by codebook compression using minimum redundancy, variable-length bit codes. The step of classifying each block comprises classifying each block according to a plurality of categories, and the step of classifying each block typically comprises classifying each of the blocks in one of four categories: null blocks exhibiting little or no change from the higher level or previous frame, uniform blocks having a standard deviation less than a predetermined threshold, uniform chroma blocks having a significant luminance component to the standard deviation, but little chrominance deviation, and pattern blocks having significant data in both luminance and chrominance standard deviations.
  • [0017]
    In a presently preferred aspect of the method, the number of bits to be preserved for each component of the block are determined after each block is classified, and the number of bits for the Y and Cr/Cb components is reduced independently for each classification. The texture map of the block is also preferably matched with one of a plurality of common pattern maps for uniform chroma and pattern classified blocks. All components of each block are preserved for pattern blocks; the mean luminance and chrominance, standard deviation luminance, and a selection map are preserved for uniform chroma blocks; and the mean values are preserved for all three color components for uniform blocks.
  • [0018]
    The step of filtering currently preferably comprises evaluating each individual pixel as a target pixel and a plurality of pixels in close proximity to the target pixel to determine an output value for the target pixel, and in a currently preferred aspect of the method of the invention, the step of filtering comprises evaluating a sequence of five pixels, including two pixels on either side of the target pixel and the target pixel itself, for each target pixel.
  • [0019]
    In a presently preferred embodiment, the step of filtering comprises determining an average of the data for a window of the pixels immediately surrounding the target pixel for those pixels surrounding the target pixel that are within a specified range of values, according to the following protocol: if all five pixels are within the specified range, the output target pixel is determined to be the average of the four pixels in a raster line on each side of the target pixel; if the two pixels on either side are within a specified range and both sides themselves are within the range, the target pixel is determined to be impulse noise, and the filtered output target pixel data is determined to be the average of the two pixels on each side of the target pixel; if the two pixels on either side of the target pixel and the target pixel itself are within a specified range, and the other two pixels on the other side are not within the specified range, the target pixel is determined to be an edge pixel, and the output target pixel is determined to be the average of the two pixels on the matching side that fall within the specified range; if the five pixels are all increasing or decreasing, or are within a small range to account for ringing or pre-emphasis typically found in analog video signals, the target pixel is treated as being in the midst of a blurred edge, and the output target pixel is then determined to be the average of two pixels on whichever side of the target pixel is closest in value to the target pixel; and if the five pixels in the window do not fit into any of the prior cases, the output target pixel is unchanged.
  • [0020]
    A currently preferred aspect of the method of the invention further comprises replacing the background in the color image being compressed with a solid color, in order to reduce noise in the image, and to increase the visual quality of the compressed image. A significant part of replacing background in the color image being compressed with a solid color comprises calibrating an initial chromakey value and color of the background color. In one presently preferred embodiment, the step of calibrating comprises capturing at least one calibration image of the background prior to capturing an image with a subject of interest in place, consisting substantially of background pixels, and determining the average and standard deviation of the at least one calibration image to set at least an initial chromakey color and range. In another presently preferred embodiment, the step of calibrating comprises capturing an image with a subject of interest in place, and beginning in the upper-left and upper-right corners of the image, collecting pixels down and towards the center of the image until an edge or image boundary is encountered, and determining the average and standard deviation of those pixels to set at least an initial chomakey value and range. In another presently preferred embodiment, the step of calibrating comprises manually specifying an initial chromakey value and range without respect to the properties of an individual image being captured prior to image capture.
  • [0021]
    Another preferred aspect of the method of the invention comprises converting digital color image data to the YCrCb (Luminance-Chrominance) color space. In a currently preferred embodiment, this involves converting the color image data from the RGB (Red-Green-Blue) color space. In a currently preferred aspect of the method, the step of converting digital color image data to the YCrCb (Luminance-Chrominance) color space comprises utilizing lookup tables of selected color values for color space conversion, and preferably the step of converting digital color image data to the YCrCb (Luminance-Chrominance) color space comprises utilizing nine 256-entry one-byte lookup tables containing the contribution that each R, G and B make towards the Y, Cr and Cb components.
  • [0022]
    These and other aspects and advantages of the invention will become apparent from the following detailed description and the accompanying drawings, which illustrate by way of example the features of the invention.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • [0023]
    [0023]FIG. 1 is a schematic representation of an RGB color space known in the prior art;
  • [0024]
    [0024]FIG. 2 is a schematic representation of NTSC/PAL video color system in the HVS color space known in the prior art;
  • [0025]
    [0025]FIG. 3 is a schematic representation of NTSC/PAL video color system in the YCrCb color space known in the prior art;
  • [0026]
    [0026]FIG. 4 is a schematic diagram illustrating the process of image acquisition for the method of the present invention for compressing color image data;
  • [0027]
    [0027]FIG. 5 is a schematic diagram of an overview of the method of the present invention for compressing color image data for storage and use on a smart card;
  • [0028]
    FIGS. 6 to 10 illustrate color image data preprocessing filter protocols according to the method of the present invention for compressing color image data;
  • [0029]
    [0029]FIGS. 11A to 11D show a flow chart for the color image data preprocessing according to the method of the present invention for compressing color image data;
  • [0030]
    [0030]FIG. 12 illustrates the process of chromakeying an image with a background and a subject of interest before and after chromakeying of the image;
  • [0031]
    [0031]FIG. 13 is a flow chart of the steps for calibration of the chromakey color and range according to the method of the present invention for compressing color image data;
  • [0032]
    [0032]FIG. 14 is a diagram illustrating the automatic chromakey process according to the method of the present invention for compressing color image data;
  • [0033]
    [0033]FIGS. 15A to 15C show a flow chart for multilevel encoding of color image data according to the method of the present invention for compressing color image data;
  • [0034]
    [0034]FIG. 16 is a flow chart illustrating the steps of encoding a bitstream according to the method of the present invention for compressing color image data;
  • [0035]
    [0035]FIGS. 17A to 17E show a flow chart for codebook compression according to the method of the present invention for compressing color image data;
  • [0036]
    [0036]FIGS. 18A to 18D show a flow chart for encoding pattern maps according to the method of the present invention for compressing color image data; and
  • [0037]
    [0037]FIG. 19 shows a flow chart for encoding luminance or chrominance values by codebook lookup according to the method of the present invention for compressing color image data.
  • DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
  • [0038]
    While BTC statistical encoding can be used to reduce the number of bits required to digitally encode image data, the BTC method is limited to simple encoding of grayscale images. Digital color image compression methods typically use floating point arithmetic or 16-bit integer arithmetic, which are quite wasteful of computing power, particularly for encoding of color image data on smart cards and databases. Noise can also seriously interfere with the efficiency of the image compression process, and although preprocessing filters can be used to remove noise, too much filtering can make the decompressed image cartoon-like, while too little filtering may not be sufficient to improve compression performance.
  • [0039]
    As is illustrated in the drawings, which are presented for purposes of illustration and are not intended to limit the scope of the invention, the present invention accordingly provides for a method for digital compression of color images such as color identification photographs for storage of the color images in smart cards and databases. Digital color image data typically from a video camera or an existing digitized photograph are first converted from the RGB (Red-Green-Blue) color space to the YCrCb (Luminance-Chrominance) color space.
  • [0040]
    At (or prior to) compile time, nine 256-entry one-byte lookup tables of selected color values are prepared containing the contribution that each R, G and B make towards the Y, Cr and Cb components, for i=0.255, as follows:
  • [0041]
    1) RY[i]=2240.299i/225≅0.263i
  • [0042]
    2) GY[i]=2240.587i/225≅0.516i
  • [0043]
    3) BY[i]=2240.114i/225≅0.100i
  • [0044]
    4) RCr[i]=2550.7130.701i/225≅0.441i
  • [0045]
    5) GCr[i]=2550.713−0.587i/225≅−0.369i
  • [0046]
    6) BCr[i]=2550.713−0.114i/225≅0.072i
  • [0047]
    7) RCb[i]=2550.564−0.299i/225≅−0.149i
  • [0048]
    8) GCb[i]=2550.564−0.587i/225≅−0.292i
  • [0049]
    9) BCb[i]=2550.5640.886i/225≅0.441i
  • [0050]
    Once completed, the table can be used to convert a pixel from RGB to YCrCb as follows:
  • Y=RY[r]+GY[g]+BY[b]
  • Cr=RCr[r]+GCr[g]+BCr[b]
  • Cb=RCb[r]+GCb[g]+BCb[b]
  • [0051]
    This method requires 8304 bytes of constant ROM, six 8-bit additions and nine table lookups. The nine table lookups might require a 16-bit addition each, but more likely, the microcontroller could handle the lookup through an opcode or built-in addressing mechanism.
  • [0052]
    The color image digital compression method of the invention has unique requirements, since it is to be used for storage of color identification photographs on smart cards and databases. First, the information density is very high, since all of the features of a human face are represented in 6,000 to 7,000 pixels, ignoring the background. Secondly, small block encoding techniques, upon which the present invention is based, are particularly friendly to sharp edges. Third, the method of the invention is targeted for low computational power devices, making floating point, 25-pixel convolutions impractical. Due to bandwidth restrictions of the analog video signal, edges of objects are usually quite blurry. Conventional low pass convolutions used for noise reduction make this problem even worse. Conventional median or directional smoothing filters are very effective at removing impulse noise, but tend to be too aggressive for this application, because features like eyes and teeth are discarded.
  • [0053]
    The method of the invention therefore implements a preprocessing filter with three goals: 1) reducing noise without removing important face features, 2) sharpen blurred edges, and 3) not to be computationally complex. The preprocessing filter utilizes a five pixel window on a single scan line to determine the output value for the center pixel. For each target pixel, a sequence of five pixels, including 2 pixels on either side of the target pixel and the target pixel itself, are evaluated. Five cases are accounted for in the following discussion, which is directed only to the component of luminance, for simplicity. All three components (YCrCb) are included in the actual filters.
  • [0054]
    Referring to FIG. 6, in order to filter data for an individual target pixel, an average of the data for the pixels immediately surrounding the target pixel is taken, for those pixels surrounding the target pixel that are within a specified range of values. If all five pixels are within specified limits, the output is the average of four pixels in a raster line (A, B, D, E), two on each side of the target (C). If the two pixels on either side are within a specified range and both sides themselves are within the range, the target pixel is treated as impulse noise. As is illustrated in FIG. 7, the filtered output target pixel data is the average of the four pixels (A, B, D, E) on each side of the target pixel (C). Referring to FIG. 8, if the two pixels on either side of the target pixel and the target pixel itself are within a specified range, the target pixel (C) is considered to be an edge pixel. The output target pixel (C) is the average of the two pixels (A, B or D, E) on the matching side. If the five pixels are all increasing or decreasing (or are within a small range to account for ringing or pre-emphasis typically found in analog video signals), the target pixel is considered to be in the midst of a blurred edge. As is shown in FIG. 9, the output target pixel is then the average of two pixels (A, B) on whichever side is closest in value to the target pixel. As is illustrated in FIG. 10, if the five pixels in the window do not fit into any of the prior cases, the target is treated as being in the midst of a busy area, and the output target pixel is unchanged. The flow chart of FIGS. 11A to 11D illustrate the color image data preprocessing according to the method of the present invention.
  • [0055]
    Three methods illustrated in the flow chart of FIG. 13 are used to set the initial chromakey value and color: calibration image, automatic, and manual. In the chromakey calibration image process, also illustrated in FIG. 12 showing a picture before and after being subjected to the chromakeying process, prior to capturing an image with the subject of interest in place, one or more calibration images are captured of the background consisting substantially entirely of background pixels. The average and standard deviation of those entire images are determined, and are used to set at least an initial chromakey color and range.
  • [0056]
    In the automatic chromakey calibration process of the invention, illustrated in FIG. 14, an image is captured with the subject in place. Starting in the upper-left and upper-right corners of an image, pixels are collected down and towards the center until an edge or image boundary is encountered. The average and standard deviation of those pixels are calculated and used to set the initial chromakey value and range. For manual calibration the initial chromakey value and range are specified without respect to the properties of an individual image being captured prior to image capture.
  • [0057]
    In the multilevel statistical encoding of color image data according to the present invention, as illustrated in FIGS. 15A to 15C, 16 and 17A to 17E, the first portion of the actual process of compression typically involves dividing the color image into an array of 44 squares of pixels, and encoding each 44 square of pixels into a fixed number of bits that represent the sixteen pixels in the block. In the modified BTC algorithm the image is divided into 44 blocks of pixels, and the first sample moment (the arithmetic mean) and the second sample moment are determined. A selection map of those pixels having color values less than or greater than a quantizer set to the first sample moment is determined, along with a count of the lighter pixels. The sample variance and the standard deviation can thus be determined based upon the first and second sample moments. The mean, standard deviation, and selection map are preserved for each block. As adapted for YCrCb color, according to the method of the present invention, the first sample moment for each color component is thus determined according to the following equations: Y _ = 1 16 i = 1 16 Y i Cr _ = 1 16 i = 1 16 Cr i Cb _ = 1 16 i = 1 16 Cb i
  • [0058]
    The second sample moment is determined according to the following equations: Y 2 _ = 1 16 i = 1 16 ( Y i ) 2 Cr 2 _ = 1 16 i = 1 16 ( Cr i ) 2 Cb 2 _ = 1 16 i = 1 16 ( Cb i ) 2
  • [0059]
    The standard deviation is determined according to the following equations:
  • σY ={square root}{square root over (Y2)}−({overscore (Y)})2
  • σCr ={square root}{square root over (Cr2)}−({overscore (Cr)})2
  • σCb ={square root}{square root over (Cb2)}−({overscore (Cb)})2
  • [0060]
    The selection map mi for each block is determined as is illustrated in FIGS. 15A to 15C, where:
  • m i =Y i <{overscore (Y)},i=1 . . . 16
  • [0061]
    Referring to FIG. 15A, each 44 block of pixels is collected into a 16 element buffer, in which the index ranges from 0 to 15. In the first step, the first and second moments are determined. Squares are preferably determined by table lookup using an 8-bit table of squares rather than by multiplication. In the second step, the mean and standard deviation are determined, using a square12 function to determine the square of a 12-bit number based upon the same 8-bit table of squares above. The root function finds roots by binary search of the same 8-bit table of squares. In FIG. 15A, dY, dCr and dCb are the standard deviations for each component, and mY (mean luminance), mCr, and mCb are the arithmetic means. In the third step, illustrated in FIG. 15B, the selector map is determined from the mean luminance value mY for the selector. The one bits in the map mark those pixels that are “darker” than the mean. The signed differences are accumulated from the mean in each chrominance (Cr/Cb) channel. If the Cr channel decreases when the luminance increases, dCr is inverted. If the Cb channel decreases when the luminance increases, dCb is inverted. In the fourth step, illustrated in FIG. 15C, values are normalized. If the luminance of all pixels is equal to or slightly greater than the mean, all standard deviation values are zeroed. If all of the pixels are nearly equal, the standard deviations will all be zero, in which case the map is also zeroed. To reduce the number of possible maps from 65,536 to 32,768, if the MSB (most significant bit) map is set, the map is inverted and the dY, dCr, and dCb values are negated. Typically 8 bits are used for each of the mY, mCr, mCb, dY, dCr, and dCb values, and 16 bits are used for the map.
  • [0062]
    The second half of the compression process involves taking the fixed bit (8 and 16 bit) length blocks encoded by the previous multilevel encoding step, and compressing them using minimum redundancy, variable-length bit codes.
  • [0063]
    The basic process for compressing a single encoded block comprises three steps: classification, quantization, and codebook compression. Before beginning the compression steps, however, the parameters used for the particular block must be established. Several parameters are used to control the compression process. The parameters specify tolerances for how blocks are classified, how many bits of which component in the encoded block will be preserved, and how precisely the selection map is preserved. Different sets of parameters may be used for “key” (or “delta”) frames. For adaptive compression, each region of the image will use a different parameter set.
  • [0064]
    The basic codebook compression process consists of three steps: First, blocks are classified into four categories—null, uniform, uniform chroma, and pattern. Second, the number of bits for the Y and Cr/Cb components may be reduced, differently for each classification. Third, for uniform chroma and pattern classified blocks the texture map is tested against three groups of simpler, more common “pattern maps.” Where the pattern map is sufficiently similar to the texture map from the encoder, it is used. Otherwise the entire 16-bit texture map is kept, as is described further below.
  • [0065]
    For multilevel compression, blocks can be processed (and therefore stored or transmitted) in either block order or level order. In block order, for each top level block, a top-level block is processed followed by the lower level blocks within the top level block. This method allows adaptive compression or selective processing of top-level blocks. In level order processing, all of the blocks of the top level are processed first, then each intermediate level, followed by the lowest level processing. This method allows for progressive decoding of still images where a very low resolution image can be displayed when the top level is decoded, and progressively higher resolution images can be displayed as each intermediate level is decoded, and finally the fall resolution image can be displayed after decoding the level one data.
  • [0066]
    Adaptive compression is the process by which one section of the image is given priority over other areas. The emphasized section may be specified by the user or automatically selected according to certain predetermined criteria. The priority may either be more bits of quantization or tighter thresholds or both. In some cases, lower levels may not be processed for lesser priority blocks. Adaptive compression can thus also be implemented for storing photographic images.
  • [0067]
    Four codebooks are used in the basic compression process, one each for block classification, luminance difference, chrominance difference, and group three pattern maps, as is described further below. Different applications will have different distributions of values to be compressed.
  • [0068]
    The system of statistical encoding known as Huffman coding is used for constructing variable bit length codebooks based upon the frequency of occurrence of each symbol. For the ultimate use of this technique, a new set of codebooks would need to be constructed for each image and transmitted to the decoder. However, this process is usually not practical. The method of the present invention preferably includes several codebooks optimized for a variety of applications. Typically a single set of codebooks is used for an image, but if necessary, each set of parameters can specify different codebooks.
  • [0069]
    Once the block data and parameters have been collected, the block is classified as null, uniform, uniform chroma, or pattern. Null blocks exhibit little or no change from the higher level or previous frame. Run lengths of one to eight null blocks are collected, and no other information is preserved. The term “run length” is used here to indicate the number of conservative occurrences of null blocks before another classification occurs. Uniform blocks have a relatively low standard deviation, being less than a predetermined threshold, and are therefore relatively uniform in their change in color from the higher level or previous frame. The mean values for all three components are preserved.
  • [0070]
    Uniform chroma blocks have a significant luminance component to the standard deviation, but little chrominance deviation. The mean luminance and chrominance, standard deviation luminance, and a suitable selection map are preserved. Pattern blocks have significant data in both luminance and chrominance standard deviations. All components of the block are preserved. An additional classification, called an escape code, is also used to navigate the compressed bitstream.
  • [0071]
    After the block is classified as null, uniform, uniform chroma, or pattern, the number of bits to be preserved for each component of the block is set as follows:
    {overscore (Y)} {overscore (C)}r {overscore (Cb)} σY σCr σCb MAP
    Null 0 0 0 0 0 0 0
    Uniform bYU bCU bCU 0 0 0 0
    Uniform Chroma bYU bCU bCU bYP 0 0 Yes
    Pattern bYU bCU bCU bYP bCP bCP Yes
  • [0072]
    For uniform chroma and pattern blocks, the selection map is preserved along with the color data. Three groups of common selection maps are identified by the compression method of the invention. The first two groups are fixed while the application developer can select from several codebooks for the third group. If a suitable match cannot be found in the three groups, the entire texture map is preserved.
  • [0073]
    The following notation is used when identifying selection maps:
    0 b14 b13 b12 For 0 1 1 1 = 7528H
    b11 b10 b9 b8 example 0 1 0 1 in hexidecimal
    b7 b6 b5 b4 0 0 1 0 notation.
    b3 b2 b1 b0 1 0 0 0
  • [0074]
    Since the selection map is normalized in the encoding step so that the MSB is zero, each map actually represents two.
    Group Members Implied Maps Encoding
    1 OOFFH 3333H FFOOH CCCCH 3 bits
    2 OFFFH 7777H FOOOH 8888H 4 bits
    1111H OOOFH EEEEH FFFOH
    3 By Codebook typically
    5 to 9 bits
    4 Actual Texture Map 17 bits
  • [0075]
    Since the decoded colors from a block depend upon the number of bits in the map, if a map that is substituted has a different number of bits, the standard deviation components of the block are adjusted. For each individual block, the bitstream is written as is illustrated in FIG. 16.
  • [0076]
    The classification codebook contains twelve entries, eight run lengths of null blocks and one each for uniform, uniform chromas, and pattern blocks, plus an entry for preceding escape codes. Escape codes are dependent upon the implementation and can be used to signal the end of an image, end of a block run, skipping to a different block, and the like.
  • [0077]
    The luminance and chrominance codebooks contain the most often observed delta values—the luminance typically including +25 to −25 and chrominance from −6 to +6. For values that need to be coded and are not found in the selected codebook, an “other” entry at +128 is used, followed by the value, using the number of bits to which the value was quantized, as illustrated in FIG. 19.
  • [0078]
    Typical codebooks are shown in the following tables.
    Sample Luminance Codebook
    Value Bits Pattern Value Bits Pattern
    −4 3 000 −10 6 000
    −8 4 0101 9 6 0101
    −7 4 0100 10 6 0100
    −6 4 1010 11 6 1010
    −5 4 0110 12 6 0110
    −3 4 1110 13 6 1110
    3 4 1001 −16 7 1001
    4 4 0010 . . . . . . . . .
    −9 5 10111 −22 10 1111000111
    5 5 11010 −21 10 1111111101
    6 5 10110 −20 10 1111000110
    7 5 01111 −18 10 1100001001
    8 5 10000 −19 11 11111110101
    −11 6 001100 −17 12 111111101001
  • [0079]
    Sample Chrominance Codebook:
    Value Bits Pattern Value Bits Pattern
    −6 9 000000000 0 1 1
    −5 8 00000001 1 3 001
    −4 7 0000011 2 6 000011
    −3 6 000010 3 7 0000001
    −2 4 0001 4 8 00000101
    −l 2 01 5 8 00000100
    6 10 0000000011
  • [0080]
    It should be readily apparent that the method and system of the invention is also applicable to grayscale images, and other monochromatic images and chromatic image systems with pixels having scalar values. It will be apparent from the foregoing that while particular forms of the invention have been illustrated and described, various modifications can be made without departing from the spirit and scope of the invention. Accordingly, it is not intended that the invention be limited, except as by the appended claims.
Patent Citations
Cited PatentFiling datePublication dateApplicantTitle
US5467413 *May 20, 1993Nov 14, 1995Radius Inc.Method and apparatus for vector quantization for real-time playback on low cost personal computers
US6026217 *Jun 21, 1996Feb 15, 2000Digital Equipment CorporationMethod and apparatus for eliminating the transpose buffer during a decomposed forward or inverse 2-dimensional discrete cosine transform through operand decomposition storage and retrieval
US6356588 *Apr 17, 1998Mar 12, 2002Ayao WadaMethod for digital compression of color images
US6441865 *Sep 5, 1997Aug 27, 2002Eastman Kodak CompanyPhotographic system using chroma-key processing
US6597860 *Aug 11, 1998Jul 22, 2003Samsung ElectronicsDigital camcorder apparatus with MPEG-2 compatible video compression
Referenced by
Citing PatentFiling datePublication dateApplicantTitle
US6958772 *Jan 18, 2000Oct 25, 2005Canon Kabushiki KaishaImage sensing apparatus and image processing method therefor
US7148820 *Apr 14, 2003Dec 12, 2006Fujitsu LimitedVariable-length encoded data decoding apparatus and decoding method
US7433085Apr 15, 2003Oct 7, 2008Lexmark International, Inc.Intelligent hardware for detecting color value of an image
US7477417 *Aug 25, 2000Jan 13, 2009Dai Nippon Printing Co., Ltd.Image processing system
US7542076Aug 8, 2005Jun 2, 2009Canon Kabushiki KaishaImage sensing apparatus having a color interpolation unit and image processing method therefor
US7929026Apr 23, 2009Apr 19, 2011Canon Kabushiki KaishaImage sensing apparatus and image processing method thereof using color conversion and pseudo color removing
US8059712Jun 2, 2006Nov 15, 2011Adobe Systems IncorporatedIndependently adjusting the quality levels for luminance and chrominance channels in a digital image
US8243340 *Aug 31, 2006Aug 14, 2012Microsoft CorporationPre-processing of image data for enhanced compression
US8537891Oct 19, 2011Sep 17, 2013Adobe Systems IncorporatedIndependently adjusting the quality levels for luminance and chrominance channels in a digital image
US8634668Feb 25, 2011Jan 21, 2014Sony CorporationMethod of compression of digital images using a fixed number of bits per block
US8773443Dec 30, 2009Jul 8, 2014Nvidia CorporationCompression for co-processing techniques on heterogeneous graphics processing units
US8781236 *Nov 20, 2009Jul 15, 2014British Telecommunications Public Limited CompanyProcessing graphical data representing a sequence of images for compression
US8837006 *Jul 12, 2012Sep 16, 2014Microsoft CorporationPre-processing of image data for enhanced compression
US9147264 *Feb 23, 2012Sep 29, 2015Nvidia CorporationMethod and system for quantizing and squeezing base values of associated tiles in an image
US9171350Oct 28, 2010Oct 27, 2015Nvidia CorporationAdaptive resolution DGPU rendering to provide constant framerate with free IGPU scale up
US9384410Dec 27, 2012Jul 5, 2016Nvidia CorporationMethod and system for image compression while encoding at least one extra bit
US9819969Nov 26, 2013Nov 14, 2017Nvidia CorporationGeneralization of methods and systems for image compression while encoding at least one extra bit
US20030194012 *Apr 14, 2003Oct 16, 2003Fujitsu LimitedVariable-length encoded data decoding apparatus and decoding method
US20040207882 *Apr 15, 2003Oct 21, 2004Mohamed Nooman AhmedIntelligent hardware for detecting color value of an image
US20050285968 *Aug 8, 2005Dec 29, 2005Masami SugimoriImage sensing apparatus and image processing method therefor
US20070016081 *Jul 11, 2006Jan 18, 2007Globalmedia Group, LlcChroma-photon staining
US20070165725 *Jun 2, 2006Jul 19, 2007Balaji KrishnamurthyIndependently adjusting the quality levels for luminance and chrominance channels in a digital image
US20070195369 *Aug 31, 2006Aug 23, 2007Microsoft Corporationpre-processing of image data for enhanced compression
US20090207275 *Apr 23, 2009Aug 20, 2009Canon Kabushiki KaishaImage sensing apparatus and image processing method therefor
US20110063302 *Dec 30, 2009Mar 17, 2011Nvidia CorporationCompression for co-processing techniques on heterogeneous graphics processing units
US20110262048 *Nov 20, 2009Oct 27, 2011Barnsley Jeremy DGraphical data processing
US20120213435 *Feb 23, 2012Aug 23, 2012Nvidia CorporationMethod and system for quantizing and squeezing base values of associated tiles in an image
CN103761753A *Dec 31, 2013Apr 30, 2014浙江大学Decompression method based on texture image similarity
EP1967010A1 *Jan 25, 2007Sep 10, 2008Microsoft CorporationBayesian demosaicing using a two-color image
EP1967010A4 *Jan 25, 2007Jan 18, 2012Microsoft CorpBayesian demosaicing using a two-color image
Classifications
U.S. Classification375/240.03, 348/391.1, 375/240.24, 375/240.23, 375/240.01, 348/396.1
International ClassificationG06T9/00, H04N1/46, H04N1/41, H04N7/26, G06T1/00, H04N11/04, H04N1/60, H04N1/409, H04N9/75, H04N1/64
Cooperative ClassificationH04N19/91, H04N19/13, H04N19/60, G06T9/005, H04N1/64
European ClassificationH04N7/30E2, H04N1/64, G06T9/00S