CA2259358A1 - Method and apparatus for decoding spatiochromatically multiplexed color images using predetermined coefficients - Google Patents

Abstract

Color image decoding is achieved by a simplified method of decoding a spatially and chromatically multiplexing image plane (32), such as a plane consisting of RGB (Red-Green-Blue) pixels (22) by performing a summation of pixels of all three colors in a neighborhood of a missing pixel. The decoding process has applications in decoding images made by a data processor, made by an imaging device with a mosaic color filter, or made by a multi-sensor CCD imaging device with a sensor offset (800). Various techniques of entropy reduction, smoothing and speckle reduction may be incorporated into the coefficient pattern. The coefficient pattern may be generated automatically using a process of correlated decoding or may be developed by hand using a process of trial and error.

Description

CA 022~93~8 1998-12-24 W O 98/00980 PCTrUS97/12548 s METHOD ANI) APPARATUS FOR DECODING SPATIOCHROMATICALLY
- MULTIPLEXED COLOR IMAGES USING PREDETERMINED COEFFICIENTS

~C~PnUND OF THE l~v~h.lON
This invention relates to ~eco~ patiochromatically multiplexed digit~zed color ~mages with applLcations to - _~s~ion, transmission, reception, storage, and ~ p.~ing image qual~ty.
Related background material and di~closure are found in coinvented and coa~signed U.S. Patent No. 5,398,066, (the '066 patent) and coinvented and coassignod U.S. application serial number 08/401,825, filed Harch 10, 1995, both incG~yGLated herein by reference for all purpo~es.
Other background information and de~cription~ of the prior art may be found in the references cited and submitted along with those applications.
A~ the above referenced document~ thoroughly de~cribe, in digital image processing, a representation of an image is typically stored and transmitted as an array of numerical value~. The image ie dLvided up into a grid. Each small ~quare in the grid is referred to as a pixel.
The intensity of the image at each pixel i~ translated into a numerical value which iB stored in an array. The array of numerical values representing an image is referred to as an Lmage plane.
Black and white ~gray scale) images are commonly represented as a two-dimensional array where the location of a pixel value in the array corresponds to the location of the pixel in the image. Each location in the array for gray scale images can commonly store a number, for example, an integer value of between 0 and 255 (an 8-bit binary number). This means that there can be 1 of 256 different gray levels displayed at each pixel in the image.
Color images are commonly represented by three two-dimensional arrays. Each array (or plane) lepr~acnts one of the primary colorL3, e.g., red, green, or blue. The planes overlap so that each pixel in the displayed image display~ a composite of a red, green, and blue value at that pixel. In a common 24-bit color system, each pixel in each of the three planeq can store a value of between O and 255. This means that there can be 1 of 2563 or 16 million different colors displayed at each pixel. Typical digital color images can range in size from 107 bit~/image (a TV frame) to 10'~ bits/image (a satellite image) thus posing problems for efficient storage and transmission.
In practice the number of bits required to represent the information in realistic digital images may be greatly reduced without significant 1068 in perceived quality by taking advantage of the fact that in ordinary images the pixel value~ tend to be strongly redundant in three domains: spectral (becau6e pixel values from different spectral bands-e.g., RG~-are generally highly correlated); spatial (because ..... . . .... ... .. ~ .. . . . . .

CA 022~93~8 1998-12-24 W O 9~ 9~ PCTrUS97tl2548 neighboring pixel~ al~o tend to be highly correlated); and, for dynamic image~, temporal (because con~ecutive frames tend to be very similar).
Image compression technique~ can reduce the number of bits required to represent images by removing these redundancie~.
The above cited references discus~ one type of prior art digital color image ~ystem having a ~ingle array CCD-type cameras with a mosaic color filter covering the CCD array. TheQe cameras, by their inherent nature, produce a ,cp.eeentation of an image that contains just one color r _nent at every pixel. The arr~n~r --t of the c~ ,_r~nts ie dete ined by the mosaic pattern in the filter. Digital images produced by ~uch ~ystems are referred to as being ~patiochromatic~lly multiplexed.
The above cited reference~ also di~cu~s two other means of producing a spatiochromatically multiplexed digital image plane, (hereinafter referred to as an ~ plane) either by decoding a 3-plane image as described in the '066 patent or by u~ing a multiple array CCD-type cameras with array offset and interpreting the multiple array CCD image as a single spatiochromatically multiplexed plane and then decoding that plane to produce a resolution enhanced image as described in the '825 application.
However the M plane i8 produced, and whatever the pattern of spectral pixels in the M plane, the M plane must generally be decoded before viewing to recreate full multi-spectral image planes. Prior art methods for decoding ~patiochromatically multiplexed digital images have commonly required that the image first be decoded as a YIQ JPEG image before being reconstructed a8 a full three-plane RGB image, as discu66ed in previously cited references.
The inventors of the present invention described in the '066 patent a method and ~ystem for decoding ~patiochromatically images directly, without conversion to another color representation such as YIQ.
This decoding has proven and been de~cribed as useful for any spatiochromatically multiplexed plane, regardless of how produced, including multiplexed planes using just two spectral components.
While this has been shown to be a simplification and improvement over systems that require YIQ conversion, even in this decoding, a multi-pass mathematically process i9 required to decode high-quality images. This process can be complex to implement and can consume a large amount of computer resources.
What is need is a method and system capable of more quic~ly and efficiently decoding a ~patiochromatically multiplexed image plane to derive a full multi-spectral image. Preferably, the method will be generalizable to advantageously decode a number of different types of spatiochromatically multiplexed planes while requiring a minimum of calculations and processing.

CA 022~93~8 1998-12-24 W O98/OOg80 PCTrUS97/12548 SUMMARY OF THE l~v~.ION
Accordinq to the invention, a ~patially and chromatically multiplexed multi-~pectral image compri~ing at least two digitized multi-apectral plane~ consisting, for example, of RGB (Red-Green-81ue), is decoded into a multi~pectral image u~ing a ~implified correlated decoding method. The simplified correlated decodin~ method con~ists of deteL ining a miasing pixel value in one ~pectral plane by computing a wei~htod sum of pixel values in multlple spectral planes in the neighborhood of the missing pixel. The weighted sum iu computed using a specific pattern of value- in each spectral plane, ~aid specified pattern being po~sLbly different for different pixel position~ from the same spectral plane in the spatiochromatically multiplexed plane.
The invention may be employed in conjunction with any number of types of color imaging devices. A color imaging device may be a multi-receptor device with an off-~et among receptors in which case the invention may effectively increases the resolution of the captured image by decompressing the data from the imaging device. A color imaging device may also be an imaging device that contains only two sensors, ~ay one for R and one for G, for u~e in special application~ such as proceseing bank documents where material written or stamped in red (or another color) must be distingui~hable from other writing.
The present invention will be described with reference to the handling of reali~tic color images encoded in the RGB color primaries.
However, it will be apparent to one of ordinary skill in the art that there are alternative multi-spectral uses for the invention. One alternative would be to use the invention in a color system that employs primaries other than RGB for color repre~entations, such as systems that use cyan, magenta, and yellow. Another alternative would be to use the invention in systems that process different types of multi-spectral data, such as images from a satellite, images from infra-red detectors or images from x-ray detectors.
The invention will be better understood upon reference to the following detailed description in connection with the accompanying drawing~.
BRIEF DESCRIPTION OF THE DRAWINGS
Fig. 1 i~ a block diagram of an example of a data compre6sion system employing basic multiplexing and demultiplexing processes in which decoding according to the invention may be employed.
Fig. 2 is a block diaqram of an alternative embodiment data compre~sion sy~tem employinq a multiplexing input device.
Figs. 3A-B are diagrams of different examples of spatiochromatically multiplexed planes of digital data on which the invention may be advantageou~ly employed according to various embodiments of the invention, with an indication of a minimum cell of one particular CA 022~93~8 1998-12-24 periodic ~patiochromatically multiplexed plane of digital data u~ed as an example in illu~trating the invention.
Fig. 4 i~ a flowchart of the generalized decod;ng method according to the invention.
Fig. 5 i~ a diagram of a ~i n i cell ~howing grouping of different sample pixel pair~ for which ~imilar deco~ing patterns are used according to the invention.
Figs. 6A-C are diagram~ illu~trating the decoding of missing pixel values at various sampled pixel locat~on~ in the example cell of Fig. 5 according to an . 's'i --t of the invention.
Fig~. 7A-H are diagrams illu~trating examples of coefficient values for the decoding of mi~sing pixel values at various sampled pixel locations according to an ~ '~'i --t of the invention.
Fig. 8 i~ a schematic block diagram of a system for compression of images designed to operate in accordance with one embodiment of the invention.
Fig. 9 is a schematic block diagram of a three sensor CCD
camera with enhanced re~olution in accordance with an additional embodiment of the invention.
Fig. 10 is a block diagram of a computer sy~tem with an ability to read a computer readable medium allowing the system to operate in accord~nce with the invention.
Fig. ll shows a sy~tem block diagram of computer system lO
used to execute a software embodiment of the present invention.
DESCR~PTION OF THE PREFERRED EMBODIMENT
Definitions and TerminoloqY Related to Diqital Imaqe Processinq and Compression In the specification, the uppercase letters R, G, B, and M are used to indicate two-dimensional arrays of values representing an image or a separable component of an image. The two-dimensional arrays of values are also referred to as "planes.~ For example, the R plane refers to a two-dimensional array of values that represent the red component at every location tpixel) of an image. The letter groups RGB and YIQ indicate three-dimensional arrays of values representing an image as a composite of overlaid separable image component planes. M is used to represent a ~patiochromatically multiplexed plane.
Uppercase letters used with an asterisk (*) tsuch a~ RGB*, M*, R*, G*, or B*) are used to indicate an array of valuee representing an image or a separable component of an image after some type of processing has been done to the image values.
The lowercase letters r, g, and b are used to-refer to an individual pixel in an image plane. These letters are used with a subscript 8 (r" g" or b,) to indicate a pixel value that is directly sampled from the original image and i6 not changed. These letters are used with a ~ub6cript c (rc, gc, or bc) to indicate a pixel value that i6 CA 022~93~8 1998-12-24 W O 98/00980 PCTrUS97112548 computed by a ~_ ~r~-t of the image d~c _es~ion system of the invention.
Angle braekets ~> are u~ed to indicate a local average value computed from the pixels in a local submatrix of an image plane.
The term~ decode ke~nel or decode m~trix are u~ed to denote a pattern of samples that ~s used to determine a miesing pixel value~ and a-~ociated coefficient~. Decode ~ernel- are moved around an M plane and rotated with the ce~te~ of the kernel placed over the pixel for which minaing values are being decoded. There are a number of different decode kernels for pixel~ in an M plane, depen~in~ on where the pixels are located within the ~ ni cell.

Svstem Overview Fig. 1 i~ a block diagram of a specific = ~ of an image compre~sion and decompres~ion ~ygtem of one type in which the present invention may be employed. This ~y~tem ie presented for illustration purposes only, and it should be understood that the techniques of the invention may be u~ed with different types of ~patiochromatically multiplexed images.
A~ shown in the figure, a full color scene 10 is presented to image capturing means 20. Capture means 20 captures a multi-spectral image with data from a plurality of discrete spectral c- ~ncnts (e.g. R, G, & B) captured at every picture element (pixel) location. Capture means 20 may be a digital scanner coupled to a random access memory, or it may be any type of analog or digital camera coupled to a storage means such as a computer memory or a magnetic or optical recording medium. Capture means 20 may also be a means for receiving an image that had previously been stored in a random access memory, on video tape, or a laser disk, etc., or for receiving an image created by a computer. The repre~entation of the image in the image capture means 20 in this example is a three-plane RGB array, indicated by icon 22.
Once the image i8 present in the capture means 20, it is presented to image multiplexer 30, which constructs a new data representation of the image 32, called an M plane (for 'multiplexed~
plane), by, in one embodiment, extracting at every pixel information regarding just one 5pectral component of the image. Multiplexer 30 may operate in detail as described in the '066 patent. For an image made up of three separable spectral component~, multiplexer 30 therefore ~compresses" the data needed to represent the source image to 1/3 the size of the original data ba5ed on a three-plane Bource representation. This compressed d-ata is then received by transmission or storage means 40, which can be any means known for transmitting or storing electronic information. After transmission or storage, decoder 50 decodes the multiplexed plane u6ing the method of the invention to recover RGB~ plane 52, which is a clo6e approximation of the full data set initially captured CA 022~93~8 1998-12-24 WO 98/00980 PCTAUS97112~48 ~n capture mean~ 20. ~hat data net i~ pre~ented to diQplay device 60 which di~plays the data for viewing.
Fig. 2 i~ a block diagram of an alternative ~ystem in which the invention may be employed u~ing a multiplexing input device 70.
MultLplexing input device 70 can be any mean~ known for dlrectly capturing a multiplexed plane from an image, ~uch a~ a CCD ~en~or camera with a mo~aic color filter, or, in an embodiment de~cribed more fully below, a multi ~--rDr CCD camera with an off-set among the sen~ors. In this ~ rt, the image never exi~t~ a~ a full RGB plane before being multiplexed into an M plane. The M-plane output of multiplexing input device 70 is directly input to ntorage or tran~mis~ion means 40.
Demultiplexer 50 receives thi~ output and demultiplexes it to produce RGB~
plane~ 52 which are a clore approximation of what a full RGB
representation of the ~mage would have looked like. In one example, the device u~es a unique mo~aic filter arra-3~ --t and processing to achieve high levels of image compression and superior image epLodlction with a minimum of computational overhead. In another example, the device usea a multi-sensor CCD camera with an off-set among sensors as a multiplexing input device.
The M Plane The present invention is directed to decoding an M plane ~uch as 32. This plane may have any number of different patterns of pixels, with a primary characteristic being that the plane is spatially multiplexed. In general, at each location in the H plane, a pixel value for just one spectral component is present, and which spectral component is present depend~ on the location of the pixel in the M plane. (The invention may also have application~ to hybrid M planes, in which each location includes values from more than one spectral plane.) When the M plane is expanded, the spectral value present at a given location in the M plane is sometimes referred to as the sampl ed value, because that value typically repre~ents a directly 6ampled value from the full spectral image.
Fig. 3A is a diagram of an 8 x 8 pixel region in a spatiochromatically multiplexed image plane such as M plane 32. This diagram represents one example of a type of spatiochromatically multiplexed image on which the present invention may be effectively employed. The letters R, G and B in Fig. 3A each represent the location of one pixel in the M plane and indicate the color component stored in that pixel. For example, R pixel 120 stores a value r" which is the sampled value of the red component from that location in the original source RGB planes. Likewise, G pixel 123 store8 the sampled green component g,from the RGB plane at that location- B pixel 128 stores a value b which, according to the '066 patent can be either the average b,of the blue values in a submatriX of the original image or a sampled value b, from the original image, ag described in the '066 patent. This particular CA 022~93~8 1998-12-24 pattern and ~ome related patterns are referred to as Chromoplex(TM) pattern~ by the a-~ignQe~ of the pr..eent invention.
M plane 32 uses l/3 of the data space to represent the image as did the original RGB planes, becal~e each pixel R, G, or B in the M-plane stores the value of only one of the 3 RGB planes and discards the value of the other two plane~. Thus 3 to l data c~ e~sion iB achieved from the spatiochromatic multiplexing.
Figs. 3B illustrates other poseible configurations for M plane 32 which may al~o be decoded according to the pre~ent invention. These include the well-known Bayer-pattE~rn, field-~taggered 3G CFA, line-~taggered 3G CFA, interline geometry CFA, modif~ed Bayer, and green checker field seguence. Other pattern~ can al~o be decoded by the invention, such as pattern~ in use by Sony Corporation, the Sanyo pattern of lines of alternating color, or any other pattern for constructing a epatially multiplexed M plane.
In the case where an M-plane is produced directly by a CCD-type device with a mosaic filter, such as a CCD camera or flat bed or other type scanner, Figs. 3A-B would also represent the patterns of the red, green and blue filter~ that would be used in a filter mosaic of a multiplexing input device ~uch a~ 70.
Different factor~ will underlie the form of the sampling patterns as shown in Figs. 3. If the final image is intended to be a real-world image intended for viewing by humans, G may dominate the chosen M-plane pattern, followed by R samples, because there is evidence that the human eye is more sensitive to the G primary than to the R primary, and much less sensitive to the B primary. Patterns designed to optimize for human vision therefore may include more G and R values than B values.
Where the image will be quantized by JPEG or any other compression scheme, the chosen pattern is affected by the requirements of the scheme. For example, in JPEG, the fact that one strict aspect of the standardized JPEG scheme is the basic 8 x 8 size of pixel blocks, which defines the range of the ba8i8 functions for the cosine transform. An even submultiple of the standard is therefore desired so that any possible phase-induced effects of multiplexing will not be present with re~pect to the JPEG basis functions required to transform the image into the spatial-frequency domain.
M-plane images used to process two color bank-documents, or for satellite or X-ray image8 will have different optimal M-plane patterns depending on the application.
The Minimum Cell One characteristic that various types of M-plane patterns will have in common is that they are periodic, me--ning there is some basic minimum pattern that is repeated as often as needed to make up the full M-plane. This minimum pattern is referred to as a minimum cel~, and one example of a minimum cell i8 shown as 122 in Fig. 3A and Fig. 5.

CA 022~93~8 1998-12-24 W O 98100980 PCT~US97/12548 According to on- Fdi --t, the praE~t invention simplifiec the proces~
of correlated decoding by analyzing th$g ~ini cell to determine appropriate coefficients to u~e in decoding mi~sing pixel values, and then uses tho~e eame coefficient~ on co~ff~onding pixel locations throughout the M-plane.

Imace Decodino In general, image decoding con~i~t~ of a ~coder ~uch as 50 eYa i ni~g each pixel po~ition in the M plane in turn. As previously described and a~ ~hown ~n Fig-. 3 and 4, the ~ plane contaLns a value repre~enting just one ~pectral plane at that location. Decoding the M
plane consi~ts of dete~ ~ning an approx~mation for the miYsing spectral ~ ~ar,rnts at each location. AccordLng to the previously cLted patents, this i~ accomplished by a proces~ of correlated decoding, where the difference between the value pre~ent at a given location and surrounding values in its epectral plane is correlated to the difference between a calculated value in a different spectral plane and valueR in the other spectral plane.
The previously referenced patents described several alternative methods for decoding an M plane. These methods incorporated a num~er of different possible processes to enhance the quality of the overall image, including reducing color speckles, blurriness, and pattern noi~e. In the earlier cited patents, after the initial approximation of a missing pixel, high quality decoding required a number of passes through the decoded planes, with each pixel and its neighborhood visited numerous times before obtaining the final decoded pixel values.
The previously di~clo~ed decoding had several discrete steps that had to be done in order, each one over the entire image, because each step was dependent on the steps that preceded it. This is not an optimal type of algorithm because each step's results must be saved in a buffer before moving on to the next step. This is expensive in terms of time and/or buffer memory. According to an embodiment of the present invention, these multiple passes through the M plane are eliminated, and a decoder such as 50 may accompli~h effective decoding in just one pass using an expanded coefficient matrix to compute a weighted sum for each missing pixel value.
Fig. 6 is a flow chart of a general procedure of M plane decoding function of an embodiment of the invention. At a current location at which mi~sing pixel values will be decoded, a demultiplexer first determines which pixel values are missing at that location (S4).
According to a specific embodiment, the demultiplexer then determines which position in the minimum cell is being decoded and selects an appropriate sampling pattern for that location (S6) to determine missing values. The demultiplexer then retrieved a matrix of signed weighting factors for that sampling pattern (S10). The demultiplexer then determine4 a value for the misRing pixel by performing a summation of the CA 022~93~8 l998-l2-24 W O 98/00980 PCT~US97/12548 weight-d ~ampled values, taking the ~ampled value~ from variou~ spectral planes in accordance with the retrieved matrix, applying the weighting coefficient~, and performing a sum to obtain the mi~eing pixel value (S12). She computed value i~ then ~tored in the ap~,op~iate location in the recon~tructed image plane ~S14). Deco~;n7 of one miseing pixel value at one location in the M plane now complete, the method loope to the next mi~sing pixel value (S24).

SDec~fic ExamDle~ of Decodincl SamD1e Pattern~
According to the invention, the decoding matrices for a given p~xel location w~ll be based on the ~pecific arrAn~ -rt of sampled values in the l~n; cell of the M plane around that location. In eome configuratione of ;n; cell~, there will be ~ome pixel location~ within the cell that have the same sampling pattern around them and therefore the ~ame decoding matrix. Other pixel locatione may have a decoding matrix that is ea6ily derivable from that for another locations, such as by a rotation of the matrix.
As an example, Fig. S ~hows inil cell 122 from a Chromaplex(TM) pattern, with eight pair~ of pixel locations shown at the bottom, with the two pixel locations in any pair having the same decoding - matrix. Ae shown in Fig. S, there are four paire of equivalent G
po~ition~, three pair~ of equivalent R position~, and 1 pair of equivalent B positions. For each location, two decoding patterns are needed to decode the two missing pixel values at those locations.
Figq. 6A-6C ehow one specific example of a eet of decoding patterns for the RGB M plane with the minimum cell 122 6hown in Fig. S.
In Fig 6A, the two patterns shown 308 and 309 decode a missing R and a miseing B pixel value at a location where there is a gampled G
pixel value (indicated with the letter G in a circle. This po~ition is also designated the center of the decode kernel or matrix). The letters R, B, and G in the decode matricee indicated sampled values in the M plane that are ueed to compute the mi~sing values according to a summation technique such as described below. The blank boxes indicated locations in the M plane that are not u~ed to decode the missing pixel value~ at the indicated current location.
The two kernels 308 and 309 are the sampled patterns for the missing values at the two G locatione shown in 301. When either of those locations is the current location in the decode routine, the kernel 308 is placed with the kernel center over that location and samples are taken from the M plane as indicated by the kernele. Theee 6amples then have applied to them a weighting coefficient aB described below and a 6ummation is formed to determine the miesing pixel value.
Pattern 308 and 309 are rotated in order to determine sampling patterne for other G locatione in minimum cell 122. Patterns 308 and 039 are rotated 90 degrees clockwi8e for the two G locations indicated by 302, CA 022~93~8 1998-12-24 W O 98/00980 PCTrUS97/12548 180 degr-es clockw~-e for the two G locations indicated by 303, and 270 degi~s clockwi-e for the two G locations indicated by 304.
Figs. 6B and 6C aimilarly indicate decodinq patterns for missing values at locations holding a ~ampled R and B value respectively.
Note that the two ~ values ahown in 321 are identical ln terms of their surroun~ing sampled value patt-rn, and therefore there i~ no need to rotate the pattern for different B locations.

SDecific ExamDles of Decodina Coeffic$ents The patterns ~hown in Flgs 6A-6C de~cribe which sampled values in a neighborhood of a missing pixel value will be u~ed in deteL ining that mi-sing value, but they do not ~pecify what will be done with those ~amples to get the missing value.
One example of a method of decoding the missing pixel values i~ shown in Figs 7A-7H. Pigs 7A-7H show epecific examples of coefficient values that may be used with the ~codi ng patterns ~hown in Figs. 6A-6C to achieve a missing pixel value by performing a simple weighted summation in a neighborhood surrounding the missing pixel. These numerical coefficient values may be selected or tuned by human estimate and trial and error, or they may be developed by an automatic procedure de~cribed below.
As an example, Fig. 7A ~hows coefficients 408 that may be used to determine a missing red pixel value at po~itions with a G sampled value using pattern 308. As can be ~een, the greatest single contribution comes from the value of the green sampled value at that location, which has a 2S positive coefficient of 0.8, and surrounding R, G, and B values are assigned positive and negative coefficients to correlate the missing R
value with what is happening in the other spectral planes in it~
neighborhood. The r~ -ining example coefficient patterns similarly can be used with the patterns ehown in figureQ 6A-6C, with coefficient 409 corresponding to pattern 309, 418 to 318 and ~o on. The coefficient values are rotated along with the patterns as shown in Figures 6A - 6c.
A consideration of the patterns and coefficients just discussed with the correlated decoding with speckle correction and other techniques as disclosed in the '066 patent will show that the computation using the present method can be accomplished much more quickly and require less computations per decoded pixel than the full '066 method. Also the decoding according to the present invention may be done in just one pass through a large image, rather than the multiple passes required by the '066 patent.
Generation of Decodina Coefficients and Patterns The decoding patterns and coefficients shown in Fig~. 7A-H may be determined according to the invention by a variety of methods. As one example, the patterns and coefficients may be selected by trial and error, with an experienced spectral engineer u6ing knowledge about the particular M plane pattern to be decoded and the desired quality of the resultant CA 022~93~8 1998-12-24 W 098/00980 PCTrUS97/12548 decc~ image to ~el-ct in~tLal approp~iate coefficients, and then iteratively ~ecoding a number of ~ample images and adjusting coefficients as needed. For unique patterns and for ~pecific applications, such as two color imaging for processing of bank document~, this ~by hand~ approach may be particularly appropriate.

General PrinciDle~ for Sv~tematic D~c~'~nq According to another 5 _di --t of the invention, ~eCo'ing patterns are dete~ d by following a multi-pass correlated deco~ing process ~imilar to that de~crlb-d in the '066 patent. How-ver, in the current invention, this multi-pa~s proces~ i- done only once, on just one a ini cell of a particular M plane pattern, and then the coefficients derived can be applied using a ~imple summation to any image that is enco~ed with that M plane pattern.
To determine decoding coefficient~, one more systematic method according to the invention first begins by dete ining the cmallest po~sible 2-dimensional array, centered on the to-be decoded location in the M plane, that contains at least one sample from each spectral plane.
In the pattern shown in Fig. 3A, thi~ would be a 3 X 3 array.
Next, rough coefficient values for the contributions of these surrounding sample values (between -l and l) are selected for these values, using an approximation of the procedure of correlated decoding described in the earlier cited '066 patent. This process is applied with two colors at a time, one color corresponding to the center and the other color corresponding to a circularly-symmetric array of samples around the center, even if some of the ~amples in the surrounding array are not of the chosen color.
If the image has more than two spectral components, it i6 nece~ary to specify several subsets of coefficients in order to adjust the rough value~ of coefficients corresponding to sample~ of the higher order spectral plane~. These subsets of coefficients are calculated by applying the step above using a minimum cell centered on those 6amples and selecting a different pair of colors. Thus, additional sample positions beyond tho~e within the initial minimum cell will become part of the decoding pattern. Some pO8itiOns will have two or more coefficients, one being a rough value and one or more values computed for the subsets. At every pixel in the decoding pattern, all re8ulting coefficients should be linearly combined into a single one to produce the net effect upon the ~ample in that pixel.
Additional and optional procedures to improve the quality of the decoding, like speckle correction, are applied over the sampling patterns, which will be affected again in 90me of the positions that already have a coefficient.

CA 022~93~8 1998-12-24 W O 98l00980 PCTrUS97/12548 SDecific Im~lementation of Pecodin~ Method According to the invention, the coefficient~3 also may be determined more precinely, following a method similar to the one discloQed in the '066 pat-nt, with nome refiq --ts as described below, to determine a decodinq kernel. Ac an example, a proce~ for dete. ining the coefficient pattern 419 i8 deacrib-d below. This example will be uced becauce it results in a nmall ~co~i n~ kernel.
To decode a blue value at a red ~ample according to one ~ 'odi~~-t, fir~t apply the following equation on a 5X5 neighborhood, auch ar: b - R - F~+ B~.
In order to generalize the method, the equation may be written with explicit offoet coordinate~ from the pixel being decoded as follows:
b~o~ -- R~o)--[R~o) + ~.20) + 4R~ 2) + 4R~o,+2) + 4R~+2,o) 1/5 + IB ~ ) + B (+~,+,)+ bo ~,~.+~) + bo ~+l,.l) ]/4~
lS Multiplying through by 20 to make the coefficients integer yields:
20b(o,o) = 20R~o,o~ - 4R~o,o~ ~ 4R~.2~o\ ~ 4R~o~ 2) ~ 4Rto~+2) ~ 4R~+2~o~+ 5Bo~, "
+ 5B(+1,+l) + 5bo~ +1) + 5bO~+I-I)' Gathering together like terms to get the weighted equation:
20b(o,o) = 16R~oo) - 4R(,2,o) - 4R(O,-2~ - 4R~o,+2~ - 4R(+2,o~ + 5B(,~
5B(+, +" + 5bo~,) +" + 5bo~+l 1)' Comparing the coordinates in this equation to the M plane shown in 319, it i8 ~3een that some of the blue values from the equation are actually located on red samplet3 in the M plane. At these pixels, the blue values also must be decoded, as follows:
20b~,o) = 16R~o,o, - 4R(,2,.2, -- 4R~.Z,+2, -- 4R~+2"2, - 4R~+2~+2) + 5B~ 2~o) + 5B~+2~o) +
5B~o,2~ + 5B~o,+2)~
The coordinates in this equation are relative to the pixel being decoded. Because blue values at bo o,+,) and bo~+l~,l) in the original equation are being decoded, add this coordinate shift to the bo equation:
20bo(l,+1) = 16Rm+o - 4R~3,~) - 4R~,3,+3) - 4R,t,",) - 4R~+,,+3) + 5B~3,+,) + 5B~+~,+~
+ 5Bo"o + 5B~+3);
20b~+~,~) = 16R~+,.,)- 4Rm"3) - 4R(~,+~ - 4R~+3,3) - 4R~+3,+,) + 5B~) + 5B~+3,) +
5B~+,,3) + 5B~+~+~.
Now substitute these equations back into the original equation making ~ure to preserve the proper weights and collect like terms to obtain the final convolution kernel:
20b(o,o) = 16R~o,o) ~ 4R~,2,o) ~ 4R(o,.2) - 4R(o,+2) - 4R(+2o~ + 5B~,,) + 5B(+~+~) +
5bo~,l+l) + 5b~+");
20b(o,o) = 16R~oo) - 4R~2,o~ - 4R(o,,2, - 4R(o,+2, - 4R~+2o, + 5B~,,) + 5B~+,+,) +
5[~16R~,+~) - 4R,3,,) - 4R~3,+3) - 4R(+~ - 4R~+~+3~ + 5B(3+,) +
5B(+~+~ + 5BO,~) + 5B(~,+3~/20] + 5[{16R(+~) - 4R(,,3~ - 4R~,,+,) - 4R~3,3)- 4R~+3+~) + 5B~,,,,) + 5B(~3,~) + 5B(+, O) + 5B(+~+~)~/20]
Multiply throu5h by 4 to keep all of the coefficients integer:

CA 022~93~8 1998-12-24 W 098/00980 PCTrUS97/12S48 80b(o,o) = 64R~o,o, ~ 16R,.,,o) ~ 16R~o,.2) - 16R~o~2) - 16R(t2,o) + 20B(,~"o +
20B(~ ) + 16~o - 4R(3O - 4R(3+3) - 4R(+~O - 4R(~+3) +
5B(3~l) + 5B(-~+~) + 5B~o + 5B(~3) + 16R~+~) - 4R~3) - 4R(~O
~ 4R(+33) ~ 4~+3+~) + SB(~,) + SB(~3~) + SB(+~3) + SB("+~) S Now gather up the like term~, and printing terms out according to their row~ in the decode matrix 419.
80b~o) s -4~3~) + SB(a+"~ 4~a+3) - 4~3) ~ 30B(,,) + 12~,+,) + 5B(,+
- 16R,o,.2,+ 64R~o~o) ~ 16R~o+
+ 5B(+~3) + 12~") 1 30B(+,+o - 4~tl+
- 16R~+2,o) _ 4~+33) + 5B(+3O - 4~+3")-Divide all coefficient~ in the equation~ by B0 to get the coefficients shown in 419.
The above i~ ~u~t one example of the computation of the convolution kernels. Actual kernels used in real world applications may involve many more ~teps, which i~ not problematic because the computation need be done only once for a particular pattern and then the coefficients will be used to decode actual image~. The actual decoding of a pixel in the '066 patent al~o involved many more steps for some of the refinement p. ocedures .
In actual practice, some approximations are made to large convolution kernels which resulted from large neighborhood operations such a~ combining the final ~tronq plane deco~ing step with the pattern reduction/resaturation ~tep. These kernels turned out to be large (45X45), but dropped off to small values quickly and therefore were clipped arbitrarily to keep their size small but remain functional. This is why the convolution kernels of the present invention are close, and in fact perceptually indistingui8hable to the human eye, but not numerically equivalent to the full process described in the '066 patent.
To test the accuracy of the calculations, in one embodiment, the sum of the coefficients in a particular kernel fcr sample values in the eame spectral plane as the missing value should be one, and the individual sums of the coefficient values in each of the other spectral planes should be zero.

Specific Circuit Embodiments Fig. 8 is a block diagram of a general purpo6e system designed to practice the demultiplexing and decoding functions of the invention.
Fig. 8 ~hows a general purpoBe system deBi~ned to receive, decompress and di6play images repre6ented a5 a compreBsed data 8et. Interface circuit 918 receives the signals from storage or tran~mission device 40 and stores the compres~ed data set in random acce6s memory (RAM) 920.
Central procesBing unit (CPU) 923 may be either a standard microprocessor which is part of a general purpo~e computer or a specially CA 022~93~8 1998-12-24 W 098/00980 PCTrUS97112548 con~tructed mic~o~oc_~-or. Memory 925 conta$nH in~tructions that direct CPU 923 in proce~sing of a compre~ed data ~et according to the invention.
AB CPU 923 performs these decoding functions, the decoded data is stored in RAM 920. Once the decoding functions are complete, a reeulting RGB
decoded image in RA~ 920 i~ tran-mitted to di~play driver 928 for di-play on output device 930.

ImProvinq the resolution of 8 multi-CCD camera ~n a further a~pect of the invention, the invention may be u~ed to increa~e the re~olution of a multi-~pectral imaging device that uses multiple CCD ~ensor~ by applying the ilp;oved method of the invention to a ~ystem like th-t di~cu~ed in the '825 patent application. In one rpecific example of this ~ s'i --~, as shown in Fig. 9, a digital camera employs two or more separate CCD arrays to capture separate and complete apectral cl ~nent planes with the ~ensors deliberately offset one from another by a fraction of a pixel ~for example, one half a pixel), with a possible filling method to construct an M plane having a higher resolution than any of the individual CCD arrays. ~he M plane could then be decoded according to the methods of the invention.
The Invention as Embodied in a computer readable medium Fig. 10 illustrates an example of a computer system used to execute the method of present invention when implemented in general purpose software according to one embodiment. Fig. 10 shows a computer system 10 which includes a monitor 703, screen 705, cabinet 707, keyboard 709, and mouse 711. Mouse 111 may have one or more buttons such as mouse buttons 113. Cabinet 107 is shown housing a disk drive 715 for reading a CD-ROM or other type disk 117. Cabinet 707 also houses familiar computer c~ tonents (not shown) such as a processor, memory, disk drives, and the like, as well as an adaptor 1 for connection to a network medium 15.
Fig. 11 shows a system block diagram of computer system 10 used to execute the software of the present invention. As in Fig. 10, computer system 10 includes monitor 703 and keyboard 709. Computer system 10 further includes subsystems such as a central processor 722, system memory 724, I/O controller 726, di~play adapter 728, serial port 732, disk 736, network interface 738, adaptor 1 and speaker 740. Computer readable media Euch as memory, hard disks, floppies, C~-~OMs, tapes, flash memory, and the like may be used to store a computer program including computer code that implements the present lnvention. Other computer systems suitable for u~e with the present invention may include additional or fewer subsystems. For example, another computer system could include more than one processor 722 (i.e., a multi-processor SyEtem) or a system may include a cache memory.
Arrows such as 722 represent the syEtem bus architecture of computer Eyste~ 10. However, these arrows are illustrative of any interconnection scheme serving to link the subsystems. For example, W O 98/00980 PCT~US97/12548 cpe~r 740 could be connecLed to the other subsystem~ through a port or have an internal direct connec~ion to central proces~or 722. Computer ~ystem 10 shown in Fig. 11 iB but an example of a computer system suitable for use with the pre~ent invention. Other configurations of ~ubsystems suitable for u~e with the ~,e~- L invention will be readily apparent to one of ordinary ~kill in the art.
The invention has now been explained with reference to cpecific ~ . Other ~ t~ will be apparent to tho~e of ~kill in the art. It i~ therefore not intended that thi~ invention be limited, except as indicated by the appended claims.

. ,

Claims (2)

WHAT IS CLAIMED IS:

1. An image decoder capable of decoding a multi-spectral multiplexed image into multiple spectral planes comprising:
an image storage capable of receiving and storing a spatiochromatically multiplexed image plane comprised of a plurality of pixels;
memory for storing a plurality of decoding patterns of weighted coefficients for decoding said spatiochromatically multiplexed image plane;
a processor for determining missing pixel values in said multiple spectral planes by applying said patterns of coefficients to said multiplexed plane to compute a weighted summation to determine the value of said missing pixels.

2. The decoder according to claim 1 wherein said plurality of patterns comprise different patterns of coefficients for decoding each missing pixel value at different locations in a minimum cell of said multiplexed plane and wherein said coefficients include positive and negative coefficients.