US 20090263017 A1 Abstract A method of color reconstruction includes a first process for a first pixel and a second process for the first pixel. The first process includes extracting a first kernel from a multi-color matrix, generating first variance weights from the first kernel, and generating a first color based on the first variance weights and adjacent pixel values of the first color. The second process includes extracting a second kernel from the multi-color matrix, generating second variance offsets from the second kernel, and generating a second color based on the second variance offsets and an adjacent pixel of the second color.
Claims(12) 1. A method for object-based color reconstruction in a multicolor matrix-based sensor arrangement comprising color sensors that have one first luminance component sensed at a relatively higher spatial frequency and two further chrominance components sensed at relatively lower spatial fancies, for a particular pixel not sensed in said first luminance component estimating its first color value through determining local gradients among various first luminance component values a said method being characterized by executing the following steps:
and in accordance with such local gradients executing such estimating through along relatively stronger edge informations, interpolating with a relatively greater weight factor, in favor over interpolating along relatively weaker edge informations with a relatively lesser weight factor, and for a particular pixel not sensed in a particular further chrominance component value estimating that further chrominance component's value in a direction along with relatively smaller differences evaluated in said first luminance component. 2. A method as claimed in 3. A method as claimed in 4. A method as claimed in Clam 2, when said first luminance component's value is estimated on the basis of a 5×5 pixel kernel centered on said particular point.5. A met as claimed in 6. A method as claimed in 7. A method as claimed in 8. A computer program comprising program instruction for controlling a computer to implement a method according to one of 9. A computer program product as being represented with a tangible read-only computer memory medium or being carried by an electrical signal, and comprising program instructions for controlling a cower to implement a method according to one of 10. An apparatus being arranged for implementing a method as claimed in 11. An apparatus according to 12. An image facility comprising an image forming facility for forming an image on an as claimed in Description The priority benefits of the Oct. 10, 2002 filing dates of U.S. provisional applications Ser. Nos. 60/417,142 and 60/417,152 are hereby claimed. 1. Field of the Invention The present invention relates to a method and apparatus for object-based color reconstruction in a multi-color based sensor arrangement through maintaining coherence within objects and along edges, and a computer program and computer program product for controlling such method, and an image sensing facility comprising such apparatus. In particular, the invention relates to a method for executing object-based color reconstruction in a multi-color matrix-based sensor arrangement that estimates color value through determining local gradients among various luminance component values assessed. 2. Description of Related Art Multi-color matrix-based sensor arrangements that include color sensors that have one first luminance component sensed at a relatively higher spatial frequency and two further chrominance components sensed at relatively lower spatial frequencies are coming in rapidly expanding use. Fields of application include digital cameras, digital cinematography, video cameras, scientific imaging, broadcast cameras, surveillance, security monitoring, and various others. Now generally, each object point translates into a single-color of the corresponding image pixel. Inasmuch as, in the resulting image each pixel lacks two of the three sensor colors, reconstruction of three-color pixels requires an appreciable amount of processing. However, in earlier realizations, the resulting reconstructed array often had artificially colored, noise-enhanced, or blurred edges. In consequence, there exists a growing need to provide for an algorithmic procedure that will provide high-quality estimations for the color values of pixels that were originally sensed in a single color only, whilst requiring to access only a minimal memory facility and minimal power consumption, needing to access only a small number of image lines/columns at a time, and being independent of other DSP-based (i.e., Digital Signal Processing-based) algorithms, such as those for removing false colors or correcting color balance, overall influencing of hue or luminance, or others. As will be discussed more extensively hereinafter, the present embodiment generally utilizes only a small data kernel of optimally 5×5 pixels for Green, or minimally 3×3 pixels for Red and Blue. It will generally minimize perceptional error, it reconstructs frequencies above the Nyquist frequency, it has much lower color aliasing than many other algorithms, it will hardly expand existing defects, and it will only be marginally affected by image noise. United States Patent Application Publication 2002/0063789 A1 to Acharya et al and published on May 30, 2002 discloses a Color Filter Array and Color Interpolation Algorithm, and uses a classification to determine which pixels it will use in the interpolation. The present invention in contradistinction uses a soft decision method that is useful for adaptive adjusting for variations in system noise. Moreover, the reference singles out a very particular matrix design with a very high percentage of 75% Green pixels sensed. The present invention is much wider applicable. Moreover, the inventor has found that correct application of the algorithm will not remove fundamental signal noise. Rather, it will prevent enhancing of the noise. In particular, the inventor has recognized the corresponding behavior of the gradients among the various sensed colors. In consequence, amongst other things, it is an object of the present invention to use such corresponding behavior to attain an improved result against an investment of only a limited amount of processing complexity. By itself, co-pending patent application by the present inventor, titled “Method And Apparatus For Adaptive Pixel Correction Of A Multi-Color Matrix,” identified by Attorney Docket No. 12546 and assigned to the same assignee as the present application relates to the adaptive correcting of defective pixels and is incorporated herein by reference. The adaptive pixel correction of a multi-color matrix is based on soft decisions across the various color planes, and in particular based on taking into account the spatial response pattern of the human visual system. The result of the co-pending application may be taken as a starting point for applying the present invention, but such represents no express restriction. A method for object-based color reconstruction in a multi-color matrix-based sensor arrangement is based on color sensors that have one first color sensed at a relatively higher spatial frequency and two further colors sensed at relatively lower spatial frequencies. In particular, the method executes the following steps. A particular pixel not sensed in said first color has its first color value estimated through determining local gradients ( Now therefore, according to one of its aspects the invention a method includes a first process for a first pixel and a second process for the first pixel. The first process includes extracting a first kernel from a multi-color matrix, generating first variance weights from the first kernel, and generating a first color based on the first variance weights and adjacent pixel values of the first color. The second process includes extracting a second kernel from the multi-color matrix, generating second variance offsets from the second kernel, and generating a second color based on the second variance offsets and an adjacent pixel of the second color. The invention also relates to a computing machine that includes a processor and a memory. The memory stores a first module for controlling the processor to perform a first process for a first pixel. The first process includes extracting a first kernel from a multi-color matrix, generating first variance weights from the first kernel, and generating a first color based on the first variance weights and adjacent pixel values of the first color. The memory further stores a second module for controlling the processor to perform a second process for the first pixel. The second process includes extracting a second kernel from the multi-color matrix, generating second variance offsets from the second kernel, and generating a second color based on the second variance offsets and an adjacent pixel of the second color. The invention also relates to a computer readable medium having stored thereon a plurality of modules for controlling a processor. The plurality of modules includes a first module and a second module. The first module is for controlling the processor to perform a first process for the first pixel. The first process includes extracting a first kernel from a multi-color matrix, generating first variance weights from the first kernel, and generating a first color based on the first variance weights and adjacent pixel values of the first color. The second module is for controlling the processor to perform a second process for the first pixel. The second process includes extracting a second kernel from the multi-color matrix, generating second variance offsets from the second kernel, and generating a second color based on the second variance offsets and an adjacent pixel of the second color. Further advantageous aspects of the invention are as disclosed and claimed herein. The invention will be described in detail in the following description of preferred embodiments with reference to the following figures wherein: Now, the intended end result of the reconstruction embodiment is three complete images of Red, Green, and Blue pixel values, respectively, where the original recording did yield less than three values for each respective pixel, according to the pixel pattern for the respective single color. It is known from advance that at each pixel location, two values must be estimated. In practice, the application most favored is to a Bayer pattern wherein the green pixels form a chessboard pattern, while the non-green fields have alternating red and blue values in the diagonal directions. Therefore, first the Green matrix will be completed. In this respect, Likewise, Then, in a relatively simplistic implementation a local edge direction is determined and a Green value for the missing central point is estimated as follows: if δ A fuller realization as an algebraic expression for the interpolated green pixel is: wherein λ The algebraic expressions offer more benefits than just for presentation. The non-linear combination of the edge indicators can be adjusted with the value of k to either emphasize edge clarity in low-noise situations, or rather avoid the enhancing of the noise while still maintaining minimal color aliasing in high-noise situations. Therefore, k has a two-sided adaptive effect on the algorithm. With small values (between 1 and 2) noise will be adaptively minimized whilst ensuring that edges will have none to minimal color aliasing. On the other hand, with larger values for k in low-noise situations, more emphasis can be put on removing all color aliasing whilst enhancing edge continuity and sharpness. Further, the use of five arrows in After having completed the Green array G For a Blue sample, one approach is to choose among the four diagonally adjacent Red values, whilst using the respective Green differences shown in i.e. add the minimum Green difference to the Red value, and correspondingly: A corresponding version applies to the missing Blue values at pixels that were sampled Red. Now, although the color reconstruction according to the present invention's embodiment will generally be correct, the particular arrangement of the Bayer matrix, in combination with non-ideal optical image low-pass filtering and system noise can cause some false color responses. Heuristically, the following description is given. The sampling of a scene through a Bayer matrix will cause two major sampling errors, as follows: (a) The relative displacement between green, red and blue samples introduces a phase error between the various colors; (b) The sampling frequencies of Red and Blue are relatively lower than that of Green. This is perceptually advantageous, because the human eye's chroma and hue resolution is relatively less than its luminance resolution. On the other hand, this introduces different aliasing responses for Green versus for Red/Blue, in particular, inasmuch as Green will attain a better high-frequency response. Although this may cause false-color spikes and the like, such phenomenon is relatively rare. Such spikes can be considered high-frequency chromatic points, and low-pass filtering would generally remove them. In fact, a chroma image can be estimated through two Green differences In this respect, Now, block If not, the system will undertake to reconstruct an appropriate Green value for the pixel in question, as follows. First, in block If in block Generally, after numerous rounds through the reconstructing columns, the answer in block As an example of the above described embodiment, a method may be regarded as having first and second processes for a first pixel. For example, in the Bayer patterned multi-color matrix, if the first pixel is a Blue pixel, then the first process computes a Green value for the pixel and the second process computes a Red value for the pixel. The first process includes extracting a first kernel from a multi-color matrix. In the example discussed herein, the first kernel is a 5 by 5 sub matrix of the Bayer pattern centered on the exemplary Blue pixel. The first process further includes generating first variance weights (e.g., λ The generating of the first variance weights includes first determining horizontal and vertical gradient value averages (e.g., δ The first variance weights include horizontal and vertical interpolation weights (e.g., λ the vertical interpolation weight is calculated based on where δ The second process includes extracting a second kernel from the multi-color matrix. In the example discussed herein, the second kernel is a 3 by 3 sub matrix of the Bayer pattern centered on the exemplary Blue pixel. The second process further includes generating second variance offsets The second variance offsets are generated from the second kernel. In the example discussed herein, the second kernel is a 3 by 3 sub matrix of the Bayer pattern centered on the exemplary Blue pixel. As discussed with respect to the first process, a Green value has been calculated to become associated with the exemplary Blue pixel, and associated with each of the four Red pixels at the four corners of the second kernel, as discussed herein. Also see More specifically, generating the second variance offsets includes determining diagonal gradients between pixels in the second kernel (e.g., [i, j] to [i−1, j+1], [i, j] to [i+1, j+1], [i, j] to [i+1, j−1] and [i, j] to [i−1, j−1]) and then determining gradient values of the first color (Green in this example) corresponding to the gradients. Then, generating a second color (Red, in this example) includes choosing a minimum value of the gradient values (e.g., Δg However, if Δg or if Δg or if Δg On the other hand, if the center of the first and second kernels were a Red pixel, the second process would determine a Blue value for the center pixel. The method repeats for every pixel (except pixels on the edge of the matrix where a full kernel cannot be extracted) so as to determine all three color planes for each pixel. For a second pixel, the method further includes a third process. The third process includes extracting a third kernel (e.g., a 3 by 3 sub matrix centered on the second pixel) from the multi-color matrix (e.g., the Bayer multi-color matrix described in this example). The third process further includes generating third variance offsets from the third kernel (e.g., Δg The value of the predetermined exponent, k, is determined in step After a full RGB image is recovered in block The processes described above for determining Red and Blue values require that Green values be determined for all pixels in the 3 by 3 kernel used to determine the Red values or the kernel used to determine the Blue values. As can be observed in In the event that the processes are implemented on a programmable computing machine (such as a general purpose processor or DSP), the details of above described method are included in program modules stored on a computer readable media. The computer readable media has stored thereon a plurality of modules for controlling a processor. The plurality of modules include first and second modules. The first module controls the processor to perform a first process for a first pixel. The first process includes extracting a first kernel from a multi-color matrix, generating first variance weights from the first kernel, and generating a first color based on the first variance weights and adjacent pixel values of the first color. The second module controls the processor to perform a second process for the first pixel. The second process includes extracting a second kernel from the multi-color matrix, generating second variance offsets from the second kernel, and generating a second color based on the second variance offsets and an adjacent pixel of the second color. Having described preferred embodiments of a novel method and apparatus for reconstructing pixel color values (which are intended to be illustrative and not limiting), it is noted that modifications and variations can be made by persons skilled in the art in light of the above teachings. It is therefore to be understood that changes may be made in the particular embodiments of the invention disclosed which are within the scope and spirit of the invention as defined by the appended claims. Having thus described the invention with the details and particularity required by the patent laws, what is claimed and desired protected by Letters Patent is set forth in the appended claims. Referenced by
Classifications
Rotate |