US 20060082649 A1
Briefly, in accordance with a preferred embodiment of the present principles, simulation of a block of film grain for addition to a block of an image occurs by first establishing at least one image parameter in accordance with at least one attribute of the block. Thereafter, a block of film grain is established in accordance with the image parameter. Deblocking filtering can be applied to the film grain block.
1. A method for simulating a film grain block for addition to a block of an image, comprising the steps of:
establishing at least one parameter at least in part in accordance with an attribute of the block; and
establishing at least one block of film grain in accordance with the at least one parameter.
2. The method according to
3. The method according to
providing a plurality of film grain patterns for selection; and
selecting from among the plurality of film grain patterns a source pattern;
selecting from the source pattern a block of film grain.
4. The method according to
5. The method according to
6. The method according to
7. The method according to
generating first and second pseudo-random numbers;
generating first and second offsets within the film grain pattern based on the first and second pseudo-random numbers; and
extracting the film grain block at a location in the film grain pattern specified by the first and second offsets.
8. The method according to
9. The method according to
seeding a shift register with a seed value selected in accordance with a color component of the macroblock, and
extracting a set of most significant bits as the first random number; and
extracting a set of least significant bits as the second random number.
10. The method according to
11. The method according to
12. The method according to
13. A method for simulating film grain in a 4:2:0 chroma format in at least one block of an image, comprising the steps of:
establishing at least one film grain parameter in 4:4:4 chroma format in accordance with at least one attribute of the at least one block; and
deriving a film grain block from a database of film grain patterns by randomly selecting the film grain block from the database in accordance with the at least one film grain parameter;
14. The method according to
15. The method according to
16. Apparatus for simulating a film grain block for addition to a block of an image, comprising:
a film grain simulator for establishing at least one parameter at least in part in accordance with an attribute of the block, and for establishing at least one block of film grain in accordance with the parameter
17. The apparatus according to
18. The apparatus according to
a database providing a plurality of film grain patterns for selection;
means for selecting from among the plurality of film grain patterns a source pattern; and
means for selecting from the source pattern a block of film grain.
19. The apparatus according to
20. The apparatus according to claim method 19 further including means for thresholding the average pixel intensity in accordance with upper and lower pixel intensity thresholds contained in supplemental information accompanying the image.
21. The apparatus according to
22. The apparatus according to
means for generating first and second pseudo-random numbers;
means for generating first and second offsets within the film grain pattern based on the first and second pseudo-random numbers; and
means for extracting the film grain block at a location in the film grain pattern specified by the first and second offsets.
23. The apparatus according to
24. The apparatus according to
a shift register seeded with a value selected in accordance with a color component of the macroblock, for yielding a set of most significant bits as the first random number and a set of least significant bits as the second random number.
25. The apparatus according to
26. The apparatus according to
27. The apparatus according to
This application claims priority under 15 U.S.C. 119(e) to U.S. Provisional Patent Application Ser. No. 60/619,655, filed Oct. 18, 2004, the teachings of which are incorporated herein.
This invention relates to a technique for simulating film grain in an image.
Motion picture films comprise silver-halide crystals dispersed in an emulsion, coated in thin layers on a film base. The exposure and development of these crystals form the photographic image consisting of discrete tiny particles of silver. In color negatives, the silver undergoes chemical removal after development and tiny blobs of dye occur on the sites where the silver crystals form. These small specks of dye are commonly called ‘grain’ in color film. Grain appears randomly distributed on the resulting image because of the random formation of silver crystals on the original emulsion. Within a uniformly exposed area, some crystals develop after exposure while others do not.
Grain varies in sizes and shapes. The faster the film, the larger the clumps of silver formed and blobs of dye generated, and the more they tend to group together in random patterns. The grain pattern is typically known as ‘granularity’. The naked eye cannot distinguish individual grains, which vary from 0.0002 mm to about 0.002 mm. Instead, the eye resolves groups of grains, referred to as blobs. A viewer identifies these groups of blobs as film grain. As the image resolution becomes larger, the perception of the film grain becomes higher. Film grain becomes clearly noticeable on cinema and high-definition images, whereas film grain progressively loses importance in SDTV and becomes imperceptible in smaller formats.
Motion picture film typically contains image-dependent noise resulting either from the physical process of exposure and development of the photographic film or from the subsequent editing of the images. The photographic film possesses a characteristic quasi-random pattern, or texture, resulting from physical granularity of the photographic emulsion. Alternatively, a similar pattern can be simulated over computed-generated images in order to blend them with photographic film. In both cases, this image-dependent noise is referred to as grain. Quite often, moderate grain texture presents a desirable feature in motion pictures. In some instances, the film grain provides visual cues that facilitate the correct perception of two-dimensional pictures. Film grain is often varied within a single film to provide various clues as to time reference, point of view, etc. Many other technical and artistic uses exist for controlling grain texture in the motion picture industry. Therefore, preserving the grainy appearance of images throughout image processing and delivery chain has become a requirement in the motion picture industry.
Several commercially available products have the capability of simulating film grain, often for blending a computer-generated object into a natural scene. Cineon® from Eastman Kodak Co, Rochester N.Y., one of the first digital film applications to implement grain simulation, produces very realistic results for many grain types. However, the Cineon® application does not yield good performance for many high-speed films because of the noticeable diagonal stripes the application produces for high grain size settings. Further, the Cineon® application fails to simulate grain with adequate fidelity when images are subject to previous processing, for example, such as when the images are copied or digitally processed.
Another commercial product that simulates film grain is Grain Surgery™ from Visual Infinity Inc., which is used as a plug-in of Adobe® After Effects®. The Grain Surgery™ product appears to generate synthetic grain by filtering a set of random numbers. This approach suffers from disadvantage of a high computational complexity.
None of these past schemes solves the problem of restoring film grain in compressed video. Film grain constitutes a high frequency quasi-random phenomenon that typically cannot undergo compression using conventional spatial and temporal methods that take advantage of redundancies in the video sequences. Attempts to process film-originated images using MPEG-2 or ITU-T Rec. H.264|ISO/IEC 14496-10 compression techniques usually either result in an unacceptably low degree of compression or complete loss of the grain texture.
Thus, there exists a need for a technique simulating film grain, especially a technique that affords relatively low complexity.
Briefly, in accordance with a preferred embodiment of the present principles, simulation of a block of film grain for addition to a macroblock of an image occurs by first establishing at least one image parameter at least in part in accordance with at least one attribute of the macroblock. Thereafter, a block of film grain is established in accordance with the image parameter.
To understand the technique of the present principles for simulating a bit-accurate film grain pattern comprised of individual film grain blocks, a brief overview of film grain simulation will prove helpful.
The overall management of film grain requires the transmitter 10 (i.e., the encoder) provide information with respect to the film grain in the incoming video. In other words, the transmitter 10 “models” the film grain. Further the receiver 11 (i.e., decoder) simulates the film grain according to the film grain information received from the transmitter 10. The transmitter 10 enhances the quality of the compressed video by enabling the receiver 11 to simulate film grain in the video signal when difficulty exists in retaining the film grain during the video coding process.
In the illustrated embodiment of
A film grain modeler 16 accepts the input video stream, as well as the output signal of the film grain remover 14 (when present). Using such input information, the film grain modeler 16 establishes the film grain in the incoming video signal. In its simplest form, the film grain modeler 16 could comprise a look up table containing film grain models for different film stocks. Information in the incoming video signal would specify the particular film stock originally used to record the image prior to conversion into a video signal, thus allowing the film grain modeler 16 to select the appropriate film grain model for such film stock. Alternatively, the film grain modeler 16 could comprise a processor or dedicated logic circuit that would execute one or more algorithms to sample the incoming video and determine the film grain pattern that is present.
The receiver 11 typically includes a video decoder 18 that serves to decode the compressed video stream received from the transmitter 10. The structure of the decoder 18 will depend on the type of compression performed by the encoder 12 within the transmitter 10. Thus, for example, the use within the transmitter 10 of an encoder 12 that employs the ITU-T Rec. H.264|ISO/IEC 14496-10 video compression standard to compress outgoing video will dictate the need for an H.264-compliant decoder 18. Within the receiver 11, a film grain simulator 20 receives the film grain information from the film grain modeler 16. The film grain simulator 20 can take the form of a programmed processor, or dedicated logic circuit having the capability of simulating film grain for combination via a combiner 22 with the decoded video stream.
Film grain simulation aims to synthesize film grain samples that simulate the look of the original film content. As described, film grain modeling occurs at the transmitter 10 of
The film grain simulation technique of the present principles enables bit-accurate film grain simulation and has applications to consumer products, such as HD DVD players for example. Other potential applications could include set top boxes, television sets, and even recording devices such as camcorders and the like. Film grain simulation occurs after decoding the video bit-stream and prior to pixel display. The film grain simulation process requires the decoding of film grain supplemental information transmitted in the SEI message. Specifications affecting the film grain SEI message ensure the technology will meet the requirements of high definition systems in terms of quality and complexity.
The value of the parameters conveyed in an ITU-T Rec. H.264|ISO/IEC 14496-10 film grain characteristics SEI message follow these constraints:
The parameter model_id specifies the simulation model. It shall be 0, which identifies the film grain simulation model as frequency filtering.
The parameter separate_colour_description_present_flag specifies whether the color space in which the film grain parameters are estimated is different from the color space in which the video sequence has been encoded. This parameter equals 0, which indicates that the color space for film grain simulation is the same than for encoding.
The parameter blending_mode_id specifies the blending mode used to blend the simulated film grain with the decoded images. This parameter equals 0, which corresponds to the additive blending mode.
The parameter log2_scale_factor specifies the logarithmic scale factor used to represent the film grain parameters in the SEI message. This parameter lies in the range [2, 7] to ensure the film grain simulation can occur using 16-bit arithmetic.
The parameters intensity_interval_lower_bound[j][i] and intensity_interval_upper_bound [j][i] specify the limits of the intensity interval i of color component j for which film grain parameters have been modeled. For all j and i, intensity_interval_lower_bound [j][i+1], this parameter remains greater than intensity_interval_upper_bound[j][i] since multi-generational film grain is not allowed.
The parameter num_model_values_minus1[j] specifies the number of model values present in each intensity interval for color component j. For all j, this parameter lies in the range [0, 2], which specifies that band-pass filtering and cross-color correlation are not supported.
The parameter comp_model_value[j][i] specifies the film grain intensity for color component j and intensity interval i. For all j and i, this parameter lies in the range [0, 255] to ensure film grain simulation can be performed using 16-bit arithmetic.
The parameter comp_model_value[j][i] specifies the horizontal high cut frequency that characterizes the film grain shape for color component j and intensity interval i. (The horizontal high and low cut frequencies, together with the vertical high and low cut frequencies that describe the properties of a two-dimensional filter that characterizes the desired film grain pattern).For all j and i, this parameter lies in the range [2, 14], which includes all the required grain patterns.
The parameter comp_model_value[j][i] specifies the vertical high cut frequency that characterizes a film grain shape for color component j and intensity interval i. For all j and i, this parameter shall be in the range [2, 14] which includes all the required grain patterns. For the combination of all the color components j and intensity intervals i in an SEI message, the number of different pairs (comp_model_value[j][i], comp_model_value[j][i]) remains not greater than 10.
All the other parameters in the film grain SEI message specified by the ITU-T Rec. H.264|ISO/IEC 14496-10 standard have no constraint according to this specification.
In accordance with the present principles, bit accurate film grain simulation occurs in the current picture unless the parameter film_grain_characteristics_cancel_flag equals unity or the frame range specified by the parameter film_grain characteristics_repetition_period becomes exhausted. The current specifications of the ITU-T Rec. H.264|ISO/IEC 14496-10 standard allows the simulation of film grain in all color components. Film grain is simulated and added to color component c of the decoded image if the parameter comp_model_present_flag[c] equals unity in the film grain SEI message. Bit-accurate film grain simulation occurs by specifying: a database of film grain patterns; a uniform pseudo-random number generator; and a precise sequence of operations. Film grain simulation typically occurs independently for each color component.
As discussed above, the SEI message contains several parameters, including intensity_interval_lower_bound[c][i] and intensity_interval_upper_bound[c][i] parameters where i ranges from 0 to the value of the parameter num_intensity_intervals_minus1[c]. These SEI message parameters undergo a comparison against the average pixel intensity value computed during step 102 for a color component c of each non-overlapping 8×8 pixel block in the decoded image stored in a display buffer 102 following decoding by the decoder 101. For each non-overlapping 8×8 pixel block from color component c of the decoded image, computation of the average value occurs during step 102 in the following manner:
The value of i for which the macroblock average pixel intensity value remain not less than intensity_interval_lower_bound[c][i] and not greater than intensity_interval_upper_bound[c][i] serves as the basis for selecting the film grain parameters for the film grain simulated for the current block in the image. If no value exists that fulfills the condition, no film grain simulation will occur for the current block.
The film grain parameter selection process typically includes the step of scaling the cut frequencies when processing chroma components (c=1, 2) in order to adapt to the 4:2:0 chroma format as follows:
comp_model_value[c][s]=Clip(2, 14, (comp_model_value[c][s]<<1))
comp_model_value[c][s]=Clip(2, 14, (comp_model_value[c][s]<<1))
Step 104 initiates establishing a film grain block, typically although not necessarily 8×8 pixels in size. The step of establishing a film grain block of 8×8 pixels involves retrieving a block of 8×8 film grain sample from a film grain database 105, and scaling the sample to the proper intensity, although scaling while desirable need not necessarily occur. The database 103 typically comprises 169 patterns of 4,096 film grain samples, each representing a 64×64 film grain pattern. The database 105 stores the values in 2's complement form, ranging from −127 to 127. Synthesis of each film grain pattern typically occurs using a specific pair of cut frequencies that establish a two-dimensional filter that defines the film grain pattern. The cut frequencies transmitted in the SEI message enable access of the database 105 of film grain patterns during film grain simulation.
The scaled cut frequencies (comp_model_value[c][s] and comp_model_value[c][s]) determine which pattern of the database serves as the source of film grain samples. Two randomly generated values serve to select an 8×8 block from the pattern selected in accordance with the cut frequencies. These random values used to select the 8×8 pixel film grain block represent a horizontal and vertical offset within the 64×64 pixel pattern and are created using the following procedure:
i_offset=(MSB16(x(k, ec)) % 52)
i_offset+=m & 0x0008
j_offset=(LSB16(x(k, ec)) % 56)
j_offset+=n & 0x0008
where x(k, ec) indicates the k-th symbol of the sequence x of pseudo-random numbers initiated with the seed ec, MSB16 and LSB16 denote the 16 most significant bits and 16 least significant bits, respectively, and (m,n) are the coordinates of the current 8×8 block in the decoded image. For the i_offset, the first equation generates a pseudo-random value uniformly distributed in the range [0,51], the second equation restricts that value to multiples of 4, and the last equation adds 8 to i_offset when m %16 equals 8. Equivalent operations are performed for the j_offset.
A uniform pseudo-random number generator 106 provides the pseudo-random numbers used to select the 8×8 pixel block. Referring to
The pseudo-random value x(k, ec), created using the pseudo-random number generator 106 undergoes updating every 16 pixels (horizontally) and every 16 lines (vertically) of the image. The same pseudo-random number x(k, ec) is used in each non-overlapping area of 16×16 pixels of the decoded image. As illustrated in
The random number generator 106 has different seeding depending on the color component (c) to which film grain is being added. Upon receipt of a film grain SEI message, the seed e1, used for simulating film grain on the first color component, typically has a value of unity. The seed e2, used for simulating film grain on the second color component, typically has a value of 557,794,999; whereas the seed e3, used for simulating film grain on the third color component, typically has a value of 974,440,221.
scale_factor=BIT0(x(k, ec))==0 ? comp_model_value[c][s]:—
comp_model_value[c] [s] 
for (i=0 . . . 7, j=0 . . . 7)
During step 108, deblocking filtering occurs between each film grain block created during step 104 and a previous block 109 to ensure the seamless formation of film grain patterns. Deblocking filtering applies only to the vertical edges between adjacent blocks. Assuming simulation of film grain blocks in raster scan order and that the left-most pixels of the current_fg_block lie adjacent to the right-most pixels of the previous_fg_block, deblocking filtering typically occurs by means of a 3-tap filter with coefficients 1,2,1 (not shown) as follows:
for(i=0, j=0 . . . 7)
At the end of the film grain simulation process, the deblocked film grain block undergoes blending with the corresponding decoded image block via a block 110 and the result undergoes clipping to [0, 255] prior to display in the following manner:
for (i=0 . . . 7, j=0 . . . 7)
display_image[c][m+i][n+j]=Clip(0, 255, decoded_image[c][m+i][n+j]+fg_block[i][j])
where (m,n) are the coordinates of the top-left corner of the block, decoded_image[c][x][y] is the decoded pixel value at coordinates (x,y) of color component c and display_image[c][x][y] is the video output at the same coordinates.
A switching element 111 controls the passage of the deblocked film grain block to the block 110 under the control of a control element 112. The control element 112 controls the switching element responsive to whether the SEI message parameter film_rain_characteristics_cancel_flag equals unity or the frame range specified by the parameter film_rain_characteristics_repetition_period has been exceeded which dictate whether film grain simulation should occur as discussed above.
The foregoing describes a technique for simulating film grain that has application in consumer electronic devices, such as set top boxes, HD-DVD players, television sets, and camcorders. The relatively low cost of random access memory readily permits incorporation of the film grain database 105 within a memory element. The combination of one or more of a microprocessor, programmable gate array and dedicated logic circuit, depicted generally in block 114 in