US 20050073529 A1
A system and methods for creating user definable windows for applying image processing, particularly useful in a post-production color correction equipment. The system allows a user of an image processing system such as a scene by scene color corrector to define a color region of interest in an image for purposes of applying image processing only to the region of interest, with automatic tracking of that region of interest over a plurality of frames in a scene. The user defined window, comprising a closed polygon, is converted on a frame-by-frame basis to a matte that is keyed with image processing such as color correction values. The user defined windows can be edited and stored and recalled for use in other frames. The color region of interest is automatically tracked from a beginning frame to the ending frame of a scene, with a vector window defined for each frame, so as to adapt to and compensate for changes in shape, size, and geometry of the region of interest during the succession of frames in the scene.
1. In a scene by scene color corrector, a process for automatically applying color correction to a selected region in plurality of frames in a scene, comprising:
(a) in response to user commands, defining a selected region in a first frame of the plurality of frames;
(b) generating a user defined window for the first frame corresponding to the selected region;
(c) for a subsequent frame of the plurality of images, generating a user defined window for the subsequent frame based on a color parameter associated with a prior frame;
(d) applying color correction to areas within the user defined window region in the subsequent frame.
2. The process of
3. The process of
4. The process of
determining a seed point derived from the user defined window of the prior frame;
starting at a point associated with the seed point, searching in the frame according to a predetermined methodology for picture elements that fall within a predetermined range of the color parameter;
defining a boundary based on a point in the frame at which the located picture elements fall outside of the predetermined range of the color parameter; and vectorizing the boundary to derive the user defined window.
5. The process of
6. The process of
7. The process of
8. The process of
9. In a scene by scene color corrector, a system for automatically applying color correction to a plurality of frames in a scene, comprising:
a component that defines a region in a frame in accordance with a color parameter of the frame;
a component that utilizes a region from a prior frame to define a region in a subsequent frame of the scene; and
a component that applies color correction to areas within the regions of the frames over the selected plurality of frames in the scene.
10. The system of
11. The system of
a component for determining a seed point derived from the prior frame;
a search component operative, starting at a point associated with the seed point, for searching in the subsequent frame according to a predetermined methodology for picture elements that fall within a predetermined range of the color parameter;
a component for defining a boundary based on a point in the subsequent frame at which the located picture elements fall outside of the predetermined range of the parameter; and
a vectorizing component for vectorizing the boundary to derive the region for the subsequent frame.
12. The system of
13. The system of
14. The system of
15. The system of
This application is a divisional of application Ser. No. 09/282,913, filed Mar. 31, 1999, now U.S. Pat. No. ______, entitled AUTOMATIC REGION-OF-INTEREST TRACKING FOR A COLOR CORRECTION SYSTEM, which is incorporated herein in its entirety by reference.
The present invention relates generally to image processing, and more particularly relates to systems and methods that allow a user of an image processing system such as a scene by scene color corrector to define a color region of interest in an image for purposes of applying image processing only to the region of interest, with automatic tracking of that region of interest over a plurality of frames in a scene.
In certain types of image processing systems, especially color correction systems employed in post-production equipment, system operators observe an image on a video monitor, adjust color and other parameters of the image until it is aesthetically satisfactory, store the parameters in system memory, and apply the parameter corrections to a sequence of images forming a scene. Various types of image processing are often employed to create, enhance, compress, filter, or otherwise modify characteristics of an image.
In a video signal color correction system, color corrections of motion picture film and/or video tape are typically made on a scene-by-scene basis. A “scene” is a sequential collection of images shot from the same camera, having the same viewpoint, composed in a certain way, etc. A system operator or “colorist” views a selected exemplary image or frame from a scene. The colorist then applies color corrections via a control panel to adjust video parameters such as hue, saturation, luminance, etc. of a frame image being viewed. The correction settings are stored in system memory.
After the colorist is satisfied with the adjustments he or she has made to the selected frame, the correction system, which is typically computer-controlled, applies the stored color corrections to each frame in the scene one at a time. The color-corrected frames of the scene are then recorded on film or videotape. The steps are repeated for other scenes in the film or video tape, often with different correction settings stored for different scenes. This process creates a color-corrected master film or video tape that reflects color adjustments to all frames in all scenes.
Devices are known in the art for defining a region of an image for applying one set of color corrections, with other regions of the image receiving another, separate set of color corrections. Such devices are useful when it is desired to isolate a certain area of an image to receive special image processing. For example, in color correcting a soft drink commercial it may be desirable to isolate the beverage can from the remainder of the image so as to make the colors of the beverage can stand out from the rest of the image.
Certain prior art image processing computer software, e.g. ADOBE PHOTOSHOP™, manufactured by Adobe Systems Corporation, Mountain View, Calif., employs edge detection algorithms for detecting color regions and creating paths or Bezier curves between points. Such edge detection algorithms rely upon color differences between regions to define the paths and produce odd results if there are discontinuities in the edges of a region, or if the colors defining the regions are not sufficiently dissimilar. The PHOTOSHOP™ software, for example, vectorizes an entire color region and provides no control over the number of points in the region. There is no way to modify a defined region except by changing region qualification parameters. For these and other reasons, such software is not suitable for real time applications such as a scene by scene color corrector.
A system for creating user definable windows for applying image processing is described in U.S. patent application Ser. No. 08/912,662, filed Aug. 8, 1997 entitled “User Definable Windows for Selecting Image Processing Regions.” This application is owned by the same assignee as the present invention. In this system, the luminance values of a key image are captured in a key frame buffer. The key image is edited in a “paint” mode to enhance the luminance values or eliminate regions of the image, to define a region of interest. The key image is then converted to a 1-bit image. The 1-bit image is vectorized to derive a user defined window. The user defined window, comprising a closed polygon, is converted on a frame-by-frame basis to a matte that is keyed with image processing such as color correction values. User defined windows can be edited and stored and recalled for use in other frames. A user defined window for a first frame of a scene can be transformed or morphed into a user defined window for a final frame of a scene, with the system automatically interpolating between the beginning window and the ending window so that the window is automatically moved, resized, and changed in geometry during the succession of frames in a scene.
Although this system provides remarkable improvements in isolating regions for receiving color correction in a scene-by-scene color corrector, only the luminance of the target image is employed to isolate a region of interest. Moreover, the system as described in this patent is not capable of automatically tracking a region of interest based on its color parameters (hue, saturation, and/or luminance) over a number of frames—the system therein described “morphs” the polygon from a first frame to a final frame of a scene.
There is a need, however, for a system that isolates a region of interest by its color parameters, e.g. hue, saturation, and/or luminance, or red/green/blue values, and employs those color parameters to establish and track the region of interest over number of frames in a scene, to effect automatic tracking of the region of interest over a plurality of frames.
The present invention is particularly adapted for use in connection with the system described in U.S. patent application Ser. No. 08/912,662, filed Aug. 8, 1997, entitled “User Definable Windows for Selecting Image Processing Regions,” the disclosure of which is incorporated herein by reference and made a part hereof.
The system in this patent employs the concept of a key well known in the television arts to define regions for receiving image processing. A good example of the use of a key is that of a television weatherman superimposed on a map. The weatherman is typically shot with a video camera standing in front of a solid blue color background. Other equipment automatically generates the video weather map. Image processing equipment is readily able to discern the outline of the weatherman relative to the blue background by use of color values. The image of the weatherman, which is called a “key”, is overlaid on the map by essentially providing transparency for the blue regions of the key outside the weatherman's body. This allows the weather map to show through in the blue areas, and superimposes the weatherman on top of the map.
Related principles are employed in the present invention for defining a user defined key corresponding with a predetermined region of an image and utilizing the key to derive a user defined window. Image processing such as a set of color corrections is applied within the region defined by the key-derived user defined window. A separate second type of image processing, for example a second set of color corrections, is applied in regions outside the user defined window.
Briefly described, the present invention comprises a system and methods that allow creation of user definable windows for selecting image processing regions. These windows can vary in geometry, size, and location during a number of frames in a scene. The invention is particularly useful for allowing a colorist in a video image color correction system to define windows for selecting regions for color correction. The user typically starts by capturing an exemplary image or frame of a scene for use as a “key image” or “key frame”. The key image is manipulated by adjusting parameters such as gain, clip, and limit to derive one or more key regions of the image to serve as keys. The key regions are vectorized and stored in a computer system's memory in association with the key frame. The vectorized keys, which are then called “user defined windows”, are then used during a plurality of frames for defining regions to receive image processing such as color correction, data compression, noise reduction, etc.
For use in a scene having a number of frames, the user defines a window for a first frame of the scene and a window for the last frame of the scene. The user may use the last frame as a key image and derive another key for defining the final window, can use and modify the window for the first frame for defining the final window, or can use modify any other previously defined window.
The system described in the referenced patent then interpolates or “morphs” the window of the first frame of the scene into the window of the final frame of the scene by linear interpolation across each frame of the scene. The windows are generated on a real time, frame by frame basis so that the color correction or other image processing can be viewed by the colorist on a video monitor. The windows thus change during the sequences of frames in the scene to compensate for changes in the size, geometry, and position of the area of interest in the scene.
In a system constructed in accordance with the present invention, the operator uses a pointer to select a point or group of points (a region) having desired color properties. The system automatically calculates the mean of the red, green, and blue values of each pixel of the selected point or points. The system then searches for other points having red, green, and blue values within a predetermined variance. The system then defines a boundary that separates qualified points from the rest, and vectorizes this boundary to obtain a closed polygon. This polygon defines a user defined window as in the incorporated patent.
The operator has the capability to adjust the initial mean value and the variance for each of the red, green, and blue channels. This allows inclusion of more or fewer points from the collection of qualifying points, and a new polygon or user-defined window can be defined.
In operation, the operator selects at least one region of a starting image frame. The operator also selects an ending image frame. The system can then be commanded to search for similarly colored regions in the intervening frames and create user-defined windows for each intervening frame by applying the selection criteria (for example, within a predetermined variance of the mean red, green, and blue). This allows creation of a new, different user defined window for each frame of the scene, automatically, which tracks the selected color region of interest regardless of geometric shape or location in the image.
According to another aspect of the invention, the operator can define plural regions of interest in each frame, with each region having different selection criteria. The regions need not have the same characteristics, such as color or shape.
After the regions of interest are isolated and vectorized to obtain user defined windows, the characteristics of the user defined windows can be edited to allow the user to adapt the window to the outline of the region of interest. Modification of a user defined window comprises adjusting the number of points representing a window, and adjusting the location of one or more selected points of a window.
Advantageously for use in a real time processing environment such as a scene-by-scene color corrector, the operation of automatically tracking the color region of interest from during playback of the scene is carried out at a real time rate. Thus, the user can adjust aspects of the image processing for exemplary frames of a scene and observe the effects of the image processing during playback of the scene on a real time basis.
These and other features and advantages of the present invention may be more clearly understood and appreciated from a review of the following detailed description of the disclosed embodiment and by reference to the appended drawings and claims.
Referring now to the drawings, in which like numerals indicate like elements or steps throughout the several figures,
Although the present invention will be described in the context of a scene by scene digital color corrector, it will be understood that the inventions described herein are not limited to video color correction and are equally useful in other types of systems where there is a need for application of image processing such as image compression, image filtering, noise suppression, animation, and any other type of processing. In particular, the invention is useful in any image application where there is a need to define a region of an image for receiving one type of data processing and another region for receiving a different type of data processing. However, in the context of the disclosed embodiment, the inventions are particularly useful in a scene by scene color corrector where a plurality of frames comprising a scene must be viewed and adjusted to create or modify a video program such as a movie, commercial, television program, etc.
As used herein, the term “key” is used to describe a predetermined region of an image that is used to define or derive a user defined window which is stored in a vector format, as well as the video matte produced by devectorizing a user defined window which is employed to selectively apply image processing to image regions in accordance with the matte.
The source of images 32 may be a telecine, video tape player, or a digital video storage medium. The source is connected to an Input circuit board 31 in the color correction system 30. It will be understood that the image signals internal to the color correction system 30 are digital, and are stored and manipulated in a digital video format such as the International Radio Consultant Committee (CCIR) promulgated Recommendation No. 601-1 which defines a standard set of digitized color signals. CCIR Recommendation 601-1 (1986) is hereby incorporated by reference. It is known in the art to take CCIR 601-1 digitized signal streams, convert same to corresponding RGB signals (in analog or digital domains), and then to perform the primary and secondary color corrections and modifications on the resultant signals. The corrected or modified signal can then be passed through a signal matrix and reconverted to a digital bit stream through use of an analog to digital converter. The standard signals defined in Recommendation 601-1 essentially consist of a luminance signal Y and two color difference signals (R-Y) and (B-Y). It is well known that, since the luminance signal contains information on levels of red, green and blue (RGB), the three standard signals can be used to reproduce the RGB levels for any given set of samples.
An Output circuit board 35 is also provided as a part of the color correction system 30, and typically converts the digital signals within the color correction system back into analog form for display. Thus, the Output circuit 35 is coupled via a video signal cables 37 to a video monitor 40, to a utilization device 42 such as a video tape player or other image storage device, and to the workstation 12 so that the results of image processing (e.g. color correction) can be viewed by the operator on the workstation monitor. It will also be understood that the corrected digital signals can be stored in digital form in a digital data storage device coupled to the system.
The preferred workstation 12 is an INDY™ model workstation manufactured by Silicon Graphics, Inc. (SGI). The preferred workstation includes an R4600 RISC processor operating in excess of 133 MHz, internal 24-bit XL color graphics (with RGB single or double buffer color indexes), a digital video input port capable of storing 640×480×30 fps to memory, 2 serial and 1 parallel data port, an Ethernet network port, a fast SCSI-2 data port, at least 32 megabytes of RAM, and a 1.0 or larger Gb hard disk. Optionally, a different SGI model workstation could be employed, e.g. one including a video adapter card that allows support and use of multiple monitors. Further details of the preferred workstation are available in the literature supplied by the manufacturer.
The preferred workstation 12 includes on-board display driver circuit, namely, the 24-bit XL color graphics circuitry with RGB double buffer color indexes, which provides video signals to at least one workstation monitor 15. A separate, second video monitor 40 is utilized in the preferred embodiment to display a target image, that is, a full screen representation of the video image being processed in the system. If a workstation model is employed that supports multiple monitors, the second monitor 40 may be connected to the workstation 12.
The preferred graphics tablet 19, which is particularly useful for certain manipulations of key images and vectors as will be described, is a Wacom ArtPad™ or ArtZ™ graphics tablet manufactured by Wacom Technology Corporation, Vancouver, Wash. 98661.
The preferred color correction system 30 is a Da Vinci RENAISSANCE 8:8:8™ digital color corrector manufactured by Da Vinci Systems, Inc., Ft. Lauderdale, Fla. This color correction system includes various circuit board components that are employed to carry out digital video signal processing, and is coupled to the control panel 25. The preferred RENAISSANCE 8:8:8™ control panel 25 comprises three separate panels each having a combination of dedicated keys, soft keys, and soft knobs that provides many simultaneous functions utilized in the color correction process. A dedicated keyboard 50 contains both dedicated and soft keys for system feature control, display/list management, source and destination motion control, editing, special effects features, and the like. This keyboard also contains a slide-out alphanumeric keyboard (not shown) to enter names, comments, and other important session information. Included on this control panel is a “User Definable Windows” (UDW) button or key 51 that is utilized to activate the functions described herein.
It will be understood that the preferred RENAISSANCE 8:8:8™ system includes software that is operative for controlling the image source and destination, and reading and storing frame numbers in association with user-defined scene identifiers or names. Thus, the system is able to provide displays of scene lists that facilitate the user in associating scene identifiers or names with exemplary images from the scene.
The control panel 25 further includes a joyball panel 53, which includes knobs, a trackball 54 for manipulating a cursor, keys for source setup, primary color balance, color grabbing, trackball control, and other effects control. When the windows feature of the present invention is activated, movement of the trackball 54 causes generation of a cursor 60 that is displayed on the video monitor 40. The cursor 60 is utilized to manipulate and select portions of video images displayed on the monitor, as will be described.
Finally, the control panel 25 includes a soft panel 57, which contains knobs and softkeys for dedicated 6-color secondary enhancement, special effects control, dedicated memory keys, and other functions not forming a part of the present invention. It will be understood that the provision of primary and secondary color correction, and sets of primary and secondary color correction, one for regions inside a user defined window and one for regions outside a user defined window, can be set and stored in memory by manipulation of controls on the soft panel 57.
The color correction system 30 includes several circuit boards that provide the functions required to carry out color correction or other types of image processing in the digital domain, and to control input from the image source 32 and utilization or storage in the utilization device 42. These circuits include a real time controller (RTC) circuit 70 with network interface that provides a data communication network interface to the data communication network 22 coupling the color correction system 30 to the workstation 12 and color correction system control panel 25.
The real time controller (RTC) 70 includes a dedicated microprocessor for handling real time events. Real time events occur at the rate of one every 30 millisecond (ms), so as to maintain an event rate corresponding to at least a 33 frames per second (fps) display rate. Data comprising an events list is provided to the RTC 70 via the data communications network 22; this events list is provided to control various functions of components in the system, for example providing a set of color corrections for each frame of a specified scene being processed. In particular, the events list as employed in the present invention includes a data structure that identifies a particular frame number and scene number, information that specifies one or more user defined windows for the particular frame, information identifying color corrections to be applied to areas inside the one or more windows of the particular frame, and information identifying color corrections to be applied outside the window of the particular frame, and other information.
The events list in the preferred embodiment is a data structure having frame numbers as a time reference. The data structure comprises a sequence of data blocks, each block being related to a beginning frame number and an ending frame number. Each block is called a “scene” which corresponds to a continuous segment of film or video shots. Each block also contains a set of color correction parameters and window parameters. The color correction parameters include known parameters such as red gain, red gamma, red black, etc., for each RGB color channel. The window parameters include window shape, window size, window location, and other information.
A user definable windows (UDW) generator board 75 constructed as described herein includes a dedicated high speed microprocessor that converts vectorized user defined windows into a video matte or key on a real time basis at the real time event rate. The RTC circuit 70 provides the vector representations of the windows to the UDW board 75 at the real time event rate.
A key frame buffer (KFB) circuit board 78 stores a video image from the image source 32, or from any other digital storage system, in a high speed memory. This image is then utilized in accordance with methods described herein to derive a vector representation of any user defined windows. Keys for vectorizing may be obtained directly from the image source 32 or be retrieved from prestored images stored in the memory of the workstation 12.
Color correction circuit 80 carries out digital color correction and other video parameter adjustment functions associated with the color correction system 30. The block in
As will be understood by those skilled in the art, the preferred color correction system 30 is operative to receive video images from an image source such as telecine 32, digitize the video images, capture a digitized image in the KFB board 78, display the captured image on the video monitor 40, receive color corrections entered by the colorist or operator via the control panel 25, impose such color corrections upon video images as they are transferred through the system, and provided corrected output signals via the output board 35 which are employed or stored in a utilization device 42 and/or simultaneously displayed on the video monitor 40.
In carrying out the operations for applying image processing to user defined windows, a system operator or user interacts with the system through the control panel 25, and via the user interface elements of the workstation 12, namely, keyboard 20, mouse 18, graphics tablet 19, and workstation monitor 15. In particular, the workstation 12 executes computer programs for carrying out the computer-implemented methods described herein, and generates displays on the monitor 15 containing windows with user interface elements such as informational displays, control buttons, slider bars, data entry fields, image display areas, etc.
The reader should be careful here to distinguish user-interface “windows” displayed on the monitor 15 generated from execution of computer software for carrying out the invention, from the “user defined windows” for applying image processing. During system use, the workstation monitor 15 will typically be displaying one or more user-interface oriented windows for carrying out the methods as described herein. One such user-interface window is a control window 44, which comprises a display of various buttons and slider controls for manipulation of the key image. Activation of certain commands causes the display of other, secondary windows.
One of the secondary windows is a View/Edit Window 46, shown overlapping the control window 44 in
The system operator manipulates controls in the Control Window 44 (of which there are several types), and interacts with the images, vector windows or keys, etc. through various View/Edit windows 46 of different types.
It should be noted that the video images displayed in the video monitor 40 are preferably displayed within a particular View/Edit window 46. Since the images provided from the image source are digitized for processing in the preferred RENAISSANCE 8:8:8™ processing system, the system readily generates digitized video images and provides them in an appropriate format, such as Apple Computer's Quicktime™ video format, MPEG I or II, or other digital video format via the network 22 to the workstation 12. Accordingly, it will be appreciated that any displays described herein where a video image is shown on the video monitor 40 are typically also displayed (albeit in reduced size) on the workstation monitor 15. Furthermore, any of the functions described in conjunction with a video monitor including generation of a cursor of a display of overlays of user defined window over a video image, display of a key, etc. can also be supplied in a window in the workstation display 15.
Thus, it is to be expected that as more and more image processing of film and video is conducted in the digital domain, as opposed to the traditional analog domain, methods of the present invention will be utilized in strictly digital systems for image storage, retrieval, manipulation, display, and output.
In the disclosed embodiment, however, the results of image processing are viewed on the video monitor 40, while the user interface and controls aspects of the image processing and the invention are carried out through the workstation 12 and its associated monitor 15.
As partly described in the referenced patent, the system operates in a plurality of different primary modes. One is a keystore mode that allows capture, editing and manipulation of 10-bit and 1-bit key images based on luminance capture, another is a vector editing mode that allows manipulation of user-defined vectorized windows dry from key images in the key store mode, and another is a color region of interest (ROI) mode that allows definition of regions by color information and manipulation of the vectorized regions of interest.
Turn next to
It should be understood that in the following several discussions of methods and processes involved in the present invention, numerous details are provided such as computer display system elements, aspects of computer program user interfaces, display screens and formats, sample data, etc. in order to provide an understanding of the invention. However, those skilled in the art will understand that the present invention may be practiced without the specific details. Well-known circuits, programming methodologies, and structures are utilized in the present invention but are not described in detail in order not to obscure the present invention.
Certain of the method descriptions which follow are presented in terms of exemplary display images, algorithms, and symbolic representations of operations of data within the computer's memory. As will be known to those skilled in the programming arts, and particularly those skilled in object-oriented programming methodologies, these algorithmic descriptions and graphic exemplary displays are the means used by those skilled in the art of computer programming and computer construction to convey teachings and discoveries to others skilled in the art.
For purposes of this discussion, an “algorithm” is generally a sequence of computer-executed steps leading to a desired result. These steps require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, or optical signals that are capable of being stored, transferred, combined, compared, or otherwise manipulated. It is conventional for those skilled in the art to refer to these signals as bits, values, elements, symbols, characters, images, terms, numbers, or the like. It should be kept in mind, however, that these and similar terms should be associated with appropriate physical quantities inside the computer and that these are merely convenient labels applied to these physical quantities that exist within the computer.
It should also be understood that manipulations within the computer are often referred to in terms such as adding, comparing, moving, etc. which are often associated with mental operations performed by a human operator. It must be understood that no involvement of a human operator is necessary or even desirable in many aspects of the present invention, since the operations described herein (except those specifically identified as origination with the operator or user) are machine operations performed in conjunction with a human operator or user that interacts with the computer. The machines used for performing the operation of the present invention, as will be understood, include general purpose digital computers or other similar computing devices.
Furthermore, it should be kept in mind that there is a distinction between the methods, steps, or operations carried out by a computer, and the method of computation itself. The present invention does not involve a method of computation. The present invention rather relates to methods, steps, or operations for a computer and processing electrical or other physical signals to generate desired physical signals and display results and interactions. As illustrated in
It is assumed in
At step 101, the user enters a command to capture a key frame from the image source; this key frame is captured and stored in digital form in the KFB circuit 78. A key frame at this stage in the disclosed embodiment for color region of interest tracking is a 24-bit color image, with 8 bits each of red (R), green (G), and blue (B) video channels. The key frame image, a color image in the disclosed embodiment, is displayed on the video monitor 40 as it is viewed from the image source; after capture it is displayed in a miniaturized version in an image area in a control window 44 on the workstation monitor.
At step 102, the system determines if a user defined window (UDW) from a preceding frame is available. In the case when the operator is first selecting a color region of interest, there will not be a prior user defined window in existence. On the other hand, after the operator has selected a color region of interest for a given frame, a user defined window associated with this color region of interest will be in existence and stored in the system memory. This prior-frame user defined window is employed as a starting window for a subsequent frame. In accordance with methods described herein, the beginning user-defined window from a prior frame is used to define a search area wherein the system begins to search for pixel color values within the predetermined variance of the mean color established for that prior user defined window.
If no user defined window from a preceding frame is available, then at step 104, the user employs a tool to select a point or points in the color region of interest. As will be described, this feature is activated by clicking on a “color grabber” button that causes display of a predetermined cursor.
At step 106, the system computes the mean of the points selected by the operator and displays this mean as a color in a predetermined display region on the display screen 46.
Step 110 shows that the operator can adjust the mean and the variance of the color value so as to provide an adjusted mean.
Using the adjusted mean and variance, at step 112 the system automatically determines the color boundaries of the region of interest by conducting a search algorithm to isolate the x and y coordinates of picture elements that meet the selected criteria of being within a predetermined variance of the mean color value.
Then, at step 118, this region, identified as a color region of interest, is vectorized to obtain a user-defined window (UDW). At step 120, this UDW can be edited and at step 114 stored and retrieved.
At step 118, the isolated key image, which is also called a “key”, is vectorized at the workstation to obtain a user defined window (UDW) defined in terms of points, with vectors extending between the points. The term “vectorized” means that a continuous polygon of n points has been drawn around the region of interest, to obtain a set of points which can be connected with (and displayed as) a set of lines. In this disclosed embodiment, UDW's are defined in terms of points within an image space having 586 horizontal lines, with 720 pixel locations per line, where the last point of a set of points defining a window connects to the first point.
After the key image has been vectorized at step 118 to obtain a vector representation of a window, the window can be stored in the workstation's memory as indicated at the optional step 114. Further, a prestored vector window can be retrieved from memory, edited, and re-stored if desired.
At step 120, the user may edit the UDW if necessary or desired. The UDW editing process essentially entails operations of adding, deleting, and/or moving a point, a selected group of points, or an entire window. Windows for editing can be derived directly from the vectorizing step at 118, by free hand drawing with a pointing device (e.g. mouse, graphics or digitizer tablet), or by retrieving a prestored window from storage in the workstation 12, as indicated at 114. Edited windows can also be stored in the workstation's memory.
After the user has created a suitable UDW, it is used to apply image processing, as indicated generally at step 122. In the disclosed system for color correction, using the window to apply processing involves devectorizing the window for a frame of video to thereby obtain a digital key, and employing the key as a mask or matte for applying color corrections. Regions corresponding to one or more windows receive a predetermined set of color corrections, while other regions receive another set of color correction.
For use in a series of frames of a scene, the user associates a first user defined window with a first frame of a scene, typically edits the first frame's window or defines a new window for the final frame of the scene, and then designates the windows as corresponding to the beginning and ending frames of the scene. The preferred system is then automatically operative to employ the first user defined window and automatically track the color region of interest there between during the succession of frames in the scene in order to provide a progressively changing window throughout the frames of the scene. Thus, step 124 in the disclosed embodiment involves applying color correction on a frame by frame basis to regions of each frame in accordance with one or more user defined windows for each frame.
Other types of image processing that can be applied in accordance with user defined windows include image compression, image filtering, animation creation, noise reduction, and colorization. Other types of image processing applications will occur to those skilled in the art.
Referring first to
After the vectorized regions have been modified to the operator's satisfaction, the user
The vector window(s) 160 are displayed on the workstation monitor, and are preferably overlaid on the key image so the user can visually verify the accuracy of the user-defined window relative to the key image. In the example shown, the vector window comprises a plurality of points connected by piece-wise linear segments. The user may select more or fewer points, as desired, so that the vector window matches the area of interest as closely as possible. The vector window may be edited by adding or deleting points, as desired.
In operation of the system to define windows for a plurality of frames of a scene, the user will define a window for the first frame of the scene and the last frame of the scene, as will be described.
The vectorized user defined window(s) 160 are utilized in the UDW circuit board to create a digital matte that is electronically mixed with the original image to define a region for applying color correction. Stated in other words, the open regions 165 in
In addition, the present invention provides for a softness filter region 170, which is shown as a white outline around the processing region 165. As described in the referenced patent, color corrections are applied in the softness filter region 170 gradually tapering from zero correction at the boundary adjacent the non-processed region 162 (zero percent) to 100 percent processing at the boundary of the processing region 165. The width or gradient of the softness filter region 170 is selectively adjustable by the user.
With the foregoing in mind about the method for creating a user defined window associated with a predetermined color region of interest, turn next to
Using the example of the bottle and glass, assume that frame 1 of a scene having n frames shows the bottle 150 and glass 155 in an initial position of the screen, as displayed on the video monitor 40 and in the view/edit window 46. As the scene progresses through its frames until the final frame n the bottle is gradually tilted forward, the cork removed, moved in position toward the upper center of the screen from the left of the screen, and changed in orientation from a side view of the bottle to a top view, as the bottle is turned towards the view, with the contents being poured into the glass. Thus, the bottle changes its position, size, and geometry (shape) from frame one to frame n. However, the mean color of the bottle does not change as the bottle changes its position, size, and geometry.
With the present invention, the user-defined window 160 effectively transforms its shape from the elongate bottle to a circle throughout the n frames. The system is automatically operative to detect the mean color in the region of interest defined by the starting color region key 160 a detect that predetermined color in the successive frames 2, 3, . . . n etc. and to vectorize the color region of interest so as to define the regions for applying image processing. At any point along the way, the operator may freehand draw a window, derive a new key from the video image of a frame, add a new key or region of interest, use a prestored key such as a circle or square, or may use a prestored window derived from a prior frame and edit the characteristics to define a region such as that shown at 160 n.
In order to effect an orderly transition between geometry over a number of frames, the user may want to maintain a correspondence between the points of the starting window 160 a and the ending window 160 n. Typically, this correspondence is maintained by providing the same number of points in the ending key window or vector 160 n as in the starting key window or vector 160 a. The number of points is selectably adjustable by the operator.
In the referenced patent, there is a mapping between predetermined points and a starting and ending key window. In the present invention, there is no need to maintain a correspondence between particular points in the key windows, and the key windows are employed merely to define the color region of interest.
A preferred approach when a user-defined window changes shape, size, and position is for the operator to work backwards from a window for a final frame and edit the window of the final frame to derive a window for the first frame by adjusting the color parameters so that the mean hue and variance from the mean effectively capture the region of interest throughout the plurality of frames. In this approach, the operator views the last frame of a scene and determines the number of points required to represent the user-defined window for the region of interest. This number of points is then noted as a minimum number of points for use as the starting window. The user then views the window for frame 1 and adjusts the color parameters as appropriate.
It should be understood at this juncture that the number of frames defining any given “scene” is completely up to the discretion of the operator. If the transformation of the region of interest between a starting window and an ending window is not acceptable, the user may define a subscene comprising a smaller number of frames that provides a more satisfactory transition. The capability of assigning any number of frames to a scene is provided in the preferred RENAISSANCE 8:8:8™ color correction system.
Of course, the ultimate worse case operation is to create and assign a user defined window to each frame individually, but in most cases this will not be necessary. The present invention allows storage of a plurality of user defined windows, each having a number of points, limited only by the total number of points defining such windows for each frame of a program. As described in greater detail below, the total number of points in the user defined windows is limited to a number that allows the UDW board 75 to devectorize all windows and create the digital matte on a real time basis, which is 200 in the disclosed embodiment.
The user definable window (UDW) generator circuit 75 employed in the preferred system shown in
The softness filter 300 employed in the UDW circuit 75 is also described in the referenced patent, and will not be described further herein.
Furthermore, in the present invention, a bounding rectangle 174 is defined that encloses the entire polygon 160 and defines an outer search limit.
The searching methodology is as follows: beginning at the seed point 172 of the vectorized region of interest 160 from the preceding frame, the RGB values of the pixel at the seed point are examined to determine whether or not they satisfy the criteria of being within the predetermined variance of the mean value of the color of interest. In most cases, this will result in a “yes” determination. The system begins by searching first to the left of the seed point until it encounters a predetermined color boundary defined by a predetermined color value outside of the mean and variance. This point is identified as a left-most color transition edge. In like manner, the system searches from the seed point to the right to determine a color transition edge, to the top, and then to the bottom to determine four color transition edges. These four points then define initial boundaries for further searching.
Those skilled in the art will understand that a number of different algorithms for searching can be employed, including but not limited to an inverted spiral search and a simple progressive scan in x and y from the seed point.
Furthermore, the preferred algorithm will employ methods that terminate the search in the event that the maximum geometric boundary such as that of the bounding rectangle 174 are exceeded to change search methodology to that of random searching or in the worse case a progressive scan of every value in the array.
Furthermore, the vectorized regions of interest and their respective centers of gravity can be employed to determined a trajectory, velocity and acceleration for the center of gravity. Such methodologies can be employed to project the direction and rate of change of the region of interest so as to locate a seed point for searching of the next frame at a position within a subsequent frame corresponding to that of the velocity and/or acceleration of the region of interest across one or more frames.
The software primarily executes in four basic modes—a keystore mode 360, a vector mode 365, an edit window mode 368, and a color region of interest (ROI) mode 369. Each mode results in the generation of particular user interface displays or windows (as contrasted with user defined windows) on the workstation monitor 15—control windows and view/edit windows—with corresponding displays on the video monitor 40, which collectively allow user interaction with the system. Within the principal modes, certain types of operations are possible which are appropriate for the task at hand.
The three modes of operation of keystore mode 360, vector mode 365, and edit window mode 368 are described in the referenced patent and will not be repeated herein. Thus, the discussion which follows is primarily focused on the color ROI mode 369.
Operation within the various modes results in display of various user interface or control windows such as an image capture window 44 and view/edit windows 46 on the workstation monitor.
In each control window 44, selection between the keystore mode, the vector mode, the edit window mode, or the color ROI mode is made by pressing (i.e. clicking with the mouse) a Key Store button 370, a Vector button 375, an Edit Window button 376, or a Color ROI button 377. Selection of these buttons by the operator activates the corresponding mode.
Still referring to
Other details of the buttons in the image capture control window 44 a in
Referring now to
In connection with automatic region of interest tracking, a number of controls are provided. First, a color grabber button 412 when depressed causes display of a “color grabber” cursor icon as shown at 153.
A mean color display region 415 provides a visual display of the average or mean color as determined by the system, in response to selection of a point or points by the operator.
A plurality of variance sliders 416, one for each of the red, green, and blue channels, is provided to adjust the variance from the mean. The operator can adjust the variance between a minimum of zero and a maximum of 256 possible values (an 8-bit variance is provided). A selected variance of 20 is shown in
A first frame control button 417 is provided to allow the operator to select and/or rewind to the first frame of a scene for manipulation. An ending frame control button 418 is provided to allow the operator to select and/or advance to the ending frame of a scene.
A play button 420 is provided for commanding the system to display the frames in the scene for viewing by the operator. A “start auto tracking” button 422 is provided to allow the operator to command the system to begin with a region of interest defined in a first frame and track the particular region of interest in successive frames, one frame at a time.
A plurality of points controls 424 are provided to allow operator adjustment of the number of points (Points) in the closed polygon comprising the vectorized key such as 160, and the minimum distance between points (Dist).
A Detect button 425 is provided so that the operator can command the system to initiate the process of calculating the mean and detect colors in the image within the predetermined variance of the mean.
A Done button 426 is provided so that the operator can signal to the system that operation with this particular window in this mode are completed.
Similarly, the pixel at location is shown in a 9×9 pixel window. The system is automatically operative to computer the mean color value of the pixels within the 9×9 window.
Alternatively, the user can define a predetermined radius around a selected point, as shown at the point (x3, y3). The point (x3, y3) is shown within a portion of the glass 155, with a circle defined by a four-pixel radius around the point. If the operator selects the “Define Radius” button 432, the system determines the average color value within a radius of n pixels measured from the center point (as defined by a click).
For this purpose, a Define Radius button 432 is provided in the window 46. Activating this button causes display of a predetermined dialog box that allows the operator to set the radius around a selected point. The point is selected by a click operation when in the “Define Radius” mode.
A “Define Box” button 433 is provided so that the operator may define a predetermined square box comprising a predetermined number of pixels around a selected point as shown for the points (x1, y1) and (x2, Y2).
Finally, an Add Region button 434 is provided. Actuating this button allows the operator to “freeze” and save a particular vectorized key such as that shown at 160, and activate a mode wherein a new mean value can be selected so as to allow creation of another, second vectorized key for another region of interest in the same image.
Other tools, not forming a part of the present invention, are also provided in the preferred system so as to allow other types of image manipulation, some of which is described in the referenced patent.
From the foregoing, those skilled in the art will appreciate that a number of different tools are provided in a system constructed in accordance with the present invention that allow the user to create and edit user defined windows—either windows created from an immediately preceding vectorizing operation carried out on a key image, or windows previously created during prior operations and stored in the memory of the workstation. After any desired window editing operations, the user defined window (or windows, as plural windows are permitted in the preferred embodiment) is associated with a selected frame of a scene, and the controls in the control screen 44 activated to cause image processing to occur in accordance with the commands issued, e.g. apply color correction.
Starting at step 1001, the first step taken is to initialize the system and load the software for the workstation 12 by allocating a region in memory for handling the software processes and storing relevant data. At step 1003, the image capture window (or other suitable starting display window) such as that shown in
At step 1004, the program enters an infinite loop of getting a user command or receiving an event that drives subsequent actions. Those skilled in the art will understand that the preferred software is event-driven and actions occur as a result of a user typing a command, pointing with the cursor and clicking on an object (such as a control), clicking and dragging, etc.
At decision step 1005, the inquiry is made whether the Key Store button has been depressed. If so, process 1020 is executed and the system enters the key store mode. Steps associated with the key store mode of operation are described in the referenced patent.
If the Key Store button has not been pressed, at decision step 1012 the inquiry is made whether the Vector button has been pressed by the user. If so, process 1030 is executed to enter the Vector mode. Steps associated with the vector mode of operation are described in the referenced patent. If not, at decision 1035 the inquiry is made whether the Edit Window button has been pressed by the user. If so, process 1040 is executed and the system enters the edit window mode. Steps associated with the Edit Window mode of operation are described in the referenced patent.
If at step 1035 the Edit Window button has not been pressed, control passes to decision 1045, where the inquiry is made whether the Color ROI button has been pressed. If so, the process 1100 is executed. Steps associated with the Color ROI mode of operation are described in connection with
If at decision 1109 the Capture button 413 has been pressed (
If at decision 1112 the “color grabber” tool button 412 (
If at decision 1117 a first frame command has been provided by depressing the First Frame control button 417, a First Frame routine is carried out at 1119. steps associated with the First Frame routine are described in connection with
If at decision 1121 an Ending Frame command is provided by depressing the Ending Frame control button 418 (
If at decision 1125 an Autotrack command is provided by depressing the Start Autotracking button 422 (
If at decision 1130 a Variance Adjust command has been provided by actuation of one of the RGB variance sliders 416 (
If at decision 1133 the Play button 420 (
If at decision 1140 the Define Radius button 432 (
If at decision 1145 the Define Box button 433 (
If at decision 1150 the Add Region button 434 (
At decision 1155, the inquiry is made whether the Detect button 425 has been actuated. If so, a Detect routine 1157 is executed. The Detect routine is operative to utilize the predetermined mean color value and detect all pixels having a color value within the indicated variance as controlled by the RGB variance sliders and include those within an area which is then vectorized. Steps associated with the Detect routine are described in connection with
At decision 1160, the inquiry is made whether another command that is appropriate for the window 46 in
If on the other hand the cursor is in the image area, the inquiry is made at decision 1207 as to whether the mouse button is down. If not, no particular action is taken and the program loops back to decision 1203. If the mouse is down, control passes to step 1209, since a potential click or drag operation has been indicated. The instantaneous value of the red, green, and blue color values at the location of the cursor as indicated by the variable (xn, yn) is stored in an array for use in computing the mean color value. Control passes to decision 1212.
At decision 1212, the inquiry is made as to whether the mouse button has been released (mouse up). If so, the “yes” branch is taken to step 1215, and the mean is computed for all of the values in the array that have accumulated since the mouse button was depressed. At step 1217, the computed mean is converted into a color and displayed within the mean color region 415 on the view/edit window 46 (
If at decision 1212 the mouse is still depressed, the “no” branch is taken to decision 1220. At 1220, the inquiry is made whether the cursor has been moved. If not, control passes back to decision 1203. If so, control passes to step 1222 where an index (i) for tracking the possible movement of the cursor within the region of interest is incremented so as to add another value for purposes of computing the mean. Control then passes back to decision 1203.
The process 1115 continues to execute until the user presses the Done button, or actuates another command.
It will be understood that the steps shown in
At step 1505, the frame count index i is incremented to advance to the next frame. At step 1507, the vector key for frame i−1 is retrieved so as to obtain a starting key for beginning the search algorithm to locate the color region of interest. At step 1508, the system displays frame i of the scene. At step 1511, the vector key for frame i−1 is overlaid on the image for viewing by the operator. At step 1515, the system determines the center of gravity for the vector key for frame i−1 to determine the seed point.
At step 1520, the system begins at the seed point to execute an algorithm for searching for color values that are within the predetermined variance of the mean color. This results in determination of a region of pixels, not necessarily continuous or contiguous, that can be satisfactorily vectorized.
Then, at step 1525, the region of interest identified for all color values meeting the criteria of values within the variance of the mean color are vectorized. At step 1530, the vector key for the current frame i is then stored. Control passes to decision 1535, where the inquiry is made whether the final frame of the scene for autotracking has been reached. If not, control passes back to step 1505 and the next frame is processed. The final frame has been reached, the “yes” branch is taken and the routine exits.
At step 1605, an inquiry is made as to whether the variance for the selected channel is at its maximum value. If so, at step 1607 the maximum variance value (256) is displayed on the screen and employed as the maximum variance.
If at decision 1605 the variance is not at its maximum, the inquiry is made at step 1609 whether the variance is at a minimum value. If so, the minimum variance (0) is displayed. If the variance is not at its minimum value, then control passes to step 1615, and the adjusted variance value is displayed and employed as the selected variance value.
From the foregoing, it will be understood and appreciated that the present invention allows an operator to identify a region of interest by hue, saturation, and/or luminance, and particularly RGB color information so as to identify a color region of interest, automatically establish a vector key for the region of interest that can be selectably adjusted so as create a region for applying image processing such as color correction and autotrack this region of interest based on the mean color of the region of interest over a number of frames in a scene.
In view of the foregoing description of the preferred embodiments of the present invention, other embodiments will suggest themselves to those skilled in the art. Therefore, the scope of the present invention is to be limited only by the claims set forth below.