WO1993017517A1

WO1993017517A1 - Image editing system, for inserting part of a video image on a background

Info

Publication number: WO1993017517A1
Application number: PCT/US1993/001703
Authority: WO
Inventors: Arthur M. Blank
Original assignee: Imageware Software, Inc.
Priority date: 1992-02-25
Filing date: 1993-02-25
Publication date: 1993-09-02
Also published as: JP3936387B2; JPH07507664A; JP2004166221A; US5687306A; US5345313A; JP3936680B2

Abstract

A system and method for editing digital images in three dimensions includes a computer (130) for storing a digital image of an object (112, 156) and a background (114) as well as at least one additional background image (145, 157). The periphery, or edge (146), of the object has a first hue, and the surrounding background has a second hue. Based upon the difference between the hues and predetermined hue difference, the computer locates the edge of the object and removes portions of the image (i.e, the background) that are outside the edge. Then, the object can be combined with a preselected one of the other background images so as to form a composite image.

Description

IMAGE EDITING SYSTEM, FOR .INSERTING PART OF A VIDEO IMAGE

A BACKGROUND

Background of the Invention

Field of the Invention

The present invention relates to image processing an more particularly, to systems for editing digital images.

Background of the Technology

A large number of applications require combining o video image with another video image, i.e., a televisi broadcast of a weather person in front of weather maps duri a weather report, so as to produce a composite video imag One well-known technique for producing composite video imag is commonly referred to as "chroma-key". The chroma-k technique is so named because it uses the chroma or col information portion of a television signal as a "key" control the formation of a composite image. A chroma-k device is essentially a video multiplexer which selects video input signal by feeding the chroma signal from one the two video inputs to decision logic.

In the case of the above-mentioned weather repo application, a television camera is directed at a weath reporter standing in front of a vertical sheet, called matte, which has a predetermined blue hue or color shade. H is the dimension of color that is referred to a scale perceptions ranging from red through yellow, green, and blu and circularly back to red. The image of the weather report and matte foreground source is provided to the chroma-k device, which is programmed to select all of the video signa received from the camera, except those video signals th represent portions of the image having the predetermined bl hue. Thus, the chroma-key device effectively separates t image of the reporter from the image of the matte.

At the same time that the previously described operati takes place, a video image of a weather map or satellite vi of the earth, either of which may be superimposed with ci names, high/low temperatures, and the like, is provided to t chroma-key device as a background source. The decision log of the chroma key device selects the background source video output wherever a blue hue is detected in the foreground source and presents the resulting background and foreground as a composite picture on a studio monitor that can be viewed by the reporter. The reporter can then point to positions on the matte that correspond to geographic locations on the background source and the viewer of the television program sees a composite image of a reporter and a weather map. Such a composite image is the desired output of a chroma-key device. However, as was noted in U.S. Patent No. 4,811,084 to

Belmares-Sarabia, et al., a major disadvantage of a chroma-key system is that false keys can be produced. For example, weather reporters wearing blue or striped-suits, and even blue eyes, may cause a chroma-key system to produce an incorrectly spliced composite. Also, a chroma-key device is used too large a distance between the reporter and the matte causing reflections resulting from false keying, hence restraining the movements of the reporter.

To overcome the problems inherent in chroma-keying, the Belmares-Sarabia, et al., patent discusses a device for video color detection that does not depend on a single color for keying. For example, such a device is also said to discriminate among similar hues by limiting the bandwidth of the hues and/or hue and saturation combinations that can be recognized by the device.

The device disclosed in Belmares-Sarabia, et al. , uses an analog process to multiplex television signals. Analog processes, however, are not as versatile in combining images as are digital processes, which can be programmed to apply sophisticated image processing algorithms to a digitized image so as to alter or edit an image. Thus, it would be an advancement in the technology to provide a digital image editing system which can strip a digital image of an object from a background and combine the digital object with a different digital background or backgrounds (composite) without suffering from the above-mentioned problems and limitations. Other ways of producing a composite image include imag editing software programs running on a Macintosh^® compute available from Apple Computer, Inc. , or a PC type compatibl computer available from IBM and other companies. Thes programs are exemplified by Picture Publisher^® produced b Micrografx, Inc. for the PC and Adobe Photoshop™ produced b Adobe Systems Incorporated for the Macintosh. Pictur Publisher is a registered trademark of Micrografx, Inc. Adob Photoshop is a trademark of Adobe Systems Incorporated. Macintosh is a registered trademark of Apple Computer, Inc.

These programs enable the user to place one object imag in front of a background scene and to cut and remove th object image. However, these programs are limited to workin with one object only. These programs cannot build a set o layers of objects and backgrounds and allow the user to mov an object to different depths or layers into the composit image. In other words, a person could not be placed behind fence but in front of a house of the background scen simultaneously. Thus, it would be an advancement i technology to provide a system which could place an object i front or behind any other object or the background at an desired depth into the composite image. It would also b desirable to make part of an object which was moved into particular layer to be transparent based on a desire attribute of the object. For example, one attribute of th object is hue, which is the perceived color shade. The leave of a tree, which have a particular hue range of green, coul be set to be transparent rather than opaque. Thus, the sk would then be seen between the branches of the tree, wherei previously, the leaves had blocked the sky.

When producing a composite image involving the face of person, one may desire to remove the original person's fac and replace it with the face of another person. Frequentl the two faces will not be the same size, e.g., one face is closeup and the other is not, and the person trying to mak the composite image will have to reduce or enlarge th replacement face to fit. This may involve numerous trials t achieve a pleasing appearance. Thus, it would be desirable to provide a system which can automatically size the replacement face to provide a natural appearance without a trial and error process. Another aspect to consider when replacing one face for another is the coloration of the skin. The person making the composite image may be placing a fair complexioned face to replace a dark complexioned face. The rest of the body, such as hands, arms and legs, may be visible in the original image. The composite would therefore not appear natural after the replacement was done. Thus, it would be desirable to provide a system which can automatically match the skin tone of the replacement face to that of the original face without any manual intervention to provide a natural appearance for the resultant composite image.

A further aspect to consider when replacing one face for another in creating a composite image is the positioning of the replacement face. This positioning may involve an iterative process to try different placements to achieve a pleasing and natural appearance. Thus, it would be desirable to provide a system which can automatically position the replacement face at the appropriate location to produce a pleasing appearance for the resultant composite image.

It is often the case that an object is imaged under one lighting condition and is then overlaid on a background that was imaged under another lighting condition. Consequently, the composite image may look artificial. Thus, it would be a further advantage if the editing system could establish the same lighting conditions, or "gamma", for the entire composite image. For example, it may be desirable to have an object that was imaged under fluorescent light inserted into a background that was imaged under full daylight and have the composite image maintain the same lighting condition. The lighting condition of the composite image could even be a third condition such as moonlight.

Accordingly, a need exists to provide a digital image editing system which can separate the digital image of an object from a background against which the object was imaged It is a further purpose of the present invention to provide digital image editing system which can automatically size position, and layer the digital image of a replacement objec or multiple objects into a predetermined background at desired depth, and then match the lighting conditions of th replacement object with one or more original objects and th background, and to provide a digital image editing syεstem tha is easy to implement and cost-effective to use.

Summary of the Invention The present invention satisfies the above-mentioned need and includes a system and method for selectively combinin digital images. The system includes a computer, which i connected to a device, such as a video camera, that ca generate a signal representing an image. An object, fo example, a human, is positioned in front of the video camera and the video camera generates a signal representing th object and the background that is behind the object. Accordingly, the signal from the camera includes a object component representative of the image of the object an a background component representative of the image of th background. The object has an edge, and the computer detect the edge of the object and separates portions of the imag that are outside the edge of the object (i.e., the backgroun component) from portions of the image that are inside the edg (i.e., the object component).

In one embodiment, the background has a single continuou hue, and based upon the difference in hue between the objec and background, the computer determines the location of th edge of the object. Based on this determination, the compute removes the background component from the signal.

More particularly, the video camera produces throug digitization a digital signal that is comprised of a pluralit of pixels that are arranged in rows. Each pixel has a hu gamma and each hue gamma has a corresponding numeric valu which represents how light or dark the hue is. The compute determines the location of the edge of the object by first ascertaining the numeric value of the hue gamma of a first pixel that is located at the end of the top row of the video window (i.e. , is located at the periphery of the video image) . The computer also determines the numeric value of the hue gamma of an adjacent second pixel in the row, and then compares the difference between the gammas to a predetermined difference.

When the difference between the hue gamma of the first pixel and the hue gamma of the second pixel is less than the predetermined difference, the computer compares the hue gamma of the second pixel with the hue gamma of a third pixel that is located in the same row as the first and second pixels and is adjacent to the second pixel, and so on. When the difference between any two pixels exceeds the predetermined value, this indicates the presence of a portion of the edge of the object. The computer then stores the location of the edge portion in memory, and then repeats the process described above for the row immediately below the top row of pixels. If desired, the computer can simultaneously perform a process like the one described above, but working from another direction, such as upwardly from the bottom row of pixels.

The computer continues the process, i.e., working inwardly from the boundary of the video window in the pixel- by-pixel comparisons, until the entire edge of the object has been mapped. After mapping the edge of the object, the computer is ready to "strip" (i.e., remove) the background component from the object component by setting all background pixels to a preselected transparent value. In another embodiment, the background has a white substrate with a plurality of boxes printed thereon. Each box contains a plurality of dots, and the processor has a map of the background stored in memory. When the object is positioned in front of the background, the object masks some of the vertices of the boxes and some of the dots within the boxes. Consequently, the processor can determine the position of the object by comparing the digital video signal with the stored map of the background to thereby determine whic vertices and dots have been blocked by the object. Once th location of the object has been determined, the processo strips the background image away from the object image. After stripping away the background image from the objec image, the computer can, if desired, integrate the objec image into a preselected background image that is differen than the background against which the object was imaged. Mor particularly, the computer can have one or more preselecte background images stored in the memory of the computer. Th computer can selectively digitally combine the objec component of the signal with multiple signals which ar representative of multiple layers of these stored backgrounds together with optional text entered by an operator or user o the system and present the composite image on a video display Thus, the image of the object can be combined, if desired with a preselected background that is different from th background against which the object was imaged, user tex added, if desired, and the composite image displayed. Thi combining of images can be accomplished real-time by a operator, who can view the video display and position an orient the image of the object on the preselected backgroun by means of a mouse, keyboard, joystick or other input devic that is operably connected to the editing system. Stated differently, the computer can essentially functio as an image combiner by stripping, from a digital vide signal, the image of an object from the image of th background in front of which the object was positioned. Th processor then combines the -image of the object with preselected image, and may accomplish the image combining i response to real-time operator-generated signals Specifically, stored in the memory of the processor is a two dimensional digital image representation of a three dimensional field of view. The processor then blends th object into the selected background so that the object appear to be integrated into the three-dimensional field of view. Preferably, to further blend the object into the preselected image, the processor averages the hue of edge of the object and the hue of the portion of the preselected background that is contiguous to the edge. The processor then adjusts the hue of the edge of the object to equal the* averaged hue.

Additionally, the processor of the present invention can "adjust the gamma of one or both of the object and the preselected stored background, to make the integrated image appear as if the object was imaged under the same lighting conditions as the preselected stored background. For example, the processor can ascertain the gamma of the preselected background, and then adjust^ the numeric value of the hue of the pixels that make up the image of the object as appropriate to make the object appear as though it was imaged under the same lighting conditions under which the preselected background was imaged.

The system of the present invention can also include a video printer electrically connected to the processor for generating a picture of the object. Also, the system can include a currency acceptor which is operably engaged with the processor for accepting currency and activating the system in response to the insertion of currency into the acceptor. Thus, the system can, in one application, be used in conjunction with electrical imaging booths that are installed in public places for use by the public.

In addition to the system described above, a method is also disclosed for video editing, i.e., for integrating the image of an object, e.g., a person, into a preselected two- dimensional representation of a three-dimensional space. In accordance with the method of the present invention, a first digital image that includes an object having a first hue and a background having a second hue has an edge defined by the difference in hue between the object and the background. This edge is detected, and the portion of the image outside the edge (i.e., the background) is removed from the image. The object is then overlayed on a preselected background, to produce an integrated image of the object and preselect background.

The system of the present invention automatically siz the object image to naturally integrate into the composi image. The computer compares the original background obje size to that of the replacement object and adjusts the siz if necessary, of the replacement object.

The system of the present invention automatical positions the object image to naturally integrate into t composite image. The computer utilizes the address of predetermined location on the original background object a transfers that address to a predetermined location of t replacement object.

These and other objects and features of the prese invention will become more fully apparent from the followi description and appended claims taken in conjunction with t accompanying drawings, in which like numerals refer to li parts.

Brief Description of the Drawings

Figure 1 is a perspective view of a presently preferr embodiment of the imaging system of the present inventio with portions cut away and shown in phantom, and with a hum subject sitting in front of a background of a single hue; Figure 2 is a block diagram illustrating the componen of the imaging system of Figure 1;

Figures 3a, 3b and 3c are a top-level flow diagram of t imaging system, used in conjunction with the monochro background of Figure 1; Figures 4a, 4b, 4c, 4d, 4e, 4f, 4g, and 4h are a sequen of images exemplary of those produced by the imaging system performing the process steps shown in Figure 3;

Figure 5 is a perspective view of an embodiment of t apparatus imaging system, with portions cut away and shown phantom, of the present invention, and a human object standi in front of a background of a single hue; Figure 6 is a perspective view of a coin-operated electronic imaging apparatus, which is another embodiment of the system of the present invention, and which has a checkerboard background, with portions broken away for clarity;

Figure 7 is a block diagram showing the overall processing steps of the system of the present invention, when the checkerboard background shown in Figure 6 is used;

Figure 8 is a schematic diagram graphically showing a video image in its various phases during the process shown in Figure 7;

Figure 8 is a flow diagram of the function for fuzzing the edge of an object of the control flow, shown in Figure 7;

Figure 9 is a flow diagram of the coarse and refined function for removing portions of the checkerboard background of the control flow of Figure 7;

Figure 10 is a flow diagram of the strip function defined as block 234 in Figure 3;

Figure 11 is a flow diagram of the sizing function defined as block 260 in Figure 3;

Figure 12 is a flow diagram of the skin match function defined as block 264 in Figure 3;

Figure 13 is a flow diagram of the background gamma function defined as block 266 in Figure 3; Figure 14 is a flow diagram of the pixel enhancement function defined as block 268 in Figure 3;

Figure 15 is a flow diagram of the position function defined as block 270 in Figure 3;

Figure 16 is a flow diagram of the layering function defined as block 274 in Figure 3;

Figures 17a and 17b are a flow diagram of the top-level gamma function defined as block 262 in Figure 3;

Figure 18 is a flow diagram of the fuzz function corresponding with block 530 in Figure 17; Figure 19 is a flow diagram of the blend function corresponding with block 540 in Figure 12; Figure 20 is a flow diagram of the change gamma functio corresponding with block 550 in Figure 17;

Figure 21 is a flow diagram of the transparency/opacit function corresponding with block 560 in Figure 12; and Figure 22 is a flow diagram of the gradient sharpenin function defined as block 446 in Figure 14.

Detailed Description of the Preferred Embodiment Referring initially to Figure 1, a presently preferre embodiment of a digital image editing system of the presen invention is generally indicated at 100. It is to b understood that the imaging system 10 can be used in virtuall any application where it is desirable to separate an objec from a background in a digital image, and then combine th object with a different background to form a composite image

One such application is shown in Figure 1, which show that the system 100 can be mounted by brackets 102 within a automated electrical imaging system 104. The apparatus 10 includes a video camera 106, such as a model VKC-360 camer available from Hitachi Corp., and which is electricall connected to the system 100.

Still referring to Figure 1, a video monitor or displa 110, such as a Kodak .model 1310 RGB/CGA touch screen display is mounted in the apparatus 104. The monitor 110 i electrically connected to the system 100 for displaying video image, such as the image of a person 112. As shown, th person 112 using the system (the user) in Figure 1 is sittin in front of a momochrome background 114, which can be an desired color. As shown in Figure 1, the apparatus 104 also includes payment-receiving device 116, such as a model OB-A4 devic made by Rowe. The presently preferred embodiment uses th device 116 to accept currency as payment. However, othe embodiments may accept other forms of payment, such as credi cards and tokens. Further, the apparatus 104 includes printer 118 which is filled with blank sheets of card stoc 120. Following deposit of an appropriate amount of currenc in the device 116, or following other initiation, the image of the person 112 can be printed on one of the sheets 120 and dispensed into a printer bin 122. In one presently preferred embodiment, the printer 118 is a model SV6510 color printer available from Kodak.

Referring now to Figure 2, the system 100 is shown to include various electronic components. As indicated in Figure 2, the system 100 includes a digital computer 130, preferably an IBM PC compatible having a 80386 microprocessor operating at 33 MHz and having eight Megabytes (Mb) of memory. As shown, the computer 130 is electrically connected to the video camera 106 and payment adapter 116 for receiving input signals therefrom. Also, the computer 130 is electrically connected to a suitable graphics or video interface card 132, preferably a Targa^®+ 16-32, available from Truevision having two Mb of video memory. Targa is a registered trademark of Truevision, Inc. The video memory on the graphics interface card 132 may store, at various times, a digital representation of part of the person 112 (Figure 1) , a background scene, and instructions screen information for the user. The graphics interface card 132 in turn is electrically connected to the touch screen video monitor 110. The user of the system 100 can respond to prompts given by the system by either touching or touching and then moving, (dragging) a location on the video monitor screen. A RS232 (serial format) digital connection from the video monitor 110 to the computer 130 then provides for the transfers of the user input to the system 100.

The computer 130 connects to the printer 118 via a Centronics compatible interface. The printer 118 is further connected to the video monitor 110 via an analog interface. Figure 2 also shows that if desired, an electronic storage device 134, such as a hard disk drive, can be connected to the computer 130. In the presently preferred embodiment, the hard disk 134 has a capacity of 120 Mb.

Now referring to Figures 1, 3a, 3b and 3c, the operation of one presently preferred embodiment of the imaging system 100 will be described. Figures 3a,b,c show the main or to level control flow for the system 100. For the specif embodiment shown in Figure 1, execution begins at a start st 202 of Figure 3a and proceeds to a step 204 wherein a set background scenes is displayed on the video monitor 1 (Figure 1) to attract the attention ("attract" mode) of potential user of the system 100. Periodically the comput 130 proceeds to a decision step 206 to check if someone h either touched the monitor screen 110 (Figure 1) at appropriate location or inserted payment into the payme adapter 116 (Figure 1) . If there is no user input at st 206, the computer 130 loops back to step 204 to continue t "attract" mode.

When there is user input at step 206, a decision step 2 then checks whether the user inserted payment into the payme adapter 116, in which case the computer 130 proceeds to decision step 210. At step 210, the payment is tested determine if it is the proper amount and is genuine. In t preferred embodiment, the system 100 accepts only a fi dollar bill. However, a configuration file (not shown) us by the computer 130 allows a technician to alter the type payment that is accepted, e.g., five one dollar bills. step 210, if the payment is rejected for any reason, t computer loops back to step 204 to begin again. If step 208 determines that the user payment has not be made, the computer 130 proceeds to step 212. At step 212, voice file from a sound board in the computer 130 and voi driver activate to instruct the user to insert five dollar Sound boards, such as Sound Blaster^® from Creative Labs Thunder Board™ from Media Vision are readily available. Sou Blaster is a registered trademark of Creative Labs, In Thunder Board is a trademark of Media Vision, Incorporate A video clip, having instructions on how to make payment, stored on the storage disk 134 (Figure 2) and is shown on t monitor screen 110 for about fifteen seconds. A check is ma at a decision step 214 during the fifteen seconds to determi if payment has been made. If not, the computer loops back step 204 to begin again. If payment is made at step 214, step 210 tests for correct payment. If the payment is correct, the computer moves to step 216 wherein a RS232 acknowledgement is sent to the computer 130. The main control flow leaves Figure 3a through the off-page connector A 218 and resumes again at step 220 in Figure 3b.

At step 220, live video with a digital mask 140 (Figure 4a) is shown on the video display 110. A voice file from the computer 130 activates to give the user instructions. Figure 4a illustrates what the user 112 may see on the video display 110. A box 142 delineates the edge of the digital mask 140 and defines a specific region where the head of the user 112 needs to be. Voice instructions prompt the user 112 to sit back or to raise or lower the provided stool (not shown) if the head is not within the box 142. When the user's head fits properly within the box 142, further processing is not necessary to shrink the head size. The digital mask 140 is a monochrome hue that is configurable by a technician.

Moving from step 220 to step 222 (Figure 3b) , the computer 130 activates the video camera 106 (Figure 2) and a video signal is captured representative of an object, e.g., the person or user 112, and a background against which the object was- imaged, e.g., the continuous hue background 114 (Figure 4a) . The computer 130 proceeds to a decision step 224 to determine if this is the first picture or image captured. If so, the computer 130 loops back to step 220 and step 222 to capture a second pose of the user 112. When step 224 determines that two images are captured, the computer 130 proceeds to step 226 wherein both video images are digitized. Of course, the system could be modified to receive and process only one or more than two images.

As shown in Figure 4a, only a portion 142 of the camera view is digitized for processing by the computer 130. Ordinarily, the signal from the camera 106 is an analog signal, and is digitized into a two-dimensional matrix of pixels or image 144 (shown in Figure 4b) by a video frame grabber on the Truevision Targa+ card 132. In the presently preferred embodiment, the matrix of pixels 144 is 512 X 486 which is a digital video rectangular standard, although othe matrix sizes can be used such as 640 X 480 or higher.

After both images have been digitized, the computer move to step 228 wherein both images are displayed on the vide display 110 (Figure 1) and a voice file instructs the user 11 to pick one of the two images to be used for the rest of th process and to touch that image on the.monitor screen 110 a step 230. The selected image 144 is stored in the vide memory of the graphics interface card 132 (Figure 2) . Afte the user 112 makes the necessary selection at step 230, th computer 130 proceeds to step 232. At step 232, a set o different background scenes are displayed on the video displa 110. For example, the background scenes could be of sport team members in various poses with their equipment, eithe individually or in groups. In the preferred embodiment, th computer 130 displays a preselected set of background scenes but in other embodiments, the user may be able to select different set to choose among, e.g., another sports team Other embodiments may allow the user to pick which person from a group of people in a scene, that the user would lik his picture to replace. This choice may be done by pointin to a person, pointing to the name or title of a person, o other ways of selecting the location for replacement. Th user 112 is prompted at step 232 to select one desire background scene by touching the appropriate scene on th video screen 110 (Figure 1) .

While the user is deciding which background scene t select, the computer executes a function 234 to strip portion of the image selected by the user at step 230. Function 23 will be explained in detail hereafter. When the user 11 touches the monitor screen 110 (Figure 1) at step 236 t select the desired background scene, an interrupt is sent t the computer 130 to denote that the selection has been made In the meantime, when the function 234 returns from execution the computer 130 moves to step 238 wherein the results o function 234, i.e., the image of the user's head and nec along with the stripped background pixels, is written to the video memory on the graphics interface card 132 (Figure 2) . When the interrupt indicating the selection of the background scene from step 236 is received by the computer 130, the control flow 200 continues at step 240.

At step 240, a personal computer keyboard and text entry screen along with instructions to the user 112 are displayed on the video monitor 110. A voice file from the computer 130 activates to give the user oral instructions. The user is prompted to enter text, such as the user's name or another name, to personalize the final composite image. While the user 112 is thinking about the text to enter, the computer 130 moves to step 242 where the background scene selected by the user at step 236 is sent to a frame store in a memory of the printer 118. As shown in Figure 4e, in the preferred embodiment, the background scene sent to the printer 118 will be missing the head and neck of a preselected person 145 in the original computer-stored scene. In other embodiments, the person in the original computer-stored background scene that is chosen to be replaced by the user (from multiple people in the scene) will be sent to the printer frame store without the head and neck. While the background is being written to the printer frame store at. step 242, the user can press locations on the touch screen monitor 110 to choose characters composing the desired text at step 244. A location on the touch screen monitor 110 is provided to indicate completion of the text upon which the computer proceeds to a decision step 246. At step 246, a check is done to determine if a valid text entry has been made. If not, the computer 130 loops back to step 242 where locations are provided on the touch screen monitor 110 to correct the text entry.

After a valid entry has been determined at step 246, the computer 130 moves to step 248 wherein the image of the face (Figure 4b) selected by the user 112 at step 230 is displayed on the video monitor 110 along with the original monochrome background present when the user's picture was taken. After completion of step 248, the main control flow 200 leaves Figure 3b through the off-page connector B 250 and resumes a step 252 in Figure 3c.

At step 252, the computer 130 draws a horizonta reference line on the video display 110 and prompts the use 112 to touch and drag (move while touching) the reference lin over the pupils of the eyes. At step 254 the user 112 center the horizontal line over the pupils and presses a butto location on the touch screen monitor 110 to signal completio of the step. Moving from step 254 to step 256, the compute 130 draws a reference cross on the video display 110 an prompts the user 112 to touch and drag the reference cross t the bottom of the chin and centered on the neck. At step 25 the user 112 centers the reference cross on the middle of th neck and bottom of the chin, and then presses a butto location on the touch screen monitor 110 to signal completio of step 258. The monitor 110 has a display similar to Figur 4g but including the monochrome background 114.

Upon completion of step 258 in Figure 3c, the compute calls a function 260 to match the size of the user's face, a determined by steps 252 through 258 above, to the size of th face to be replaced of the previously selected person 14 (from step 236) . The function 260 will be described belo after the remainder of the main control flow 200 is described After the function 260 returns, the computer calls a functio 262 to change various gamma values.

The overall gamma includes many attributes: hue (H) saturation (S) , lightness (L) , intensity (I) , contrast (C) red (R) , green (G) , blue (B) , and combinations thereof such a HSL, HSI, HSC and RGB. The top-level gamma function 262 ca change any combination of the gamma attributes by: pixel, are of the image or the entire image. For example, the user ca change the hue, saturation and intensity of an area of th image. Other operations done by the top-level gamma functio include: strip, fuzz, blend, transparency/opacity, and pixe enhancement. These functions, and the apparatus and metho for accomplishing them, are disclosed hereafter in thi document. For example, the user could choose to only enhanc the pixels of a particular hue of blue, and blend the pixels of a certain saturation level. Any combination of gamma attributes and operations could be done.

To more fully understand the application of the top-level gamma function 262, as well as other aspects of the invention, one should appreciate that the composite final image is handled by the computer 130 on a layer basis, where each pixel on a layer has X,Y Cartesian coordinates. Thirty two layers are used in the presently preferred embodiment, but in other embodiments the numbers of layers may be greater, with the maximum number only being limited by the size of the memory. The final composite image is viewed with the layers stacked on top of each other. The layer number provides a Z coordinate with the original background scene having a Z coordinate of zero. Objects in the original background scene can be chosen to have a higher or the same priority or similar and hence be assigned to a higher numbered layer and Z coordinate. Other objects, such as the image of the user 112, can be assigned a Z coordinate and be placed in front of or behind objects (depending on their Z coordinate) from the background scene which were previously moved from layer zero.

As an example, consider a final composite image having four layers (Z=0 to 3). as shown in Figure 4h. If a pixel 150 at a particular X,Y cartesian coordinate address of the top layer (Z coordinate = 3) has an attribute of transparent, then a pixel 151 at the same X,Y coordinate on the layer below (Z=2) will be seen if the attribute is opaque. However, if the pixel 151 of layer Z=2 is also transparent, then a pixel 152 at the same X,Y coordinate on layer Z=l will be seen if it has an attribute of opaque and so on. For a pixel 153 on the background or Z=0 layer to be seen, all pixels on higher numbered layers for that X,Y coordinate address must all have an attribute of transparent.

In further explanation of Figure 4h, several objects have been moved from the original background layer (Z=0) : two kneeling players 154 to the top layer (Z=3) and a hat 155 to the Z=l layer. The head of the user 112 is placed as an object 156 on the Z=2 layer. The person 157 in the backgroun scene whose head is to be replaced has the head area 158 se transparent. The final layered composite image 15 illustrates the user's head 156 wearing the hat 155 on th body of the person 157.

All the previously mentioned gamma attributes an operations can be done on a layer by layer basis. Fo example, the user can strip pixels of a particular hue of re from layer 1, 2 and 3, followed by sharpening pixels of certain value of intensity on layers 0 and 1, and finishin with fuzzing all pixels with a desired saturation level o layers 1 and 3. Details of the top-level gamma function 26 will be given below.

After a return from the function 262, the computer 13 calls the function 264 to match the skin of the user 112 t the skin of the selected background person 145 (Figure 4e) The function 262 will be described below.

After the function 264 returns, the computer 130 call the function 266 to match the gamma of the user image and th gamma of the selected background scene (Figure 4e) , a selected at step 236. The function 266 will be describe below.

In the presently preferred embodiment, after the functio 260 returns, the main control flow 200 will move to functio 268 and bypass functions 262, 264, and 266 (this bypass is no shown in Figure 3c) . However, it can be appreciated that th functions 262, 264, and 266, are included in othe embodiments.

After the function 266 returns, the computer 130 call the function 268 for pixel enhancement. The function 268 wil be described below.

After the function 268 returns, the computer 130 calls function 270 to position the object into the selecte background scene (Figure 4e) . The function 270 will b described below.

Upon return from the function 270, the computer 130 wil display, at step 272, a diversion screen, e.g., a team logo on the video monitor 110 along with a message that the final image will be ready soon. While the diversion screen is displayed, the computer 130 proceeds to step 274. At step 274, the computer 130 calls a function to layer the image of the user's head and neck into the background scene chosen at step 236. The function 274 will be described below.

After the function 274 returns, at step 276, the computer 130 sends the user's head and neck block, as processed by the layering function, to the frame buffer of the printer 118 in overlay mode. In overlay mode, the block currently sent to the frame buffer overwrites the previous information in the buffer starting at a defined location associated with the current block. The information previously in the frame buffer outside of the area for the block being overlaid remains unchanged. The final composite image is built up at the printer memory. Previously, the background scene was sent to the memory of the printer 118 at step 242.

Upon completion of step 276, the computer 130 moves to step 278 where the composite image, which is almost complete, is sent from the printer memory to the video display 110. The computer 130 then proceeds to call the layering function 274 again to layer the personalization text that the user entered at step 244. When the layering function 274 returns, the computer 130 proceeds to step 282 where the personalization text is sent in overlay mode to the frame buffer of the printer 118. At this point, the final composite is complete in the printer memory. The computer 130 moves to step 284 where the personalization text is sent from the printer memory to the video display 110 to show the final composite image to the user 112 for a few seconds. As the final composite image is being displayed, the computer 130 moves to step 286 where a voice file from the computer 130 activates to thank the user 112 for using the system 104. The computer 130 then signals the printer 118 at step 288 to print the final composite image on the printer card stock 120 and release the printed product into the printer bin 122. After the final composite image is printed at step 290, the main control flow 200 returns to ste 202 and begin the whole process again.

Referring now to Figure 5, another preferred embodimen of a digital image editing system of the present invention i generally indicated at 1010.

One application is shown in Figure 5 which shows that th system 1010 can be mounted by brackets 1014 within a automated postcard system 1012. Some of the followin features of system 1010 were also described in Figure l fo system 100. The apparatus 1012 includes a video camera 1016 such as a model VKC-360 camera manufactured by Hitachi Corp. and the camera 1016 is movably mounted on the apparatus 101 and is electrically connected to the system 1010. Also, th apparatus 1012 includes a handle 1018 that is connected to th camera 1016 for the purpose of manually orienting the camer 1016 in the appartaus 1012.

Still referring to Figure 5, a video monitor 1020, suc as a standard RGB/CGA display, is mounted in the apparatu 1012 and the monitor 1020 is electrically connected to th system 1010 for a displaying a video image, such as the imag of a model 1022. As shown, the model 1022 in Figure 5 i standing in front of a monochrome background 1024, which ca be any desired color..

As shown in Figure 5, the apparatus 1012 also includes currency-receiving device 1026, such as a model 0B-A4 devic made by Rowe. Further, the apparatus 1012 includes a printe 1028 which is filled with blank sheets of postcard stock 1030 Upon payment of an appropriate amount of currency in t device 1026, the image of the model 1022 can be printed on o of the sheets 1030 and dispensed. In one embodiment, t printer 1028 is a model SV6510 color printer made by Kodak.

Moreover, one or more controls 1032 are mounted on t apparatus 1012 and are connected to the system 1010. T control 1032 shown in Figure 5 is a button that provides means whereby the model 1022 can manually position and t model 1022 image onto the image of one of a plurality preselected backgrounds which is electronically stored in the system 1010.

Figure 6 shows that the system 1010 can be used in conjunction with a background which is not monochrome. More particularly, as shown in Figure 6, the system 1010 can be used with a background 1034 which comprises a monochrome (e.g., white) substrate 1036. A plurality of black boxes 1038 are printed on the substrate 1036 and, inside each square, a plurality of black dots 1040 are printed on the substrate 1036 in a predetermined pattern. As shown, the boxes 1036 establish a plurality of vertices 1042. In one presently preferred embodiment, each box 1036 has a side length of about 2 inches, and each dot 1040 has a diameter of about .025 inches. Thus, the background 1034 has a predetermined pattern, and the pattern is electronically stored in the system 1010. This background 1034 is also prestored in the system 1010.

Referring now to Figures 6, 7, and 8, the operation of the system 1010 when a checkerboard background 1034 is used can be appreciated. The operation of the computer 130 at blocks 1086 and 1088 in Figure 7 is essentially as previously described.

From block 1088, the computer 130 proceeds to block 1090. At block 1090, the computer 130, starting simultaneously at the upper left and lower right hand corner pixels of the digitized video image, performs a "coarse background strip". Specifically, the computer 130 compares the vertices 1042 of the stored image 1054 with a stored map of the background 1034 shown in Figure 6 and indicated at Roman numeral I in Figure 8. This stored background map is a map of the boxes 1038 and dots 1040 of the background 1034 in x,y, Cartesian coordinates. In other words, at block 1090, the computer 130 determines which portions of the image 1054 contain boxes 1038, and saves this portion as background. From block 1090, the computer 130 moves to block 1092, wherein the computer 130 removes portions of the image 1054 which match the background map stored in memory, based upon the comparison in block 1090. Specifically, at block 1092 the computer 130 removes those portions of the image 105 which form the edges of complete (i.e., four-sided) box 1038. Also, the computer 130 removes those portions of t image 1054 that are within complete imaged boxes 1038 (i.e. portions of the image of the white substrate 1036 and do 1040 that are within imaged boxes 1038) .

The specific coarse stripping operation of the comput 130 in block 1090 can be better seen with reference to Figu 9. Specifically, as indicated in decision block 1094 i Figure 9, the computer 130 determines whether the curre pixel, i.e., the test pixel, should be part of the edge of box 1038. As described previously, the computer 130 initiall selects the upper left and lower right pixels as the tes pixels. After performing the coarse strip test describe below on, for example, the upper left hand pixel, the compute 130 selects either the next pixel in the top row, if an ed was not detected, or the left-most pixel in the next low row, if an edge was detected, using the coarse strip functio described below. When proceeding from the lower right pixel the skilled artisan will appreciate that the sequence o selecting test pixels is a mirror image of that described fo the upper left pixel sequence. More specifically, after t lower right pixel has been tested, the computer 130 selec either the next pixel in the bottom row, if an edge was no detected, or the right-most pixel in the next higher row, i an edge was detected.

At block 1094, the computer 130 compares, with the tes pixel, the portion of the stored background map tha correlates to the test pixel, in x-y Cartesian coordinates

If this comparison indicates that the test pixel should be t same color as a box edge (i.e., that the test pixel should part of a box) , the computer 130 proceeds to decision blo 1096. At decision block 1096, the computer 130 determin whether the current pixel of the image 1054 is the same col as the edges of the boxes 1038, i.e., black. If so, t computer 130 proceeds to block 1098 to store the current test pixel as a background pixel. From block 1098, the computer 130 proceeds to block 1100, wherein the computer 130 selects the next pixel in the row of the current test pixel. If the computer 130 in decision block 1096 determines that the test pixel is not black, the computer 130 proceeds to block 1102, wherein the computer 130 maps the test pixel as an edge pixel, i.e. , identifies the test pixel as being part of the edge 1066 of the object 1022 of the digitized image 1054. From block 1102, the computer 130 proceeds to block 1100, and the next pixel is chosen as described above.

If, at decision block 1094 of Figure 9, the computer 130 determines that the current pixel should not be part of a box 1038, the computer 130 proceeds to decision block 1104. At decision block 1104, the computer 130 determines whether the test pixel is white, i.e., whether the test pixel is part of the white portion 1036 of the background 1034. If not, the computer 130 proceeds to block 1102 and operates as described above. Otherwise, from block 1104, the computer 130 proceeds to block 1098.

At this point in the process, the test pixel has either been recorded as a background 1034 pixel at block 1098 or an object 1022 pixel at block 1102. From block 1100, the computer 130 proceeds to decision block 1106, wherein the computer 130 determines whether the test pixel is beyond the last pixel in the row. If so, the computer 130 proceeds to block 1108, wherein the computer 130 selects the first pixel (i.e., left-most pixel for processes that started from the upper left corner and right-most pixel for processes that started at the lower right corner of the digitized image) pixel in the row immediately following the row that had been under test. Otherwise, the computer 130 loops back to decision block 1094 to process the new test pixel.

From block 1108, the computer 130 proceeds to decision block 1110, wherein the computer 130 determines whether the row which had been under test was the last row. Stated differently, at decision block 1110 the computer 130 determines whether the row which had been under test is t bottom row of the digitized image, if the particular proce is that portion of the processing that started at the upp left corner pixel of the digitized image. If, on the oth hand, the particular process is that portion of the processi that started at the lower right corner pixel of the digitiz image, the computer 130 determines whether the row that h been under test is the top row in the digitized image.

If the row of the current pixel is not the last row, t computer 130 loops back to decision block 1094 to process t new test pixel. On the other hand, if the computer 130, decision block 1110, determines that it has tested the la pixel in the last row, the computer 130 exits the coar background stripping function shown in Figure 9 and returns block 1112 in Figure 7.

Referring back to Figure 7 , in contrast to the coar background stripping subroutine described above, the proce undertaken by the computer 130 at blocks 1112, 1114 can thought of as a refined background stripping function. T image remaining after the computer 130 has performed t coarse background stripping defined in blocks 1090, 1092 depicted in Roman numeral III of Figure 8 , wherein only small portion of the background 1034 (i.e., that portion n within the images of complete boxes 1038) remains surroundi image of the model 1022. The small remaining portion of t background 1034 is removed by the processes undertaken blocks 1112 and 1114, wherein the remaining background ima dots are compared with the background map and removed from t digital image by the computer.130. The refined background stripping function of block 11 is more fully understood in reference to Figure 9. Starti at decision block 1116 of Figure 9, the computer 1 determines whether the current pixel should be a dot. It to be understood that in selecting the current pixel , t computer 130 undertakes a selection routine analogous to t routine described in the case of the coarse strip functio Specifically, the computer 130 starts at the upper left a lower right pixels of the remaining image 1054, and sequentially performs the test described below on a pixel-by- pixel, row-by-row basis.

Preferably, but not required, each dot 1040 consumes a single pixel in the digitized video image. To determine whether the current pixel should be a dot 1040, the computer

130 accesses the portion of the stored background map in its

•memory that corresponds to the position occupied by the current, i.e., test, pixel, to determine whether, based on the stored background map, the test pixel should be a dot. If the test pixel should be a dot according to the stored background map, the computer 130 proceeds to decision block 1118, wherein the computer 130 determines whether the test pixel is black.

If the test pixel is black or other hue determined by software, computer 130 proceeds to block 1120 and designates the test pixel as a background pixel to be removed later. From block 1120, the computer 130 proceeds to block 1122, wherein the computer 130 selects the next pixel in the row, i.e., the pixel that is immediately adjacent to the test pixel.

If, on the other hand, the computer 130 determines at decision block 1118 that the test pixel is not black, the computer 130 proceeds, to block 1124 and designates the test pixel as an object edge pixel. From block 1124 the computer 130 proceeds to block 1122 and selects the next pixel.

If the computer 130 at block 1116 determines that the current pixel should not be a dot, the computer 130 proceeds to decision block 1126, wherein the computer 130 determines whether the test pixel is white. If not, the computer 130 proceeds to block 1124 to designate the pixel as an object edge pixel. Otherwise, the computer 130 proceeds to block 1120 and proceeds as previously discussed to block 1122.

From block 1122, the computer 130 proceeds to decision block 1128, wherein the computer 130 determines whether the new test pixel is beyond the last pixel in the row of the remaining digitized image. If the test pixel is not beyond the last pixel in the row, the computer 130 loops back to decision block 1116. Otherwise, the computer 130 proceeds block 1130 wherein the computer 130 moves to the next row the remaining digitized image and selects the first pixel that row. From block 1130, the computer 130 proceeds to decisi block 1132 to determine whether the test pixel is the la pixel in the last row of the remaining digitized image. not, the computer 130 moves back to decision block 111 Otherwise, the computer 130 exits the function of Figure 9 a returns to block 1114 in Figure 7.

At block 1114, the computer 130 removes portions of t image 1054 which are represented by pixels that we designated as background pixels in block 1112. After t remaining background has been removed, the image 1054 appea as shown in the diagram designated by Roman numeral III Figure 8.

The computer 130 proceeds to block 1134 in Figure wherein a background 1133 against which the operator (e.g the model 1022) of the system 1010 desires to place the ima of the object which can be selected from the memory of t computer 130. This background selection can be made appropriately operating the control 1032 so that the operat can interactively choose the background against which t operator wants the image to be positioned, or the backgrou can be selected automatically by the computer 130.

From block 1134, the computer 130 proceeds to block 11 in Figure 7 wherein the operator of the system 1010 can orie the object as desired on the chosen background by viewing t composite image on the monitor 1020 and appropriate manipulating the control 1032 (Figure 5) . The operator c also position the object as desired on the background, i.e can superimpose the object on any portion of the background desires. This orientation can be accomplished by contro (not shown) or can be accomplished automatically by t computer 130.

Referring again to Figure 7, the computer 130 proceeds block 1138 wherein the computer 130 fuzzes the edges of t object as described with respect to Figure 8. From block 1138, the computer 130 proceeds to block 1140. At block 1140, the computer 130 blends the edges of the object 1022 with the surrounding background to sharpen the edge of the object as shown schematically at Roman numeral IV in Figure 8.

The blending function 1140 is more fully understood by reference to Figure 19. The computer 130 begins at block 1142 in Figure 19 wherein the computer 130 selects an object edge pixel and determines the hue gamma value of the edge pixel. The selection of the object edge pixel can be made at a software determined address, e.g., the upper left edge pixel. From block 1142, the computer 130 proceeds to block 1144 and determines the hue of the background pixel that is immediately adjacent to and below the selected edge pixel in the same row as the edge pixel. Then the computer 130 proceeds to block 1146 wherein the computer 130 determines the average hue of the two pixels. From block 1146, the computer 130 proceeds to block 1148, wherein the computer 130 sets the hue of the edge pixel to equal the average hue that was determined in block 1146.

From block 1148, the computer 130 proceeds to block 1150, wherein the computer 130 selects the edge pixel that is immediately adjacent to the pixel whose hue has just been set at block 1148, and loops back to block 1142. If the pixel under test was the last edge pixel to be blended with the background, the computer 130 exits the function shown in Figure 14 and proceeds to block 1152 in Figure 7.

At block 1152 of Figure 7, the computer 130 varies the hue gamma value of selected portions of the object of the video image with the hue gamma value of selected portions of the background in the composite video image to make it appear as though the object were imaged under the same lighting condition as the background against which the object has been digitally superimposed. The details of the process carried out by the computer 130 at block 1152 is more fully understood in reference to Figure 20. Starting at block 1154 in Figure 20, the computer 1 measures the hue gamma of the object 1022 (Figure 6) at software determined address within the object. The softwa determined address can be a predetermined portion of t object such as the hand, face or clothing of the person th was imaged, and can be defined in terms of relati coordinates. From block 1154, the computer 130 proceeds block 1156 wherein the computer 130 measures the hue gamma the background at a predetermined software determined addres Next, the computer 130 proceeds to block 1158 wherein t computer 130 chooses which lighting condition is desired. other words, the computer 130 at block 1158 determines wheth the hue gamma of the object will be changed to match the h gamma of the background or whether the hue gamma of t background will be changed to match the hue gamma of t object. This determination by the computer 130 can accomplished in response to an interactive command by t operator of the system 1010, i.e., the operator of the syst 1010 can decide whether he wishes to change the lighti condition of the background to match the lighting conditio under which he was imaged, or whether he wishes to change t apparent lighting conditions under which he was imaged match the lighting conditions of the background. Furthermor the apparent lighting conditions of the object and backgrou can both be changed to match a third lighting condition desired.

From block 1158, the computer 130 proceeds to decisi block 1160 wherein the computer 130 determines whether t object hue is to be changed. If so, the computer 130 procee to block 1162 wherein the computer 130 sets the gamma of t object to equal the background hue gamma. Otherwise, t computer 130 proceeds to block 1164 wherein the computer 1 sets the gamma of the background object to equal the gamma the object. Now referring to Figure 10, the function 234 (Figure 3 for stripping portions of an image will be described. T function 234 starts and moves to a step 310 wherein t computer 130 selects the upper left and lower right pixels of the video image 144 passed to the function as shown in Figure 4b. It is to be understood, however, that the principles of the process described below, which is executed by the computer 130, can be applied when selecting only a single pixel as a starting point or when selecting more than two pixels as simultaneous starting points. For example, as shown at Figure 4c, the computer 130 could execute the process described below starting at all four corners of the image 144 simultaneously. The process described below can be coded in any suitable computer language, e.g. C.

Accordingly, for disclosure purposes, the process below will be described with reference to use of the upper left pixel of the digitized video image 144 as the starting point. It will be appreciated that the principles of the process will be substantially identical for other starting points, e.g., the lower right pixel of the video image 144, including instances wherein the computer 130 starts at two or more points simultaneously. The control of the parallel processing for multiple processing regions is known to those skilled in the technology.

Step 312 indicates that the computer 130 compares the gamma of the upper left pixel (target pixel) with the gamma of the next pixel in the top row of the video image 144 (Figure 4b) . As is used irt the technology, the gamma of a particular pixel is a numeric value that refers to one or more of the particular data attributes which characterize the pixel. For example, some video image pixels have attributes that relate to the hue, intensity, lightness, saturation, and contrast of the portion of the image represented by the pixel. Accordingly, such pixels will have respective numeric "gammas" that represent each one of the above-listed attributes of the particular pixel.

For purposes of the present invention, the gammas that are compared between pixels are the hue gammas of the pixels, although other gammas may be used. In the present embodiment, the hue gamma of each pixel is an integer value from zero (0) to two hundred fifty-five (255) , with the hue gamma val indicating the hue of the portion of the image represented the pixel. In the event that the video image is in black-an white, the hue gamma of a pixel will represent the shade gray of the pixel, or a gray scale value.

It will accordingly be appreciated that in the case the continuous hue background 114 (Figure 4b) , adjace background pixels will have substantially identical hue gam values, with the particular value depending on the color the background 114. For example, if the background w defined as 255, e.g., saturated blue, then the backgrou pixels would typically not deviate more than 20 to 40. Thu the computer 130 assumes that the upper left pixel of t video image 144 is a background pixel, and uses this pixel a standard pixel. The computer 130 compares its hue gam with the hue gamma of the immediately adjacent pixel (whi can accordingly be considered the test pixel) that is in t same row as the target pixel, to determine whether t immediately adjacent pixel (i.e., test pixel) is also background pixel. This step is represented at a decision st 314 in Figure 10.

More specifically, as indicated at decision step 314, t computer 130 compares the difference between the hue gammas the upper left corner pixel (i.e., the target pixel) and t immediately adjacent pixel (i.e., the test pixel) with predetermined difference. When the predetermined differen is greater than the difference between the hue gammas of t two pixels, indicating that the test pixel has approximate the same hue as the target pixel and is therefore flagged a background pixel to be acted upon later, the computer 1 proceeds from step 314 to a decision step 316 in Figure 1 At step 316, the computer 130 determines whether the te pixel is the last pixel in the row. If there are more pixe in the row, the computer 130 advances to step 318, wherein t computer 130 sets the old test pixel to be the new targ pixel, and selects the next pixel in the row as the new te pixel. The computer 130 then continues to step 312 determine the difference between the hue gamma of the target pixel and the hue gamma of the test pixel, as described above, and compares this difference to the predetermined difference at decision step 314. On the other hand, when the computer 130 determines that the test pixel is the last pixel in the row at decision step 316, the computer 130 proceeds to a decision step 320 that determines whether the last row of the image 144 (Figure 4b) has been reached. Stated differently, at decision step 320, the computer 130 determines whether the row which had been under test is the bottom row of the digitized image, if the particular process is that portion of the processing that started at the upper left corner pixel of the digitized image. If, on the other hand, the particular process is that portion of the processing that started at the lower right corner pixel of the digitized image, the computer 130 determines whether the row that had been under test is the top row in the digitized image. If not, the computer 130 proceeds to step 322 wherein the computer 130 sets the target pixel to be the last pixel in the row and the test pixel to be the first pixel in the next immediately lower row. The computer 130 then loops back to step 312 to begin comparing the pixels of the next row. The computer 130 resumes the gamma test comparison described above. Thus, the computer 130 reads and tests the pixels of the background component of the video image 144 (Figure 4b) in a pixel-by-pixel, row-by-row sequence.

When the computer 130 determines, at decision step 314, that the difference between the target pixel and the pixel under test exceeds the predetermined difference, indicating that the test pixel is not a background pixel and is therefore representative of an edge 146 of the image of the user 112 (Figure 4c) which has been imaged against the monochrome background 114 (Figure 1) , the computer 130 stores the location of the test pixel in memory. Stated differently, the computer 130 maps the test pixel as a portion of the edge 146. As indicated in Figure 5, the computer 130 then proceeds to step 320 and resumes processing as described above. The computer 130 follows the process described abov working simultaneously from the upper left and lower rig hand corner pixels, until the computer 130 determines that more pixels remain to be tested, as indicated by a positi test at decision step 320. At the point wherein no mo pixels remain to be tested, the computer 130 moves to st 324, wherein the computer 130 "floods" the background (i.e. turns all pixels that were outside the edge 146 transparen by setting the hue gamma of the background pixels to zero (0) In other words, the computer 130 removes portions of the ima 144 (Figure 4b) which are represented by pixels that we designated as background pixels 114 in step 314. Alternatively, each background pixel could be turn transparent as soon as the computer 130 determines that t particular pixel is indeed a background pixel. The comput 130 then moves to step 326 and returns to the calling progra At this point in the process, the digitized video image 1 appears as schematically shown at Figure 4d.

Thus, the computer 130 effectively removes substantial only the portion of the video image 144 that is outside t edge 146 (i.e., the background component of the image) , a leaves intact the entire portion of the image 144 that inside the edge 146 (i.e., the object component of the image) Consequently, portions of the image 144 inside the edge 1 can have the identical hue as the background 114, witho being removed from the image 144. Further, no particula predetermined hue is required to be used as the backgrou hue, in contrast to the chroma-key technology. Instead, a hue may be used as the background hue. From the disclosure above, it will be appreciated th the predetermined hue gamma difference is selected to sufficiently large to avoid edge mapping due to a backgrou test pixel that has a hue gamma which is marginally offs from the hue gamma of the target pixel. On the other han the predetermined hue gamma difference is selected to sufficiently small to accurately detect the presence of object edge pixel, even when the edge pixel has a hue gam value that is relatively close to the value of the background hue gamma. The precise value of the predetermined hue gamma difference can vary from application to application, depending upon lighting conditions, location, subject matter, etc. In one presently preferred embodiment, the predetermined hue gamma difference is 20.

Referring to Figure 11, the function 260 (Figure 3) for matching the size of the object to the size of the selected background object will be described. In the preferred embodiment, the object is the head and neck of the user 112, and the selected background object is the head and neck of the background person 145 (Figure 4e) that will be replaced.

The function 260 starts and moves to a step 342 wherein the computer 130 captures the face height of the user 112. Referring to Figure 4f, a portion of the background scene chosen by the user at step 236 (Figure 3b) is shown, including the background person 145 whose face and neck (shown by dashed lines) is to be replaced. A horizontal line 160 is centered over the pupils of the eyes at a coordinate y₀. A cross symbol 162 is centered over the middle of the neck and at the bottom of the chin at a coordinate y,. A line 164 demarcates the edge of the clothing, e.g., a uniform, from which point downward the user's neck is assigned a lower priority layer than the uniform. Thus, the uniform may be seen to cover part of the user's neck in the final composite image. The face height is the absolute difference between the two coordinates y₀ and y₁. The face height information of the background person, or for each background person in background scenes where a choice of multiple people is given, has been previously calculated and stored in a file header for a file that contains the image of the background scene. The file header format for the presently preferred embodiment is encrypted and is unique to this embodiment. The header is encrypted by XORing with a pseudo-random function. The file header contains RGB information, location of the eyes, location of the chin (cross symbol) , the face height, and so on to define the characteristics of the face or object in file image.

Referring to Figure 4g, an image of the user's face neck is shown. A horizontal line 170 is centered over pupils of the eyes at a coordinate y₀' . A cross symbol 172 centered over the middle of the neck and at the bottom of chin at a coordinate y₁' . The user's face height is absolute difference between the two coordinates y₀' and y₁ The user's face height information is stored in a file hea for a file that contains the image of the user's head neck.

Now returning to Figure 11, after the completion of s 342, the computer advances to step 344. At step 344, computer 130 compares the user's face height captured at s 342 to the face height of the background person at location chosen by the user at step 236 (Figure 3b) . If, a decision step 346, the face height of the user is le e.g., the user 112 is a child, the computer moves to a s 348. At step 348, the computer 130 will proportionately s up the image of the user's face and neck until the face hei of the user equals the face height of the background pers and then returns at step 352. If, at a decision step 346, face height of the user 112 is greater than that of background person, the computer 130 moves to a step 350. step 350, the computer 130 will proportionately size down up the image of the user's face and neck, until the f height of the user equals the face height of the backgro person, and then returns at step 352. However if, at decision step 346, the face height of the user 112 is equal that of the background person, the computer 130 returns step 352 to the calling main flow 200 (Figure 3) .

Referring to Figure 12, the function 264 (Figure 3c) matching the object or user person's skin to the skin of selected background person 145 (Figure 4e) will be describ The function 264 starts and proceeds to a step 370 wherein area of the background person, that is to be replaced in composite image, is assigned to be measured. Moving to s 372, the computer 130 measures three gamma attributes in the assigned area: overall hue, overall saturation, and overall intensity. Next, at step 374, the computer 130 assigns an area of the user's skin to be measured. The computer 130 knows where the eyes of the user 112 are on the image shown in Figure 4g because of the line 170. The computer assigns the area on the forehead of the image of the user 112 just above the eyes. This area is then measured at step 376 for the same attributes: hue, saturation, and intensity. Proceeding to step 378, the computer 130 then determines the skin areas of the image of the user's face by using the attributes measured at step 376, which describe the user's skin, to be compared against upon calling the strip function 234. The function 234, in this call, will flag the skin pixels of the image of the user's face. The hair, eyebrows, eyes, lips, beard and/or mustache (if present) , and so forth will not be flagged. Then, upon return from the function 234, the computer 130 proceeds to step 10382. At step 10382, the flagged skin pixels identified by the function 234 are set to the values of the attributes measured at step 372 of the background person's skin, and then the function 264 returns at step 10384 to the calling main flow 200 (Figure 3) .

Referring to Figure 13, the function 266 (Figure 3c) for matching the object or user gamma to the gamma of the selected background scene (Figure 4e) will be described. The function 266 can make it appear as though the object were imaged under the same lighting conditions as the background into which the object will be digitally layered. The function 266 starts and proceeds to a step 400 wherein the computer 130 determines the type of lighting used for the background scene (Figure 4e) . The lighting conditions are determined by measuring the gamma attributes of hue, saturation, and contrast at a software determined area of the background scene. Moving to step 402, the computer 130 measures the same gamma attributes as at step 400, but for the user image 144 (Figure 4d) .

Next, the computer 130 proceeds to step 404 wherein the computer 130 chooses which lighting condition is desired. In other words, the computer 130 at step 404 determines wheth the hue gamma of the object will be changed to match the h gamma of the background or whether the hue gamma of t background will be changed to match the hue gamma of t object. This determination by the computer 130 can accomplished in response to an interactive command by the us of the system 100, i.e., the user of the system 100 can deci whether he wishes to change the lighting condition of t background scene to match the lighting conditions under whi he was imaged, or whether he wishes to change the appare lighting conditions under which he was imaged to match t lighting conditions of the background scene. Furthermore, t apparent lighting conditions of the object and backgrou scene can both be changed to match a third lighting conditi if desired (this choice is not shown in Figure 8) .

From step 404, the computer 130 proceeds to a decisi step 406 wherein the computer 130 determines whether t object gamma is to be changed. If so, the computer 13 proceeds to step 408 wherein the computer 130 sets the gam attribute values for hue, saturation, and contrast of t object to equal the background scene gamma attribute valu for hue, saturation, and contrast. Otherwise, the comput 130 proceeds to step 410 wherein the computer 130 sets t gamma of the background scene to equal the gamma values of t object. Upon completion of either step 408 or step 410, t computer 130 proceeds to step 412 and returns to the ma calling flow 200.

Referring to Figure 14, the function 268 for enhanci the pixels of an image will be described. The function 268 called by either the main flow 200 (Figure 3c) or the to level gamma function 262 (Figure 17) . The function 268 star and proceeds to a step 420 wherein the computer 130 begi processing the image at a predetermined location, e.g., X, coordinates 0,0. Moving to step 422, the computer 1 determines the hue value of each pixel on the current row attempt to locate an edge of the object in that row. If edge is found, as determined by a decision step 424, t computer 130 proceeds to a step 426. The edge is located if the computer 130 determines that the hue of the pixel has a major change compared to the previous pixel. In the preferred embodiment, such a change would be found if the value of hue changed by an amount of 80 or more, where the full range of hue values runs from 0 to 255. Of course, the change threshold could be any value within the above-indicated range, as selected for the particular application of the system.

At step 426, the computer 130 selects the three pixels just outside the edge of the object and uses them as edge pixels in performing a blend operation. The blend operation is performed by the computer 130 in steps 428, 430, and 432 wherein the aforementioned three pixels are blended to the background layer directly below the current object layer. At step 428, for each of the three pixels, the computer 130 determines the hue of the background pixel in the layer beneath the object pixel and then averages the hue value of each of the three background pixels with the corresponding object pixel at step 430. Moving to step 432, the computer 130 sets the transparency or opacity of each of the three object layer edge pixels according to a blend factor that is found in the file header. The blend factor, having a value from zero (opaque) to 127 (transparent) , is used for dynamic variable transparency of the object, and is selected from within this range of values by the programmer using trial and error in view of what the results are desired to be. Upon completion of step 432, the computer moves to a step 434, wherein the computer 130 uses the edge found on the current row, as determined at step 422, and selects the two pixels inside the object edge for a fuzz operation. The fuzz operation is performed by the computer 130 in steps 436, 438, and 440 wherein the edge of the object is smoothed. At step 436, for each of the two aforementioned pixels, the computer 130 determines the hue of the three pixels immediately adjacent them on the same row. Then, as indicated at step 438, the computer 130 determines the average hue value of the three-pixel interval. Moving to step 440, the computer 130 sets the hue value of each of the two aforementioned pix equal to the average determined in the previous step 438.

Upon completion of step 440, the computer loops back step 422 to attempt to locate another edge in the current r If another edge is found, as determined at the decision s 424, the previously described process is performed. Howev if the end of the current row is reached without locating edge, the computer 130 moves to a decision step 1302 determine whether the last row of the image has just b processed. If not, the computer 130 proceeds to step 1304 selects the next row for processing and then loops back step 422 to locate an edge. However, if the last row has b processed, as determined by decision step 1302, the compu calls a gradient sharpening function 1306 to enhance object edges.

The file header has predetermined values for the amo of sharpening to be done. For example, the image could h no sharpening done on the edges but full sharpening done the center, 100% with a linear scale in between the center borders. Other percentages can be used as a sharpeni factor. For example, the image can have 30% sharpening left edge, 20% at center and 80% at right edge with a lin scale in between the center and borders. The function 13 will be described hereinbelow. After the function 1306 h returned back to the function 268, the computer 130 exits pixel enhancement function 268 at a step 1308.

The details of the process carried out by the compu 130 at function 270 of Figure 3c are more fully understood reference to Figure 15. The function 270 for positioning object or user 112 into the selected background scene (Fig 4e) will be described. The function 270 starts and proce to a step 460 wherein the computer 130 gets the address of cross symbol 172 (Figure 4g) on the user image. This sym is at the middle of the neck and bottom of the chin on user image as placed by the user 112 at step 258 of Figure 3 The address of the cross symbol 172 is retrieved from the f header of the user image file. The computer 130 then proce to step 462 and retrieves the address of the cross symbol 162 (Figure 4f) for the selected person 145 of the background scene. This address is retrieved from the file header of the background scene image file. The computer then sets the address of the user's cross symbol 172 equivalent to the address of the cross symbol 162 of the selected person 145 in the background scene. Therefore, when the composite image is layered together, the image of the user 112 will be at the correct location. The computer 130 returns at step 464 to the calling main flow 200 (Figure 3) .

The details of the process carried out by the computer 130 at function 274 of Figure 3c are more fully understood in reference to Figure 16. The function 274 for layering an object or personalization text into the selected background scene (Figure 4e) will be described. The function 274 is called twice by the main control flow 200. The first call in the preferred embodiment is to layer the image of the user's head and neck, such that user image will result being slid down, for example, a jersey or uniform, in the final composite image. The jersey has a higher priority than the user image, such that the user's neck and head become visible at the top of the jersey. Both the jersey and the user image have a higher priority than the background scene..

The second call in the preferred embodiment is to layer the personalization text which the user has entered at step 2130 (Figure 3b) . The text has the highest priority and, therefore, will always be visible in the final composite image.

The function 274 starts and proceeds to a decision step 480 wherein a determination is made whether this routine is a function call, as in the preferred embodiment, or is a user call, for an alternate embodiment. In the preferred embodiment, the computer 130 proceeds to step 482 wherein the item, e.g., text or an object, or the area of an image to be layered is identified in the argument for the function call. The computer 130 then accesses the information corresponding to the item or area and proceeds to step 488. If however, at step 480, the call to this routine is a user call, th computer 130 proceeds to a step 484. At step 484, an area o object in the background scene can be defined by the user 11 to be moved to another layer. At step 486, the selected are or object is stored to memory along with information for name layer, position within the layer, and alpha-bit flag used t set transparency bits.

Upon completion of either step 486 or 482, the name layer, position within the layer, and alpha-bit flag, al corresponding to the item or area, are written to a linke list by the computer 130 at step 488. The alpha-bit fla denotes transparency or opacity of the item or area. Th position is stored in X,Y coordinates.^' After completio of step 488, the computer 130 moves to a decision step 490 an checks to see if an option is invoked, in an alternat embodiment, to stitch, or link together, multiple objects o the same layer. If so, at option step 492, the compute stitches the objects desired on the same layer together int a layer collection. After completion of option step 492 o if the decision step 490 is false, the computer 130 moves t step -494. At step 494, the computer 130 invokes the linke list and looks for the transparency alpha-bit flags. At ste 496, for the item pointed to by the linked list, the compute 130 displays the object or area defined by name, layer, an position within the layer on the monitor 110 (Figure 2) .

In an alternate embodiment, a check is made at an optio decision step 498 to determine if the user 112 (Figure 1 desires to reset the transparency alpha-bit flag. If so, th computer moves to option step 500 wherein the transparenc flag is reset to opaque by use of a pointing tool, e.g. mouse, or by area definition using X,Y cartesian coordinates Upon completion of step option 500 or if the decision step 49 is false, the computer 130 moves to an option decision ste 502. In an alternate embodiment, step 502 determines if th user 112 wants to stitch an object or objects on one laye together with the background. If so, at option step 504, th computer 130 stitches the objects on one layer together wit the background into one collection and then the objects are removed from the linked list. The collection may include objects, an area, backgrounds, text, and so forth. In the presently preferred embodiment, a file format for storing the background, or the collection as a new background, has a file extension ".spo". At the completion of option step 504 or if decision step 502 is false, the computer 130 returns at step 506 to the calling main flow 200 (Figure 3) .

Referring to Figures 17a and 17b, the top-level gamma function 262 (Figure 3c) will now be described. Although the preferred embodiment disclosed herein does not include the ^■tebp-level gamma function 262, an alternate embodiment may include this function. The function 262 may be utilized in two ways: as a function call, wherein attributes to be changed are preselected, or as a user call, wherein attributes to be changed are selected by the user 112. When function 262 is accessed by a user call, the system 100 operates as a highly interactive and very powerful image editing tool.

The function 262 starts and moves to a decision step 514 to determine if the routine 262 is called as function. If so, the computer 130 proceeds to a step 516 to access preset gamma attributes, layer, and object or area selections, e.g., all pixels on layer one with a hue of blue at a value range of 10 to 75. The computer 130 then moves from step 516 and proceeds through the off-page connector C 580 and resumes again at step 582 on Figure 17b. However, if the routine 262 is a user call as determined at step 514, the computer 130 proceeds to a decision step 520 wherein the computer 130 checks to determine if a strip operation is to be done. The strip operation in this function may be done on an object or on the background scene. If so, the computer 130 proceeds to step 522 wherein the user 112 selects the gamma attributes, with corresponding values, and/or a layer and an object or an area on the layer to be changed. At the completion of step 522, the computer 130 proceeds to step 524 and enters the selections from step 522 into a matrix. At a decision step 526, the computer 130 checks to see whether the user 112 wants to undo the operation selected at step 520. If so, at step 528, the computer 130 will negate the operation selected at step 520.

The x-axis of the matrix mentioned above includes the gamma attributes: hue (H) , saturation (S) , lightness (L) , intensity (I) , contrast (C) , red (R) , green (G) , blue (B) , and the combinations HSL, HSI, HSC and RGB. The y-axis of the matrix includes the operations done by the top-level gamma function: strip, fuzz, blend, change gamma, transparency/opacity, and pixel enhancement. Associated with the matrix is a z-axis that corresponds to the layer number of a composite image. In the presently preferred embodiment, the z-axis utilizes 32 layers. The matrix permits one or multiple operations on one or multiple gamma attributes to be performed on one or multiple layers of the composite image in any combination.

After completion of step 528, or if decision step 526 result is false, or if the decision step 520 result is false, i.e., strip was not selected, the computer 130 proceeds to a decision step 530 wherein a check is made to determine if a fuzz operation is to be done. The fuzz operation may be done on an object or on the background scene. If so, the computer moves to a set of steps essentially the same as steps 522 through 528 as described above and then proceeds to a decision step 540. If the check to determine if the operation is to be done at step 530 (and steps 540, 550, and 560 below) is false, the computer 130 continues to the next decision step to test for the next possible operation.

At step 540, the computer 130 checks to determine if a blend operation is to be done. The blend operation in this function may be done on the background scene or on an object on the adjacent layer. If so, the computer moves to a set of steps essentially the same as steps 522 through 528 as described above and then proceeds to a decision step 550. At step 550, the computer 130 checks to determine if a gamma change operation is to be done. The gamma operation in this function may be done on an object and/or the background scene. If so, the computer moves to a set of steps essentially the same as steps 522 through 528 as described above and then proceeds to a decision step 560.

At step 560, the computer 130 checks to determine if a transparency/opacity operation is to be done. The transparency/opacity operation in this function may be done on the background scene or on an object on the adjacent layer. If so, the computer moves to a set of steps essentially the same as steps 522 through 528 as described above and then proceeds to a decision step 570.

At step 570, the computer 130 checks to determine if a pixel enhancement operation is to be done. The pixel enhancement operation in this function may be done on an object and/or the background scene. If so, the computer moves to a set of steps essentially the same as steps 522 through 528 as described above and then proceeds to step 582 via the off-page connector C 580 to Figure 17b. If the pixel enhancement operation was not selected at step 570, the computer 130 also proceeds to step 582 via the off-page connector c 580.

The next three steps (582, 584, 586) of function 262 are recursively performed by the computer 130 for each preset (from step 516) or selected operation (from steps 520 to 570) in preparation for step 590. At step 582, the computer 130 performs a layer limit address test. In the presently preferred embodiment there are 32 layers. The computer 130 tests to see if it is at the top layer (Z=3l) or at the bottom layer (Z=0) . Moving to step 584, the computer 130 prioritizes the layers and operations . The first operation that is prioritized is transparency/opacity. Layer zero (Z=0) is the original background scene. As an example, for an area on layer zero to be opaque, nothing in the same area on the 31 layers above layer zero can be opaque, so therefore that area would be set transparent in each of the 31 layers. As another example, if layer seven (Z=7) is made semi-transparent, then all the layers above layer seven (Z=8 to 31) must be transparent, and the layers below (Z=0 to 6) layer seven are adjusted in transparency/opacity to account for the sem transparency of layer seven. In another example, the t layer (Z=3l) is made opaque, so nothing needs to be changed the other 31 layers. The other preset or selected operatio are then also prioritized. At step 586, the computer 1 looks for the layer that is affected for each preset selected operation. There could be six different laye corresponding to the six operations from steps 520 to 570 the priority list. The steps 582 through 586 are recursive performed until the layer limit address test is satisfied f each operation.

At step 590, the computer 130 goes through a process schedule the execution of the operations selected in steps 5 through 570, or from step 516, as described above, on a lay by layer basis. At a decision step 592, the computer 1 checks to see if the operation scheduled for execution is t strip operation. If so, at step 594, the computer 1 performs the strip on a layer (chosen for execution by st 590) using the method that starts at the four corners of t layer and advances through the rows, as previously describ in conjunction with function 234. If the test at step 592 false (not strip), one operation, e.g., fuzz, as scheduled the computer 130, is- executed on a layer (chosen for executi by step 590) using the row by row method of execution. Aft all the rows of a layer have been processed at step 596, t computer 130 advances to a decision step 598 and chec whether more layers are to be processed. If so, the comput 130 loops back to step 590 to schedule the next selected lay with the same operation as previously executed, e.g., fuz One operation is done on all desired layers before moving a decision step 600 wherein a check is made to determine additional operations are to done. If so, the computer 1 loops back to step 590 wherein the next selected n operation, e.g., blend, is scheduled to be executed. If a operations are complete at step 600, the computer returns step 602 to the calling main flow 200 (Figure 3) . The top-level gamma function 262 can be used to create interesting composite images. As an example, the user desires the image of his head to replace the image of the head of a hockey player on a hockey playing card. The hockey player has a big scar on his face which the user wants to have on the image of his face in the resultant composite image. Using the top-level gamma function 262, the image of the scar can be isolated and made opaque. The rest of the image of the hockey player's head is made transparent. The image of the user's head is made opaque except for the area where the scar is, which is made transparent. In the resultant composite image playing card, the scar appears on the image of the user's head which is on the image of the hockey player's body.

Four functions execute the operations as selected by steps 530, 540, 550 or 560 of the top-level gamma function 262 (Figure 17) . The actual scheduling and call for execution of these functions is carried out by steps 590 and 596. However, for sake of reference, the operation selected by step 530 is referred to as a fuzz function 530' , the operation selected by step 540 is referred to as a blend function 540', the operation selected by step 550 is referred to as a change gamma function 550' , and the operation selected by step 560 is referred to as a transparency/opacity function 560' . These four functions will be described hereinbelow. The operation selected by step 520 is performed by the strip function 234 as described in conjunction with Figure 5. The operation selected by step 570 is performed by the pixel enhancement function 268 as described in conjunction with Figure 14.

After the background component 114 of the digital image 144 (Figure 4b) has effectively been removed, the component representing the image of the user 112 remains (Figure 4d) , and has a relatively "fuzzy" edge, characterized by a pixel width of about one or two pixels. Accordingly, to produce a digital image having a smooth edge the digital computer 130 executes the fuzz function 530' as shown in Figure 18. The fuzz function 530' may be called by either the pixel enhancement function 268 (Figure 14) or the top-level gamma function 262 (Figure 17). The function starts at a step 530 and proceeds to a step 680 wherein the computer 130 select one of the edge pixels (i.e., a "test" pixel) by using software determined address (e.g., by selecting the uppermos left pixel) and determines its hue at step 682. Next, a indicated at step 684, the computer 130 selects the edg pixels which are immediately adjacent the test pixel for • three-pixel interval on one row and determines the hue o these pixels. Then, as indicated at step 686, the computer 13 determines the average hue gamma value of the three pixels As indicated at step 688, the computer 130 then sets the hu gamma value of the test pixel to be equal to the average valu calculated in step 686. The computer 130 proceeds to decision step 690, wherein the computer 130 determines whethe the test pixel is the last edge pixel to be processed in th fuzz function 530'. If not, the computer 130 proceeds to ste 692, wherein the computer 130 selects one of the edge pixel that is immediately adjacent the test pixel, designates thi adjacent pixel as the new test pixel, and loops back to ste 682. If, at step 690, the computer 130 determined that th test pixel was the last edge pixel, the computer 130 exits th fuzz function 530' at a step 694.

The computer 130 blends the surrounding background to th edges of an object or an area on the layer adjacent t background scene layer in function 540'. The blend functio 540' is more fully understood by reference to Figure 19. T blend function 540' may be called by either the pixe enhancement function 268 (Figure 14) or the top-level gam function 262 (Figure 17) . The computer 130 starts at st 540' and proceeds to step 700 wherein the computer 130 selec an object edge pixel as the test pixel. The selection of t object edge pixel is made at a software determined address e.g., the left edge pixel of the first row of the objec Moving to step 702, the computer 130 determines the hue gam value of the test pixel. From step 702, the computer 13 proceeds to step 704 and determines the hue of the backgrou pixel that is on the layer immediately below the selected test pixel in the same row as the test pixel. Then the computer 130 proceeds to step 706 wherein the computer 130 determines the average hue of the two pixels. From step 706, the computer 130 proceeds to step 708, wherein the computer 130 sets the transparency/opacity of the test pixel according to the blend factor in the calling function.

From step 708, the computer 130 proceeds to a decision step 710, wherein the computer 130 determines whether there are other edges in the current row; that is, whether the test pixel is the last edge pixel to be processed in the current row. If there is at least one additional edge in the current row, the computer moves to step 712 and selects the next edge pixel. This pixel is designated as the new test pixel, and then the computer 130 loops back to step 702. However, if there are no more edge pixels in the current row as determined at step 710, the computer 130 advances to a decision step 714 wherein a check is made to determine if the last row of the object has been processed. If not, the computer 130 proceeds to step 716, wherein the computer 130 selects the next row and the first edge pixel in that row. This pixel is designated as the new test pixel, and then the computer loops back to step 702. If, at step 714,. the computer 130 determined that the last row of the object has just been processed, the computer 130 exits the blend function 540* at a step 718.

Referring to Figure 20, the function 550* for changing the gamma attributes of an image will be described. The function 550' is called by the top-level gamma function 262 (Figure 17) . A list of the gamma attributes that may be changed was disclosed in conjunction with Figure 17. The file header may contain information about the area or object for which a gamma attribute or multiple attributes is to be changed. Depending on the complexity of the background, the header may contain additional information necessary to properly achieve photo-realistic events. The change gamma function allows the user to change one or more gamma attributes on a whole layer or on an area or object on the layer. For example, the saturation attribute can be pumped from a value of 40 to a value of 80 on the whole layer.

The function 550' starts and proceeds to a step 73 wherein the computer 130 begins processing at a softwa determined location of the image. This location may be at t X,Y coordinates of the top left corner of an object or area i one embodiment or it may be the entire layer. Moving to ste 732, the computer 130 changes the pixel value of the selecte attribute, e.g., hue. The computer 130 then moves to decision step 734 to determine if additional preselecte pixels (of the object, area, or layer) in the current row a to have an attribute changed. If so, the computer 130 move a step 736 and advances to the next pixel in the current row The computer 130 then loops back to step 732. If there a no additional preselected pixels in the current row a determined by step 734, the computer 130 moves to a decisio step 738 to determine if the last row of the selected area ha been processed. If not, the computer 130 moves to step 740 advances to the first pixel on the next row of the selecte area, and then loops back to step 732. However, if the las row of the selected area, object, or layer has been processed the computer moves to a decision step 742 wherein determination is made whether there are any more attributes t change. If so, the computer 130 moves to step 7130, gets th next gamma attribute for change, and then loops back to ste 730. If there are no further attributes to change, a determined by step 742, the computer 130 exits the chan gamma function 550' at a step 746.

Referring to Figure 21, the transparency/opacity functio 560' will be described. The function 560' is called by t top-level gamma function 262 (Figure 17). The function 560 starts and proceeds to a decision step 770 wherein t computer 130 determines whether the function is to performed on an object or on a background scene. If it i determined at step 770 that an object is to be processed, t computer 130 moves to step 772 to begin processing at t first row of the object as defined in the file header. T computer 130 selects an object edge pixel on the current row as the test pixel. The selection of the object edge pixel is made at a software determined address, e.g., the left edge pixel of the first row of the object. Moving to step 774, the computer 130 determines the transparency/opacity (T/0) value of the test pixel. From step 774, the computer 130 proceeds to step 776 and determines the T/0 of the background pixel that is on the layer immediately below the selected test pixel in the same row as the test pixel. Then the computer 130 proceeds to step 778 wherein the computer 130 determines the average T/0 of the two pixels. From step 778, the computer 130 proceeds to step 780, wherein the computer 130 sets the transparency/opacity of the test pixel according to the blend factor in the calling function. From step 780, the computer 130 proceeds to a decision step 782, wherein the computer 130 determines whether there are other edges in the current row; that is, whether the test pixel is the last edge pixel to be processed in the current row. If there is at least one additional edge in the current row, the computer moves to step 784 and selects the next edge pixel. This pixel is designated as the new test pixel, and then the computer 130 loops back to step 774. However, if there are no more edge pixels in the current row as determined at step 782, the computer 130 advances to a decision step 786 wherein a check is made to determine if the last row of the object has been processed. If not, the computer 130 proceeds to step 788, wherein the computer 130 selects the next row and the first edge pixel in that row. This pixel is designated as the new test pixel, and then the computer loops back to step 774. If, at step 786, the computer 130 determines that the last row of the object has just been processed, the computer 130 moves to a step 790. At step 790, the computer 130 initializes further processing by accessing the first pixel of the first row of the area made transparent by use of the strip function 234 (Figure 10) .

If the decision step 770 determination is that the function 560' is processing the background scene, the computer 130 moves to a step 800 and accesses the first pixel of t first row of the image as the test pixel. After completion step 800, the computer 130 moves to a step 802 wherein t file header is utilized to determine which of hue, red, gree blue (RGB) , and intensity tests are to be run. A combination or all five tests can be run. The test is determine the hue, red, green, blue, or intensity value of t pixel. For each test, a range is kept in the file head along with information that denotes whether the test on t test pixel is for values that are inside the range or f outside the range. The range and the inside/outsi information can be different for each of the five tests. an example, a predetermined limit on hue values may be fr zero to 255 and the test may look for pixels that fall outsi the hue value range of 50 to 100. These pixels will ha their alpha-bit flag set. Thus, of the steps 804, 808, 81 816, and 820, only those tests selected to run, according the file header, will have their respective ranges checke So, for the example just above, the computer 130 moves to decision step 804 and determines whether the hue value of t test pixel is outside the range specified in the header, e. 50 to 100. If so, the computer 130 sets the alpha-bit flag transparent at step .806. However, if the pixel value does n match the range specification in the file header for hue, determined by step 804, the computer 130 moves to the ne decision block as selected in the file header, if any, amo steps 808, 812, 816, and 820. A set of steps similar steps 804 and 806 above is executed for each test selected the file header. At the conclusion of the one to five tests above, t computer 130 moves to a decision step 824 and determines there is at least one additional pixel in the current row be tested. If so, the computer 130 advances to step 82 accesses the next pixel in the current row, and loops back step 802 to repeated the above-described process. If the la pixel on the current row has been processed, as determined step 824, the computer 130 moves to a decision step 828 determine if the last row has just been processed. If not, the computer 130 moves to step 830, advances to the next row of the image, and loops back to step 802. If the last row has been processed, as determined by step 828, the computer 130 exits the transparency/opacity function 560' at step 832.

After completion of step 790, the computer 130 moves to step 802. Steps 802 to 832 are then similar to that described above except that for steps 806, 810, 814, 818, and 822, the alpha-bit flag is set opaque rather than transparent, because the area to be processed is already transparent (when step 802 is reached via step 790) . As an example, setting the alpha- bits opaque allows the user to make a drop shadow in ^"the shape of the user's head, which is known as shadow preservation.

Referring to Figure 21, the gradient sharpening function 1306 will be described. The function 1306 is called by the pixel enhancement function 268 (Figure 14) . A s a n example, the function 1306 may be used to locate the eyes on an image of a face so that the eyes could be enhanced while the skin is left unchanged. The function 1306 starts and the computer 130 proceeds to a step 850 and accesses the first pixel of the first row as the test pixel of the image that is passed to function 1306 from function 268. Moving to step 852, the computer 130 measures the hue of the test pixel and advances to a decision step 854. At step 854, the computer determines whether the hue value falls outside a range specified by the file header, e.g., 110 to 150. If so, e.g., the value is less than 110 or greater than 150, the computer 130 moves to step 856 wherein the alpha-bit flag is set for that pixel. At the completion of step 854 or if the hue value does fall in the range, as determined at step 854, the computer 130 moves to a decision step 858. At step 858, the computer determines if there is at least one additional pixel left in the current row. If so, the computer 130 moves to step 860, accesses the next pixel in the current row and loops back to step 852. However, if the computer 130 determines, at step 858, that the last pixel in the current row has been processed, a decision step 862 determines whether the last row has just been processed. If not, the computer 130 moves step 864, accesses the next row, and loops back to step 852 process the new row.

If the determination is made, at step 862, that the la row has been processed, the computer 130 moves to a step 8 to look for the first row having a pixel with the alpha-b flag set. Moving to step 872, the computer 130 does ed enhancement on the pixel. The computer 130 performs one, t or three operations on the pixel, depending on the hue val of the pixel and a set of ranges for the hue value a associated operations which are kept in the file header. T three operations are: saturate the pixel, shift the hue (ma the blacks blacker and the whites whiter) , and shift t intensity. A set of hue value ranges are set up in the fi header such that, for example, if the measured hue val exceeds the file header value by more than 40, the saturati operation is done; if the measured value exceeds the head value by more than 70, the shift intensity operation is don and if the measured value exceeds the file header value more than 100, the saturation and the shift hue operations a done. The hue value ranges are background specific, and a set by the programmer to achieve the desired results based that background. For example, if the skin color of t background object is green, the range will be selected so th the skin color of the object being placed in the backgrou will also match this green color.

Moving to a decision step 874, the computer 1 determines whether there is at least one more pixel in t current row that has the alpha-bit flag set. If so, t computer 130 proceeds to step 876, accesses the next pixel the current row having the alpha-bit set, and loops back step 872. However, if decision step 874 is false, t computer 130 moves to a decision step 878 to determine if t last row of the image has just been processed. If not, t computer 130 moves to a step 880, advances to the next r having a pixel with the alpha-bit set, and loops back to st 872. If the last row has been processed, as determined step 878, the gradient sharpening function 1306 returns at step 882 to the pixel enhancement function 268.

The software described herein is written in the "C" language and was translated from source code to machine- readable object code using a Photo Novelty Programming Language (PNPL) compiler developed by ImageWare Software, Inc., formerly Practically Perfect Productions, Inc. of San -Diego, CA. Nonetheless, one skilled in the technology will recognize that the steps in the accompanying flow diagrams can be implemented by using a number of different compilers and/or programming languages.

The image editing system described herein finds application in many environments, and is readily adaptable for use therein. For example, the system finds use in generation of composite photographs where portions of a person's image may replace those of another person's image, such as on a sports team photo. It can be used for production of postcards depicting a person in an exotic location. It also can be used in applications such as production of driver's licenses or security access cards where a person's image is combined with selected other information on a card. Accordingly, the claims are to be interpreted to encompass these and other applications of the invention within their scope and are not to be limited to the embodiments described herein. While the above detailed description has shown, described and pointed out the fundamental novel features of the invention as applied to various embodiments, it will be understood that various omissions and substitutions and changes in the form and details of the device illustrated may be made by those skilled in the art, without departing from the spirit of the invention.

Claims

I CLAIM:

1. A method for selectively combining digital images i three dimensions, comprising the steps of: providing an image of a background, havin components of said background assigned to one of plurality of X-Y planes that are overlayed on each othe in a direction so as to define relative positions of sai components with respect to three dimensions; providing an image of an object to be combined wit the image of the background to form a composite image wherein the position of said image of said object i defined in the X, Y and Z directions with respect to sai plurality of said X-Y planes; and combining the image of the background with the imag of the object to form a composite image wherein a portio of a component which is closer than the object to viewing location in the Z direction and which correspond to the position of a first portion of the object in th

X-Y plane will obscure said first portion of the objec from view and wherein a portion of a component which i further than the object from the viewing location in th

Z direction, and which corresponds to the position of second portion of the object in the X-Y plane will b obscured from view.

2. The method of Claim 1, wherein the step of providin an image of a background comprises the step of providing digitized image of a background, wherein the step of providin an image of an object comprises providing a digitized image o an object, and wherein the combining step comprises the ste of combining said digitized images to form a digitize composite image.

3. The method of Claim 2, wherein the digitized imag of the object has a plurality of pixels, and wherein only portion of the digitized image of the object is to be combine with the digitized image of the background, and wherein th portion of the image of the object to be combined with th digitized image of the background, and wherein the portion o the image of the object to be combined with the image of the background is bordered by an object edge having a first hue, and an object background having a second hue and wherein, prior to the combining step, the method comprises: locating said object edge by comparing, for object background pixels and some of, but not all, the object pixels, the hues of the pixels with a predetermined hue; and removing the object background from the image of the object using the located edge.

4. The method of Claim 3 in another embodiment the method further comprising the steps of: averaging the hue of the object and the second digital image along the edge of the object, thereby producing a blended object in the second digital image; and setting one of: the first hue in at least a portion of the blended object and a third hue in the second digital image, to equal the other hue.

5. The method of Claim 3, wherein prior to the combining step, the method in another embodiment comprises the steps of: measuring the gamma of the object in a first hue; measuring the gamma of the background in a second hue; and shifting the gamma of a selected one of the object or the background to substantially equal the gamma of the other.

6. The method of Claim 1, wherein, prior to the combining step, the method comprises the steps of: providing a representation of size of a selected portion of the background image; providing a representation of size of a portion of the object image that is to be combined with the background image; comparing the representation of size of the portion of the object image with the representation of the size of the selected portion of the background image t determine their relative sizes; and changing the size of the image of the object to b combined in response to the results of the compariso step.

7. The method of Claim 6, wherein the selected portio of the background image is that portion of said backgroun image which is to be replaced by the object to be combined.

8. The method of Claim 7, wherein the selected portio of the background image includes an image of a first person' head, and wherein the portion of the object image to b combined includes an image of a second person's head, an wherein the representation of size of the selected portion o the background image comprises the distance between selecte features of the first person's head, and wherein th representation of size of the object image that is to b combined comprises the distance between selected features o the second person's head.

9. The method of Claim 8, wherein the distance betwee selected features of the first and second person's head comprise, respectively, the distance between the bottom of th chin and a line which extends between the pupils of the eye of the head.

10. A system for combining images in three dimensions comprising: a computer having a memory for storing a firs digital image having a plurality of pixels, wherein th first digital image includes an object bordered by a edge having a first hue, and a background having a secon hue; means for providing an image of a background having components of said background assigned t one of a plurality of X-Y planes that are overlaye on each other in a direction so as to defin relative positions of said components with respec to three dimensions; means for providing an image of an object to be combined with the image of the background to form a composite image, wherein the position of said image of said object is defined in the X, Y and Z directions with respect to said plurality of said X-Y planes; means for locating the edge in the first digital image by comparing, for background pixels and some of, but not all, the object pixels, the hues of the pixels with a predetermined hue; and means for removing the background from the first digital image using the located edge.

11. The system of Claim 10, additionally comprising means for combining the object with a second digital image, wherein said means for combining the image of the background with the image of the object to form a composite image wherein a portion of a component which is closer than the object to a viewing location in the Z direction and which corresponds to the position of a first portion of the object in the X-Y plane will obscure said first portion of the object from view and wherein a portion of a component which is further than the object from the viewing location in the Z direction, and which corresponds to the position of a second portion of the object in the X-Y plan will be obscured from view.

12. The system of Claim 11, wherein the combining means includes means for averaging the hue of the edge and the hue of the second digital image so as to generate an averaged hue, wherein the averaged hue replaces the edge and thereby produces a blended object.

13. The system of Claim 12, wherein at least a portion of the blended object has a first gamma and at least portion of the second digital image has a second gamma, and additionally comprising means for setting one of: the first gamma and the second gamma, to equal the other gamma.

14. The system of Claim 10, wherein each hue is characterized by a numeric value.

15. The system of Claim 14, wherein the background includes a plurality of two-dimensional boxes, each box having a plurality of dots therein, the computer having an image the background stored in memory, wherein the means f locating the edge compares the hue of pixel in the fir digital image with the hue of a corresponding pixel of background image stored in the memory, the predetermined h for each comparison thus being the hue of the correspondi pixel in the background image.

16. The system of Claim 14, wherein the means f locating the edge determines the difference between t numeric value of one pixel and the numeric value of contiguous pixel, and compares the difference to predetermined difference, the predetermined hue for ea comparison thus being the hue of the adjacent pixel.

17. The system of Claim 10, wherein the means f removing the background from the first digital image sets a background pixels to a preselected transparent value.

18. The system of Claim 10, additionally comprising video camera connected to the computer, the camera generati a signal which is stored as the first digital image.

19. The system of Claim 16, additionally comprising video monitor connected to the computer for displaying t first digital image.

20. The system of Claim 10, additionally comprising printer connected to the computer for generating a hard co image, the system also comprising a currency accept connected to the computer for accepting currency a activating the system in response thereto.

21. A gamma correlator, comprising: means for measuring a gamma of an object in first image and for measuring the gamma of background in a second image; and means for shifting the gamma of a selected o of the object or the background to substantial equal the gamma of the other.

22. An image combiner, comprising: a computer for receiving a sign representative of a first digital image, the sign including an object component representative of an object, the object being bordered by an edge having a first hue, the signal also including a background component representative of a background having a second hue, wherein the computer includes means for separating the background from the object by determining the location of the edge of the object and then removing all portions-of the first digital image which are outside the edge from all the portions of the first digital image which are inside the edge.

23. The image combiner of Claim 22, wherein the first digital image has a periphery, and the computer further comprises: means for determining the difference between the hue of a portion of the periphery and a predetermined hue; means for comparing the difference to a predetermined difference; wherein when the difference exceeds the predetermined difference, the computer identifies the location of the portion of the periphery as the location of a portion of the object, and then compares the hue of other portions of the periphery with the predetermined hue; and wherein when the difference is less than the predetermined difference, the computer compares the hue of inwardly succeeding portions, relative to the periphery, of the first digital image with the predetermined hue until the difference between a portion and the predetermined hue exceeds the predetermined difference; until the entire edge of the object has been identified.

24. The image combiner of Claim 23, wherein the background has a single continuous hue, and the predetermined hue is the hue of the background.

25. The image combiner of Claim 23, wherein th background includes a plurality of two-dimensional boxes, eac box having a plurality of dots therein, the computer having a image of the background stored in memory, the stored imag corresponding to the first digital image, wherein the compute compares the hue of at least some of the first video imag with the hue of the corresponding portions of the image of th background stored in memory to determine the location of th edge of the object.

26. The image combiner of Claim 23, wherein the compute receives a second digital signal representative of preselected background, and the computer overlays the objec onto the preselected background.

27. The image combiner of Claim 26, wherein th preselected background has at least one hue, and the compute averages the hue of the object and the hue of the preselecte background to generate an averaged signal representative of a averaged hue, the computer changing the hue of the edge of th object to equal the averaged hue, to thereby produce a blende object.

28. The image combiner of Claim 27, wherein at least portion of the blended object has a first gamma and at leas portion of the preselected background has a second gamma, an the computer sets one of: the first gamma and the secon gamma, to equal the other gamma.

29. The image combiner of Claim 23, wherein the digita signal comprises a plurality of pixels, each pixel having hue and each hue having a corresponding numeric value, and th background is a continuous hue, wherein the compute determines the location of the edge of the object b determining the difference between the numeric value of on pixel and the numeric value of an adjacent pixel, an comparing the difference to the predetermined difference.

30. The image combiner of Claim 29, wherein the compute removes the background component from the object component b setting all background pixels to a preselected transparen value.

31. The image combiner of Claim 22, further comprising a video camera for generating the signal, the computer being electrically connected to the video camera for receiving the signal therefrom.

32. The image combiner of Claim 31, further comprising a video monitor electrically connected to the computer for displaying the image of the object.

33. The image combiner of Claim 32, further comprising a video printer electrically connected to the computer for generating a print of the object, the system also comprising a currency acceptor operably engaged with the computer for accepting currency and activating the system in response thereto.

34. A method for sizing a selected digitized object image to match a selected digitized background image, to form a composite image, comprising the steps of: providing a representation of size of a selected portion of said background image; providing a representation of size of a portion of said object image that is to be combined with said background image; comparing the representation of size of the portion of said object image with the representation of the size of the selected portion of said background image to determine their relative sizes; and changing the size of said image of the object to be combined in response to the results of a comparison step.

35. The method of Claim 34, wherein the selected portion of said background image is that portion of said background image which is to be replaced by said object to be combined.

36. The method of Claim 35, wherein the selected portion of said background image includes an image of a first person's head, and wherein the portion of said object image to be combined includes an image of a second person's head, and wherein the representation of size of the selected portion of said background image comprises the distance between selected features of said first person's head, and wherein the representation of size of said object image that is to b combined comprises the distance between selected features o said second person's head.

37. The method of Claim 36, wherein the distance betwee selected features of said first and said second person's head comprise, respectively, the distance between the bottom of chin and a line which extends between the pupils of eyes of head.

38. The method of Claim 34, wherein said object image i combined with said background image through layerin comprising the steps of: performing a layer limit address test; prioritizing layers; selecting said layers in a layer by layer basis; positioning said object image into said selecte background image; calling a layering function using the main contro flow; identifying said object or objects from the argumen in said layering function; writing said object to a linked list, said lis including name, layer, position within said layer, an alpha-bit flag of said object or objects; stitching said object or objects into one collectio on one layer; removing said objects from said linked list; displaying said objects; and stitching said objects into said background.

39. The method of Claim 38, wherein said positionin further comprises the steps of: getting the address of a cross symbol at the middl of the neck and bottom of the chin on said object image retrieving said address of said cross symbol; and setting said address equivalent to a cross symbol o said background image.

40. A method of layering in three dimensions on object image or images and background image or images comprising the steps of: providing said background, having components of said background image assigned to one of a plurality of X-Y planes that are overlayed on each other in a direction so as to define relative positions of said components with respect to three dimensions; providing an object image to be combined with said background image to form a composite image, wherein the position of said object image is defined in the X, Y and

Z directions with respect to said plurality of said X-Y planes; combining said background image with said object image to form a composite image wherein a portion of a component which is closer than said object image to a viewing location in said Z direction and which corresponds to the position of a first portion of said object in said X-Y plane will obscure said first portion of said object image from view and wherein a portion of a component which is further than said object image from the viewing location in said Z direction, and which corresponds to the position of a second portion of said object images in said X-Y plane will be obscured from view; performing a layer limit address test; prioritizing layers; selecting said layers on a layer by layer basis; calling a layering function using the main control flow, identifying said object images from the argument in said layering function; writing said object images to a linked list, said list including name, layer, position within said layer, and alpha- bit flag of said object images; stitching said or objects images into one collection on one layer; removing said object images from said linked list; displaying said object images; and stitching said object images into said backgroun images.