WO1996029675A1 - Aesthetic imaging system - Google Patents

Aesthetic imaging system

Info

Publication number
WO1996029675A1
WO1996029675A1 PCT/US1996/003678 US9603678W WO9629675A1 WO 1996029675 A1 WO1996029675 A1 WO 1996029675A1 US 9603678 W US9603678 W US 9603678W WO 9629675 A1 WO9629675 A1 WO 9629675A1
Authority
WO
WIPO (PCT)
Prior art keywords
digital image
image
user
monitor
patient
Prior art date
Application number
PCT/US1996/003678
Other languages
French (fr)
Inventor
Ray A. Linford
Perin Blanchard
Original Assignee
Virtual Eyes, Incorporated
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Family has litigation
First worldwide family litigation filed litigation Critical https://patents.darts-ip.com/?family=23606955&utm_source=google_patent&utm_medium=platform_link&utm_campaign=public_patent_search&patent=WO1996029675(A1) "Global patent litigation dataset” by Darts-ip is licensed under a Creative Commons Attribution 4.0 International License.
Application filed by Virtual Eyes, Incorporated filed Critical Virtual Eyes, Incorporated
Priority to AU54251/96A priority Critical patent/AU5425196A/en
Priority to EP96911333A priority patent/EP0815533A1/en
Priority to JP8528545A priority patent/JPH11503540A/en
Publication of WO1996029675A1 publication Critical patent/WO1996029675A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/002D [Two Dimensional] image generation
    • G06T11/20Drawing from basic elements, e.g. lines or circles
    • G06T11/203Drawing of straight lines or curves
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/002D [Two Dimensional] image generation
    • G06T11/60Editing figures and text; Combining figures or text
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99951File or database maintenance
    • Y10S707/99952Coherency, e.g. same view to multiple users
    • Y10S707/99953Recoverability
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99951File or database maintenance
    • Y10S707/99952Coherency, e.g. same view to multiple users
    • Y10S707/99954Version management

Definitions

  • This invention generally relates to computer imaging programs and, more specifically, to a method and apparatus for manipulating digital photographs.
  • Existing aesthetic imaging systems typically use a number of tools to allow a physician to manipulate a pre-operative image of a patient to illustrate an intended post-operative result.
  • the tools allow the physician to manipulate the preoperative image during a consultation with a patient. By manipulating the image with the patient in attendance, the patient receives immediate feedback from the displayed results.
  • the use of the editing tools should be as unobtrusive as possible.
  • a physician would like the patient to focus on the end results of the surgery, rather than the technologic wizardry used to demonstrate those results on the video monitor.
  • the editing tools used in existing aesthetic imaging systems typically hinder rather than help the physician in demonstrating the results that may be achieved through cosmetic surgery.
  • a disadvantage of existing aesthetic imaging programs is that a physician or facilitator in a pre-operative consultation typically must go back and forth through many windows-based menus in order to edit an image. Cycling between the various menus to invoke the tools necessary for a consultation is disadvantageous in that it is time consuming. For example, some physicians schedule a follow-up visits for patients to allow the physician time to edit the images. More important, however, is that the process is distracting to the patient and tends to make the pre-operative consultation all the more mystifying. As a result of the disadvantages associated with prior art systems, some patients lose interest or become frustrated with the interview, both of which may reflect back on the physician.
  • a further disadvantage of existing aesthetic imaging systems is that it is impossible for a physician or facilitator to display different combinations of the edits that they have performed.
  • existing aesthetic imaging programs as a physician edits a patient's image, the physician's edits are added to the preexisting edits of the image. Most programs are only capable of showing two version of the patient's image; the unedited, original version, and the final edited version incorporating all of the physician's changes. It is therefore difficult for the physician to show various combinations of the edits that had been performed. For example, a physician may edit an image to remove wrinkles around a patient's eyes and to narrow the patient's nose.
  • Existing aesthetic imaging programs only allowed the physician to simultaneously show all of these changes.
  • a still further disadvantage of existing aesthetic imaging systems is that the systems allow a physician to perform nearly flawless editing of a patient's image.
  • the edits performed by a physician on an aesthetic imaging system are often unobtainable results that cannot be achieved when actual surgery is performed.
  • the physician is especially skilled at using the aesthetic imaging system, it is difficult to show the patient achievable results, which typically fall within a range somewhere between the original patient image, and the optimum results as displayed by the edited image on the screen. It therefore would be advantageous to develop an aesthetic imaging system that allowed a physician to display more realistic results that are achievable through surgery.
  • the invention is an aesthetic imaging system for use in editing digital images.
  • the aesthetic imaging system includes a unique user interface that allows edits to be performed more efficiently and with less confusion to the patient.
  • a method of editing a digital image comprised of a plurality of color pixels in an aesthetic imaging system including a processor, a memory, a monitor, and a pen and cooperating tablet for controlling a cursor displayed on the monitor.
  • the pen has a depressable tip and a side button, each of which include an on status and an off status, wherein the position of the pen tip relative to the tablet determines the position of the cursor on the monitor.
  • the method comprises: (a) evaluating the following variables: (i) the status of the tip of the pen; (ii) the status of the side button on the pen; and (iii) movement of the pen tip relative to the tablet; (b) actuating a freehand drawing mode if a first set of variables are present, wherein movement of the pen relative to the tablet edits pixels that are located at positions corresponding to the position of the cursor; and (c) actuating a curve drawing mode if a second set of variables are present, wherein a line segment is displayed between two endpoints and movement of the pen relative to the tablet stretches the line segment, forming a curve and editing pixels that are located at positions corresponding to the position of the curve.
  • method further includes: (a) actuating a freehand undo mode if a third set of variables are present, wherein movement of the pen relative to the tablet restores pixels that are located at positions corresponding to the position of the cursor to their pre-edited color; and (b) actuating a curve undo mode if a fourth set of variables are present, wherein a line segment is displayed between two endpoints and movement of the pen relative to the tablet stretches the line segment, forming a curve and restoring pixels that are located at positions corresponding to the position of the curve to their pre-edited color.
  • the freehand draw mode is actuated if the tip of the pen is depressed and pressure is maintained while the tip is moved a predetermined distance.
  • the curve draw mode is actuated if the tip of the pen is depressed and released within a predetermined distance.
  • the curve draw mode is actuated by: (a) establishing a first endpoint at the position of the pen when the second set of variables are present; and (b) monitoring the status of the tip of the pen and establishing a second endpoint at the position of the pen if the tip is toggled from an off state to an on state.
  • an improved prioritize feature for viewing an image.
  • a user may identify several areas in a modified patient image containing edits that alter the image from the original image. As each area is identified by the user, an identifying tag is assigned to each of the areas. When desiring to show various combinations of the edits that have been performed on the image, the user may select the areas to display using the identifying tags. A user may therefore quickly cycle through various permutations of the procedures that have been edited for patient display.
  • an improved user interface is provided to minimize the distraction of a patient as the patient is watching the image being edited.
  • a menu bar on the top of the display is removed during most editing, so that only the image of the patient is displayed.
  • the bar itself is transparent to allow the patient to see the image through the menu bar. Only the commands and the outline of the menu bar are presented in a contrasting color, minimizing the overall visual impression created by the menu bar.
  • a warp tool is described that allows a user to quickly and easily manipulate various features in an image. To use the warp tool, a user first encircles a portion of the image to be edited.
  • the user may tip the pen to designate a stretch point within the selected area.
  • the image is then stretched as if pulled from the stretch point. Areas of the image in the direction of stretch are compressed, and areas away from the direction of stretch are expanded. Areas surrounding the warping area are automatically adjusted to ensure that there is no discontinuities with the warping area.
  • the manipulation of the image is performed in real-time, allowing a patient to see the warping as it is being performed by a user.
  • An advantage of the tools and features described herein are that they improve the overall experience of a patient during a preoperative visit with a physician.
  • the powerful tools in the aesthetic imaging system allow the physician to easily manipulate the patient's image in response to feedback provided by the patient.
  • the aesthetic imaging system interface also allows the patient to focus on the image being manipulated, rather than on the aspects of the aesthetic imaging system that allow the manipulation.
  • the end result is an improved preoperative visit that provides a more realistic impression of the results that a physician may achieve through surgery.
  • FIGURE 1 is a block diagram of an aesthetic imaging system in accordance with the invention.
  • FIGURE 2 is a block diagram illustrating various buffers used by the aesthetic imaging system to store and manipulate data
  • FIGURE 3 is a flow chart illustrating an exemplary routine by which digital images may be viewed and edited using the aesthetic imaging system
  • FIGURE 4 A is a flow chart of an exemplary routine for photographing patients in accordance with the invention
  • FIGURE 4B is a flow chart of an exemplary routine for calculating a checksum value and comparing the calculated value to a previously stored value to determine if an image has been altered
  • FIGURE 5 is a flow diagram of an exemplary routine for implementing a combination tool for use with various drawing (draw) tools in accordance with the invention
  • FIGURE 6 is a pictorial representation of an image to be edited
  • FIGURES 7A-7E are pictorial representations of editing an image using a prior art imaging program
  • FIGURES 8A-8E are pictorial representations of using the aesthetic imaging system to accomplish the identical edits shown in FIGURES 7A-7E;
  • FIGURE 9A is a flow chart of an exemplary routine of a contour tool in accordance with the invention.
  • FIGURES 9B-9C are pictorial representations illustrating the function of the contour tool of FIGURE 9 A;
  • FIGURES 9D-9G are pictorial representations illustrating exemplary edits that may be accomplished using the contour tool
  • FIGURE 10 is a flow chart of an exemplary routine for implementing an autoblend tool in accordance with the invention.
  • FIGURE 11 is a pictorial representation of a user interface for implementing the autoblend tool of FIGURE 10;
  • FIGURE 12A is a flow chart of an exemplary routine illustrating a cutout tool in accordance with the invention.
  • FIGURE 12B is a flow diagram of an exemplary routine illustrating a rotate tool in accordance with the invention.
  • FIGURE 13 is a flow chart of an exemplary routine for viewing images in accordance with the invention.
  • FIGURES 14A-14D are pictorial representations illustrating the effects of a compare feature in accordance with the invention;
  • FIGURES 15A-15C illustrate a split image option of viewing images in accordance with the invention
  • FIGURE 16 is a pictorial representation illustrating the use of a translucent image to allow a patient to accurately position themselves in order to capture a second image having the same location and orientation as an original stored image;
  • FIGURE 17 is a pictorial representation illustrating a compare image wherein a presurgical image of a patient is compared side-by-side with a postsurgical image having the same location and orientation;
  • FIGURES 18A-18C are pictorial representations illustrating the use of a warp shape tool to edit a patient's image
  • FIGURES 19A-19E are pictorial representations illustrating the function of the warp tool;
  • FIGURE 20 is a pictorial representation illustrating the use of a transparent menu bar when viewing an image of a patient; and
  • FIGURE 21 is a flow chart of an exemplary routine for implementing a zoom viewing feature in accordance with the invention.
  • the aesthetic imaging system 20 includes an imaging program 21 that runs on a processing unit 22 controlled by an operating system 24.
  • a memory 26 is connected to the processing unit and generally comprises, for example, random access memory (RAM), read only memory (ROM), and magnetic storage media such as a hard drive, floppy disk, or magnetic tape.
  • RAM random access memory
  • ROM read only memory
  • magnetic storage media such as a hard drive, floppy disk, or magnetic tape.
  • the processing unit and memory are typically housed within a personal computer 28 which may be, for example, a MacintoshTM, International Business Machines (IBMTM) or IBM-compatible personal computer.
  • the operating system 24 may be DOS based or may incorporate a windowing environment such as Microsoft WindowsTM or OS/2TM.
  • the aesthetic imaging system also includes an image capture board 30 that is coupled to the processing unit 22, a monitor 32, video source 34, and printer 36.
  • the video source, monitor, and printer are coupled to the processing unit 22 through the image capture board 30.
  • the video source may include one or more video cameras, a VCR, a scanner, or similar source for providing digital images to be edited by the aesthetic imaging system.
  • the aesthetic imaging system further includes a pointing device, which is preferably a stylus (pen) and tablet 38, that is connected to the processing unit 22.
  • the aesthetic imaging system may include a modem 40 to provide on-line capabilities to users of the system, such as technical support and teleconferencing.
  • the image capture board 30 has a plurality of buffers in high-speed memory, e.g., RAM, that are used by the imaging program 21 to provide very fast response times to image edits.
  • high-speed memory e.g., RAM
  • four buffers are illustrated for use in explaining the operation of the aesthetic imaging system. These include an original image buffer 50, a modified image buffer 52, a current image buffer 54, and a working buffer 56.
  • Suitable image capture boards for use in the aesthetic imaging system include the Targa +64 and Targa 2000 boards, distributed by Truevision, Inc. of Indianapolis, Indiana. The buffers are discussed in regard to a single pose only, such as a profile or front view of a person.
  • the original image buffer 50 contains an unedited digital image, for example, a side profile picture of a potential patient.
  • the modified image buffer 52 contains any edits made to a copy of the original image.
  • the modified image buffer is updated during a save and after each session.
  • the current image buffer 54 contains information identical to the modified image buffer upon beginning a session. Thereafter, edits made to the current image are saved in the working buffer 56 as an overlay to the current image. During a save, the contents of the current image buffer 54 are copied to the modified image buffer 52, and the working buffer 56 is cleared.
  • Stylus The "pen” that may be used to select menus, modify images, and carry out other commands in the program.
  • the stylus controls the cursor, just as a mouse pointing device does on a personal computer.
  • Tablet or Pad
  • the electronic notepad used in conjunction with a stylus The pen must be held relatively close to the pad in order for the pen to communicate with the tablet. Unlike a mouse, the tablet follows an X / Y grid that matches the monitor, i.e., if the pen is positioned at the top left corner of the tablet, the cursor is displayed at the top left corner on the monitor.
  • Selecting Selecting also referred to as “tipping” or “pressing" the tip of the pen briefly onto the tablet. This selects a command or affects the drawing tool, depending on the current procedure being implemented.
  • FIGURE 3 illustrates an exemplary routine for implementing the imaging program 21 in accordance with the invention.
  • a system startup is performed wherein the computer looks for peripheral devices that are connected to the aesthetic imaging system, the memory is tested, and any other startup procedures needed to get the system up and running are implemented.
  • the imaging program displays a main menu, which provides access to the various features of the imaging program. Specifically, the main menu includes the following options: Storage, Camera, View, Shape, Draw, Print, And Exit. Those options that are pertinent to the invention are described in further detail below.
  • a test is. made to determine if the Camera option has been selected from the main menu, indicating that the user wants to take a picture of a patient. If the Camera option has been selected, a routine to implement this command is called at block 66. A suitable subroutine for this task in illustrated in FIGURE 4. Upon return from the Camera routine, the program loops to block 62.
  • the original image is copied to the original image buffer 50 and the modified image is copied to both the modified and current image buffers 52 and 54. It will be appreciated that the number of images that may be loaded at one time will be limited, in part, by the capacity of the frame grabbing board.
  • the program then loops to block 62. If the Storage option was not selected, a test is made at block 74 to determine if the Draw option has been selected from the main menu. If the Draw option has been selected, a draw tool routine is called at block 76. The program then loops to block 62. A suitable routine for implementing the Draw option is shown in FIGURE 5. If the Draw option was not selected, a test is made at block 78 to determine if the Shape option has been selected.
  • a shape tool routine is invoked at block 82.
  • a suitable routine for implementing the Shape option is shown in FIGURE 10. Otherwise, at block 82 a test is made to determine if the View option has been selected. If the View option has been selected, a view subroutine is invoked at block 84.
  • FIGURE 13 An appropriate routine for the View option is shown in FIGURE 13.
  • the program then loops to block 62.
  • a test is made to determine whether the Exit option has been selected from the main menu. If not, the program loops to block 62. Otherwise, any edits to the image are saved at block 88. At this point in the program, the image in the current image buffer 54 is saved to the modified image buffer 52, and the working buffer 56 is cleared. The program then terminates. Taking Pictures of a Patient Using an Inverse Image
  • FIGURE 4A illustrates an exemplary user interface that utilizes a video camera for acquiring a digital image of a patient. It is noted a scanner or other input device may also be used to input an image into the aesthetic imaging system.
  • the solid blocks indicate user interface options presented to the user by the aesthetic imaging system and the dashed blocks represent system responses to the decisions made.
  • a patient is positioned in front of the video camera.
  • an inverse or "mirror" image of the patient's image will be displayed on the monitor, as indicated at block 101.
  • the inverse image is computed using data from the original image, and is representative of how patients see themselves day to day when looking into a mirror.
  • digital images are comprised of pixels or picture elements. It is known to those skilled in the art how digital image pixels may be manipulated to create an image that is the inverse of the original. Displaying an inverse image of a patient is advantageous when taking pre- and post-surgical pictures of patients because it allows patients to more easily center or otherwise position themselves on the monitor. Without the pixel manipulation, the input from a camera or other digital device may create confusion when positioning a patient. Under normal viewing, if a patient appears left of center in the monitor, they are in reality too far to the right. In this instance, a typical patient's reaction is to move even further to the right. With a mirror image displayed, the tendency of most patients is to naturally adjust to the desired position.
  • the displaying of a mirror image is particularly important when taking post ⁇ surgical pictures.
  • post-surgical pictures it is advantageous to have the patient in exactly the position they were in when taking the pre-surgical picture.
  • the aesthetic imaging system will preferably display a translucent inverse image of the pre-surgical picture on the monitor, and then overlay an inverse image of the picture currently being taken.
  • a translucent patient image 370 in this case a patient's profile, is displayed on the monitor.
  • the translucent image is the preoperative image taken prior to undergoing a surgical procedure.
  • a "live" video image 372 of the patient is also displayed under the translucent inverse image.
  • the aesthetic imaging system of the present invention may generate a side-by-side display of the two images to allow a patient to easily and accurately compare a presurgical picture with a post-surgical picture.
  • the left half of the monitor may display the presurgical image 370
  • the right half of the monitor may display the post-surgical image 372. Allowing a patient to view the two images side-by-side in precisely the same orientation provides the patient with an accurate impression of the results achieved by surgery.
  • the image is focused and sized at block 102 by using the aesthetic imaging system to adjust the electronic controls on the video camera.
  • the tip of the pen is pressed anywhere on the tablet to freeze the digital image onto the monitor.
  • the user makes a determination if the image currently displayed on the monitor is acceptable. If the image is not acceptable, the routine loops to block 100. If the image is acceptable, an appropriate command is entered at block 108 and the image is stored in nonvolatile memory for future viewing.
  • a test is made to determine if an exit or other similar command has been entered by the user, i.e., if any more pictures are to be taken. If additional pictures are to be taken, the program loops to block 100.
  • a checksum value (described below) is calculated by the imaging program for each (original) image that has been stored.
  • the imaging program stores each image and its associated checksum value. The routine then returns to block 68 of FIGURE 3. Determining Authenticity Using Checksum Values
  • the checksum value is an addendum to an original stored image that is used to determine its authenticity when the image is subsequently displayed or printed.
  • Those skilled in the art will recognize that there are a number of methods of implementing such a checksum procedure. For example, one checksum computation is to add up the grayscale values for one of the colors, i.e., red, green, or blue, for each pixel comprising the image. Assuming a screen of 640 by 480 pixels and 256 colors per pixel, the checksum values would range from 0 to (640)(480)(255). When an image is recalled for display or to be printed, the checksum value is recalculated.
  • the newly calculated checksum value will be equivalent to the addendum value, and the image is certified as being unaltered. If the image has been modified, the checksum values will vary, indicating the image has been modified. In this instance, an indication of the fact that the image has been modified may be displayed or printed with the image, if desired.
  • the authentication of an original image using a checksum value is ideal for situations in which physicians display before and after pictures of a patient who has undergone cosmetic surgery. In some instances, viewers are skeptical as to whether an "after" image is really representative of a patient's appearance after surgery. This is in reaction to beliefs that post-surgical images have been altered to make the patients look better. For example, there may be temptation to slightly fade wrinkles or otherwise edit features that the physician was attempting to address in a surgery. Using the described checksum feature, the post-surgery image can be verified as an authentic, unaltered image based upon the addendum value, and the veracity of the image is not questioned. This is beneficial to physicians when illustrating post- surgical results during lectures or in other teaching situations.
  • FIGURE 4B illustrates an exemplary routine for determining whether or not an original image, i.e., pre- or post-surgical image, has been altered in accordance with the invention.
  • This routine may be invoked whenever a pre- or post-surgical image is displayed on a monitor or printed on a page.
  • a test is made to determine whether the image to be displayed is portrayed as an "original image" that was not modified, e.g., a before or after picture. If the image is supposed to be an original image, a current checksum value for the image is calculated at block 115.
  • the calculated checksum value for the image is compared to the checksum that was stored when the image was acquired by the aesthetic imaging system, e.g., when the picture was taken.
  • the calculated checksum value is compared to the stored checksum value to see if they are equivalent. If the two values are not equivalent, at block 118 an icon is added to the image, e.g., displayed or printed along with the image, indicating that the image has been altered. If the checksum values are equivalent, an icon is added to the image at block 119 verifying the authenticity of the image. Once either icon has been added to the image, or if the image being printed or displayed is not an original image, the program terminates. As will be appreciated, the same checksum computational method must be used on each image, i.e., when an original image is acquired and when an image is to be displayed, or the comparison will be meaningless.
  • a checksum is contemplated in the exemplary routine for determining whether an original image has been altered
  • other techniques may be used to detect when an original digital image has been modified. For example, a flag or other marker may be uniquely associated with the original digital image. If the flag or other marker is absent from the digital image being displayed, the digital image is a copy that is presumed to have been changed. Alternatively, two versions of the image may be stored, included an unaltered original and a copy. The two images may be compared in order to determine whether modifications have been made to the copy.
  • the imaging program includes a unique combination draw (CD) feature that generally works with all of the drawing tools.
  • CD combination draw
  • the CD feature allows a user to freehand draw, use curves to edit an image, as well as undo using either freehand or curves, without having to invoke a separate menu for each item.
  • the airbrush tool is selected as the drawing tool, although it is to be understood that the CD tool generally applies to all of the drawing tools.
  • the aesthetic imaging system prompts the user to choose a color from a color palette that appears on the monitor. A color is selected using the pen. After selecting a color, a side bar menu is displayed. The user may select from a number of options on the side bar, including brush size and intensity, or select away from the side bar menu to remove the menu from the screen. In the latter case, the system defaults are used.
  • the user To draw a curve, the user must set a first anchor point by selecting with the pen. Thereafter, as the user moves or "floats" the pen across the tablet, a green line will extend from the first anchor point to the current position of the pen. In a desired location, a second anchor point is set by selecting with the pen. Once both anchor points have been established, the green line appears on the monitor as a segment between the two anchor points.
  • the user floats the pen across the tablet. The system will display a curved line bending and moving with the movement of the pen. The pen movement (top to bottom, or side to side) determines the arc of the curve. As the curve moves across the image, the image is edited in accordance with the selected draw tool and draw tool options.
  • FIGURE 5 illustrates an exemplary routine for implementing the CD feature of the imaging program.
  • the draw tool group includes: Airbrush, Tint, Texture, Blend, Undo, and Contour.
  • a color With airbrush, a color must be chosen. After a color has been chosen, a side bar menu is displayed at block 122 which illustrates other options that may be invoked for the airbrush tool. These typically include selecting a brush size, brush intensity, and other miscellaneous options. Sample side bar menus for various shape tools are described below. The draw tool side bar menus are similar to these.
  • the imaging program begins a curve mode by establishing a first endpoint, as indicated at block 132, and drawing a line on the monitor from the endpoint to the current pen position, indicated at block 134.
  • a test is made to determine if the tip has been pressed. The imaging program at this point is looking for a second endpoint to be entered. If not, the program loops back to block 136 to await the input.
  • the user can go back to the beginning of the routine using the cancel button if the user has changed his or her mind, although this is not shown in the flow diagram.
  • the first cancel would place the routine at block 124, the second at either block 122 or 124, depending on the draw tool selected, and subsequent cancels would forward the routine to the main menu.
  • the second endpoint is established at the tip position, and a line segment is drawn on the monitor from the two endpoints, as indicated at block 138.
  • the imaging program is in "curve draw mode" as indicated at block 140, and the user can make any edits desired using a curvilinear line segment having the established endpoints.
  • the routine loops to block 124 while in this mode.
  • a test is made at block 146 to determine if the user wishes to exit. This test generally refers to the user again depressing the side bar, and thus "backing out” of the drawing routine. In this case, the draw routine returns to block 78 of FIGURE 3. Otherwise, the routine branches to block 124.
  • FIGURE 6 an original (unedited) image 130 that is representative of an image displayed on a monitor is illustrated in FIGURE 6.
  • a main menu 132 is displayed across the top of the image to allow a user to select editing, viewing and printing options, as discussed in FIGURE 3 and accompanying text.
  • the main menu 132 is from an embodiment of the aesthetic imaging system 20.
  • FIGURES 7A-7E and 8A-8E contrast exemplary steps taken to make identical edits to the image 130.
  • the steps required to modify the image using a relatively advanced prior art imaging system are illustrated in FIGURES 7A-7E. These steps are modeled after a prior art imaging system that has been distributed by Mirror Image Technology, Inc., a division of Virtual Eyes, Incorporated, the assignee of the present invention.
  • the steps required using an embodiment of the aesthetic imaging system 20 in accordance with the invention are illustrated in FIGURES 8A-8E.
  • each set of drawings illustrates examples of edits to a patient's nose, chin, and neck regions. The edits are for use in explaining the invention only, and merely exemplary in nature.
  • FIGURE 7 A the curve option of an airbrush tool is used to modify the bridge of a patient's nose.
  • a resultant curve 134 is displayed having anchor points (endpoints) 136 and 138.
  • FIGURE 7B a freehand motiQn is used to eliminate a portion of the tip of the patient's nose.
  • FIGURE 7C an undo tool is used to replace a portion of the bridge of the patient's nose that was removed by the edits performed in FIGURE 7 A.
  • FIGURE 7D the neck area of the patient has been edited using the curve option of an airbrush tool.
  • the resultant curve 140 has anchor points 142 and 144.
  • FIGURE 7E an undo tool is used to add back a portion of the neck area that was removed in FIGURE 7D.
  • FIGURE 7A Step SI move pen to draw on main menu
  • Step S2 select draw
  • Step S3 move pen to airbrush
  • Step S4 select airbrush
  • Step S5 move pen to curve
  • Step S6 select curve
  • Step S7 select an airbrush color
  • Step S8 move pen to the first anchor point position
  • Step S9 select at the position to establish the anchor point 136;
  • Step S10 move pen to the second anchor point position
  • Step Sll select at the position to establish the anchor point 138;
  • Step S12 move pen to bend the curve 134 into the bridge of the nose
  • FIGURE 7B Step S 13 press the side button on the pen to exit to the main menu; Step S14 move pen to draw; Step SIS select draw; Step S16 move pen to airbrush; Step S17 select airbrush; Step S18 move pen to freehand; Step S19 select freehand; Step S20 select a color for the airbrush tool; Step S21 use a rubbing motion with the pen to make the freehand edit;
  • Step S22 press the side button on the pen to exit to the main menu
  • Step S23 move pen to draw; Step S24 select draw; Step S2S move pen to undo; Step S26 select undo; Step S27 move pen to curve; Step S28 select curve; Step S29 use a rubbing motion with the pen to undo the previous edit;
  • Step S30 press the side button on the pen to exit to the main menu
  • Step S31 move pen to draw; Step S32 select draw; Step S33 move pen to airbrush; Step S34 select airbrush; Step S35 move pen to curve; Step S36 select curve; Step S37 select color to be used by the airbrush tool;
  • Step S38 move pen to the first anchor point position
  • Step S39 select at the position to establish the anchor point 142;
  • Step S40 move pen to the second anchor point position; Step S41 select at the position to establish the anchor point 144; Step S42 move pen to bend the curve 140 toward the neck, thereby making the edit shown.
  • FIGURE 7E Step S43 press the side button on the pen to exit to the main menu; Step S44 move pen to draw; Step S4S select draw; Step S46 move pen to undo;
  • Step S47 select undo; Step S48 move pen to freehand; Step S49 select freehand;
  • Step S50 use a rubbing motion with the pen to undo a portion of the previous edit.
  • Step S51 press the side button to exit back to the main menu
  • the steps required to perform the same edits using the aesthetic imaging system 20 are now described. With reference to FIGURE 8A-8E, the steps required to perform the edits include:
  • FIGURE 8A Step Nl move pen to draw
  • Step N2 select draw
  • Step N3 move pen to airbrush
  • Step N4 select airbrush
  • Step NS elect any color for the airbrush tool
  • Step N6 move pen to the first anchor point position
  • Step N7 select at the position to establish the anchor point 136;
  • Step N8 move pen to the second anchor point position
  • Step N9 select at the position to establish the anchor point 138;
  • Step N10 move pen to bend the curve 134 into the bridge of the nose;
  • FIGURE 8B Step Nil pressing the tip of the pen against the tablet and use a rubbing motion to make the freehand edit.
  • FIGURE 8C Step N12 pressing the tip of the pen and the side button simultaneously, and maintain pressure while rubbing in the area to be undone;
  • FIGURE 8D Step Nil move pen to the first anchor point position; SStteepp NN1122 select at the position to establish the anchor point 142; Step N13 move pen to the second anchor point position; Step N14 select at the position to establish the anchor point 144; Step N15 move pen to bend the curve 140 toward the neck, thereby making the edit shown.
  • FIGURE 8E Step N 16 press tip of pen and the side button simultaneously, while rubbing to undo a portion of the previous edit;
  • Step N17 release pressure on the pen and side button, and press the side button to return to the main menu.
  • FIGURE 9A illustrates a user interface for a contour tool for use in editing images in accordance with the invention.
  • the contour tool is invoked from the main menu of the imaging program, as indicated at block 200.
  • the contour tool has similarities to a blend tool, but utilizes pixel manipulation to pull pixels from one area to another. For example, the tool works great for chin and lip augmentations.
  • a side bar menu is displayed by the aesthetic imaging system, as indicated at block 201.
  • the point size for the tool may be selected from the side bar menu.
  • an opacity percentage is entered by the user. If the opacity is at 100 percent, any areas affected by the edit are completely covered by the replacement pixels. As the percentage is reduced, more and more of the original pixels will remain, creating a blending of the replacement and prior pixels.
  • Anchor points are selected at block 202. The selection is accomplished as described in blocks 128 and 130 of FIGURE 5. As also described, once the anchor points are selected, a line is displayed between the points by the aesthetic imaging system, as indicated at block 205.
  • a curve having the anchor points as endpoints is positioned along a part of the body, e.g., lips or chin, to be edited.
  • the body part is then edited by dragging the curve in the direction in which a body part is to be augmented. Edits made in block 206 are saved at block 208 by pressing and then releasing the tip of the pen. The program then terminates.
  • FIGURES 9B-9C further describe the operation of the contour tool, by illustrating how pixels are replicated from one area of an image to another.
  • the image areas described are for exemplary purposes only, and are simplified for clarity in this discussion.
  • an area 209 of an image is comprised of red 218R, green 218G, blue 218B, and yellow 218Y areas separated by boundary lines 210, 212, 214, and 216. It is assumed that a pair of anchor points 218 and 219 have been established by a user along the boundary 212, wherein the aesthetic imaging system will display a line segment 220 between the two anchor points.
  • FIGURE 9C it is assumed that the user has moved the midsection of the line segment 220 to the right.
  • the blue area 218B has been stretched into the yellow area 218Y.
  • This area is bounded by the line segment 220 (now curved) and the boundary line 214.
  • the green area 218G has been stretched into the blue area 218B.
  • This area is bounded by a curved line segment 221 and the boundary line 212.
  • the red area 218R has expanded into the green area 218G; this area is bounded by a curved line segment 222 and the boundary line 210.
  • the newly defined areas will be comprised of the color being expanded.
  • the area bounded by segments 220 and 221 will be blue; the area bounded by segments 221 and 222 will be green; and the area bounded by segment 222 to the left edge of the diagram will be red.
  • the opacity level is less than 100 percent, pixels from the underlying image areas that are being written over by the newly defined areas will be blended into the newly defined areas.
  • the area bounded by segments 220 and 221 will still be primarily blue, but the portion of this area bounded by the segment 220 and the boundary line 214 may have a yellow tinge; and the portion of this area bounded by the boundary line 212 and segment 221 may have a green tinge.
  • the opacity percentage is dropped, the effects on these areas will be even greater.
  • FIGURES 9B and 9C While somewhat simplistic, the illustration in FIGURES 9B and 9C describes the function of the contour tool and the effect on more complicated pixels patterns will be appreciated by those skilled in the art.
  • the area bounded by segment 220, boundary line 214, segment 221, and boundary line 212 will not be affected by changes in the opacity setting.
  • FIGURES 9D-9G illustrate edits to a patient's lips 224 using the contour tool.
  • FIGURE 9D is a "before" picture without any modifications.
  • the user To edit the right half of the patient's upper lip, the user first selects the desired end points surrounding the feature to be modified.
  • a first anchor point 374 is designated near the middle of the upper lip, and a second anchor point 376 is designated at the right outer margin of the upper lip.
  • a line 378 is displayed between the points by the aesthetic imaging system. By floating the pen over the tablet in a direction generally indicated by arrow 380, the line is bent to form a curve 382 that approximates the contour of the feature being edited.
  • the shape of the curve is set by tipping the pen. As shown in FIGURE 9E, the user then selects a point along the portion of the curve by tipping the pen at the desired location. By floating the pen above the tablet, the user can stretch the selected feature in the manner described above. In FIGURE 9E, the user floats the pen in a direction generally indicated by arrow 384 to enlarge the patient's right upper lip. As the upper lip is expanded, the pixels forming the upper lip are replicated to expand the portion of the Up, while the pixels outside of the upper lip are deleted.
  • FIGURE 9E the left side of the person's upper Up has been edited using the contour tool.
  • FIGURE 9F both the left and right sides of the person's upper Up are shown modified using the contour tool.
  • FIGURE 9G the lower Up has been edited.
  • these edits are accompUshed very quickly, in part because the augmented area will automatically match the area around it.
  • the features of the contour tool make it much easier to perform augmentations than is currently available using prior art imaging tools. Editing Using Shape Tools
  • FIGURES 10-12B are directed to the shape tool features of the aesthetic imaging system. Similar to the draw tool, the imaging program has a combination feature that is generally available with any of the shape tools. This feature, called the autoblend feature, allows the user to easily move and paste shape tool edits, with or without blending the edges around the edit.
  • FIGURE 10 illustrates an exemplary routine for implementing the autoblend feature of the imaging program.
  • the shape tool group generally includes: Warp, Stretch, Copy, Cutout, Rotate, Freeze Compare, and Resize.
  • the routine of FIGURE 10 is invoked after a shape tool has been selected at block 80 of FIGURE 3.
  • the side bar menu for the selected shape tool is displayed at block 230. Exemplary side bar menus for various shape tools are set forth below. The side bar menus are illustrated in an effort to further detail options available for a given tool, and may be especially helpful for those unfamiUar with imaging packages.
  • the black dots are brush size options that allow a user to choose the thickness of a shaping tool.
  • the zoom option allows a user to look at an image in greater detail.
  • the aesthetic imaging system displays a square overlaid on the image. The square can be positioned by the user with the pen. After positioning the square, that area of the image will be magnified when the pen is selected. Canceling with the pen will display normal viewing mode.
  • the undo option allows a user to undo edits to an image.
  • the compare option allows a user to transition between before and after images.
  • the spUt image option allows a user view before and after images side by side.
  • the inverse option creates the mirror image of all or only a portion of an image that has been designated by the user.
  • the blend tool wUl blend edits with the surrounding area. Many of the options shown are also implemented as separate tools under View in the main menu. These are described in greater detail below.
  • the side bar menus available for the drawing tools are similar to the shape tool side bar menus shown. They do, however, typically include an option wherein the user may choose the intensity or opacity of a color used in conjunction with a draw tool.
  • the user is prompted to designate an area of the image to be edited. In a preferred embodiment, this is accomplished by pressing down on the pen and freehand drawing an area, e.g., a circle, that is to be subject to the edit.
  • the imaging program contains a unique feature wherein if a partial area is designated and the pen subsequently released, the drawing area will automatically be formed into a contiguous area by the imaging program.
  • a test is made to determine if an area has been designated by the user. If not, the routine loops back to block 234 and awaits a designation.
  • any edits to the designated area of the image are performed in accordance with the selected shape tool, as indicated at block 236.
  • Two exemplary shape tools for editing an image are iUustrated in FIGURES 12A and 12B.
  • a test is made to determine if editing of the designated area is complete. In one embodiment, this involves testing for when the user "selects" with the pen anywhere on the tablet. The routine remains at block 238 until editing is complete (or the user exits using the side button).
  • an autoblend box is displayed in the vicinity of the edited area, as indicated at block 240.
  • a test is made to determine if the tip of the pen has been pressed against the tablet. If not, the routine loops, testing for this occurrence (or an exit command from the user). After the pen tip has been pressed, the imaging program calculates the location of the pen relative to the autoblend box.
  • FIGURE 11 illustrates an example of an autoblend box 250 that may be drawn by the aesthetic imaging system in accordance with the invention.
  • the autoblend box 250 may be used to: (1) move an edited area, (2) paste the edited area onto the image while blending the edge created between the edited area and the rest of the image, and (3) paste the edited image without blending. While the autoblend box 250 uses the conventions set forth below, those skilled in the art will appreciate that other conventions may be used without departing from the scope of the invention.
  • the “move" area is designated with reference numeral 254.
  • a test is made at block 244 to determine if the pen tip has been pressed at a location outside of the autoblend box. If the pen tip was pressed at a location outside the autoblend box, the edit area is pasted with a blending of the edges at block 260. Otherwise, a test is made at block 262 to determine if the tip location was within the approximate three-, six-, nine-, and twelve-o'clock areas of the autoblend box. A pressing of the tip within any of these areas results in the edited area being pasted without blending, as indicated at block 264.
  • a selection in a location in the autoblend box apart from the three-, six-, nine-, and twelve-o'clock areas will allow the image to be moved. In this case, the edited area will track movement of the pen as long as the tip remains pressed.
  • the routine loops to block 242. If a paste has been accompUshed using blocks 260 or 264, a test is made at block 268 to determine if the user wishes to exit the shape routine, e.g., by pressing the side button. If not, the routine loops to block 232 where a new area of the image may be considered. Otherwise, the routine returns to block 82 of FIGURE 3.
  • FIGURES 12A and 12B lustrate two exemplary shape tools that are available when using the aesthetic imaging system.
  • a cutout tool is unique in that a user can select an area of the image to be cut out, thereby creating a "hole" in the image, and an identical image underneath the cutout image can then be moved in all directions as it is viewed through the hole.
  • the cutout feature is especially useful for profile views including chin augmentation, brow lifts, and maxillary and mandibular movement; and frontal views, including otoplasty, brow lift, Up augmentation, nasal base narrowing, and maxillary and mandibular movement.
  • the current image is copied to a working buffer.
  • a working buffer As is discussed in FIGURE 10, when the cutout subroutine is called the user has defined an area of the image to be edited.
  • a boundary is created around the designated area designated in block 234 of FIGURE 10.
  • the working area is displayed inside the boundary, and the current image displayed outside the boundary. In this manner, the image in the working buffer can be moved relative to the image in the current buffer until the desired alignment has been achieved.
  • the program then returns to the routine of FIGURE 10.
  • the edit may be frozen by selecting with the pen. Thereafter, the autoblend box is displayed. Selecting within the area 252 of the autoblend box aUows the designated area to be moved. Selecting anywhere outside the autoblend box will make the edit permanent, with automatic blending. Selecting within the box at the three-, six-, nine- or twelve-o'clock positions (areas 254) wiU make the edit permanent, without blending.
  • a rotate tool in accordance with the invention is particularly useful when editing profile views including the nasal tip, mandible, maxilla and brow areas; and frontal views including nares, brows, and the corners of the mouth.
  • the user has defined an area of the image to be edited.
  • the area designated in block 234 of FIGURE 10 is shown in phantom.
  • the imaging program waits for the user to enter an axis of rotation. An axis is then entered by the user, as indicated at block 279.
  • a display Une emanating from the axis point is displayed on the monitor, as indicated at block 280. Also, the number of degrees of rotation is displayed. The position of the pen dictates the degree of rotation. As the pen is moved away from the axis point, the display line will lengthen, providing the user greater control of the rotation of the designated area.
  • the system waits for the user to enter a desired degree of rotation. The degree of rotation is entered by the user by selecting with the pen, as indicated at block 283. Once the degree of rotation is entered, the designated area is redrawn onto the current image, as indicated at block 284.
  • the routine returns to FIGURE 10.
  • the autoblend box is displayed. Selecting within the autoblend box allows the designated area to be moved. Selecting anywhere outside the autoblend box will make the edit permanent, with automatic blending. Selecting within the box at the three-, six-, nine-, and twelve-o'clock positions will make the edit permanent without blending.
  • FIGURES 18A-18D disclose the capabiUties of the warp shape tool in the aesthetic imaging system.
  • the warp shape tool is a powerful tool that allows users to easily edit a patient's features. Similar to the contour tool, the warp tool allows a user to select and manipulate a feature of the patient's image, with the aesthetic imaging system automatically redrawing the area surrounding the manipulated feature as the manipulation is being performed. As shown in FIGURE 18 A, the user first defines an image to be manipulated by the warp tool by encircling the portion of the image to be edited as shown by dotted line 386.
  • a user may tip the pen to designate a stretch point within the selected area.
  • a stretch point 388 has been designated near the top portion of the patient's right upper Up.
  • the user may float the pen in the desired direction that they would like to stretch the image.
  • the portion of the image that is located at the stretch point is puUed in the direction that the user floats the pen, with the area surrounding the stretch point either expanded or compressed. That is, the area in the direction that the user floats the pen is compressed, and the area away from the direction that the user floats the pen is expanded.
  • the amount of expansion or compression is dictated by the distance of the area away from the stretch point, in a manner discussed in greater detail below.
  • the image is manipulated in real time, so that the user is presented with a seamless and continuous stretching or movement of the selected feature.
  • the technique used to implement the warp tool is portrayed in
  • FIGURES 19A-19E A user first defines a manipulation area 420 to be edited by circling the desired feature of the patient. As shown in enlarged detail in FIGURE 19B, when the manipulation area has been selected, the aesthetic imaging system creates a pixel map of a warping area 422 that completely encompasses the manipulation area.
  • the warping area is preferably rectangular and approximately 33% larger in area that the manipulation area. It w l be appreciated, however, that the warping area may be differently shaped or sized depending on the particular appUcation and available system hardware.
  • the user selects a stretch point 424 within the manipulation area by tipping the pen at a desired location within the area.
  • the aesthetic imaging system maps four rectangles 426a, 426b, 426c, and 426d in the warping area. One corner of each rectangle is defined by the stretch point, and the diagonally opposing corner is defined by a corner of the warping area. Perpendicular lines 427 drawn through the stretch point intersect the warping area boundary at points 428a, 428b, 428c, and 428d.
  • the user may float their pen to manipulate the selected feature, for example in the general direction indicated by arrow 430.
  • the rectangles in the warping area are distorted.
  • the stretch point has been moved upwards and to the left in the warping area, to an intermediate location 432.
  • the aesthetic imaging system determines the shape of four quadrilaterals 434a, 434b, 434c, and 434d, with sides that extend from the intermediate location 432 of the stretch point to original points 428a, 428b, 428c, and 428d.
  • the pixels in the original rectangles 426a, 426b, 426c, and 426d are then mapped into the quadrilaterals.
  • the aesthetic imaging system also remaps the warping area, creating four new rectangles 436a, 436b, 436c, and 436d based on the ending location 438 of the stretch point.
  • a second stretch point may then be selected within the manipulation area and the process repeated, with the second warping transforming rectangles 436a, 436b, 436c, and 436d that resulted from the first warp.
  • the user may then select a third or additional warp point to further manipulate the image. Each manipulation is performed without the user having to redefine the manipulation area.
  • FIGURES 18A-18C The flexibUity and power of aUowing multiple warps on a selected region is demonstrated in FIGURES 18A-18C.
  • a user designates a stretch point 388 and floats the pen in a direction generaUy indicated by arrow 390. Floating the pen upwards and outwards generally causes the upper right portion of the Up to be expanded in that direction.
  • the resulting manipulation is shown in FIGURE 18B, wherein the right Up has been made fuller.
  • the area surrounding the Up is expanded or compressed to ensure that there are no discontinuities between the edited Up and the surrounding face.
  • the user selects a second stretch point within the manipulation area to further stretch the selected feature.
  • a stretch point 392 may be positioned on the upper portion of the left Up, and the pen floated in a direction generally designated by the arrow 394. As shown in FIGURE 18C, this generally causes the upper left portion of the Up to be made fuUer to match the upper right portion of the Up. Again, during the warping the aesthetic imaging system automatically expands or contracts the surrounding unmanipulated area to ensure that there are no discontinuities between the upper left and the unedited portion of the face.
  • the warp tool with multiple stretch points is a very powerful tool as it allows the user to quickly manipulate an image with a minimum use of drawing tools or piecemeal editing. Because the warp tool performs the manipulation in real-time, the edits are accompUshed very quickly and fiuidly. A user may therefore generate a desired image in a ininimal amount of time. Viewing an Image
  • FIGURE 13 Ulustrates an exemplary routine for implementing the view features of the imaging program.
  • the soUd blocks indicate user interface options presented to the user by the aesthetic imaging system and the dashed blocks represent system responses to the decisions made.
  • the view group includes: Compare, Prioritize, Split Image, Mirror Image, and Restore to Original, as weU as other options including Zoom and Emboss.
  • a test is made to determine if the Compare option has been selected.
  • the Compare option allows a modified image to be compared to the original image so that a viewer can more readily see the changes. Specifically, as the pen is floated from the top to the bottom of the tablet, the user will see one image transition or "morph" into the other.
  • the morphing is accompUshed by overlaying the original image with the modified image, and varying the opacity of the modified image.
  • the modified image is opaque, only the modified image may be viewed by a user.
  • the modified image is completely transparent, only the original image may be viewed by a user. In between these two extremes, varying amounts of the edits made to the image will be apparent to the user.
  • the feedback to the patient as the original image morphs into the modified image is much more powerful than a side-by-side comparison of the two images.
  • a user can press the tip of the pen to freeze an image displayed on the monitor at a point anywhere from zero to 100% of the transition from the original to the modified image.
  • Freezing an image at a partial transition is extremely helpful where edits have been performed on an image that are not reaUstically achievable in surgery, but an achievable result Ues somewhere between the original and the modified image. For example, it is easy to edit a blemish on a face so that area resembles the surrounding skin and thus becomes invisible. However, the total removal of the blemish may not be realistic. In this case, a transition of that area toward the original image will slowly "fade in" the blemish. A physician may then freeze the fading process at a desired point to provide a reaUstic image of what surgery can achieve to the patient.
  • FIGURE 14A illustrates a modified profile image 302 of a patient that includes a rhinoplasty procedure (nose) 304, a chin augmentation procedure 306 and a submental lipectomy procedure (neck) 308.
  • the boundaries that have been edited are illustrated by dashed lines 304a, 306a, and 308a, corresponding to the patients original nose profile, chin, and neck, respectively.
  • the user can designate one area on the modified image, and Ulustrate transitions between the original and modified images at that area only by floating the pen. Any areas not selected will continue to be displayed as the original image.
  • a test is made at block 309 to determine if the entire image is to be compared or only certain portions of the image, i.e., using the Prioritize option. If less than the entire image is to be compared, the user is prompted to enter the area or areas that are to be compared at block 310. A user may then define one or more "priority areas" by freehand circling the desired area. When the priority areas or defined, or if all of the edits are to be reviewed during the comparison, at block 311 the user is prompted to float the pen in a vertical motion on top of the tablet to transition between the original and modified images, in accordance with the Compare feature discussed above. An Ulustration will clarify this point.
  • a first priority area 312 has been defined that corresponds generally to the nose. Given this selection, the nose area only wiU transition from original to modified as the pen is moved, with the rest of the image being displayed unedited. Thus, the modifications to the chin and neck no longer are shown.
  • a second priority area 314 has been defined that corresponds generaUy to the chin. The first priority area 312 has been kept. Given these selections, the nose and chin areas only will transition from original to modified as the pen is moved, with the rest of the image being displayed unedited. Thus, the modifications to the neck are not iUustrated.
  • a third priority area 316 has been defined that corresponds generaUy to the neck, along with the former designations. Given these selections, all three priority areas 312, 314, and 316 transition with movement of the pen. Again, the undesignated portions of the image are displayed in an unedited form, even if parts of the image outside the priority areas have been edited (no edits are shown). Because edits have not been made, from a user's standpoint the transition in FIGURE 14D appears to be a comparison between the original and modified images. While preferably the user defines the priority areas on the image being edited, it will be appreciated that the priority areas may also be automaticaUy defined by the aesthetic imaging system.
  • a comparison may be made between an original image stored in a buffer and the edited image that has been modified by the user. Any areas containing differences between the original and the edited image may be highUghted by the aesthetic imaging system, and a priority area automaticaUy defined for each area containing differences. Whether a priority area is defined may also depend on the number of differences between the original image and the edited image.
  • a user when a user defines one of the priority areas in the Prioritize option, the user is prompted to enter a textual identifier for the defined area. For example, after circling the area in FIGURE 14B and defining a first priority area 312, the user would be prompted to enter a textual identifier corresponding to the first priority area.
  • the user may assign a descriptive identifier related to that priority area, such as "nose", or the user may assign a non-related textual identifier such as "areal”. Textual identifiers assigned by the user in this manner are displayed to the user in a submenu of the Prioritize option.
  • the user may then select from the submenu of the Prioritize option those areas that they wish to display from the Ust presented to them.
  • the prioritize submenu corresponding to FIGURE 14D may present the user with a choice of "nose,” "chin,” and "neck.”
  • the user could simultaneously display only the effects of the nose and neck procedures.
  • the user could deselect neck from the prioritize submenu, and instead select the chin and nose. This would allow the user to display only the effects of the chin and nose procedures using the compare option.
  • assigning textual identifiers to each of the defined priority areas provides greater flexibiUty to a user, since the user does not have to redefine each of the priority areas that are to be displayed each time. A user may therefore quickly cycle through various permutations of the procedures that have been edited for patient display.
  • the priority area could be highUghted or otherwise encircled with a contrasting color to indicate to the user the area of the image that corresponds to that textual identifier.
  • a user may also point to specific areas of the displayed image and have the textual identifier corresponding to that area appear to the user. Displaying the correspondence between priority area and identifier would allow a user to rapidly determine the available priority areas that may be shown to a patient when the user has not recently worked with or otherwise viewed the image.
  • a test is made to determine if the user wishes to save a transitional or morphed view of an image. If a transitional view is to be saved, the user may establish the percentage transition, i.e., anywhere from zero to 100 percent transition (zero percent being the original image and 100 percent being the edited image), by floating the pen up or down above the tablet to estabUsh the view, and the pressing the tip of the pen against the tablet to freeze the transitional image, as indicated in block 322. If the tip is pressed again, the frozen image is saved.
  • the save options are available with or without the priority areas in effect.
  • the Compare option allows a user to compare a modified image with any edits made to the modified image during the current editing session, i.e., before the changes are permanently saved to the modified image.
  • this embodiment of the Compare option contrasts the image in the current image buffer 54 with the image in the modified image buffer 52.
  • this embodiment of the Compare option may also be used in conjunction with the Prioritize option to aUow the user to select priority areas for comparison. In this case, the priority areas transition from the modified to the current image, while the modified image only is displayed in the other (nonselected areas) areas.
  • a test is made to determine whether the user wishes to view a spUt image.
  • the Split Image option is used on a frontal picture only, and allows a patient to see his or her asymmetries. If a spUt image view is desired, the user is prompted to select an image, e.g., original or modified, at block 330. At block 332, a vertical centerline is displayed on top of the selected image. The user is then prompted to position the centerline at the location desired, as indicated at block 334. TypicaUy, the centerline wUl be positioned to dissect the image into equal halves, using the nose and the eyes as reference points.
  • the aesthetic imaging system displays two images, one showing the left halves pieced together and the other the right halves pieced together. Specifically, the aesthetic imaging system will produce an inverse image of the left (right) half and then add it to the left (right) half.
  • FIGURES 15A-15C illustrate the resultant images that are displayed when the
  • FIGURE 15 A a frontal image 350 of a patient is shown, including a centerline 352 that has been positioned at the center of the patient by a user.
  • FIGURE 15B is an iUustration of the left halves of the image after being pieced together by the aesthetic imaging system, as indicated by reference numeral 354.
  • FIGURE 15C is an iUustration of the right halves of the image, as indicated by reference numeral 356.
  • FIGURES 14A-14D When viewing an image on the monitor, it is important that a patient remains focused on the image being manipulated rather than on the features of the aesthetic imaging system.
  • the menu bar normally located across the top of the display is therefore removed. Only the image of the patient is kept on the screen, ensuring that the patient remains focused on the image being manipulated. In certain situations where it is necessary to display a menu bar, however, it is advantageous to minimize the visual appearance of the menu bar.
  • FIGURE 20 is a representative image 410 of a patient with a menu bar 412 located across the top of the screen 414.
  • the menu bar is preferably translucent to allow the user to view the patient's image through the menu bar.
  • the text 416 and the line 418 indicative of the menu bar are preferably presented in a contrasting, yet muted, color to allow the user to read the commands.
  • the text and the line outlining the menu bar may be presented in an off- white. WhUe editing the image with a patient present, the patient is therefore not overly distracted when the menu bar periodically appears at the top of the screen. Even when the menu bar is present for extended periods of time, a patient is not distracted since it does not visuaUy stand-out from the patient's image. At the same time, a skilled user is provided with sufficient information about the menu choices to allow them to choose the appropriate menu options.
  • the menu bar 412 may be moved by a user to different locations on the screen 414.
  • the menu bar may be moved by a user to the bottom of the screen. Particularly when editing the face of a patient on the screen, the majority of the editing will take place on the upper two-thirds of the screen. Locating the menu bar on the bottom of the screen therefore positions the menu bar away from the area on which the patient should remain focused. It will be appreciated that techniques for moving a menu bar to various locations on a screen are known to those skiUed in the art of designing user interfaces.
  • FIGURE 21 is a flow chart of an exemplary routine for implementing a zoom feature in the aesthetic imaging program.
  • the zoom feature allows a user to increase the scale of the image to better view a selected area and to improve the abiUty of the user to edit fine details in the image.
  • a zoom point is selected by a user.
  • the zoom point identifies the center of the image to be expanded under the control of the user.
  • the picture is adjusted to position the zoom point at the center of the monitor. Centering the picture ensures that as the image is enlarged, the portion of the image surrounding the zoom point will be displayed.
  • the user is allowed to input a desired magnification factor.
  • the magnification factor is selected by floating the pen from the bottom (minimum magnification) to the top (maximum magnification) of the tablet.
  • the image on the monitor is correspondingly magnified and redisplayed at a block 406.
  • a test is made to see if the user has frozen the image by pressing or tipping the pen. Once the image is frozen at a desired magnification, a user can manipulate the image using the array of drawing tools described above.
  • the abiUty to magnify the image increases the quaUty of the editing that may be achieved.
  • a smoothing function may be incorporated in the zoom feature to ensure that as the image is magnified it does not become "pixelly" or grainy.
  • the smoothing function may be implemented in software.
  • the smoothing function is implemented in hardware, such as a smoothing feature provided in the Targa 2000 board described above and incorporated in the aesthetic imaging system.
  • feedback may be provided to the user in the form of a numerical display on the image to indicate the approximate magnification as the user floats the pen from the bottom to the top of the tablet.
  • Other means can also be implemented to allow the user to select the desired magnification, including a pull-down menu or numerical entry.
  • emboss viewing option is very helpful in allowing a user to discern wrinkles or other skin imperfections in a displayed image.
  • the user By selecting the emboss option, the user causes an image of the patient to be displayed in a gray scale.
  • the emboss option displays an image that is similar to an etching made of a three-dimensional raised surface. A two dimensional image is portrayed, with the depth of the raised surface indicated by a darker shade of gray.
  • the emboss option removes any deceptive information conveyed by the color or shading of the skin of the patient and allows any raised or depressed areas to be clearly highUghted. When viewing wrinkles or other imperfections on a patient, the emboss option therefore clearly identifies the raised or depressed features over the smooth skin of the patient.
  • the emboss option is implemented using a function provided in the Targa 2000 board described above and incorporated in the aesthetic imaging system.
  • the user may manipulate the offset of the emboss by floating the pen over the tablet.
  • Floating the pen parallel to the x-axis of the tablet adjusts the dx parameter in the emboss function
  • floating the pen parallel to the y-axis adjusts the dy parameter in the emboss function. It has been found that various features on the skin of a patient can be brought into greater clarity by adjusting the offsets of the embossed image. A user may therefore focus on the desired feature that they wish to show a patient. When the desired offsets are selected, the user may freeze the offsets by tipping the pen on the tablet. A user may then adjust the scale parameter by floating the pen parallel to the x-axis of the tablet.

Abstract

aisclosed is an aesthetic imaging system (20) for use in editing digital images. The aesthetic imaging system includes an imaging program (21) that runs on a personal computer (28) having an image capture board (30), a monitor (32), a video source (34) for providing digital images to be edited by the aesthetic imaging system, and a pen and tablet (38) for use in editing the images. The imaging program includes a unique combination draw tool that includes a freehand draw mode, a curve mode and an undo mode that are available without cycling through menus. The combination draw tool may be used with any of the draw tools. Another feature of the imaging program is autoblend, a rectangular user interface that is invoked by each of the shape tools. The autoblend interface simplifies editing when using shape tools by consolidating the move, paste and blend, and paste without blending commands into a single, convenient interface.

Description

AESTHETIC IMAGING SYSTEM
Relationship to Other Applications This application is a continuation-in-part of U.S. patent application Serial No. 08/406,201, filed March 17, 1995, the benefit of the filing of which is hereby claimed under 35 U.S.C. § 120.
Field of the Invention This invention generally relates to computer imaging programs and, more specifically, to a method and apparatus for manipulating digital photographs.
Background of the Invention The digital age continues to present additional opportunities for visual communication using computers. As an example, digital photographs are routinely being manipulated to produce a desired effect or result in the magazine and film¬ making industries. In the medical field, computer-based imaging has and continues to gain acceptance in a clinical setting as a viable communications tool between plastic or "cosmetic" surgeons and potential patients.
People are with increasing frequency consulting physicians about cosmetic surgery. While in many cases the patients considering cosmetic surgery have an impressive understanding of the procedures available and medical terms used to describe these procedures, it is apparent that the slightest miscommunication may result in dire consequences. This has promoted the use of computer imaging to facilitate communication between the physician and prospective patient. Specifically, high-end aesthetic imaging systems allow a physician to take pre-operative digital images of the patient, e.g., including profile and frontal views. The images are stored in memory in the computer where they can then be edited. Using feedback from the patient, the edited images are useful in fully understanding the procedures desired. The visual support provided by a computer-based imaging system is extremely valuable on both sides. A cosmetic surgeon can more readily understand what patients hope to achieve by a cosmetic surgical procedure; and patients can view a detailed visual representation of predicted results, including both the benefits and limitations of the procedure.
Existing aesthetic imaging systems typically use a number of tools to allow a physician to manipulate a pre-operative image of a patient to illustrate an intended post-operative result. Preferably, the tools allow the physician to manipulate the preoperative image during a consultation with a patient. By manipulating the image with the patient in attendance, the patient receives immediate feedback from the displayed results. For a successful preoperative consultation, the use of the editing tools should be as unobtrusive as possible. During the consultation, a physician would like the patient to focus on the end results of the surgery, rather than the technologic wizardry used to demonstrate those results on the video monitor. Unfortunately, the editing tools used in existing aesthetic imaging systems typically hinder rather than help the physician in demonstrating the results that may be achieved through cosmetic surgery.
Among others, a disadvantage of existing aesthetic imaging programs is that a physician or facilitator in a pre-operative consultation typically must go back and forth through many windows-based menus in order to edit an image. Cycling between the various menus to invoke the tools necessary for a consultation is disadvantageous in that it is time consuming. For example, some physicians schedule a follow-up visits for patients to allow the physician time to edit the images. More important, however, is that the process is distracting to the patient and tends to make the pre-operative consultation all the more mystifying. As a result of the disadvantages associated with prior art systems, some patients lose interest or become frustrated with the interview, both of which may reflect back on the physician.
A further disadvantage of existing aesthetic imaging systems is that it is impossible for a physician or facilitator to display different combinations of the edits that they have performed. In existing aesthetic imaging programs, as a physician edits a patient's image, the physician's edits are added to the preexisting edits of the image. Most programs are only capable of showing two version of the patient's image; the unedited, original version, and the final edited version incorporating all of the physician's changes. It is therefore difficult for the physician to show various combinations of the edits that had been performed. For example, a physician may edit an image to remove wrinkles around a patient's eyes and to narrow the patient's nose. Existing aesthetic imaging programs only allowed the physician to simultaneously show all of these changes. If, for example, the patient wished to view the changes to the eyes without the changes to the nose, it was difficult for the physician to easily separate out the sequence of edits that had been performed to adjust the patient's nose. An improved aesthetic imaging system in which a physician can more easily edit pre-operative images in response to a patient's suggestions and inquiries would be extremely advantageous.
A still further disadvantage of existing aesthetic imaging systems is that the systems allow a physician to perform nearly flawless editing of a patient's image. Unfortunately, the edits performed by a physician on an aesthetic imaging system are often unobtainable results that cannot be achieved when actual surgery is performed. Unless the physician is especially skilled at using the aesthetic imaging system, it is difficult to show the patient achievable results, which typically fall within a range somewhere between the original patient image, and the optimum results as displayed by the edited image on the screen. It therefore would be advantageous to develop an aesthetic imaging system that allowed a physician to display more realistic results that are achievable through surgery.
Summary of the Invention
The invention is an aesthetic imaging system for use in editing digital images. The aesthetic imaging system includes a unique user interface that allows edits to be performed more efficiently and with less confusion to the patient.
In one aspect of the invention, a method of editing a digital image comprised of a plurality of color pixels in an aesthetic imaging system is disclosed. The aesthetic imaging system including a processor, a memory, a monitor, and a pen and cooperating tablet for controlling a cursor displayed on the monitor. The pen has a depressable tip and a side button, each of which include an on status and an off status, wherein the position of the pen tip relative to the tablet determines the position of the cursor on the monitor. The method comprises: (a) evaluating the following variables: (i) the status of the tip of the pen; (ii) the status of the side button on the pen; and (iii) movement of the pen tip relative to the tablet; (b) actuating a freehand drawing mode if a first set of variables are present, wherein movement of the pen relative to the tablet edits pixels that are located at positions corresponding to the position of the cursor; and (c) actuating a curve drawing mode if a second set of variables are present, wherein a line segment is displayed between two endpoints and movement of the pen relative to the tablet stretches the line segment, forming a curve and editing pixels that are located at positions corresponding to the position of the curve.
In another aspect of the invention, method further includes: (a) actuating a freehand undo mode if a third set of variables are present, wherein movement of the pen relative to the tablet restores pixels that are located at positions corresponding to the position of the cursor to their pre-edited color; and (b) actuating a curve undo mode if a fourth set of variables are present, wherein a line segment is displayed between two endpoints and movement of the pen relative to the tablet stretches the line segment, forming a curve and restoring pixels that are located at positions corresponding to the position of the curve to their pre-edited color.
In other aspects of the invention, the freehand draw mode is actuated if the tip of the pen is depressed and pressure is maintained while the tip is moved a predetermined distance. In another aspect, the curve draw mode is actuated if the tip of the pen is depressed and released within a predetermined distance.
In a further aspect of the invention, the curve draw mode is actuated by: (a) establishing a first endpoint at the position of the pen when the second set of variables are present; and (b) monitoring the status of the tip of the pen and establishing a second endpoint at the position of the pen if the tip is toggled from an off state to an on state.
In another aspect of the invention, an improved prioritize feature is described for viewing an image. A user may identify several areas in a modified patient image containing edits that alter the image from the original image. As each area is identified by the user, an identifying tag is assigned to each of the areas. When desiring to show various combinations of the edits that have been performed on the image, the user may select the areas to display using the identifying tags. A user may therefore quickly cycle through various permutations of the procedures that have been edited for patient display.
In still another aspect of the invention, an improved user interface is provided to minimize the distraction of a patient as the patient is watching the image being edited. Preferably, a menu bar on the top of the display is removed during most editing, so that only the image of the patient is displayed. When the menu bar must be displayed, the bar itself is transparent to allow the patient to see the image through the menu bar. Only the commands and the outline of the menu bar are presented in a contrasting color, minimizing the overall visual impression created by the menu bar. In yet another aspect of the invention, a warp tool is described that allows a user to quickly and easily manipulate various features in an image. To use the warp tool, a user first encircles a portion of the image to be edited. Once the area has been selected, the user may tip the pen to designate a stretch point within the selected area. As the user floats the pen over the tablet, the image is then stretched as if pulled from the stretch point. Areas of the image in the direction of stretch are compressed, and areas away from the direction of stretch are expanded. Areas surrounding the warping area are automatically adjusted to ensure that there is no discontinuities with the warping area. The manipulation of the image is performed in real-time, allowing a patient to see the warping as it is being performed by a user.
An advantage of the tools and features described herein are that they improve the overall experience of a patient during a preoperative visit with a physician. The powerful tools in the aesthetic imaging system allow the physician to easily manipulate the patient's image in response to feedback provided by the patient. The aesthetic imaging system interface also allows the patient to focus on the image being manipulated, rather than on the aspects of the aesthetic imaging system that allow the manipulation. The end result is an improved preoperative visit that provides a more realistic impression of the results that a physician may achieve through surgery.
Brief Description of the Drawings The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same becomes better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:
FIGURE 1 is a block diagram of an aesthetic imaging system in accordance with the invention;
FIGURE 2 is a block diagram illustrating various buffers used by the aesthetic imaging system to store and manipulate data;
FIGURE 3 is a flow chart illustrating an exemplary routine by which digital images may be viewed and edited using the aesthetic imaging system; FIGURE 4 A is a flow chart of an exemplary routine for photographing patients in accordance with the invention;
FIGURE 4B is a flow chart of an exemplary routine for calculating a checksum value and comparing the calculated value to a previously stored value to determine if an image has been altered; FIGURE 5 is a flow diagram of an exemplary routine for implementing a combination tool for use with various drawing (draw) tools in accordance with the invention;
FIGURE 6 is a pictorial representation of an image to be edited; FIGURES 7A-7E are pictorial representations of editing an image using a prior art imaging program;
FIGURES 8A-8E are pictorial representations of using the aesthetic imaging system to accomplish the identical edits shown in FIGURES 7A-7E;
FIGURE 9A is a flow chart of an exemplary routine of a contour tool in accordance with the invention;
FIGURES 9B-9C are pictorial representations illustrating the function of the contour tool of FIGURE 9 A;
FIGURES 9D-9G are pictorial representations illustrating exemplary edits that may be accomplished using the contour tool; FIGURE 10 is a flow chart of an exemplary routine for implementing an autoblend tool in accordance with the invention;
FIGURE 11 is a pictorial representation of a user interface for implementing the autoblend tool of FIGURE 10;
FIGURE 12A is a flow chart of an exemplary routine illustrating a cutout tool in accordance with the invention;
FIGURE 12B is a flow diagram of an exemplary routine illustrating a rotate tool in accordance with the invention;
FIGURE 13 is a flow chart of an exemplary routine for viewing images in accordance with the invention; FIGURES 14A-14D are pictorial representations illustrating the effects of a compare feature in accordance with the invention;
FIGURES 15A-15C illustrate a split image option of viewing images in accordance with the invention;
FIGURE 16 is a pictorial representation illustrating the use of a translucent image to allow a patient to accurately position themselves in order to capture a second image having the same location and orientation as an original stored image;
FIGURE 17 is a pictorial representation illustrating a compare image wherein a presurgical image of a patient is compared side-by-side with a postsurgical image having the same location and orientation; FIGURES 18A-18C are pictorial representations illustrating the use of a warp shape tool to edit a patient's image FIGURES 19A-19E are pictorial representations illustrating the function of the warp tool;
FIGURE 20 is a pictorial representation illustrating the use of a transparent menu bar when viewing an image of a patient; and FIGURE 21 is a flow chart of an exemplary routine for implementing a zoom viewing feature in accordance with the invention.
Detailed Description of the Preferred Embodiment An aesthetic imaging system 20 in accordance with the invention is illustrated in FIGURE 1. The aesthetic imaging system 20 includes an imaging program 21 that runs on a processing unit 22 controlled by an operating system 24. A memory 26 is connected to the processing unit and generally comprises, for example, random access memory (RAM), read only memory (ROM), and magnetic storage media such as a hard drive, floppy disk, or magnetic tape. The processing unit and memory are typically housed within a personal computer 28 which may be, for example, a Macintosh™, International Business Machines (IBM™) or IBM-compatible personal computer. When used with IBM and IBM-compatible personal computers, the operating system 24 may be DOS based or may incorporate a windowing environment such as Microsoft Windows™ or OS/2™.
The aesthetic imaging system also includes an image capture board 30 that is coupled to the processing unit 22, a monitor 32, video source 34, and printer 36. The video source, monitor, and printer are coupled to the processing unit 22 through the image capture board 30. The video source may include one or more video cameras, a VCR, a scanner, or similar source for providing digital images to be edited by the aesthetic imaging system. The aesthetic imaging system further includes a pointing device, which is preferably a stylus (pen) and tablet 38, that is connected to the processing unit 22. In addition, the aesthetic imaging system may include a modem 40 to provide on-line capabilities to users of the system, such as technical support and teleconferencing.
The image capture board 30 has a plurality of buffers in high-speed memory, e.g., RAM, that are used by the imaging program 21 to provide very fast response times to image edits. With reference to FIGURE 2, four buffers are illustrated for use in explaining the operation of the aesthetic imaging system. These include an original image buffer 50, a modified image buffer 52, a current image buffer 54, and a working buffer 56. Suitable image capture boards for use in the aesthetic imaging system include the Targa +64 and Targa 2000 boards, distributed by Truevision, Inc. of Indianapolis, Indiana. The buffers are discussed in regard to a single pose only, such as a profile or front view of a person.
The original image buffer 50 contains an unedited digital image, for example, a side profile picture of a potential patient. The modified image buffer 52 contains any edits made to a copy of the original image. The modified image buffer is updated during a save and after each session. The current image buffer 54 contains information identical to the modified image buffer upon beginning a session. Thereafter, edits made to the current image are saved in the working buffer 56 as an overlay to the current image. During a save, the contents of the current image buffer 54 are copied to the modified image buffer 52, and the working buffer 56 is cleared.
Prior to discussing the aesthetic imaging system in further detail, a compendium of terms used in the application may be helpful:
Image A digital photograph or picture of a patient.
Stylus The "pen" that may be used to select menus, modify images, and carry out other commands in the program. The stylus controls the cursor, just as a mouse pointing device does on a personal computer.
Tablet, or Pad The electronic notepad used in conjunction with a stylus. The pen must be held relatively close to the pad in order for the pen to communicate with the tablet. Unlike a mouse, the tablet follows an X / Y grid that matches the monitor, i.e., if the pen is positioned at the top left corner of the tablet, the cursor is displayed at the top left corner on the monitor.
Floating Moving the pen to move the cursor, without actually touching the tablet.
Selecting Selecting (also referred to as "tipping" or "pressing") the tip of the pen briefly onto the tablet. This selects a command or affects the drawing tool, depending on the current procedure being implemented.
Cancel Using the side button on the pen to execute a command.
Moving Pressing the tip of the pen on the tablet, releasing it, then moving it across the tablet.
Pressing or Tipping & Dragging Pressing the tip of the pen, then dragging it across the tablet while maintaining pressure. When using drawing and shaping tools, this turns the cursor into a drawing tool, enabling the user to draw freehand objects or "brush" the image in any manner.
FIGURE 3 illustrates an exemplary routine for implementing the imaging program 21 in accordance with the invention. At block 60, a system startup is performed wherein the computer looks for peripheral devices that are connected to the aesthetic imaging system, the memory is tested, and any other startup procedures needed to get the system up and running are implemented. At block 62, the imaging program displays a main menu, which provides access to the various features of the imaging program. Specifically, the main menu includes the following options: Storage, Camera, View, Shape, Draw, Print, And Exit. Those options that are pertinent to the invention are described in further detail below.
At block 64, a test is. made to determine if the Camera option has been selected from the main menu, indicating that the user wants to take a picture of a patient. If the Camera option has been selected, a routine to implement this command is called at block 66. A suitable subroutine for this task in illustrated in FIGURE 4. Upon return from the Camera routine, the program loops to block 62.
If the Camera option was not selected, a test is made at block 68 to determine if the Storage option has been selected for the main menu, indicating that the user wants to load an image (picture) from computer storage, e.g., a hard drive. If the Storage option was selected, the image(s) to be edited or viewed during the current session are selected at block 70. At block 72, the selected images are copied to the appropriate buffers in the frame grabbing board, as described in FIGURE 2 and the accompanying text. For example, if the selected image is an original image that has not yet been edited, the original image will be copied to the original, modified, and current image buffers. If the selected image is an image that has previously been modified, the original image is copied to the original image buffer 50 and the modified image is copied to both the modified and current image buffers 52 and 54. It will be appreciated that the number of images that may be loaded at one time will be limited, in part, by the capacity of the frame grabbing board. The program then loops to block 62. If the Storage option was not selected, a test is made at block 74 to determine if the Draw option has been selected from the main menu. If the Draw option has been selected, a draw tool routine is called at block 76. The program then loops to block 62. A suitable routine for implementing the Draw option is shown in FIGURE 5. If the Draw option was not selected, a test is made at block 78 to determine if the Shape option has been selected. If the Shape option has been selected, a shape tool routine is invoked at block 82. A suitable routine for implementing the Shape option is shown in FIGURE 10. Otherwise, at block 82 a test is made to determine if the View option has been selected. If the View option has been selected, a view subroutine is invoked at block 84.
An appropriate routine for the View option is shown in FIGURE 13. The program then loops to block 62. At block 86 a test is made to determine whether the Exit option has been selected from the main menu. If not, the program loops to block 62. Otherwise, any edits to the image are saved at block 88. At this point in the program, the image in the current image buffer 54 is saved to the modified image buffer 52, and the working buffer 56 is cleared. The program then terminates. Taking Pictures of a Patient Using an Inverse Image
FIGURE 4A illustrates an exemplary user interface that utilizes a video camera for acquiring a digital image of a patient. It is noted a scanner or other input device may also be used to input an image into the aesthetic imaging system. In FIGURE 4A, the solid blocks indicate user interface options presented to the user by the aesthetic imaging system and the dashed blocks represent system responses to the decisions made. At block 100, a patient is positioned in front of the video camera. In a preferred embodiment of the aesthetic imaging system, an inverse or "mirror" image of the patient's image will be displayed on the monitor, as indicated at block 101. The inverse image is computed using data from the original image, and is representative of how patients see themselves day to day when looking into a mirror. More specifically, digital images are comprised of pixels or picture elements. It is known to those skilled in the art how digital image pixels may be manipulated to create an image that is the inverse of the original. Displaying an inverse image of a patient is advantageous when taking pre- and post-surgical pictures of patients because it allows patients to more easily center or otherwise position themselves on the monitor. Without the pixel manipulation, the input from a camera or other digital device may create confusion when positioning a patient. Under normal viewing, if a patient appears left of center in the monitor, they are in reality too far to the right. In this instance, a typical patient's reaction is to move even further to the right. With a mirror image displayed, the tendency of most patients is to naturally adjust to the desired position.
The displaying of a mirror image is particularly important when taking post¬ surgical pictures. In post-surgical pictures, it is advantageous to have the patient in exactly the position they were in when taking the pre-surgical picture. For post¬ surgical pictures, the aesthetic imaging system will preferably display a translucent inverse image of the pre-surgical picture on the monitor, and then overlay an inverse image of the picture currently being taken. As shown in FIGURE 16, a translucent patient image 370, in this case a patient's profile, is displayed on the monitor. The translucent image is the preoperative image taken prior to undergoing a surgical procedure. A "live" video image 372 of the patient is also displayed under the translucent inverse image. By moving the relative positions of the patient and the camera, a user may position the patient in the identical orientation with which they took the presurgical picture. When the patient is appropriately positioned, the tip of the pen may be pressed anywhere on the tablet to freeze the digital image on the monitor. Patients can thus easily align themselves with their former picture to achieve very similar before and after pictures.
The advantage of the translucent method of aligning a patient is shown in the side-by-side display shown in FIGURE 17. As shown in FIGURE 17, the aesthetic imaging system of the present invention may generate a side-by-side display of the two images to allow a patient to easily and accurately compare a presurgical picture with a post-surgical picture. The left half of the monitor may display the presurgical image 370, and the right half of the monitor may display the post-surgical image 372. Allowing a patient to view the two images side-by-side in precisely the same orientation provides the patient with an accurate impression of the results achieved by surgery. Upon establishing a desired position for the patient, the image is focused and sized at block 102 by using the aesthetic imaging system to adjust the electronic controls on the video camera. After any adjustments have been made to the camera, at block 104 the tip of the pen is pressed anywhere on the tablet to freeze the digital image onto the monitor. At block 106, the user makes a determination if the image currently displayed on the monitor is acceptable. If the image is not acceptable, the routine loops to block 100. If the image is acceptable, an appropriate command is entered at block 108 and the image is stored in nonvolatile memory for future viewing. At block 110, a test is made to determine if an exit or other similar command has been entered by the user, i.e., if any more pictures are to be taken. If additional pictures are to be taken, the program loops to block 100. Otherwise, at block Il i a checksum value (described below) is calculated by the imaging program for each (original) image that has been stored. At block 112, the imaging program stores each image and its associated checksum value. The routine then returns to block 68 of FIGURE 3. Determining Authenticity Using Checksum Values
The checksum value is an addendum to an original stored image that is used to determine its authenticity when the image is subsequently displayed or printed. Those skilled in the art will recognize that there are a number of methods of implementing such a checksum procedure. For example, one checksum computation is to add up the grayscale values for one of the colors, i.e., red, green, or blue, for each pixel comprising the image. Assuming a screen of 640 by 480 pixels and 256 colors per pixel, the checksum values would range from 0 to (640)(480)(255). When an image is recalled for display or to be printed, the checksum value is recalculated. If the image has not been modified, the newly calculated checksum value will be equivalent to the addendum value, and the image is certified as being unaltered. If the image has been modified, the checksum values will vary, indicating the image has been modified. In this instance, an indication of the fact that the image has been modified may be displayed or printed with the image, if desired.
The authentication of an original image using a checksum value is ideal for situations in which physicians display before and after pictures of a patient who has undergone cosmetic surgery. In some instances, viewers are skeptical as to whether an "after" image is really representative of a patient's appearance after surgery. This is in reaction to beliefs that post-surgical images have been altered to make the patients look better. For example, there may be temptation to slightly fade wrinkles or otherwise edit features that the physician was attempting to address in a surgery. Using the described checksum feature, the post-surgery image can be verified as an authentic, unaltered image based upon the addendum value, and the veracity of the image is not questioned. This is beneficial to physicians when illustrating post- surgical results during lectures or in other teaching situations.
FIGURE 4B illustrates an exemplary routine for determining whether or not an original image, i.e., pre- or post-surgical image, has been altered in accordance with the invention. This routine may be invoked whenever a pre- or post-surgical image is displayed on a monitor or printed on a page. At block 114, a test is made to determine whether the image to be displayed is portrayed as an "original image" that was not modified, e.g., a before or after picture. If the image is supposed to be an original image, a current checksum value for the image is calculated at block 115. At block 116, the calculated checksum value for the image is compared to the checksum that was stored when the image was acquired by the aesthetic imaging system, e.g., when the picture was taken.
At block 117, the calculated checksum value is compared to the stored checksum value to see if they are equivalent. If the two values are not equivalent, at block 118 an icon is added to the image, e.g., displayed or printed along with the image, indicating that the image has been altered. If the checksum values are equivalent, an icon is added to the image at block 119 verifying the authenticity of the image. Once either icon has been added to the image, or if the image being printed or displayed is not an original image, the program terminates. As will be appreciated, the same checksum computational method must be used on each image, i.e., when an original image is acquired and when an image is to be displayed, or the comparison will be meaningless.
While the use of a checksum is contemplated in the exemplary routine for determining whether an original image has been altered, it will be appreciated that other techniques may be used to detect when an original digital image has been modified. For example, a flag or other marker may be uniquely associated with the original digital image. If the flag or other marker is absent from the digital image being displayed, the digital image is a copy that is presumed to have been changed. Alternatively, two versions of the image may be stored, included an unaltered original and a copy. The two images may be compared in order to determine whether modifications have been made to the copy. The editing aspects of the invention are now described.
Editing Using Draw Tools A disadvantage of prior art aesthetic imaging systems is that a physician or facilitator may have to cycle back and forth between several menus in order to properly edit an image. In an effort to minimize the number of menus required, the imaging program includes a unique combination draw (CD) feature that generally works with all of the drawing tools. The CD feature allows a user to freehand draw, use curves to edit an image, as well as undo using either freehand or curves, without having to invoke a separate menu for each item.
The following describes an interface for the CD feature as implemented in an embodiment of the invention. For purposes of this discussion, it will be assumed that the airbrush tool is selected as the drawing tool, although it is to be understood that the CD tool generally applies to all of the drawing tools. Upon selection of the airbrush draw tool, the aesthetic imaging system prompts the user to choose a color from a color palette that appears on the monitor. A color is selected using the pen. After selecting a color, a side bar menu is displayed. The user may select from a number of options on the side bar, including brush size and intensity, or select away from the side bar menu to remove the menu from the screen. In the latter case, the system defaults are used.
To freehand draw, the user presses on the tablet with the pen tip and continues pressure while moving or "rubbing" the pen on the tablet. At this point the chosen color is written onto the image at the location on the monitor that corresponds to the pen location. Pressing the side bar while repeating the motion will allow the user to selectively remove any edits to the image using a freehand motion.
To draw a curve, the user must set a first anchor point by selecting with the pen. Thereafter, as the user moves or "floats" the pen across the tablet, a green line will extend from the first anchor point to the current position of the pen. In a desired location, a second anchor point is set by selecting with the pen. Once both anchor points have been established, the green line appears on the monitor as a segment between the two anchor points. To create a curve, the user floats the pen across the tablet. The system will display a curved line bending and moving with the movement of the pen. The pen movement (top to bottom, or side to side) determines the arc of the curve. As the curve moves across the image, the image is edited in accordance with the selected draw tool and draw tool options. With the airbrush example, the system displays the curved line repeating itself with the chosen color. Pressing the side bar while repeating the motion will allow selective removal of any edits to the image using a curve established between the anchor points. FIGURE 5 illustrates an exemplary routine for implementing the CD feature of the imaging program. In one embodiment of the invention, the draw tool group includes: Airbrush, Tint, Texture, Blend, Undo, and Contour. Once a draw tool is selected from the main menu at block 74 of FIGURE 3, the routine of FIGURE 5 begins and selections must be made, or default selections confirmed, for the selected draw tool. Specifically, at block 120 the user is prompted to enter any options that are applicable to the draw tool selected. The airbrush tool will be described as an example. With airbrush, a color must be chosen. After a color has been chosen, a side bar menu is displayed at block 122 which illustrates other options that may be invoked for the airbrush tool. These typically include selecting a brush size, brush intensity, and other miscellaneous options. Sample side bar menus for various shape tools are described below. The draw tool side bar menus are similar to these.
Once the side bar menu options are entered, a test is made at block 124 to determine if the side button has been depressed. Pressing the side button can mean a cancel/back-up or an "undo," depending upon when it is activated. It is noted that at any point during the draw routine and the other routines described below, the side button may be used to return to the main menu through multiple backup or cancel commands.
If the side button has not been pressed, a test is made at block 126 to determine if the tip of the pen has been pressed. If the tip has not been pressed, the routine loops to block 124. Otherwise, a test is made at block 128 to determine if the tip was released prior to moving the pen. If the pen has been moved while the pen tip was pressed against the tablet, the imaging program enters freehand draw mode, shown at block 130. In one embodiment of the invention, freehand draw mode is entered if the pen moves the equivalent of three or more pixels. While in freehand draw mode, freehand edits may be made to the image in a manner similar to prior art imaging programs. This mode will remain until pressure on the pen tip is released. After beginning freehand draw mode, the routine loops to block 124.
If the pen was not moved, e.g., the pen has been moved two pixels or less prior to releasing the tip, the imaging program begins a curve mode by establishing a first endpoint, as indicated at block 132, and drawing a line on the monitor from the endpoint to the current pen position, indicated at block 134. At block 136, a test is made to determine if the tip has been pressed. The imaging program at this point is looking for a second endpoint to be entered. If not, the program loops back to block 136 to await the input. As noted above, the user can go back to the beginning of the routine using the cancel button if the user has changed his or her mind, although this is not shown in the flow diagram. Specifically, the first cancel would place the routine at block 124, the second at either block 122 or 124, depending on the draw tool selected, and subsequent cancels would forward the routine to the main menu. Once the tip is pressed, the second endpoint is established at the tip position, and a line segment is drawn on the monitor from the two endpoints, as indicated at block 138. At this point in the routine, the imaging program is in "curve draw mode" as indicated at block 140, and the user can make any edits desired using a curvilinear line segment having the established endpoints. The routine loops to block 124 while in this mode.
With reference to block 124, if the side button has been depressed a test is made at block 142 to determine if the imaging program is in either the freehand or curve draw modes. If the imaging program is in either mode, an undo mode will be invoked as long as the side button remains depressed, as indicated at block 144. If at the time of depressing the side button the imaging program is in freehand draw mode, the undo will also be freehand. Selective freehand undo edits are thus available. Similarly, if at the time of depressing the side button the imaging program is in curve draw mode, the undo will be in this mode. Selective "undo" edits are then available using a curve, as opposed to freehand drawing. Releasing the side button will return the imaging program to the drawing mode that was active just prior to depressing the side button.
Once the side button is released, or if the imaging program was not in either drawing mode, a test is made at block 146 to determine if the user wishes to exit. This test generally refers to the user again depressing the side bar, and thus "backing out" of the drawing routine. In this case, the draw routine returns to block 78 of FIGURE 3. Otherwise, the routine branches to block 124.
To further exemplify the advantages of the CD draw tool described in FIGURE 5, an original (unedited) image 130 that is representative of an image displayed on a monitor is illustrated in FIGURE 6. A main menu 132 is displayed across the top of the image to allow a user to select editing, viewing and printing options, as discussed in FIGURE 3 and accompanying text. The main menu 132 is from an embodiment of the aesthetic imaging system 20.
FIGURES 7A-7E and 8A-8E contrast exemplary steps taken to make identical edits to the image 130. The steps required to modify the image using a relatively advanced prior art imaging system are illustrated in FIGURES 7A-7E. These steps are modeled after a prior art imaging system that has been distributed by Mirror Image Technology, Inc., a division of Virtual Eyes, Incorporated, the assignee of the present invention. The steps required using an embodiment of the aesthetic imaging system 20 in accordance with the invention are illustrated in FIGURES 8A-8E. Briefly, each set of drawings illustrates examples of edits to a patient's nose, chin, and neck regions. The edits are for use in explaining the invention only, and merely exemplary in nature.
A brief description of the edits follows. With reference to FIGURE 7 A, the curve option of an airbrush tool is used to modify the bridge of a patient's nose. A resultant curve 134 is displayed having anchor points (endpoints) 136 and 138. In FIGURE 7B, a freehand motiQn is used to eliminate a portion of the tip of the patient's nose. In FIGURE 7C, an undo tool is used to replace a portion of the bridge of the patient's nose that was removed by the edits performed in FIGURE 7 A. In FIGURE 7D, the neck area of the patient has been edited using the curve option of an airbrush tool. The resultant curve 140 has anchor points 142 and 144. Finally, in FIGURE 7E, an undo tool is used to add back a portion of the neck area that was removed in FIGURE 7D. These edits require the following steps:
FIGURE 7A: Step SI move pen to draw on main menu;
Step S2 select draw;
Step S3 move pen to airbrush;
Step S4 select airbrush;
Step S5 move pen to curve;
Step S6 select curve;
Step S7 select an airbrush color;
Step S8 move pen to the first anchor point position;
Step S9 select at the position to establish the anchor point 136;
Step S10 move pen to the second anchor point position;
Step Sll select at the position to establish the anchor point 138;
Step S12 move pen to bend the curve 134 into the bridge of the nose;
FIGURE 7B: Step S 13 press the side button on the pen to exit to the main menu; Step S14 move pen to draw; Step SIS select draw; Step S16 move pen to airbrush; Step S17 select airbrush; Step S18 move pen to freehand; Step S19 select freehand; Step S20 select a color for the airbrush tool; Step S21 use a rubbing motion with the pen to make the freehand edit;
FIGURE 7C: Step S22 press the side button on the pen to exit to the main menu;
Step S23 move pen to draw; Step S24 select draw; Step S2S move pen to undo; Step S26 select undo; Step S27 move pen to curve; Step S28 select curve; Step S29 use a rubbing motion with the pen to undo the previous edit;
FIGURE 7D: Step S30 press the side button on the pen to exit to the main menu;
Step S31 move pen to draw; Step S32 select draw; Step S33 move pen to airbrush; Step S34 select airbrush; Step S35 move pen to curve; Step S36 select curve; Step S37 select color to be used by the airbrush tool;
Step S38 move pen to the first anchor point position;
Step S39 select at the position to establish the anchor point 142;
Step S40 move pen to the second anchor point position; Step S41 select at the position to establish the anchor point 144; Step S42 move pen to bend the curve 140 toward the neck, thereby making the edit shown.
FIGURE 7E: Step S43 press the side button on the pen to exit to the main menu; Step S44 move pen to draw; Step S4S select draw; Step S46 move pen to undo;
Step S47 select undo; Step S48 move pen to freehand; Step S49 select freehand;
Step S50 use a rubbing motion with the pen to undo a portion of the previous edit; and
Step S51 press the side button to exit back to the main menu In accordance with the invention, the steps required to perform the same edits using the aesthetic imaging system 20 are now described. With reference to FIGURE 8A-8E, the steps required to perform the edits include:
FIGURE 8A: Step Nl move pen to draw;
Step N2 select draw;
Step N3 move pen to airbrush;
Step N4 select airbrush;
Step NS elect any color for the airbrush tool
Step N6 move pen to the first anchor point position;
Step N7 select at the position to establish the anchor point 136;
Step N8 move pen to the second anchor point position;
Step N9 select at the position to establish the anchor point 138;
Step N10 move pen to bend the curve 134 into the bridge of the nose; FIGURE 8B: Step Nil pressing the tip of the pen against the tablet and use a rubbing motion to make the freehand edit.
FIGURE 8C: Step N12 pressing the tip of the pen and the side button simultaneously, and maintain pressure while rubbing in the area to be undone;
FIGURE 8D: Step Nil move pen to the first anchor point position; SStteepp NN1122 select at the position to establish the anchor point 142; Step N13 move pen to the second anchor point position; Step N14 select at the position to establish the anchor point 144; Step N15 move pen to bend the curve 140 toward the neck, thereby making the edit shown.
FIGURE 8E: Step N 16 press tip of pen and the side button simultaneously, while rubbing to undo a portion of the previous edit; and
Step N17 release pressure on the pen and side button, and press the side button to return to the main menu. From the simplified edits shown in FIGURES 7A-7E and FIGURES 8A-8E, it will be apparent that the aesthetic imaging system 20 provides a distinct advantage over prior art systems. Specifically, patients find the continued back and forth motions required to select necessary tools from the main menu to be disconcerting. The aesthetic imaging system 20 simplifies the editing process by providing freehand, curve, and undo options in the pen commands themselves, instead of in separate pull¬ down menus as is done in the prior art. The combination of tools is extremely effective in performing edits quickly, efficiently, and nearly seamlessly, all of which benefit both patient and physician during the consultation process.
FIGURE 9A illustrates a user interface for a contour tool for use in editing images in accordance with the invention. The contour tool is invoked from the main menu of the imaging program, as indicated at block 200. The contour tool has similarities to a blend tool, but utilizes pixel manipulation to pull pixels from one area to another. For example, the tool works great for chin and lip augmentations.
After the contour tool has been selected, a side bar menu is displayed by the aesthetic imaging system, as indicated at block 201. At block 202, the point size for the tool may be selected from the side bar menu. At block 203, an opacity percentage is entered by the user. If the opacity is at 100 percent, any areas affected by the edit are completely covered by the replacement pixels. As the percentage is reduced, more and more of the original pixels will remain, creating a blending of the replacement and prior pixels. Anchor points are selected at block 202. The selection is accomplished as described in blocks 128 and 130 of FIGURE 5. As also described, once the anchor points are selected, a line is displayed between the points by the aesthetic imaging system, as indicated at block 205. At block 204, a curve having the anchor points as endpoints is positioned along a part of the body, e.g., lips or chin, to be edited. The body part is then edited by dragging the curve in the direction in which a body part is to be augmented. Edits made in block 206 are saved at block 208 by pressing and then releasing the tip of the pen. The program then terminates.
FIGURES 9B-9C further describe the operation of the contour tool, by illustrating how pixels are replicated from one area of an image to another. The image areas described are for exemplary purposes only, and are simplified for clarity in this discussion. With reference to FIGURE 9B, an area 209 of an image is comprised of red 218R, green 218G, blue 218B, and yellow 218Y areas separated by boundary lines 210, 212, 214, and 216. It is assumed that a pair of anchor points 218 and 219 have been established by a user along the boundary 212, wherein the aesthetic imaging system will display a line segment 220 between the two anchor points.
In FIGURE 9C, it is assumed that the user has moved the midsection of the line segment 220 to the right. In this instance, the blue area 218B has been stretched into the yellow area 218Y. This area is bounded by the line segment 220 (now curved) and the boundary line 214. Also, the green area 218G has been stretched into the blue area 218B. This area is bounded by a curved line segment 221 and the boundary line 212. The red area 218R has expanded into the green area 218G; this area is bounded by a curved line segment 222 and the boundary line 210.
If the opacity level is at 100 percent, the newly defined areas will be comprised of the color being expanded. Thus, the area bounded by segments 220 and 221 will be blue; the area bounded by segments 221 and 222 will be green; and the area bounded by segment 222 to the left edge of the diagram will be red.
If the opacity level is less than 100 percent, pixels from the underlying image areas that are being written over by the newly defined areas will be blended into the newly defined areas. At an opacity of 80 percent, for example, the area bounded by segments 220 and 221 will still be primarily blue, but the portion of this area bounded by the segment 220 and the boundary line 214 may have a yellow tinge; and the portion of this area bounded by the boundary line 212 and segment 221 may have a green tinge. As the opacity percentage is dropped, the effects on these areas will be even greater.
While somewhat simplistic, the illustration in FIGURES 9B and 9C describes the function of the contour tool and the effect on more complicated pixels patterns will be appreciated by those skilled in the art. The area bounded by segment 220, boundary line 214, segment 221, and boundary line 212 will not be affected by changes in the opacity setting.
FIGURES 9D-9G illustrate edits to a patient's lips 224 using the contour tool. FIGURE 9D is a "before" picture without any modifications. To edit the right half of the patient's upper lip, the user first selects the desired end points surrounding the feature to be modified. A first anchor point 374 is designated near the middle of the upper lip, and a second anchor point 376 is designated at the right outer margin of the upper lip. When the anchor points have been selected, a line 378 is displayed between the points by the aesthetic imaging system. By floating the pen over the tablet in a direction generally indicated by arrow 380, the line is bent to form a curve 382 that approximates the contour of the feature being edited. When the user has fitted the curve to the feature, the shape of the curve is set by tipping the pen. As shown in FIGURE 9E, the user then selects a point along the portion of the curve by tipping the pen at the desired location. By floating the pen above the tablet, the user can stretch the selected feature in the manner described above. In FIGURE 9E, the user floats the pen in a direction generally indicated by arrow 384 to enlarge the patient's right upper lip. As the upper lip is expanded, the pixels forming the upper lip are replicated to expand the portion of the Up, while the pixels outside of the upper lip are deleted.
In FIGURE 9E, the left side of the person's upper Up has been edited using the contour tool. In FIGURE 9F, both the left and right sides of the person's upper Up are shown modified using the contour tool. In FIGURE 9G, the lower Up has been edited. Using the contour tool, these edits are accompUshed very quickly, in part because the augmented area will automatically match the area around it. The features of the contour tool make it much easier to perform augmentations than is currently available using prior art imaging tools. Editing Using Shape Tools
FIGURES 10-12B are directed to the shape tool features of the aesthetic imaging system. Similar to the draw tool, the imaging program has a combination feature that is generally available with any of the shape tools. This feature, called the autoblend feature, allows the user to easily move and paste shape tool edits, with or without blending the edges around the edit.
FIGURE 10 illustrates an exemplary routine for implementing the autoblend feature of the imaging program. The shape tool group generally includes: Warp, Stretch, Copy, Cutout, Rotate, Freeze Compare, and Resize. The routine of FIGURE 10 is invoked after a shape tool has been selected at block 80 of FIGURE 3. The side bar menu for the selected shape tool is displayed at block 230. Exemplary side bar menus for various shape tools are set forth below. The side bar menus are illustrated in an effort to further detail options available for a given tool, and may be especially helpful for those unfamiUar with imaging packages.
Copy, Cutout, Resize Stretch, Rotate Freeze Compare
Zoom Zoom Zoom
Undo Undo Undo
Compare Compare SpUt image
SpUt image SpUt Image
Inverse Inverse
Blend Blend
The black dots are brush size options that allow a user to choose the thickness of a shaping tool. The zoom option allows a user to look at an image in greater detail. When the zoom option is invoked, the aesthetic imaging system displays a square overlaid on the image. The square can be positioned by the user with the pen. After positioning the square, that area of the image will be magnified when the pen is selected. Canceling with the pen will display normal viewing mode. The undo option allows a user to undo edits to an image. The compare option allows a user to transition between before and after images. The spUt image option allows a user view before and after images side by side. The inverse option creates the mirror image of all or only a portion of an image that has been designated by the user. The blend tool wUl blend edits with the surrounding area. Many of the options shown are also implemented as separate tools under View in the main menu. These are described in greater detail below.
It is noted that the side bar menus available for the drawing tools are similar to the shape tool side bar menus shown. They do, however, typically include an option wherein the user may choose the intensity or opacity of a color used in conjunction with a draw tool. At block 232, the user is prompted to designate an area of the image to be edited. In a preferred embodiment, this is accomplished by pressing down on the pen and freehand drawing an area, e.g., a circle, that is to be subject to the edit. In this regard, the imaging program contains a unique feature wherein if a partial area is designated and the pen subsequently released, the drawing area will automatically be formed into a contiguous area by the imaging program. At block 236, a test is made to determine if an area has been designated by the user. If not, the routine loops back to block 234 and awaits a designation.
After an area has been designated, any edits to the designated area of the image are performed in accordance with the selected shape tool, as indicated at block 236. Two exemplary shape tools for editing an image are iUustrated in FIGURES 12A and 12B. At block 238, a test is made to determine if editing of the designated area is complete. In one embodiment, this involves testing for when the user "selects" with the pen anywhere on the tablet. The routine remains at block 238 until editing is complete (or the user exits using the side button). Upon completion of the edits, an autoblend box is displayed in the vicinity of the edited area, as indicated at block 240.
At block 242 a test is made to determine if the tip of the pen has been pressed against the tablet. If not, the routine loops, testing for this occurrence (or an exit command from the user). After the pen tip has been pressed, the imaging program calculates the location of the pen relative to the autoblend box.
FIGURE 11 illustrates an example of an autoblend box 250 that may be drawn by the aesthetic imaging system in accordance with the invention. As is discussed above, the autoblend box 250 may be used to: (1) move an edited area, (2) paste the edited area onto the image while blending the edge created between the edited area and the rest of the image, and (3) paste the edited image without blending. While the autoblend box 250 uses the conventions set forth below, those skilled in the art will appreciate that other conventions may be used without departing from the scope of the invention.
To move an edited area, the user must press down on the pen anywhere inside the autoblend box, except not at the three-, six-, nine-, and twelve-o'clock positions of the box. The "move" area is designated with reference numeral 254. To paste the edited area without blending, the user must press within the outlined areas 254 at the three-, six-, nine-, and twelve-o'clock positions inside the box. Selecting anywhere outside the box, i.e., area 256, results in the edited area being pasted with blending.
With reference again to FIGURE 10, a test is made at block 244 to determine if the pen tip has been pressed at a location outside of the autoblend box. If the pen tip was pressed at a location outside the autoblend box, the edit area is pasted with a blending of the edges at block 260. Otherwise, a test is made at block 262 to determine if the tip location was within the approximate three-, six-, nine-, and twelve-o'clock areas of the autoblend box. A pressing of the tip within any of these areas results in the edited area being pasted without blending, as indicated at block 264. As indicated at block 266, a selection in a location in the autoblend box apart from the three-, six-, nine-, and twelve-o'clock areas will allow the image to be moved. In this case, the edited area will track movement of the pen as long as the tip remains pressed. After a move is completed, the routine loops to block 242. If a paste has been accompUshed using blocks 260 or 264, a test is made at block 268 to determine if the user wishes to exit the shape routine, e.g., by pressing the side button. If not, the routine loops to block 232 where a new area of the image may be considered. Otherwise, the routine returns to block 82 of FIGURE 3.
FIGURES 12A and 12B lustrate two exemplary shape tools that are available when using the aesthetic imaging system. With reference to FIGURE 12 A, a cutout tool is unique in that a user can select an area of the image to be cut out, thereby creating a "hole" in the image, and an identical image underneath the cutout image can then be moved in all directions as it is viewed through the hole. The cutout feature is especially useful for profile views including chin augmentation, brow lifts, and maxillary and mandibular movement; and frontal views, including otoplasty, brow lift, Up augmentation, nasal base narrowing, and maxillary and mandibular movement.
At block 270, the current image is copied to a working buffer. As is discussed in FIGURE 10, when the cutout subroutine is called the user has defined an area of the image to be edited. At block 272, a boundary is created around the designated area designated in block 234 of FIGURE 10. At block 274, the working area is displayed inside the boundary, and the current image displayed outside the boundary. In this manner, the image in the working buffer can be moved relative to the image in the current buffer until the desired alignment has been achieved. The program then returns to the routine of FIGURE 10.
Upon returning to block 238, the edit may be frozen by selecting with the pen. Thereafter, the autoblend box is displayed. Selecting within the area 252 of the autoblend box aUows the designated area to be moved. Selecting anywhere outside the autoblend box will make the edit permanent, with automatic blending. Selecting within the box at the three-, six-, nine- or twelve-o'clock positions (areas 254) wiU make the edit permanent, without blending. With reference to FIGURE 12B, a rotate tool in accordance with the invention is particularly useful when editing profile views including the nasal tip, mandible, maxilla and brow areas; and frontal views including nares, brows, and the corners of the mouth. As is discussed in FIGURE 10, when the rotate routine is called, the user has defined an area of the image to be edited. At block 276, the area designated in block 234 of FIGURE 10 is shown in phantom. At block 278, the imaging program waits for the user to enter an axis of rotation. An axis is then entered by the user, as indicated at block 279.
Once an axis of rotation is entered, a display Une emanating from the axis point is displayed on the monitor, as indicated at block 280. Also, the number of degrees of rotation is displayed. The position of the pen dictates the degree of rotation. As the pen is moved away from the axis point, the display line will lengthen, providing the user greater control of the rotation of the designated area. At block 282, the system waits for the user to enter a desired degree of rotation. The degree of rotation is entered by the user by selecting with the pen, as indicated at block 283. Once the degree of rotation is entered, the designated area is redrawn onto the current image, as indicated at block 284.
After the redraw, the routine returns to FIGURE 10. Upon returning, the autoblend box is displayed. Selecting within the autoblend box allows the designated area to be moved. Selecting anywhere outside the autoblend box will make the edit permanent, with automatic blending. Selecting within the box at the three-, six-, nine-, and twelve-o'clock positions will make the edit permanent without blending.
While prior art imaging programs have a rotate feature, they do not aUow a user to select the axis of rotation. The ability to select the axis is valuable in the procedures Usted above. FIGURES 18A-18D disclose the capabiUties of the warp shape tool in the aesthetic imaging system. The warp shape tool is a powerful tool that allows users to easily edit a patient's features. Similar to the contour tool, the warp tool allows a user to select and manipulate a feature of the patient's image, with the aesthetic imaging system automatically redrawing the area surrounding the manipulated feature as the manipulation is being performed. As shown in FIGURE 18 A, the user first defines an image to be manipulated by the warp tool by encircling the portion of the image to be edited as shown by dotted line 386. Once the area has been selected, a user may tip the pen to designate a stretch point within the selected area. As shown in FIGURE 18 A, a stretch point 388 has been designated near the top portion of the patient's right upper Up. Once the stretch point has beeι designated, the user may float the pen in the desired direction that they would like to stretch the image. The portion of the image that is located at the stretch point is puUed in the direction that the user floats the pen, with the area surrounding the stretch point either expanded or compressed. That is, the area in the direction that the user floats the pen is compressed, and the area away from the direction that the user floats the pen is expanded. The amount of expansion or compression is dictated by the distance of the area away from the stretch point, in a manner discussed in greater detail below. Preferably, the image is manipulated in real time, so that the user is presented with a seamless and continuous stretching or movement of the selected feature. The technique used to implement the warp tool is portrayed in
FIGURES 19A-19E. A user first defines a manipulation area 420 to be edited by circling the desired feature of the patient. As shown in enlarged detail in FIGURE 19B, when the manipulation area has been selected, the aesthetic imaging system creates a pixel map of a warping area 422 that completely encompasses the manipulation area. For computational purposes, the warping area is preferably rectangular and approximately 33% larger in area that the manipulation area. It w l be appreciated, however, that the warping area may be differently shaped or sized depending on the particular appUcation and available system hardware.
After defining the manipulation area, the user selects a stretch point 424 within the manipulation area by tipping the pen at a desired location within the area. As shown in FIGURE 19C, when the stretch point is selected, the aesthetic imaging system maps four rectangles 426a, 426b, 426c, and 426d in the warping area. One corner of each rectangle is defined by the stretch point, and the diagonally opposing corner is defined by a corner of the warping area. Perpendicular lines 427 drawn through the stretch point intersect the warping area boundary at points 428a, 428b, 428c, and 428d. After the stretch point is selected, the user may float their pen to manipulate the selected feature, for example in the general direction indicated by arrow 430.
As the user floats the pen to warp the selected feature, the rectangles in the warping area are distorted. In FIGURE 19D, the stretch point has been moved upwards and to the left in the warping area, to an intermediate location 432. Periodically during the stretch point's motion, the aesthetic imaging system determines the shape of four quadrilaterals 434a, 434b, 434c, and 434d, with sides that extend from the intermediate location 432 of the stretch point to original points 428a, 428b, 428c, and 428d. Using a bilateral transformation, the pixels in the original rectangles 426a, 426b, 426c, and 426d are then mapped into the quadrilaterals. Further movement of the stretch point repeats the process, the original set of rectangles being mapped into the new quadrilaterals that are generated by the movement. If the display is updated frequently as the stretch point is moved, the user views the mapping and remapping as a warping of the patient's feature within the manipulation area. It will be appreciated that only the portion of the warping area that coincides with the manipulation area is presented to the user. The pixels comprising the remaining portion of the warping area are used for computational purposes, but are not displayed to the user.
When the desired warping effect is achieved, the user tips the pen to fix the stretch point in a desired location. To ensure there is no discontinuity between the manipulation area and the surrounding area of the patient's image, automatic blending around the outer margins of the manipulation area is performed by the aesthetic imaging system. As shown in FIGURE 19E, the aesthetic imaging system also remaps the warping area, creating four new rectangles 436a, 436b, 436c, and 436d based on the ending location 438 of the stretch point. In a preferred embodiment of the invention, a second stretch point may then be selected within the manipulation area and the process repeated, with the second warping transforming rectangles 436a, 436b, 436c, and 436d that resulted from the first warp. After the second warping is completed, the user may then select a third or additional warp point to further manipulate the image. Each manipulation is performed without the user having to redefine the manipulation area.
The flexibUity and power of aUowing multiple warps on a selected region is demonstrated in FIGURES 18A-18C. Returning to FIGURE 18 A, a user designates a stretch point 388 and floats the pen in a direction generaUy indicated by arrow 390. Floating the pen upwards and outwards generally causes the upper right portion of the Up to be expanded in that direction. The resulting manipulation is shown in FIGURE 18B, wherein the right Up has been made fuller. The area surrounding the Up is expanded or compressed to ensure that there are no discontinuities between the edited Up and the surrounding face. In FIGURE 18B, the user selects a second stretch point within the manipulation area to further stretch the selected feature. For example, a stretch point 392 may be positioned on the upper portion of the left Up, and the pen floated in a direction generally designated by the arrow 394. As shown in FIGURE 18C, this generally causes the upper left portion of the Up to be made fuUer to match the upper right portion of the Up. Again, during the warping the aesthetic imaging system automatically expands or contracts the surrounding unmanipulated area to ensure that there are no discontinuities between the upper left and the unedited portion of the face.
The warp tool with multiple stretch points is a very powerful tool as it allows the user to quickly manipulate an image with a minimum use of drawing tools or piecemeal editing. Because the warp tool performs the manipulation in real-time, the edits are accompUshed very quickly and fiuidly. A user may therefore generate a desired image in a ininimal amount of time. Viewing an Image
FIGURE 13 Ulustrates an exemplary routine for implementing the view features of the imaging program. In FIGURE 13, the soUd blocks indicate user interface options presented to the user by the aesthetic imaging system and the dashed blocks represent system responses to the decisions made. The view group includes: Compare, Prioritize, Split Image, Mirror Image, and Restore to Original, as weU as other options including Zoom and Emboss. At block 300, a test is made to determine if the Compare option has been selected. The Compare option allows a modified image to be compared to the original image so that a viewer can more readily see the changes. Specifically, as the pen is floated from the top to the bottom of the tablet, the user will see one image transition or "morph" into the other. The morphing is accompUshed by overlaying the original image with the modified image, and varying the opacity of the modified image. When the modified image is opaque, only the modified image may be viewed by a user. When the modified image is completely transparent, only the original image may be viewed by a user. In between these two extremes, varying amounts of the edits made to the image will be apparent to the user. The feedback to the patient as the original image morphs into the modified image is much more powerful than a side-by-side comparison of the two images. When a desired comparison level is achieved, a user can press the tip of the pen to freeze an image displayed on the monitor at a point anywhere from zero to 100% of the transition from the original to the modified image. Freezing an image at a partial transition is extremely helpful where edits have been performed on an image that are not reaUstically achievable in surgery, but an achievable result Ues somewhere between the original and the modified image. For example, it is easy to edit a blemish on a face so that area resembles the surrounding skin and thus becomes invisible. However, the total removal of the blemish may not be realistic. In this case, a transition of that area toward the original image will slowly "fade in" the blemish. A physician may then freeze the fading process at a desired point to provide a reaUstic image of what surgery can achieve to the patient.
Another option that is a subset of the Compare option is the Prioritize option. The Prioritize option allows a user to designate areas that have been edited so that the user can selectively illustrate the effects of two or more procedures that have been shown to a patient. For example, FIGURE 14A illustrates a modified profile image 302 of a patient that includes a rhinoplasty procedure (nose) 304, a chin augmentation procedure 306 and a submental lipectomy procedure (neck) 308. The boundaries that have been edited are illustrated by dashed lines 304a, 306a, and 308a, corresponding to the patients original nose profile, chin, and neck, respectively. Using the Prioritize option, the user can designate one area on the modified image, and Ulustrate transitions between the original and modified images at that area only by floating the pen. Any areas not selected will continue to be displayed as the original image.
With reference again to FIGURE 13, if the Compare option has been selected, a test is made at block 309 to determine if the entire image is to be compared or only certain portions of the image, i.e., using the Prioritize option. If less than the entire image is to be compared, the user is prompted to enter the area or areas that are to be compared at block 310. A user may then define one or more "priority areas" by freehand circling the desired area. When the priority areas or defined, or if all of the edits are to be reviewed during the comparison, at block 311 the user is prompted to float the pen in a vertical motion on top of the tablet to transition between the original and modified images, in accordance with the Compare feature discussed above. An Ulustration will clarify this point.
In FIGURE 14B, a first priority area 312 has been defined that corresponds generally to the nose. Given this selection, the nose area only wiU transition from original to modified as the pen is moved, with the rest of the image being displayed unedited. Thus, the modifications to the chin and neck no longer are shown. In FIGURE 14C, a second priority area 314 has been defined that corresponds generaUy to the chin. The first priority area 312 has been kept. Given these selections, the nose and chin areas only will transition from original to modified as the pen is moved, with the rest of the image being displayed unedited. Thus, the modifications to the neck are not iUustrated.
With reference to FIGURE 14D, a third priority area 316 has been defined that corresponds generaUy to the neck, along with the former designations. Given these selections, all three priority areas 312, 314, and 316 transition with movement of the pen. Again, the undesignated portions of the image are displayed in an unedited form, even if parts of the image outside the priority areas have been edited (no edits are shown). Because edits have not been made, from a user's standpoint the transition in FIGURE 14D appears to be a comparison between the original and modified images. While preferably the user defines the priority areas on the image being edited, it will be appreciated that the priority areas may also be automaticaUy defined by the aesthetic imaging system. A comparison may be made between an original image stored in a buffer and the edited image that has been modified by the user. Any areas containing differences between the original and the edited image may be highUghted by the aesthetic imaging system, and a priority area automaticaUy defined for each area containing differences. Whether a priority area is defined may also depend on the number of differences between the original image and the edited image.
In an alternative embodiment of the invention, when a user defines one of the priority areas in the Prioritize option, the user is prompted to enter a textual identifier for the defined area. For example, after circling the area in FIGURE 14B and defining a first priority area 312, the user would be prompted to enter a textual identifier corresponding to the first priority area. The user may assign a descriptive identifier related to that priority area, such as "nose", or the user may assign a non-related textual identifier such as "areal". Textual identifiers assigned by the user in this manner are displayed to the user in a submenu of the Prioritize option. The user may then select from the submenu of the Prioritize option those areas that they wish to display from the Ust presented to them. For example, the prioritize submenu corresponding to FIGURE 14D may present the user with a choice of "nose," "chin," and "neck." By selecting the nose and neck from the prioritize submenu, the user could simultaneously display only the effects of the nose and neck procedures. After making such a comparison, the user could deselect neck from the prioritize submenu, and instead select the chin and nose. This would allow the user to display only the effects of the chin and nose procedures using the compare option. It will be appreciated that assigning textual identifiers to each of the defined priority areas provides greater flexibiUty to a user, since the user does not have to redefine each of the priority areas that are to be displayed each time. A user may therefore quickly cycle through various permutations of the procedures that have been edited for patient display.
In the alternative embodiment in which textual identifiers are assigned to each of the priority areas, it will be appreciated that various methods can be used to display to the user the correspondence between the textual identifier and the priority area. For example, when the user selects one of the textual identifiers, the priority area could be highUghted or otherwise encircled with a contrasting color to indicate to the user the area of the image that corresponds to that textual identifier. Similarly, a user may also point to specific areas of the displayed image and have the textual identifier corresponding to that area appear to the user. Displaying the correspondence between priority area and identifier would allow a user to rapidly determine the available priority areas that may be shown to a patient when the user has not recently worked with or otherwise viewed the image.
With reference again to FIGURE 13, at block 320 a test is made to determine if the user wishes to save a transitional or morphed view of an image. If a transitional view is to be saved, the user may establish the percentage transition, i.e., anywhere from zero to 100 percent transition (zero percent being the original image and 100 percent being the edited image), by floating the pen up or down above the tablet to estabUsh the view, and the pressing the tip of the pen against the tablet to freeze the transitional image, as indicated in block 322. If the tip is pressed again, the frozen image is saved. The save options are available with or without the priority areas in effect. After the save has been accompUshed, or if the user did not wish to save a transitional view of an image, the Compare option is complete and the routine branches to block 326. In an alternative embodiment of the invention, the Compare option allows a user to compare a modified image with any edits made to the modified image during the current editing session, i.e., before the changes are permanently saved to the modified image. Specifically, with reference to FIGURE 2, this embodiment of the Compare option contrasts the image in the current image buffer 54 with the image in the modified image buffer 52. As discussed above, this embodiment of the Compare option may also be used in conjunction with the Prioritize option to aUow the user to select priority areas for comparison. In this case, the priority areas transition from the modified to the current image, while the modified image only is displayed in the other (nonselected areas) areas.
At block 326, a test is made to determine whether the user wishes to view a spUt image. The Split Image option is used on a frontal picture only, and allows a patient to see his or her asymmetries. If a spUt image view is desired, the user is prompted to select an image, e.g., original or modified, at block 330. At block 332, a vertical centerline is displayed on top of the selected image. The user is then prompted to position the centerline at the location desired, as indicated at block 334. TypicaUy, the centerline wUl be positioned to dissect the image into equal halves, using the nose and the eyes as reference points. At block 336, the aesthetic imaging system displays two images, one showing the left halves pieced together and the other the right halves pieced together. Specifically, the aesthetic imaging system will produce an inverse image of the left (right) half and then add it to the left (right) half. FIGURES 15A-15C illustrate the resultant images that are displayed when the
SpUt Image option is invoked. In FIGURE 15 A, a frontal image 350 of a patient is shown, including a centerline 352 that has been positioned at the center of the patient by a user. FIGURE 15B is an iUustration of the left halves of the image after being pieced together by the aesthetic imaging system, as indicated by reference numeral 354. FIGURE 15C is an iUustration of the right halves of the image, as indicated by reference numeral 356. With the Split Image option, patients can view what they would look Uke if their faces were symmetrical. The tool is especially useful in the consultation stage because many people do not realize that the typical face is asymmetrical, and changing a face to be perfectly symmetrical, if possible, is not necessarily desirable.
Once the spUt images have been displayed, or if the SpUt Image option was not selected, a test is made at block 360 to determine if the user wishes to view an inverse or mirror image of a picture, e.g., to show patients the view they see of themselves when looking into a mirror. If an inverse image is desired, the user is prompted to select an image to be viewed at block 362. The selected image is then "flipped" using the aesthetic imaging system, and displayed on the monitor, as shown at block 364. After the image has been displayed, or if the inverse image option was not invoked, routine returns to block 86 of FIGURE 3.
When viewing an image on the monitor, it is important that a patient remains focused on the image being manipulated rather than on the features of the aesthetic imaging system. When certain viewing options are selected, for example, the prioritize option shown in FIGURES 14A-14D, or the spUt image option shown in FIGURES 15A-15C, the menu bar normally located across the top of the display is therefore removed. Only the image of the patient is kept on the screen, ensuring that the patient remains focused on the image being manipulated. In certain situations where it is necessary to display a menu bar, however, it is advantageous to minimize the visual appearance of the menu bar. FIGURE 20 is a representative image 410 of a patient with a menu bar 412 located across the top of the screen 414. To reduce the distraction caused by the menu bar, the menu bar is preferably translucent to allow the user to view the patient's image through the menu bar. The text 416 and the line 418 indicative of the menu bar are preferably presented in a contrasting, yet muted, color to allow the user to read the commands. For example, the text and the line outlining the menu bar may be presented in an off- white. WhUe editing the image with a patient present, the patient is therefore not overly distracted when the menu bar periodically appears at the top of the screen. Even when the menu bar is present for extended periods of time, a patient is not distracted since it does not visuaUy stand-out from the patient's image. At the same time, a skilled user is provided with sufficient information about the menu choices to allow them to choose the appropriate menu options.
To further reduce the distraction caused by the menu bar, preferably the menu bar 412 may be moved by a user to different locations on the screen 414. For example, the menu bar may be moved by a user to the bottom of the screen. Particularly when editing the face of a patient on the screen, the majority of the editing will take place on the upper two-thirds of the screen. Locating the menu bar on the bottom of the screen therefore positions the menu bar away from the area on which the patient should remain focused. It will be appreciated that techniques for moving a menu bar to various locations on a screen are known to those skiUed in the art of designing user interfaces.
FIGURE 21 is a flow chart of an exemplary routine for implementing a zoom feature in the aesthetic imaging program. The zoom feature allows a user to increase the scale of the image to better view a selected area and to improve the abiUty of the user to edit fine details in the image. At a block 400, a zoom point is selected by a user. The zoom point identifies the center of the image to be expanded under the control of the user. At a block 402, the picture is adjusted to position the zoom point at the center of the monitor. Centering the picture ensures that as the image is enlarged, the portion of the image surrounding the zoom point will be displayed. At a block 404, the user is allowed to input a desired magnification factor. Preferably, the magnification factor is selected by floating the pen from the bottom (minimum magnification) to the top (maximum magnification) of the tablet. As the user floats the pen over the tablet, the image on the monitor is correspondingly magnified and redisplayed at a block 406. With each redisplay of the image, at a block 408 a test is made to see if the user has frozen the image by pressing or tipping the pen. Once the image is frozen at a desired magnification, a user can manipulate the image using the array of drawing tools described above. It will be appreciated that for very fine work, such as removing small wrinkles surrounding a patient's eyes, the abiUty to magnify the image increases the quaUty of the editing that may be achieved. Several refinements of the zoom feature may be incorporated in the aesthetic imaging program to improve the results of the zoom. For example, a smoothing function may be incorporated in the zoom feature to ensure that as the image is magnified it does not become "pixelly" or grainy. The smoothing function may be implemented in software. Preferably, however, the smoothing function is implemented in hardware, such as a smoothing feature provided in the Targa 2000 board described above and incorporated in the aesthetic imaging system. Further, feedback may be provided to the user in the form of a numerical display on the image to indicate the approximate magnification as the user floats the pen from the bottom to the top of the tablet. Other means can also be implemented to allow the user to select the desired magnification, including a pull-down menu or numerical entry.
Yet another view option provided to the user in the aesthetic imaging system is an Emboss option. It has been found that the emboss viewing option is very helpful in allowing a user to discern wrinkles or other skin imperfections in a displayed image. By selecting the emboss option, the user causes an image of the patient to be displayed in a gray scale. The emboss option displays an image that is similar to an etching made of a three-dimensional raised surface. A two dimensional image is portrayed, with the depth of the raised surface indicated by a darker shade of gray. The emboss option removes any deceptive information conveyed by the color or shading of the skin of the patient and allows any raised or depressed areas to be clearly highUghted. When viewing wrinkles or other imperfections on a patient, the emboss option therefore clearly identifies the raised or depressed features over the smooth skin of the patient.
Preferably, the emboss option is implemented using a function provided in the Targa 2000 board described above and incorporated in the aesthetic imaging system. The particular function used to manipulate a bitmap contained in the Targa board buffer is: setResizerToEmbossfconst BlitListSize άsize, int tc, intdy, int scale=OxlOO) where: dx and dy = the offset of the emboss operation; and scale = a "pressure" of the emboss. After selecting the emboss option, the user may manipulate the offset of the emboss by floating the pen over the tablet. Floating the pen parallel to the x-axis of the tablet adjusts the dx parameter in the emboss function, and floating the pen parallel to the y-axis adjusts the dy parameter in the emboss function. It has been found that various features on the skin of a patient can be brought into greater clarity by adjusting the offsets of the embossed image. A user may therefore focus on the desired feature that they wish to show a patient. When the desired offsets are selected, the user may freeze the offsets by tipping the pen on the tablet. A user may then adjust the scale parameter by floating the pen parallel to the x-axis of the tablet.
Again, it has been found that various skin features can be brought into greater clarity by adjusting the pressure of the emboss. When the desired pressure results are achieved, the user may freeze the image by tipping the pen on the tablet.
While the preferred embodiment of the invention has been iUustrated and described, it will be appreciated that various changes can be made therein without departing from the spirit and scope of the invention. For example, the advantages of the aesthetic imaging system are not limited to imaging for cosmetic surgery only, but are applicable to a number of medical imaging fields, including endoscopy, pathology, and hair restoration. Those skilled in the art will recognize adaptations that may be made to accommodate these medical imaging fields without departing from the scope of the invention. Also, a different style of pointing device may be used in Ueu of the pen and tablet.

Claims

The embodiments of the invention in which an exclusive property or privilege is claimed are defined as foUows:
1. A method of editing a digital image comprised of a pluraUty of color pixels in an aesthetic imaging system, the aesthetic imaging system including a processor, a memory, a monitor, and a pen and cooperating tablet for controlling a cursor displayed on the monitor, the pen having a depressable tip and a side button, each of which include an on status and an off status, wherein the position of the pen tip relative to the tablet determines the position of the cursor on the momtor, the method comprising: (a) evaluating the following variables: (i) the status of the tip of the pen;
(ϋ) the status of the side button on the pen; and (hi) movement of the pen tip relative to the tablet;
(b) actuating a freehand drawing mode if a first set of variables are present, wherein movement of the pen relative to the tablet edits pixels that are located at positions corresponding to the position of the cursor; and
(c) actuating a curve drawing mode if a second set of variables are present, wherein a line segment is displayed between two endpoints and movement of the pen relative to the tablet stretches the line segment, forming a curve and editing pixels that are located at positions corresponding to the position of the curve.
2. The method of Claim 1, further including:
(a) actuating a freehand undo mode if a third set of variables are present, wherein movement of the pen relative to the tablet restores pixels that are located at positions corresponding to the position of the cursor to their pre-edited color; and
(b) actuating a curve undo mode if a fourth set of variables are present, wherein a line segment is displayed between two endpoints and movement of the pen relative to the tablet stretches the line segment, forming a curve and restoring pixels that are located at positions corresponding to the position of the curve to their pre- edited color.
3. The method of Claim 1, wherein the freehand draw mode is actuated if the tip of the pen is depressed and pressure is maintained while the tip is moved a predetermined distance.
4. The method of Claim 1, wherein curve draw mode is actuated if the tip of the pen is depressed and released within a predetemined distance.
5. The method of Claim 1, wherein if the second set of variables are present, the curve draw mode is actuated by:
(a) estabUshing a first endpoint at the position of the pen when the second set of variables are present; and (b) monitoring the status of the tip of the pen and establishing a second endpoint at the position of the pen if the tip is toggled from an off state to an on state.
6. In an aesthetic imaging system allowing the mampulation of digital images comprised of a pluraUty of pixels, a method of displaying an edited digital image corresponding to an original digital image wherein the edited digital image differs from the original digital image by at least one edit made to the original digital image, the aesthetic imaging system including a processor, storage means, a monitor, and means for controlling a cursor displayed on the monitor, the method comprising:
(a) displaying the original digital image on the monitor;
(b) displaying the edited digital image superimposed over the original digital image, an opacity of the edited digital image being selectable by a user to allow the user to view a composite image composed of the edited digital image and the original digital image; and
(c) allowing the user to vary the opacity of the edited digital image to display a variable amount of the at least one edit performed on the original digital image in the composite image.
7. The method of Claim 6, wherein the user may vary the opacity of the edited digital image between zero and 100 percent.
8. The method of Claim 7, wherein the user may freeze the opacity of the edited digital image at a desired value between zero and 100 percent.
9. The method of Claim 8, wherein the user may store the composite image having the opacity of the edited digital image frozen at a desired value in the aesthetic imaging system storage means for later viewing.
10. The method of Claim 7, wherein the opacity of the edited digital image is varied by manipulation of the means for controUing the cursor on the monitor.
11. The method of Claim 10, wherein the opacity of the edited digital image is varied by moving the cursor displayed on the monitor between a position adjacent the top of the monitor and a position adjacent the bottom of the monitor.
12. In an aesthetic imaging system allowing the manipulation of digital images comprised of a pluraUty of pixels and including a processor, storage means, a monitor, and means for controUing a cursor displayed on the monitor, a method of verifying that an image has not been edited or otherwise altered since being stored in the aesthetic imaging system, the method comprising:
(a) allowing a digital image to be stored in the storage means of the aesthetic imaging system and, when initially stored, calculating a first checksum value corresponding to the digital image and storing the first checksum value so that the first checksum value is associated with the digital image; (b) aUowing the digital image to be recovered from the storage means and displayed to a user;
(c) calculating a second checksum value corresponding to the digital image when the digital image is recovered from the storage means;
(d) comparing the second checksum value with the first checksum value, the second checksum value differing from the first checksum value if the digital image has been edited to change the digital image from the digital image associated with the first checksum value; and
(e) displaying to the user a first icon if the second checksum value is equal to the first checksum value and a second icon if the second checksum value is not equal to the first checksum value.
13. The method of Claim 12, wherein the digital image and the first or second icon are displayed to the user on the monitor.
14. The method of Claim 12, wherein the digital image and the first or second icon are displayed to the user by printing the digital image and the first or second icon on a print media.
15. The method of Claim 12, wherein the first and second checksum values are calculated by summing a grayscale value for each of the pluraUty of pixels comprising the digital image.
16. In an aesthetic imaging system for use in imaging patients for cosmetic surgery, a method of capturing and displaying a digital image of a patient for subsequent manipulation of the digital image using the aesthetic imaging system, the aesthetic imaging system including a video camera, a processor, storage means, a monitor, and cursor control means for moving a cursor displayed on the monitor, the method comprising:
(a) positioning the patient in front of the video camera so that an image of the patient is captured by the video camera; (b) storing the image of the patient captured by the video camera in the storage means as a digital image comprised of a plurality of pixels; and
(c) displaying the digital image of the patient on the monitor, the digital image being mirrored around a vertical axis so that a plurality of pixels forming a left half of the digital image are displayed on a right half of the monitor and a plurality of pixels forming a right half of the digital image are displayed on a left half of the monitor, thereby displaying a mirror image of the patient on the monitor.
17. The method of Claim 16, wherein the image of the patient is continuously captured, stored, and displayed to the patient so that the patient may reposition in front of the video camera and see the effect the repositioning has on the digital image displayed on the monitor.
18. The method of Claim 17, wherein a desired image may be selected and stored for subsequent manipulation by the aesthetic imaging system when a desired position of the patient is achieved.
19. In an aesthetic imaging system for use in imaging patients for cosmetic surgery, a method of capturing and displaying a second digital image of a patient to allow a comparison of the second digital image with a stored first digital image of the patient comprised of a pluraUty of pixels, the aesthetic imaging system including a video camera, a processor, storage means, a monitor, and cursor control means for moving a cursor displayed on the monitor, the method comprising: (a) positioning the patient in front of the video camera so that an image of the patient is captured by the video camera;
(b) storing the image of the patient captured by the video camera in the storage means as a second digital image comprised of a pluraUty of pixels;
(c) displaying the second digital image of the patient on the monitor; and (d) displaying the stored first digital image of the patient on the monitor so that the first digital image is superimposed over the second digital image, the first digital image being substantially translucent to allow the patient to view the second digital image through the first digital image and reposition in front of the video camera in order to approximately align the second digital image with the first digital image.
20. The method of Claim 19, wherein the image of the patient is continuously captured, stored, and displayed to the patient so that the patient may reposition in front of the video camera and see the effect the repositioning has on the second digital image displayed on the monitor.
21. The method of Claim 20, wherein a second digital image may be selected and stored for subsequent manipulation by the aesthetic imaging system when a desired position of the patient is achieved.
22. The method of Claim 21, further comprising the step of displaying the stored first digital image and the stored second digital image side-by-side on the monitor to allow an accurate comparison of the first and second digital images.
23. The method of Claim 19, wherein the first and second digital images are mirrored around a vertical axis so that a plurality of pixels forming a left half of the first and second digital images are displayed on a right half of the monitor and a pluraUty of pixels forming a right half of the first and second digital images are displayed on a left half of the monitor, thereby displaying a solid and a translucent mirror image of the patient on the monitor.
24. In an aesthetic imaging system for use in imaging patients for cosmetic surgery, a method of manipulating a digital image of a patient comprised of a pluraUty of pixels each having a pixel value, the aesthetic imaging system including a processor, a memory, a monitor, and means for moving a cursor displayed on the monitor, the method comprising:
(a) aUowing a user to select a first end point and a second end point adjacent a feature on the digital image to be edited;
(b) displaying a line on the digital image that extends between the first and second end points;
(c) aUowing the user to stretch the Une into a curve extending between the first and second end points by defining a stretch direction and stretch distance using the means for moving the cursor displayed on the monitor, the curve moving over the digital image in the stretch direction and for the stretch distance without changing the pixel values in the underlying digital image; and (d) when the curve approximates a desired portion of the feature to be edited, aUowing the user to enter an edit mode wherein further movement of the curve over the digital image causes a change in the pixel values under the curve, the pixel values under the curve being changed to reflect a value proportional to the pixel values adjacent the curve in the stretch direction and the pixel values adjacent the curve away from the stretch direction when the edit mode was entered.
25. The method of Claim 24, wherein the pixel values under the curve are changed to equal the pixel values adjacent the curve away from the stretch direction when the edit mode was entered, thereby expanding a desired portion of the feature by replicating the pixel values in the desired portion of the feature.
26. The method of Claim 24, wherein the user may select the proportion of the pixel values adjacent the curve in the stretch direction that are mixed with the pixel values adjacent the curve away from the stretch direction when the edit mode was entered in order to arrive at the pixel values under the curve.
27. The method of Claim 24, wherein the displayed line is a contrasting color to the digital image.
28. In an aesthetic imaging system allowing the manipulation of digital images comprised of a plurality of pixels, a method of displaying an edited digital image corresponding to an original digital image wherein the edited digital image differs from the original digital image by at least one edit made to the original digital image, the aesthetic imaging system including a processor, a memory, a monitor, and means for controUing a cursor displayed on the monitor, the method comprising:
(a) defining a pluraUty of regions in said edited digital image, each of the pluraUty of regions containing at least one edit that distinguishes each of the pluraUty of regions from a corresponding region in said original digital image;
(b) aUowing a user to select from the plurality of regions a subset of regions; and
(c) displaying a prioritized digital image containing only the edits of the original digital image that are present in the subset of regions.
29. The method of Claim 1, wherein the plurality of regions are defined by encircling the region on the edited digital image with the means for controlling the cursor displayed on the monitor.
30. The method of Claim 1, wherein the pluraUty of regions are automaticaUy defined by the aesthetic imaging system.
31. The method of Claim 1, further comprising the step of assigning a tag corresponding to each of the pluraUty of regions.
32. The method of Claim 4, wherein the tag is a textual identifier.
33. The method of Claim 5, wherein the subset of regions are selected from a Ust of tags corresponding to each region.
34. The method of Claim 1, wherein the original image and the prioritized image are displayed side-by-side.
35. The method of Claim 1, wherein the prioritized image is superimposed over the original image, an opacity of the prioritized image being selectable by a user to aUow the user to view a composite image composed of the prioritized image and the original image, the composite image thereby displaying a variable amount of the edits performed on the original image.
36. The method of Claim 8, wherein the opacity of the prioritized image may vary between zero and 100 percent.
37. In an aesthetic imaging system for use in imaging patients for cosmetic surgery, a method of manipulating a digital image of a patient comprised of a pluraUty of pixels, the aesthetic imaging system including a processor, a memory, a monitor, and cursor control means for moving a cursor displayed on the monitor, the method comprising:
(a) aUowing a user to select a manipulation area in a digital image by encircling the manipulation area using the cursor control means;
(b) defining a warping area that encompasses the manipulation area selected by the user;
(c) storing a copy of the waφing area in memory as an original area;
(d) aUowing the user to identify a stretch point that is contained within the manipulation area using the cursor control means;
(e) aUowing the user to move the stretch point from an initial position through a pluraUty of intermediate positions to a final position within the waφing area using the cursor control means, the movement of the stretch point defining a stretch direction from the initial position to the final position;
(f) repeatedly mapping the original area that is stored in memory to the waφing area to reflect the movement of the stretch point within the waφing area from the initial position to the final position, the mapping generally compressing an area of the image in the stretch direction and generally expanding an area of the image away from the stretch direction; and
(g) displaying the portion of the waφing area that is within the manipulation area selected by the user to reflect the stretching of the image caused by movement of the stretch point.
38. The method of Claim 10, wherein the waφing area is a rectangle.
39. The method of Claim 11, further comprising:
(a) dividing the original area into a plurality of original quadrilaterals;
(b) repeatedly dividing the waφing area into a pluraUty of waφed quadrilaterals as the stretch point is moved through the plurality of intermediate positions to the final position, each of the plurality of waφed quadrilaterals corresponding to one of the pluraUty of original quadrilaterals; and
(c) mapping a set of pixels from each of the pluraUty of original quadrilaterals to the corresponding one of the pluraUty of waφed quadrilaterals using a bilateral transform in order to display a stretching of the digital image in real-time.
40. The method of Claim 10, wherein the mapping of the original area that is stored in memory to the waφing area is performed using a bilateral transform.
41. The method of Claim 10, further comprising allowing a user to sequentiaUy identify a pluraUty of stretch points within the manipulation area, for each of the pluraUty of stretch points repeating steps (e) through (g) to manipulate the image in a desired manner.
42. The method of Claim 14, further comprising storing a copy of the waφing area as the original area following the mapping to the waφing area at the final position of each of the plurality of stretch points, so that a subsequent manipulation of the image wiU be performed on the results of an immediately preceding manipulation of the image.
43. A user interface for an aesthetic imaging system containing digitized images of a patient, the aesthetic imaging system allowing a user to manipulate a patient image to display potential results achievable through cosmetic surgery, the aesthetic imaging system including a processor, a memory, a monitor, and means for controlling a cursor displayed on the monitor, the user interface comprising:
(a) a display area for displaying a digital image of a patient comprised of a plurality of color pixels; and
(b) a command menu area for displaying a plurality of commands that may be used to manipulate the digital image of the patient, the command menu area substantiaUy overlapping the display area so that a portion of the digital image is positioned behind the command menu area, the command menu area being substantially transparent to allow the portion of the digital image positioned behind the command menu area to be visually apparent to a user, the plurality of commands in the command menu being substantially opaque and visually distinguishable from the portion of the digital image positioned behind the command menu area to allow the user to select from the pluraUty of commands.
44. The user interface of Claim 16, wherein the display area encompasses the entire screen of the monitor.
45. The user interface of Claim 17, wherein the plurality of commands in the command menu area are removed during editing of the digital image.
46. The user interface of Claim 16, wherein the command menu area may be moved by the user to a desired location on the display area to minimize the overlap of the command menu area with the display area.
47. In an aesthetic imaging system allowing the manipulation of digital images comprised of a pluraUty of pixels and including a processor, storage means, a monitor, and means for controlling a cursor displayed on the monitor, a method of verifying that an image has not been edited or otherwise altered since being stored in the aesthetic imaging system, the method comprising:
(a) aUowing a digital image to be stored in the storage means of the aesthetic imaging system and, when initially stored, generating a unique identifier for the digital image and storing the identifier so that the identifier is associated with the digital image; (b) aUowing the digital image to be recovered from the storage means and displayed to a user;
(c) determining from the identifier whether the digital image has been modified from the digital image that was initially stored; and (d) displaying to the user a first icon if the digital image has been modified and a second icon if the digital image has not been modified.
48. The method of Claim 20, wherein the identifier is a flag associated with the digital image.
49. The method of Claim 20, wherein the identifier is a copy of the digital image.
50. The method of Claim 20, wherein the identifier is a first checksum value.
51. The method of Claim 23, wherein the step of determining from the identifier whether the digital image has been modified from the digital image that was initially stored comprises:
(a) calculating a second checksum value associated with the recovered digital image; and
(b) comparing the second checksum value with the first checksum value, the second checksum value differing from the first checksum value if the digital image has been edited to change the digital image from the digital image associated with the first checksum value.
52. The method of Claim 24, wherein the first and second checksum values are calculated by summing a grayscale value for each of the plurality of pixels comprising the digital image.
53. The method of Claim 20, wherein the digital image and the first or second icon are displayed to the user on the monitor.
54. The method of Claim 20, wherein the digital image and the first or second icon are displayed to the user by printing the digital image and the first or second icon on a print media.
PCT/US1996/003678 1995-03-17 1996-03-18 Aesthetic imaging system WO1996029675A1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
AU54251/96A AU5425196A (en) 1995-03-17 1996-03-18 Aesthetic imaging system
EP96911333A EP0815533A1 (en) 1995-03-17 1996-03-18 Aesthetic imaging system
JP8528545A JPH11503540A (en) 1995-03-17 1996-03-18 Aesthetic imaging system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US08/406,201 1995-03-17
US08/406,201 US5687259A (en) 1995-03-17 1995-03-17 Aesthetic imaging system

Publications (1)

Publication Number Publication Date
WO1996029675A1 true WO1996029675A1 (en) 1996-09-26

Family

ID=23606955

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US1996/003678 WO1996029675A1 (en) 1995-03-17 1996-03-18 Aesthetic imaging system

Country Status (5)

Country Link
US (5) US5687259A (en)
EP (1) EP0815533A1 (en)
JP (1) JPH11503540A (en)
AU (1) AU5425196A (en)
WO (1) WO1996029675A1 (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1998020458A1 (en) * 1996-11-08 1998-05-14 Mirror Software Corporation Aesthetic imaging system
WO1998021695A1 (en) * 1996-11-12 1998-05-22 Mirror Software Corporation Imaging system for simulating hair styles
US7570839B1 (en) * 2005-06-15 2009-08-04 Apple Inc. Tool range selection
CN107111845A (en) * 2015-01-05 2017-08-29 莫菲斯有限公司 For the computer readable recording medium storing program for performing of the method, system and non-transitory that provide the service based on face
US11165951B2 (en) 2017-02-20 2021-11-02 Koninklijke Philips N.V. Image capturing

Families Citing this family (157)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5926576A (en) * 1994-03-30 1999-07-20 Newton; Dale C. Imaging method and system concatenating image data values to form an integer, partition the integer, and arithmetically encode bit position counts of the integer
WO1997042875A1 (en) * 1996-05-10 1997-11-20 Gunther Blaseio Method of manipulating cephalometric line tracings
JP3894376B2 (en) * 1996-05-24 2007-03-22 株式会社ニコン Information processing device
US7815436B2 (en) 1996-09-04 2010-10-19 Immersion Corporation Surgical simulation interface device and method
US6929481B1 (en) 1996-09-04 2005-08-16 Immersion Medical, Inc. Interface device and method for interfacing instruments to medical procedure simulation systems
WO1998010387A2 (en) * 1996-09-04 1998-03-12 Ht Medical Systems, Inc. Interventional radiology interface apparatus and method
US7616198B2 (en) * 1998-02-20 2009-11-10 Mental Images Gmbh System and computer-implemented method for modeling the three-dimensional shape of an object by shading of a two-dimensional image of the object
US5836877A (en) * 1997-02-24 1998-11-17 Lucid Inc System for facilitating pathological examination of a lesion in tissue
JP3912834B2 (en) * 1997-03-06 2007-05-09 有限会社開発顧問室 Face image correction method, makeup simulation method, makeup method, makeup support apparatus, and foundation transfer film
DE19711484C1 (en) * 1997-03-19 1998-09-03 Siemens Ag Computerised optimised imaging by modelling software
US5990901A (en) * 1997-06-27 1999-11-23 Microsoft Corporation Model based image editing and correction
US6009209A (en) * 1997-06-27 1999-12-28 Microsoft Corporation Automated removal of red eye effect from a digital image
JPH11120371A (en) * 1997-10-21 1999-04-30 Sharp Corp Trimming graphic display, trimming graphic display method and medium recorded with trimming graphic display control program
US6266054B1 (en) 1997-11-05 2001-07-24 Microsoft Corporation Automated removal of narrow, elongated distortions from a digital image
US6160923A (en) * 1997-11-05 2000-12-12 Microsoft Corporation User directed dust and compact anomaly remover from digital images
DE19757773A1 (en) * 1997-12-24 1999-07-01 Bernward Maehner Detailed simulation of plastic changes in human face to prepare cosmetic corrections using plastic surgery
US5986660A (en) * 1997-12-31 1999-11-16 Autodesk, Inc. Motion capture data system and display
US6044132A (en) * 1997-12-31 2000-03-28 Siemens Corporate Research, Inc. Apparatus for providing markers on an image, for use in conjunction with C-arm calibration apparatus
US6121978A (en) * 1998-01-07 2000-09-19 Ati Technologies, Inc. Method and apparatus for graphics scaling
EP1051698B1 (en) 1998-01-28 2018-01-17 Immersion Medical, Inc. Interface device and method for interfacing instruments to vascular access simulation systems
EP1103041B1 (en) 1998-01-28 2016-03-23 Immersion Medical, Inc. Interface device and method for interfacing instruments to medical procedure simulation system
US6459439B1 (en) * 1998-03-09 2002-10-01 Macromedia, Inc. Reshaping of paths without respect to control points
DE69933323T2 (en) * 1998-04-01 2007-01-11 Matsushita Electric Industrial Co., Ltd., Kadoma compiler
US6204860B1 (en) * 1998-07-02 2001-03-20 Silicon Graphics, Inc. Method and apparatus for geometric model deformation using wires
US6496594B1 (en) * 1998-10-22 2002-12-17 Francine J. Prokoski Method and apparatus for aligning and comparing images of the face and body from different imagers
WO2000031689A1 (en) * 1998-11-24 2000-06-02 Synapix, Inc. Free-form video editing system
US6686955B1 (en) 1999-01-29 2004-02-03 International Business Machines Corporation Lightweight image manipulation for maximizing the capacity of digital photo storage media
US6434269B1 (en) * 1999-04-26 2002-08-13 Adobe Systems Incorporated Smart erasure brush
US6373499B1 (en) * 1999-06-30 2002-04-16 Microsoft Corporation Automated emphasizing of an object in a digital photograph
US6293284B1 (en) 1999-07-07 2001-09-25 Division Of Conopco, Inc. Virtual makeover
US6903756B1 (en) * 1999-10-14 2005-06-07 Jarbridge, Inc. Merged images viewed via a virtual storage closet
US7418407B2 (en) 1999-10-14 2008-08-26 Jarbridge, Inc. Method for electronic gifting using merging images
US7917397B1 (en) 1999-10-14 2011-03-29 Jarbridge, Inc. Merging private images for gifting
US6750884B1 (en) 1999-10-26 2004-06-15 Red Oak Knowledge Systems, Inc. Instruction presentation apparatus
US6972775B1 (en) * 1999-11-01 2005-12-06 Medical Learning Company, Inc. Morphing patient features using an offset
US6747672B1 (en) 1999-11-01 2004-06-08 Medical Learning Company, Inc. Virtual patient hot spots
US6692258B1 (en) * 2000-06-26 2004-02-17 Medical Learning Company, Inc. Patient simulator
US7386432B1 (en) 1999-11-01 2008-06-10 Medical Learning Co., Inc./Web Simulator Web simulator
US7082227B1 (en) * 1999-11-24 2006-07-25 Baum Daniel R Producing printed images having personalized features
US7443417B1 (en) * 1999-12-13 2008-10-28 Geoffrey W Heinrich Method of performing dental work and apparatus providing vision enhancement dentistry
US6643385B1 (en) 2000-04-27 2003-11-04 Mario J. Bravomalo System and method for weight-loss goal visualization and planning and business method for use therefor
US7809153B2 (en) * 2000-04-27 2010-10-05 Inter-Images Partners, Lp System and method for assessment of health risks and visualization of weight loss and muscle gain
WO2002007000A2 (en) 2000-07-13 2002-01-24 The Belo Company System and method for associating historical information with sensory data and distribution thereof
JP2002109555A (en) * 2000-07-24 2002-04-12 Mitsubishi Electric Corp Virtual cosmetic surgery system and virtual cosmetic surgery method
US7079158B2 (en) * 2000-08-31 2006-07-18 Beautyriot.Com, Inc. Virtual makeover system and method
US20020035458A1 (en) * 2000-09-20 2002-03-21 Chang-Hun Kim Method and system for virtual surgery
JP2002109192A (en) * 2000-09-27 2002-04-12 Nec Eng Ltd Order shop system
US6748119B1 (en) * 2000-11-02 2004-06-08 Xerox Corporation Systems and methods for interactively using and training an automatic image processing technique
JP2002229955A (en) * 2001-02-02 2002-08-16 Matsushita Electric Ind Co Ltd Information terminal device and authentication system
US20040100486A1 (en) * 2001-02-07 2004-05-27 Andrea Flamini Method and system for image editing using a limited input device in a video environment
US20020181752A1 (en) * 2001-03-14 2002-12-05 Warren Wallo Method for measuring changes in portions of a human body
EP1248206A1 (en) * 2001-04-05 2002-10-09 Sun Microsystems, Inc. Method and apparatus for database table definition
US6928192B2 (en) * 2001-04-19 2005-08-09 Koninklijke Philips Electronics N.V. User interface for interactive removal of defects in an image sequence
JP4778158B2 (en) * 2001-05-31 2011-09-21 オリンパス株式会社 Image selection support device
US20020196333A1 (en) * 2001-06-21 2002-12-26 Gorischek Ignaz M. Mirror and image display system
US20030013994A1 (en) * 2001-07-11 2003-01-16 Gilles Rubinstenn Methods and systems for generating a prognosis
DE10142526C5 (en) * 2001-08-30 2006-02-16 Wella Ag Procedure for a hair color consultation
US20030065255A1 (en) * 2001-10-01 2003-04-03 Daniela Giacchetti Simulation of an aesthetic feature on a facial image
US7634103B2 (en) * 2001-10-01 2009-12-15 L'oreal S.A. Analysis using a three-dimensional facial image
US20030063102A1 (en) * 2001-10-01 2003-04-03 Gilles Rubinstenn Body image enhancement
CN100405973C (en) * 2002-03-14 2008-07-30 Netkisr有限公司 System and method for analyzing and displaying computed tomography data
JP2004005265A (en) * 2002-05-31 2004-01-08 Omron Corp Image composing method, device and system
US7348970B2 (en) * 2002-10-02 2008-03-25 Hewlett-Packard Development Company, L.P. Display system and method of using same
US7298876B1 (en) * 2002-11-04 2007-11-20 R2 Technology, Inc. Method and apparatus for quality assurance and quality control in radiological equipment using automatic analysis tools
JP4168748B2 (en) * 2002-12-20 2008-10-22 富士ゼロックス株式会社 Image processing apparatus, image processing program, and image processing method
JP4185825B2 (en) * 2003-07-01 2008-11-26 キヤノン株式会社 Coordinate input device, control method therefor, information processing device, and program
US7162063B1 (en) * 2003-07-29 2007-01-09 Western Research Company, Inc. Digital skin lesion imaging system and method
US7457670B2 (en) * 2003-08-07 2008-11-25 Production Resource Group, Llc Gobo virtual machine
US20050163379A1 (en) * 2004-01-28 2005-07-28 Logitech Europe S.A. Use of multimedia data for emoticons in instant messaging
KR100792284B1 (en) * 2004-03-10 2008-01-07 삼성전자주식회사 Method for setting driver program of image processing device and image processing system with transparent function
US7440877B2 (en) * 2004-03-12 2008-10-21 General Motors Corporation System and method for morphable model design space definition
US7658611B2 (en) * 2004-03-18 2010-02-09 Reality Engineering, Inc. Interactive patient education system
JP4218590B2 (en) * 2004-05-26 2009-02-04 ソニー株式会社 Image processing system, information processing apparatus and method, image processing apparatus and method, recording medium, and program
US20050275733A1 (en) * 2004-06-10 2005-12-15 Philip Chao Method and apparatus of rendering a video image by polynomial evaluation
JP2006004158A (en) * 2004-06-17 2006-01-05 Olympus Corp Image processing program, image processing method, image processor, and recording medium
JP2006018522A (en) * 2004-06-30 2006-01-19 Canon Inc Image edition system, method, and program
US20080001851A1 (en) * 2006-06-28 2008-01-03 Searete Llc Cosmetic enhancement mirror
US7657125B2 (en) * 2004-08-02 2010-02-02 Searete Llc Time-lapsing data methods and systems
US7876289B2 (en) * 2004-08-02 2011-01-25 The Invention Science Fund I, Llc Medical overlay mirror
US9155373B2 (en) 2004-08-02 2015-10-13 Invention Science Fund I, Llc Medical overlay mirror
US7679581B2 (en) 2004-08-02 2010-03-16 Searete Llc Medical overlay mirror
US7433504B2 (en) * 2004-08-27 2008-10-07 General Electric Company User interactive method for indicating a region of interest
US20060064310A1 (en) * 2004-09-21 2006-03-23 Crystal Image Artwurks Corp. Tattoo studio control and management system
US7885703B2 (en) * 2004-09-27 2011-02-08 General Electric Company System and method for scanning a patient
US8033832B1 (en) * 2004-12-27 2011-10-11 Stefan David B Systems and methods for performing virtual cosmetic and reconstructive surgery
US8232979B2 (en) 2005-05-25 2012-07-31 The Invention Science Fund I, Llc Performing an action with respect to hand-formed expression
US7809215B2 (en) 2006-10-11 2010-10-05 The Invention Science Fund I, Llc Contextual information encoded in a formed expression
US8823636B2 (en) 2005-03-18 2014-09-02 The Invention Science Fund I, Llc Including environmental information in a manual expression
US8640959B2 (en) 2005-03-18 2014-02-04 The Invention Science Fund I, Llc Acquisition of a user expression and a context of the expression
US8340476B2 (en) * 2005-03-18 2012-12-25 The Invention Science Fund I, Llc Electronic acquisition of a hand formed expression and a context of the expression
US8229252B2 (en) 2005-03-18 2012-07-24 The Invention Science Fund I, Llc Electronic association of a user expression and a context of the expression
US8749480B2 (en) 2005-03-18 2014-06-10 The Invention Science Fund I, Llc Article having a writing portion and preformed identifiers
US7873243B2 (en) 2005-03-18 2011-01-18 The Invention Science Fund I, Llc Decoding digital information included in a hand-formed expression
US8102383B2 (en) 2005-03-18 2012-01-24 The Invention Science Fund I, Llc Performing an action with respect to a hand-formed expression
US8290313B2 (en) 2005-03-18 2012-10-16 The Invention Science Fund I, Llc Electronic acquisition of a hand formed expression and a context of the expression
US7773242B2 (en) * 2005-04-04 2010-08-10 Xerox Corporation Automated remote correction of preflight defects
US8606950B2 (en) * 2005-06-08 2013-12-10 Logitech Europe S.A. System and method for transparently processing multimedia data
US20070052799A1 (en) * 2005-09-06 2007-03-08 International Business Machines Corporation System and method for assisting speech development for the hearing-challenged
JP4916237B2 (en) * 2005-09-16 2012-04-11 株式会社リコー Image display apparatus, image display method, program for causing computer to execute the method, and image display system
US7321372B2 (en) * 2005-09-19 2008-01-22 Lexmark International, Inc. Method for performing image editing
WO2007117586A2 (en) 2006-04-08 2007-10-18 Allan Millman Method and system for interactive simulation of materials
US8786613B2 (en) 2006-04-08 2014-07-22 Alan Millman Method and system for interactive simulation of materials and models
US7966558B2 (en) * 2006-06-15 2011-06-21 Microsoft Corporation Snipping tool
US7724287B2 (en) * 2006-07-31 2010-05-25 Hewlett-Packard Development Company, L.P. Sketch effect for digital photographs
US8257315B2 (en) 2006-10-11 2012-09-04 Ethicon Endo-Surgery, Inc. Trocar seal with retraction induced hinge
WO2008057577A2 (en) * 2006-11-08 2008-05-15 24Eight Llc Method and apparatus for recommending beauty-related products
US20080150962A1 (en) * 2006-12-20 2008-06-26 Bezryadin Sergey N Image display using a computer system, including, but not limited to, display of a reference image for comparison with a current image in image editing
US8212805B1 (en) 2007-01-05 2012-07-03 Kenneth Banschick System and method for parametric display of modular aesthetic designs
JP2008172666A (en) * 2007-01-15 2008-07-24 Kyocera Mita Corp Image forming apparatus and image forming program
JP4270404B2 (en) * 2007-01-16 2009-06-03 ソニー・エリクソン・モバイルコミュニケーションズ株式会社 Portable terminal device, display control device, display control method, and display control program
US8990681B2 (en) 2007-01-24 2015-03-24 Bluebeam Software, Inc. Method for aligning a modified document and an original document for comparison and difference highlighting
US7907794B2 (en) * 2007-01-24 2011-03-15 Bluebeam Software, Inc. Method for aligning a modified document and an original document for comparison and difference highlighting
US20080226144A1 (en) * 2007-03-16 2008-09-18 Carestream Health, Inc. Digital video imaging system for plastic and cosmetic surgery
GB2447976B (en) * 2007-03-30 2011-04-27 Sony Uk Ltd Apparatus and method of image capture
US20080282158A1 (en) * 2007-05-11 2008-11-13 Nokia Corporation Glance and click user interface
JP4924279B2 (en) * 2007-08-13 2012-04-25 セイコーエプソン株式会社 Image processing apparatus, control method thereof, and program
JP5329911B2 (en) * 2007-11-02 2013-10-30 株式会社東芝 Medical image management apparatus and medical image system
US20090124866A1 (en) * 2007-11-09 2009-05-14 The Quantum Group, Inc. Intelligent personal health profile
JP5546668B2 (en) * 2008-04-08 2014-07-09 キヤノン株式会社 Graphic drawing editing system, graphic drawing editing device, graphic drawing editing method, and program
JP2009271909A (en) 2008-04-08 2009-11-19 Canon Inc Graphics rendering editing system, graphics rendering editing device, and editing method
US8559755B2 (en) 2009-04-07 2013-10-15 Citrix Systems, Inc. Methods and systems for prioritizing dirty regions within an image
US9148618B2 (en) * 2009-05-29 2015-09-29 Apple Inc. Systems and methods for previewing newly captured image content and reviewing previously stored image content
JP2011211493A (en) * 2010-03-30 2011-10-20 Sony Corp Imaging apparatus, display method, and program
US9053563B2 (en) 2011-02-11 2015-06-09 E4 Endeavors, Inc. System and method for modeling a biopsy specimen
ES2651317T3 (en) 2011-03-01 2018-01-25 Dolphin Imaging Systems, Llc System and method to generate profile change using cephalometric monitoring data
US8650005B2 (en) 2011-04-07 2014-02-11 Dolphin Imaging Systems, Llc System and method for three-dimensional maxillofacial surgical simulation and planning
US8417004B2 (en) 2011-04-07 2013-04-09 Dolphin Imaging Systems, Llc System and method for simulated linearization of curved surface
US9032020B2 (en) 2011-12-29 2015-05-12 Google Inc. Online video enhancement
US9020192B2 (en) * 2012-04-11 2015-04-28 Access Business Group International Llc Human submental profile measurement
US20130305172A1 (en) * 2012-05-10 2013-11-14 Motorola Mobility, Inc. Pen Tool Editing Modes
US9111134B1 (en) 2012-05-22 2015-08-18 Image Metrics Limited Building systems for tracking facial features across individuals and groups
US9460462B1 (en) 2012-05-22 2016-10-04 Image Metrics Limited Monetization using video-based simulation of cosmetic products
US9129147B1 (en) 2012-05-22 2015-09-08 Image Metrics Limited Optimal automatic capture of facial movements and expressions in video sequences
US9449412B1 (en) 2012-05-22 2016-09-20 Image Metrics Limited Adaptive, calibrated simulation of cosmetic products on consumer devices
US9104908B1 (en) 2012-05-22 2015-08-11 Image Metrics Limited Building systems for adaptive tracking of facial features across individuals and groups
US9208819B1 (en) 2012-06-20 2015-12-08 Google Inc. Dynamic split-frame preview of video editing effects
US9367933B2 (en) 2012-06-26 2016-06-14 Google Technologies Holdings LLC Layering a line with multiple layers for rendering a soft brushstroke
JP5719819B2 (en) * 2012-09-28 2015-05-20 日本光電工業株式会社 Surgery support system
DE102012110491A1 (en) * 2012-11-02 2014-05-08 Carsten Dursteler Method and device for cosmetic tooth analysis and dental consultation
US9167999B2 (en) 2013-03-15 2015-10-27 Restoration Robotics, Inc. Systems and methods for planning hair transplantation
US9320593B2 (en) 2013-03-15 2016-04-26 Restoration Robotics, Inc. Systems and methods for planning hair transplantation
WO2014205632A1 (en) * 2013-06-24 2014-12-31 Adobe Systems Incorporated Gravity point drawing method
JP6311372B2 (en) * 2014-03-13 2018-04-18 オムロン株式会社 Image processing apparatus and image processing method
WO2015179876A1 (en) 2014-05-23 2015-11-26 Pathonomic Digital microscope system for a mobile device
US10387021B2 (en) 2014-07-31 2019-08-20 Restoration Robotics, Inc. Robotic hair transplantation system with touchscreen interface for controlling movement of tool
GB2531758A (en) * 2014-10-29 2016-05-04 Nokia Technologies Oy Method and apparatus for determining the capture mode following capture of the content
US9576385B2 (en) * 2015-04-02 2017-02-21 Sbitany Group LLC System and method for virtual modification of body parts
US10163262B2 (en) 2015-06-19 2018-12-25 Covidien Lp Systems and methods for navigating through airways in a virtual bronchoscopy view
US10636184B2 (en) * 2015-10-14 2020-04-28 Fovia, Inc. Methods and systems for interactive 3D segmentation
CA3133728C (en) 2015-11-02 2023-10-24 Pura Scents, Inc. Scent dispensation
USD816506S1 (en) 2015-11-02 2018-05-01 Pura Scents, Inc. Vial for a scent dispenser
USD809116S1 (en) 2015-11-02 2018-01-30 Pura Scents Dispenser
US10748312B2 (en) * 2016-02-12 2020-08-18 Microsoft Technology Licensing, Llc Tagging utilizations for selectively preserving chart elements during visualization optimizations
CN107180453B (en) 2016-03-10 2019-08-16 腾讯科技(深圳)有限公司 The edit methods and device of character face's model
CN109963205A (en) * 2017-12-25 2019-07-02 上海全土豆文化传播有限公司 Multimedia clips method and device
US10949649B2 (en) 2019-02-22 2021-03-16 Image Metrics, Ltd. Real-time tracking of facial features in unconstrained video
US11294556B1 (en) * 2021-01-05 2022-04-05 Adobe Inc. Editing digital images using multi-panel graphical user interfaces

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4434467A (en) * 1979-04-12 1984-02-28 Dale Scott Hair coloring calculator
US5060171A (en) * 1989-07-27 1991-10-22 Clearpoint Research Corporation A system and method for superimposing images
US5280570A (en) * 1992-09-11 1994-01-18 Jordan Arthur J Spectacle imaging and lens simulating system and method
US5375195A (en) * 1992-06-29 1994-12-20 Johnston; Victor S. Method and apparatus for generating composites of human faces
US5403191A (en) * 1991-10-21 1995-04-04 Tuason; Leo B. Laparoscopic surgery simulator and method of use
US5404426A (en) * 1991-05-21 1995-04-04 Hitachi, Ltd. Method of displaying hair style and apparatus for the same
US5532839A (en) * 1994-10-07 1996-07-02 Xerox Corporation Simplified document handler job recovery system with reduced memory duplicate scanned image detection
US5537662A (en) * 1992-05-29 1996-07-16 Casio Computer Co., Ltd. Electronic montage composing apparatus

Family Cites Families (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS5214112B2 (en) * 1973-02-22 1977-04-19
US4189743A (en) * 1976-12-20 1980-02-19 New York Institute Of Technology Apparatus and method for automatic coloration and/or shading of images
US4189744A (en) * 1976-12-20 1980-02-19 New York Institute Of Technology Apparatus for generating signals representing operator-selected portions of a scene
US4276570A (en) * 1979-05-08 1981-06-30 Nancy Burson Method and apparatus for producing an image of a person's face at a different age
US4488245A (en) * 1982-04-06 1984-12-11 Loge/Interpretation Systems Inc. Method and means for color detection and modification
US4602280A (en) * 1983-12-05 1986-07-22 Maloomian Laurence G Weight and/or measurement reduction preview system
EP0147493B1 (en) * 1983-12-28 1988-09-07 International Business Machines Corporation Process and equipment for the automatic alignment of an object in respect of a reference
JP2531605B2 (en) * 1984-02-24 1996-09-04 株式会社東芝 Image registration device
US4641352A (en) * 1984-07-12 1987-02-03 Paul Fenster Misregistration correction
US4975960A (en) * 1985-06-03 1990-12-04 Petajan Eric D Electronic facial tracking and detection system and method and apparatus for automated speech recognition
JP2569303B2 (en) * 1985-07-05 1997-01-08 日本電装株式会社 Image processing apparatus for performing cumulative addition of image data
US4791581A (en) * 1985-07-27 1988-12-13 Sony Corporation Method and apparatus of forming curved surfaces
US5020122A (en) * 1987-07-20 1991-05-28 Litton Industrial Automation Systems, Inc. Automatic document image revision
CA1288516C (en) * 1987-07-31 1991-09-03 Leendert M. Bijnagte Apparatus and method for communicating textual and image information between a host computer and a remote display terminal
US4853875A (en) * 1988-01-19 1989-08-01 Eastman Kodak Company Image-capable interprocessor link communications protocol
US4918611A (en) * 1988-07-21 1990-04-17 Industrial Technology Research Institute Method and apparatus for controlling laser cutting by image processing
US5133020A (en) * 1989-07-21 1992-07-21 Arch Development Corporation Automated method and system for the detection and classification of abnormal lesions and parenchymal distortions in digital medical images
JP2833137B2 (en) * 1990-04-05 1998-12-09 松下電器産業株式会社 Pattern matching method
US5022080A (en) * 1990-04-16 1991-06-04 Durst Robert T Electronic notary
US4996714A (en) * 1990-05-23 1991-02-26 Frisco Bay Industries Of Canada Limited Computerized filing and retrieval system for signature and pictorial documentation
US5465082A (en) * 1990-07-27 1995-11-07 Executone Information Systems, Inc. Apparatus for automating routine communication in a facility
US5370642A (en) * 1991-09-25 1994-12-06 Keller; Gregory S. Method of laser cosmetic surgery
US5337231A (en) * 1992-03-31 1994-08-09 General Electric Company View to view image correction for object motion with truncated data
US5553864A (en) * 1992-05-22 1996-09-10 Sitrick; David H. User image integration into audiovisual presentation system and methodology
US5358407A (en) * 1992-06-23 1994-10-25 Lainer Lisa M Appearance modification display/selection system and method
US5579471A (en) * 1992-11-09 1996-11-26 International Business Machines Corporation Image query system and method
US5426010A (en) * 1993-02-26 1995-06-20 Oxford Computer, Inc. Ultra high resolution printing method
US5431161A (en) * 1993-04-15 1995-07-11 Adac Laboratories Method and apparatus for information acquistion, processing, and display within a medical camera system
AU1839395A (en) * 1994-02-04 1995-08-29 Data Card Corporation Card creation system and method
JP2682811B2 (en) * 1994-03-22 1997-11-26 インターナショナル・ビジネス・マシーンズ・コーポレイション Data storage management system and method
US5680528A (en) * 1994-05-24 1997-10-21 Korszun; Henry A. Digital dressing room
US5515447A (en) * 1994-06-07 1996-05-07 United Parcel Service Of America, Inc. Method and apparatus for locating an acquisition target in two-dimensional images by detecting symmetry in two different directions
US5541656A (en) * 1994-07-29 1996-07-30 Logitech, Inc. Digital camera with separate function and option icons and control switches

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4434467A (en) * 1979-04-12 1984-02-28 Dale Scott Hair coloring calculator
US5060171A (en) * 1989-07-27 1991-10-22 Clearpoint Research Corporation A system and method for superimposing images
US5404426A (en) * 1991-05-21 1995-04-04 Hitachi, Ltd. Method of displaying hair style and apparatus for the same
US5403191A (en) * 1991-10-21 1995-04-04 Tuason; Leo B. Laparoscopic surgery simulator and method of use
US5537662A (en) * 1992-05-29 1996-07-16 Casio Computer Co., Ltd. Electronic montage composing apparatus
US5375195A (en) * 1992-06-29 1994-12-20 Johnston; Victor S. Method and apparatus for generating composites of human faces
US5280570A (en) * 1992-09-11 1994-01-18 Jordan Arthur J Spectacle imaging and lens simulating system and method
US5532839A (en) * 1994-10-07 1996-07-02 Xerox Corporation Simplified document handler job recovery system with reduced memory duplicate scanned image detection

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
SNAP MANUAL, ENVISION INTERNATIONAL INC., issued 1992, CAMEO, pages i-v, 1-47. *
UPDATE TO TIPS TO USER'S GUIDE, TIPS IMAGING SOFTWARE FOR THE TARGA VIDEOGRAPHIC ADAPTOR, issued 1987, TRUEVISION, pages 2-1, through 2-42 and 3-1, through 3-136. *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1998020458A1 (en) * 1996-11-08 1998-05-14 Mirror Software Corporation Aesthetic imaging system
WO1998021695A1 (en) * 1996-11-12 1998-05-22 Mirror Software Corporation Imaging system for simulating hair styles
US7570839B1 (en) * 2005-06-15 2009-08-04 Apple Inc. Tool range selection
CN107111845A (en) * 2015-01-05 2017-08-29 莫菲斯有限公司 For the computer readable recording medium storing program for performing of the method, system and non-transitory that provide the service based on face
EP3261050A4 (en) * 2015-01-05 2018-12-19 Morpheus Co., Ltd. Method and system for providing face-based services and non-transitory computer-readable recording medium
US11165951B2 (en) 2017-02-20 2021-11-02 Koninklijke Philips N.V. Image capturing

Also Published As

Publication number Publication date
US6081611A (en) 2000-06-27
US5854850A (en) 1998-12-29
AU5425196A (en) 1996-10-08
US5687259A (en) 1997-11-11
EP0815533A1 (en) 1998-01-07
JPH11503540A (en) 1999-03-26
US5978523A (en) 1999-11-02
US5825941A (en) 1998-10-20

Similar Documents

Publication Publication Date Title
US5854850A (en) Method and apparatus for selectively illustrating image modifications in an aesthetic imaging system
US6867787B1 (en) Character generator and character generating method
US5990901A (en) Model based image editing and correction
US5692117A (en) Method and apparatus for producing animated drawings and in-between drawings
US7728848B2 (en) Tools for 3D mesh and texture manipulation
JP3895492B2 (en) Image processing apparatus, image processing method, and computer-readable recording medium storing program for causing computer to execute the method
US5353391A (en) Method apparatus for transitioning between sequences of images
US7095413B2 (en) Animation producing method and device, and recorded medium on which program is recorded
US11061524B2 (en) Techniques to modify content and view content on a mobile device
WO2007083600A1 (en) Makeup simulation system, makeup simulation device, makeup simulation method, and makeup simulation program
KR20040029258A (en) Image editing method, image editing apparatus, program for implementing image editing method and recording medium recording program
US6856329B1 (en) Automated acquisition of video textures acquired from a digital camera for mapping to audio-driven deformable objects
US5289568A (en) Apparatus and method for drawing figures
JP3995114B2 (en) Switch image display method
WO1998020458A9 (en) Aesthetic imaging system
WO1998020458A1 (en) Aesthetic imaging system
WO1998021695A1 (en) Imaging system for simulating hair styles
JP3444148B2 (en) Eyebrow drawing method
JPH05233773A (en) Photograph filter metaphor for controlling digital picture processing software
US20100013838A1 (en) Computer system and motion control method
JPWO2004090213A1 (en) Knit design method and apparatus
JP3014209B2 (en) Image information presentation device
JP2002209088A (en) Image edit device and method
JP3647116B2 (en) Window hierarchy display method and system
JPH05249951A (en) Image information presenting device

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AL AM AT AU AZ BB BG BR BY CA CH CN CZ DE DK EE ES FI GB GE HU IS JP KE KG KP KR KZ LK LR LS LT LU LV MD MG MK MN MW MX NO NZ PL PT RO RU SD SE SG SI SK TJ TM TR TT UA UG US UZ VN AM AZ BY KG KZ MD RU TJ TM

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): KE LS MW SD SZ UG AT BE CH DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
WWE Wipo information: entry into national phase

Ref document number: 1996911333

Country of ref document: EP

ENP Entry into the national phase

Ref country code: JP

Ref document number: 1996 528545

Kind code of ref document: A

Format of ref document f/p: F

WWP Wipo information: published in national office

Ref document number: 1996911333

Country of ref document: EP

REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

NENP Non-entry into the national phase

Ref country code: CA

WWW Wipo information: withdrawn in national office

Ref document number: 1996911333

Country of ref document: EP