|Publication number||US7400351 B2|
|Application number||US 10/959,385|
|Publication date||Jul 15, 2008|
|Filing date||Oct 6, 2004|
|Priority date||Oct 6, 2004|
|Also published as||CN1801916A, CN1801916B, EP1648173A2, EP1648173A3, US20060072017|
|Publication number||10959385, 959385, US 7400351 B2, US 7400351B2, US-B2-7400351, US7400351 B2, US7400351B2|
|Inventors||Dongmei Zhang, Mehul Y Shah|
|Original Assignee||Microsoft Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (63), Non-Patent Citations (29), Referenced by (10), Classifications (29), Legal Events (4)|
|External Links: USPTO, USPTO Assignment, Espacenet|
The present invention generally relates to computer systems, and more particularly to a system and/or method relating to image-based video that facilitates encoding a source image associated with a motion vector.
There is an increasing use of digital photography based upon the decrease in size and cost of digital cameras as well as concurrent increases in availability, usability, and resolution capabilities. Manufacturers and the like have continuously strived to provide smaller electronics in order to satisfy consumer demand, associated with carrying, storing, and using electronic devices. Thus, digital photography has demonstrated growth and has proven to be a profitable market for both electronics and software.
A user first experiences the overwhelming benefits of digital photography upon capturing a digital image. While conventional print photography forces the photographer to wait until development of expensive film to view a print, a digital image in digital photography can be viewed within seconds by utilizing a thumbnail image and/or viewing port on a digital camera. Additionally, images can be deleted or saved based upon user preference, thereby allowing economical use of limited image storage space. In general, digital photography provides a more efficient experience in photography.
Furthermore, editing techniques available for a digital image are vast and numerous with limitations being only the editor's imagination. For example, a digital image can be edited using techniques such as crop, resize, blur, sharpen, contrast, brightness, gamma, transparency, rotate, emboss, texture, draw tools (e.g., fill or pen, add circles, boxes), insert text, etc. In contrast, conventional print photography merely enables the developer to control developing variables such as exposure time, light strength, type of light-sensitive paper, and various light filters. Moreover, such conventional print photography techniques are expensive whereas digital photography software is becoming more common on computers.
In addition to advantages relating to digital photography with respect to image capturing and developing, digital photography facilitates sharing of the taken images. Once captured, images being shared with another can accompany a story (e.g., a verbal narration) and/or physical presentation of such images. Regarding conventional print photographs, sharing options are limited to picture albums, which entail a variety of complications involving organization, storage, and accessibility. Moreover, physical presence of the album is necessary to share print photographs with another.
In view of the above benefits associated with digital photography and traditional deficiencies of print photography, digital images and digital albums correcting such deficiencies have increasingly replaced conventional print photographs and albums. In particular, image-based video provides a convenient and efficient technique for sharing digital images. Image-based video is a slide show of images with motion (e.g., panning, zooming, cross-fading, . . . ) applied to still images. An effect of utilizing image-based video is an enhanced motion video experience that enables details of high-resolution images to be better viewed on a TV screen and/or computer monitor. For example, resolution of a picture taken by a typical 3 MegaPixel digital still camera is about 2000×1500 pixels whereas resolution of a typical computer monitor is 1024×768 pixels. However, with the growing demand of consumers, high-end digital still cameras can provide pictures with much higher resolutions.
Information regarding the image-based video must be saved (e.g., during a creation process and/or at a conclusion) to a file for future playback. In one storage technique, each frame of the video is saved as a picture. However, this technique requires a substantial amount of storage space and CPU (e.g., processor) time in which information relating to the motion can be lost. In yet another more efficient storage technique, a source image and motion vectors associated with an image are encoded (e.g., the source images are encoded to provide better compression). Next, at rendering (e.g., playback) time, each encoded source image and its encoded motion vectors are decoded, allowing the generation of the output video frames.
When creating image-based video, substantial motions (e.g., panning, zooming, cross-fading, etc.) can create various problems in areas such as presentation, resolution, resolution, memory and/or processor capabilities. In any computing environment, hardware and associated components are limited; therefore efficient use is crucial to overall user experience. Substantial panning and/or zooming require a substantial portion of a source image to be encoded in video to enable display of source image content with high fidelity. For example, the size of a high-resolution source image can be m×n in pixels and a motion can be a zoom that alters a view of the source image from the entire image (m×n) to an m/8×n/8 area of such source image. Thereafter, a resulting video can be produced in p×q pixels resolution where m>p>m/8 and n>q>n/8. When fully zoomed, the picture portion of the size m/8×n/8 pixels is displayed, ideally, in the highest fidelity. In order to maintain fidelity, the entire image m×n pixels must be stored.
Based on the previous example, several problems arise during image-based video. Encoding of such a high-resolution source image requires decoding of the source image, which is time consuming to both a user and processor. Once decoded, an uncompressed image requires a substantial amount of memory to adequately store such image. In addition, each frame generated from the source image needs resizing-increasing the processor usage during resizing and rendering. The larger the source image, the higher is the CPU usage during resizing. Furthermore, a typical video system displays at a rate of 30 frames per second, allowing approximately 33 milliseconds for decoding a source image, resizing and rendering each frame to be displayed. If a system does not generate and render a frame within the allotted 33 milliseconds, motion of the video becomes choppy and viewing pleasure is deteriorated. Although processor speeds continue to increase, typical consumer processors and computing environments easily consume over 33 milliseconds when the source image is of high resolution.
In view of the above, there is a need to improve upon and/or provide systems and/or methods relating to image-based video that facilitates encoding a source image associated with a motion vector.
The following subjects a simplified summary of the invention in order to provide a basic understanding of some aspects of the invention. This summary is not an extensive overview of the invention. It is intended to neither identify key or critical elements of the invention nor delineate the scope of the invention. Its sole purpose is to present some concepts of the invention in a simplified form as a prelude to the more detailed description that is presented later.
The subject invention relates to systems and/or methods that facilitate storing image-based video via employing at least a step image. By utilizing step images, the subject invention drastically reduces an amount of storage and apportioned memory for an image-based video that produces substantial motions such as a zoom, pan, and/or a pan and zoom. The parsing of a substantial motion into smaller motions with step images enables only the step images to be stored into video rather than storing an entire source image.
In accordance with one aspect of the subject invention, a system is provided that employs a step image component to facilitate encoding a source image associated with a motion vector by utilizing at least a step image, wherein such step image provides a reduction in an amount of stored video. The system can determine a maximum step image size and a number of step images to simulate a smooth substantial motion. In order to determine the step characteristics (e.g., the maximum step image size and the number of step images), the step image component uses an input component that receives a motion parameter (e.g., a source image, a pan, a zoom, a pan and zoom, . . . ) and a computer environment parameter (e.g., processor speed, memory capacity, video display, . . . ). With step characteristics determined, the step image component can create the required step images for the motion based at least in part upon the received parameters.
In another aspect of the subject invention, the system includes a motion controller component that mitigates the video visual perception complications during a substantial zoom motion. The motion controller component utilizes a non-linear quadratic function in order to determine appropriate calculations to mitigate visual perception of acceleration and/or deceleration based upon a zoom-in and zoom-out respectively; providing the viewer with a smooth, constant motion.
In yet another aspect of the subject invention, an encoder component facilitates encoding of step images created by the step image component. The encoder component provides different specialized techniques in encoding in order to utilize appropriate sequence of step images to ensure proper decoding of such step images; providing a proper rendering of video. For example, the encoder component can help the decoder determine an appropriate image to cache with a current frame and which image to discard when a new image is decoded. In yet another aspect of the subject invention, the encoder component encodes images in an order based at least in part upon an initial generation of video frames from source images. In addition, the encoder component sets a direction of transition to a newly encoded source image in which a decoder component can accordingly decode.
The following description and the annexed drawings set forth in detail certain illustrative aspects of the invention. These aspects are indicative, however, of but a few of the various ways in which the principles of the invention may be employed and the present invention is intended to include all such aspects and their equivalents. Other advantages and novel features of the invention will become apparent from the following detailed description of the invention when considered in conjunction with the drawings.
The present invention is described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It may be evident, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing the present invention.
As utilized in this application, terms “component, system,” and the like are intended to refer to a computer-related entity, either hardware, and/or firmware. For example, a component can be, a processor, an object, and/or a computer.
Artificial intelligence based systems (e.g., explicitly and/or implicitly trained classifiers) can be employed in connection with performing inference and/or probabilistic determinations and/or statistical-based determinations as described herein. As used herein, the term “inference” refers generally to the process of reasoning about or inferring states of the system, environment, and/or user from a set of observations as captured via events and/or data. Inference can be employed to identify a specific context or action, or can generate a probability distribution over states, for example. The inference can be probabilistic—that is, the computation of a probability distribution over states of interest based on a consideration of data and events. Inference can also refer to techniques employed for composing higher-level events from a set of events and/or data. Such inference results in the construction of new events or actions from a set of observed events and/or stored event data, whether or not the events are correlated in close temporal proximity, and whether the events and data come from one or several event and data sources. Various classification schemes and/or systems (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, data fusion engines . . . ) can be employed in connection with performing automatic and/or inferred action in connection with the subject invention.
Now referring to the figures,
For example, one technique to determine a maximum step image size of a step image is to define a threshold. During a playback of image-based video, the computing environment has limited time to decode, generate, and render each frame of video. Typically, time is calculated as 1/framerate (where video is played at a rate of 30 frames per second), which provides approximately 33 milliseconds for the step image to be decoded and the frame to be rendered. Based at least in part upon the CPU (e.g., processor) capability and frame rate, maximum step image size for step images can be determined such that the CPU (e.g., processor) has sufficient time to decode the image and render the video frames. Moreover, a number of step images can be calculated based at least in part upon maximum step image size of such step image and the overall pan/zoom motion in the source image.
Furthermore, the step image component 110 creates the step images utilizing the determined maximum step image size and number of step images based at least upon the motion desired (e.g., zoom, pan, pan and zoom, . . . ), CPU capability, output video resolution and frame rate. The step image component 110 employs step images to represent a source image using a set of smaller images for a motion. In addition, the step image component 110 creates the step images such that each of the smaller images (e.g., step images) is a source image for a portion of the motion in the video and is used to generate video for that portion of motion. Since the step image component 110 creates step images that are individually smaller than the original image and the motion within the step image is correspondingly smaller, memory and processor capabilities are not compromised.
For example, a zoom motion can be initiated upon a source image starting at a full source image and zooming to a rectangular region in center of such source image. The zoom can start at a starting position (R1) and an ending position (R2) such that the starting position is the full image and the ending position is the centered rectangle of the source image. The motion from the start position (R1) and the end position (R2) can be broken down into steps. Once a maximum step image size and number of step images are determined, the step image component can create step images. For this example, motion can be broken into 2 steps. The first step is a motion from R1 to Ri, and a second step is Ri to R2 (where Ri is an intermediate rectangle between R1 and R2, the location being based at least in part upon the number of frames in the video).
Continuing with the previous example, the first step of motion (e.g., R1 to Ri) allows a scaling down of the full image to a smaller size such that Ri has substantially the same resolution (or lower) as that of video to be generated. This scaled-down image can be referred to as Step Image 1. By utilizing Step Image 1, video frames can be generated for motion from R1 to Ri without losing video quality. Similarly, a second step image can be created from the second step of motion (e.g., Ri to R2). During motion from Ri to R2, any portion of image outside rectangle R1 is not needed to generate video frames. Only a portion of the image represented by Ri is to be stored. Furthermore, the image stored can be scaled down such that rectangle R2 has the substantially similar resolution or lower as that of video to be generated. The scaled-down portion of Ri of the image can be referred to as Step Image 2. Step Image 2 is used to generate video frames for motion from Ri to R2 without losing video quality. In the above example, Step Image 1 need not be the same image as the full image because starting motion R1 can be a smaller portion of the full image and also because the portion R1 can be scaled down to an extent wherein the portion Ri has equal or lesser resolution as the output video. Moreover, Step Image 2 need not be the same as rectangle portion Ri. In other words, the source image is scaled down in order to create an image frame and the rule applies when creating such step image.
A coordinate system is defined using image coordinates such that, a rectangle (R) inside the source image 210 can be represented as:
v=[x0, y0, w, h]T
Where x0 and y0 are coordinates on the X-axis 212 and Y-axis relating to an upper-left corner of the source image 210. Furthermore, variables w and h can represent width and height of rectangle (R) inside the source image 210. This notation allows rectangle (R) position and size to be represented as follows:
During a motion from the start rectangle position 216 and end rectangle position 218, a duration can be represented by time d seconds with a video frame rate off frames per second. The number of video frames for motion is n=f×d. Unlike traditional video, image-based video only requires a source image to be encoded. Frames of video are defined as a position and size of start rectangle position 216 to the end rectangle position 218. The position and size of the rectangle within the image 210 in relation to frame i can be calculated as follows:
Although an image to be encoded may not be the full source image 210 since the start and end motion may not include the entire source image 210, a union 226 of the start position rectangle 216 and the end position rectangle 218 is encoded and stored. It is appreciated the entire union image is to be stored. Furthermore, the image to encode is scaled down without losing fidelity based at least in part upon video size and smallest size of a motion from start rectangle position 216 and end rectangle position 218.
The system 200 fails to achieve performance goals when a motion from start rectangle position 216 and end rectangle position 218 is of substantial proportions. For example, a source image can be 2560×1920 pixels (e.g., a typical 5 MegaPixel digital image). When the motion is a zoom starting from the full source image to a 320×240 rectangle in the middle of the source image, and a video size of 320×240 is used, the full image size of 2560×1920 must be encoded. Thus, a substantial amount of time and CPU (e.g., processor) time is necessary to decode the image, and to generate each video frame of size 320×240 through appropriate resizing of the source image.
Now turning to
Moreover, the analyzer component 314 computes a number of step images based in part upon a maximum step image size. For example, a number of step images can be calculated by defining the aspect ratio of the image frame as being substantially similar to an aspect ratio of the video. The analyzer component 314 can define a step factor(s), for a zooming motion with a maximum step image size (wmax step,hmax step) as follows:
With wmax step being a maximum width of a step image and wv is a video width. Similarly, the step factor can be calculated by the analyzer component 312 for a panning motion as follows:
Upon calculation of a step factor, the analyzer component 314 determines a number of step images for the video size. For a zoom-out motion from a motion rectangle R1 to a motion rectangle R2, the number of step images nz, is calculated by the analyzer component 314 as follows:
When a motion is horizontal panning from rectangle R1 to rectangle R2, a number of step image np is calculated by the analyzer component 314 as follows.
XR1 and XR2 represent an X coordinate of an upper-left corner of R1 and R2, respectively. Similarly, a Y coordinate of an upper-left corner of R1 and R2 is used by the analyzer component 314 when a panning motion is vertical as follows:
Furthermore, the analyzer component 314 can determine a number of step image n when a motion is both a zoom and pan by utilizing the following:
Once a maximum step image size of step images and number of steps for a motion are determined by the analyzer component 314, the step image generator component 316 creates step images related to a motion. In other words, the determined number of step images is created for a motion (e.g., zoom, pan, or pan and zoom). The step image generator component 316 provides a parsing of a substantial motion into smaller motions with step images which are stored into video rather than storing a substantial portion of a source image and corresponding motion vectors.
In one aspect in accordance with the subject invention, the step image generator component 316 can locate a starting and ending motion rectangle within an associated step, obtain a union rectangle, scale the union rectangle down, and calculate a starting and ending motion rectangles of a scaled down union rectangle in order to create a step image. It is to be appreciated that floating point of double precision is utilized by the step image generator component 316 avoiding complications caused by round-off error. For example, the step image generator component 316 computes Ris defining such as a portion of an image used to generate a first frame of video in the associated step. Similarly, the step image generator component 316 determines Rie to represent a portion of a source image used to generate a last frame of video in the associated step. Next, the step image generator component 316 provides a union rectangle, which is scaled down to an effect where fidelity is not affected. The scale down factor can be determined by the step image generator component 316, for example, as follows:
Where WRis is a width of a starting motion rectangle Ris, WRie is a width of an ending motion rectangle Rie, and Wvideo is a width of video to be rendered. This scale down factor can be applied by the step image generator component 316 toward an extracted union region of Ris and Rie from the source image for each step. It is to be appreciated that a scaled down factor applied to a union region is to be rounded up to the nearest integer. Once a union rectangle is scaled, the step image generator component 316 calculates starting and ending motion rectangles within the scaled down step image. The step image generator component 316 can repeat the process to create step images based upon, for example, a number of step images determined by the analyzer component 314.
The motion controller component 414 utilizes a non-linear motion in order to mitigate a motion perception described above. Conventional systems utilize linear motion defining a delta of motion for each subsequent frame as follows:
During substantial zooming motion, the motion controller component 414 achieves a perception of smooth motion by utilizing an absolute motion speed that is not constant. For example, the motion controller component 414 utilizes an absolute motion speed starting fast at first, and reduces with time for a zoom-in motion. Conversely, a zoom-out motion requires the motion control component 414 to adjust absolute motion speed to a slow speed at first, and increase with time.
In one aspect in accordance with the subject invention, the motion controller component 414 controls calculated motion speed as a quadratic function of time as follows:
Where W is a width of a motion rectangle R at time t within a motion and a, b, and c are constants. By taking derivatives of above equation with respect to time, a speed of the motion is W′ and an acceleration of the motion is W″ is calculated as follows respectively:
Using the above, the speed of motion W′ changes linearly with respect to time and the acceleration of motion W″ is constant. This gives a visual perception of smooth motion.
Furthermore, the above equation relating to W (where W is the width of the motion rectangle R) can be represented by a graph illustrated in
Now turning to
The system 600 further includes an encoder component 614 that facilitates encoding of the step images created by the step image component 610. Conventional encoding techniques are not suited for the step image component 610. For example, rendering image-based video requires a decoder to decode a source image and motion information. Generally, conventional decoders maintain two image buffers, wherein a first buffer is used for current source image frame and a second is for previously used source image frame. If there is any transition to be applied between frames, such as a cross-fade, it can be applied by using these two image buffers. By maintaining two images in buffers, the decoder is able to create transitions between subsequent images each having panning and/or zooming motion. Continuing with the example, source images I0, I1, I2, I3, . . . , In can be source images to be encoded. Using conventional techniques, images are encoded in the same order as displayed in video.
However, utilizing the step image component 610, the encoder component 614 and decoder component 616 employ special considerations for step images. For example, step images S1 0, S1 1, S1 2 are step images of image I1 and S2 0, S2 1, S2 2, S2 3 are step images of I2 as follows:
Where T1, T2 and T3 are transitions between images I0 and I1, I1 and I2 and I2 and respectively.
In one aspect in accordance with the subject invention, the encoder component 614 utilizes transition T1 involving three images between two image pairs (I0, S1 0 and (I0, S1 1). As previously stated, traditional decoders maintain two image buffers. Thus, either I0 or S1 0 is discarded when S1 1 is decoded. The encoder component 614 discards S1 0 based at least in part upon subsequent output video frames generated by step images I0 and S1 1 (created by step image component 610). Under conventional encoding/decoding schemes, the decoder maintains only the last two decoded images in memory (e.g., S1 0 and S1 0). It is to be appreciated that in such cases, the encoder component 614 can determine an appropriate image to be cached by the decoder with a current frame and which image to be discarded when a new image is decoded.
In yet another aspect in accordance with the subject invention, the encoder component 614 utilizes transition T3. Transition T3 involves three images S2 2, S2 3 and I3. The sequence is encoded by the encoder component 614 as S2 2, I3 and S2 3 rather than S2 2, S2 3 and I3. The encoder component 614 encodes images in this order based at least in part upon initial generation of video frames from source images S2 2 and I3 accordingly to produce transition T3. Thus, video frames for the transition are generated initially from source images S2 2 and I3 and later from source images I3 and S2 3, accordingly the encoder component 614 correctly determines sequencing of images for appropriate playback.
Furthermore, the encoder component 614 sets a direction of transition to a newly encoded source image in which the decoder component 616 decodes accordingly. Continuing with the above, a direction of transition is S2 2 to I3 when I3 is a newly decoded image—but the direction is from S2 3 to I3 when S2 3 is the newly decoded image.
For example, the artificial intelligence component 714 utilizes a user profile of a user to employ a maximum step image size and number of steps for a motion request by such specified user. Based upon the historic data component 716, the artificial intelligence component can infer the maximum step image size and the number of steps. The user profile can be tailored to the specific user in which optimized step characteristics (e.g., maximum step size and number of step images) can be used based upon past characteristics (e.g., processor speed, memory capacity, display device, source image size, specific source image, motion requests, . . . ). Furthermore, the historic data component 716 can provide the step image component 710 with previous maximum step image size and number of steps based upon history of such user.
At 906, the method 900 starts the creation of step images. A calculation of start and end motion rectangles are computed for each number of step images for a motion. Next at 908, a union of start and end motion rectangles is computed. At 910, a scale factor is calculated such that fidelity of the step image is not deteriorated in relation to output video resolution. Then at 912, the scale factor is applied to the extracted union region. It is to be appreciated the result of calculation 912 is rounded up to the nearest integer so as not to lose precision. Next at 914, start and end motion rectangles within the scaled down image are calculated allowing the appropriate motion to be utilized with the step image. If at 916, there are more steps for which step images are to be created, the method continues at 906, otherwise, no more step images are created.
Now turning to
In the first step of motion from start motion rectangle 1102 to first intermediate rectangle (Ri) 1106, a full image can be scaled to a smaller size such that first intermediate rectangle (Ri) 1106 has substantially similar resolution (or lower) as that of video to be generated. This scaled down image is Step Image 1. From Step Image 1, video frames for motion from start motion rectangle 1102 to first intermediate rectangle (Ri) 1106 are generated without losing video quality.
With the second step of motion from first intermediate rectangle (Ri) 1106 to second intermediate rectangle (Rj) 1108, a portion of image outside first intermediate rectangle (Ri) 1106 is not needed to generate video frames. The portion of the image to store is represented by first intermediate rectangle (Ri) 1106. Furthermore, the image to be stored can be scaled down such that the second intermediate rectangle (Rj) has substantially similar resolution (or lower) as that of the video to be generated. The scaled down portion of first intermediate rectangle (Ri) 1106 is Step Image 2, wherein video frames for associated motion from R1 to Rj can be generated without losing any video quality.
During the third step of motion from second intermediate rectangle (Rj) 1108 to end motion rectangle 1104, a portion of the image outside second intermediate rectangle (Rj) 1108 is not needed to generate video frames. Only a portion of the second intermediate rectangle (Rj) is stored. Additionally, the image that is stored can be scaled down such that second intermediate rectangle has substantially similar resolution (or lower) than the resolution of video to be generated. The scaled down image of second intermediate rectangle (Rj) 1108 is Step Image 3. From Step Image 3, video frames for motion between second intermediate rectangle (Rj) 1108 to end motion rectangle 1104 are generated without losing any video quality.
It is to be appreciated that storing of smaller step images (e.g., Step Image 1, Step Image 2, and Step Image 3) facilitates storing of video such that an entire source image need not be stored in order to generate image-based video with substantially similar quality. Moreover, the step images need not be substantially similar in size as the associated image (e.g., Step Image 1 need not be substantially similar in size as the image; Step Image 2 need not be substantially similar in size as the first intermediate rectangle (Ri) 1106; and Step Image 3 need not be substantially similar in size as the second intermediate rectangle (Rj) 1108).
The first stage of motion is from start motion rectangle 1304 to first intermediate rectangle 1308. A rectangular portion of an image—indicated by dotted lines—is calculated such that frames of video during the motion from start motion rectangle 1304 to first intermediate rectangle (Ri) 1308 can be generated from this portion. This rectangle is the union of rectangle 1304 and 1308. Thus, an associated portion of the image is stored in order to generate frames for this stage of motion. Moreover, the union rectangle is scaled down such that resolution of scaled down rectangle 1304 and 1308 is substantially similar as the resolution of output video. The scaled down union rectangle portion is Step Image 1.
The second stage of motion comprises from the first intermediate rectangle (Ri) 1308 to the end motion rectangle 1306. Similarly, the rectangular portion—indicated by dotted lines in
In order to provide additional context for implementing various aspects of the present invention,
Moreover, those skilled in the art will appreciate that the inventive methods may be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, hand-held computing devices, microprocessor-based and/or programmable consumer electronics, and the like, each of which may operatively communicate with one or more associated devices. The illustrated aspects of the invention may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. However, some, if not all, aspects of the invention may be practiced on stand-alone computers. In a distributed computing environment, program modules may be located in local and/or remote memory storage devices.
One possible communication between a client 1510 and a server 1520 can be in the form of a data packet adapted to be transmitted between two or more computer processes. The system 1500 includes a communication framework 1540 that can be employed to facilitate communications between the client(s) 1510 and the server(s) 1520. The client(s) 1510 are operably connected to one or more client data store(s) 1550 that can be employed to store information local to the client(s) 1510. Similarly, the server(s) 1520 are operably connected to one or more server data store(s) 1530 that can be employed to store information local to the servers 1540.
With reference to
The system bus 1618 can be any of several types of bus structure(s) including the memory bus or memory controller, a peripheral bus or external bus, and/or a local bus using any variety of available bus architectures including, but not limited to, Industrial Standard Architecture (ISA), Micro-Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), Card Bus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), Personal Computer Memory Card International Association bus (PCMCIA), Firewire (IEEE 1394), and Small Computer Systems Interface (SCSI).
The system memory 1616 includes volatile memory 1620 and nonvolatile memory 1622. The basic input/output system (BIOS), containing the basic routines to transfer information between elements within the computer 1612, such as during start-up, is stored in nonvolatile memory 1622. By way of illustration, and not limitation, nonvolatile memory 1622 can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory 1620 includes random access memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM).
Computer 1612 also includes removable/non-removable, volatile/non-volatile computer storage media.
It is to be appreciated that
A user enters commands or information into the computer 1612 through input device(s) 1636. Input devices 1636 include, but are not limited to, a pointing device such as a mouse, trackball, stylus, touch pad, keyboard, microphone, joystick, game pad, satellite dish, scanner, TV tuner card, digital camera, digital video camera, web camera, and the like. These and other input devices connect to the processing unit 1614 through the system bus 1618 via interface port(s) 1638. Interface port(s) 1638 include, for example, a serial port, a parallel port, a game port, and a universal serial bus (USB). Output device(s) 1640 use some of the same type of ports as input device(s) 1636. Thus, for example, a USB port may be used to provide input to computer 1612, and to output information from computer 1612 to an output device 1640. Output adapter 1642 is provided to illustrate that there are some output devices 1640 like monitors, speakers, and printers, among other output devices 1640, which require special adapters. The output adapters 1642 include, by way of illustration and not limitation, video and sound cards that provide a means of connection between the output device 1640 and the system bus 1618. It should be noted that other devices and/or systems of devices provide both input and output capabilities such as remote computer(s) 1644.
Computer 1612 can operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1644. The remote computer(s) 1644 can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor based appliance, a peer device or other common network node and the like, and typically includes many or all of the elements described relative to computer 1612. For purposes of brevity, only a memory storage device 1646 is illustrated with remote computer(s) 1644. Remote computer(s) 1644 is logically connected to computer 1612 through a network interface 1648 and then physically connected via communication connection 1650. Network interface 1648 encompasses wire and/or wireless communication networks such as local-area networks (LAN) and wide-area networks (WAN). LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet, Token Ring and the like. WAN technologies include, but are not limited to, point-to-point links, circuit switching networks like Integrated Services Digital Networks (ISDN) and variations thereon, packet switching networks, and Digital Subscriber Lines (DSL).
Communication connection(s) 1650 refers to the hardware/software employed to connect the network interface 1648 to the bus 1618. While communication connection 1650 is shown for illustrative clarity inside computer 1612, it can also be external to computer 1612. The hardware/software necessary for connection to the network interface 1648 includes, for exemplary purposes only, internal and external technologies such as, modems including regular telephone grade modems, cable modems and DSL modems, ISDN adapters, and Ethernet cards.
What has been described above includes examples of the present invention. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the present invention, but one of ordinary skill in the art may recognize that many further combinations and permutations of the present invention are possible. Accordingly, the present invention is intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.
In particular and in regard to the various functions performed by the above described components, devices, circuits, systems and the like, the terms (including a reference to a “means”) used to describe such components are intended to correspond, unless otherwise indicated, to any component which performs the specified function of the described component (e.g., a functional equivalent), even though not structurally equivalent to the disclosed structure, which performs the function in the herein illustrated exemplary aspects of the invention. In this regard, it will also be recognized that the invention includes a system as well as a computer-readable medium having computer-executable instructions for performing the acts and/or events of the various methods of the invention.
In addition, while a particular feature of the invention may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of the other implementations as may be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms “includes,” and “including” and variants thereof are used in either the detailed description or the claims, these terms are intended to be inclusive in a manner similar to the term “comprising.”
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4864516||Mar 10, 1986||Sep 5, 1989||International Business Machines Corporation||Method for implementing an on-line presentation in an information processing system|
|US4974178||Nov 20, 1987||Nov 27, 1990||Matsushita Electric Industrial Co., Ltd.||Editing apparatus for audio and video information|
|US5760788||Jul 28, 1995||Jun 2, 1998||Microsoft Corporation||Graphical programming system and method for enabling a person to learn text-based programming|
|US5973755||Apr 4, 1997||Oct 26, 1999||Microsoft Corporation||Video encoder and decoder using bilinear motion compensation and lapped orthogonal transforms|
|US6040861||Oct 10, 1997||Mar 21, 2000||International Business Machines Corporation||Adaptive real-time encoding of video sequence employing image statistics|
|US6072480||Nov 5, 1997||Jun 6, 2000||Microsoft Corporation||Method and apparatus for controlling composition and performance of soundtracks to accompany a slide show|
|US6084590||Oct 10, 1997||Jul 4, 2000||Synapix, Inc.||Media production with correlation of image stream and abstract objects in a three-dimensional virtual stage|
|US6097757||Jan 16, 1998||Aug 1, 2000||International Business Machines Corporation||Real-time variable bit rate encoding of video sequence employing statistics|
|US6108001||May 21, 1993||Aug 22, 2000||International Business Machines Corporation||Dynamic control of visual and/or audio presentation|
|US6121963||Jan 26, 2000||Sep 19, 2000||Vrmetropolis.Com, Inc.||Virtual theater|
|US6222883||Jan 28, 1999||Apr 24, 2001||International Business Machines Corporation||Video encoding motion estimation employing partitioned and reassembled search window|
|US6278466 *||Jun 11, 1998||Aug 21, 2001||Presenter.Com, Inc.||Creating animation from a video|
|US6333753||Nov 25, 1998||Dec 25, 2001||Microsoft Corporation||Technique for implementing an on-demand display widget through controlled fading initiated by user contact with a touch sensitive input device|
|US6362850 *||Aug 4, 1998||Mar 26, 2002||Flashpoint Technology, Inc.||Interactive movie creation from one or more still images in a digital imaging device|
|US6369835||May 18, 1999||Apr 9, 2002||Microsoft Corporation||Method and system for generating a movie file from a slide show presentation|
|US6469711||Jul 23, 2001||Oct 22, 2002||Avid Technology, Inc.||Graphical user interface for a video editing system|
|US6480191||Sep 28, 1999||Nov 12, 2002||Ricoh Co., Ltd.||Method and apparatus for recording and playback of multidimensional walkthrough narratives|
|US6546405||Oct 23, 1997||Apr 8, 2003||Microsoft Corporation||Annotating temporally-dimensioned multimedia content|
|US6587119 *||Sep 1, 1998||Jul 1, 2003||Flashpoint Technology, Inc.||Method and apparatus for defining a panning and zooming path across a still image during movie creation|
|US6597375||Mar 10, 2000||Jul 22, 2003||Adobe Systems Incorporated||User interface for video editing|
|US6624826||Sep 28, 1999||Sep 23, 2003||Ricoh Co., Ltd.||Method and apparatus for generating visual representations for audio documents|
|US6628303||Jul 29, 1996||Sep 30, 2003||Avid Technology, Inc.||Graphical user interface for a motion video planning and editing system for a computer|
|US6654029||May 26, 2000||Nov 25, 2003||Silicon Graphics, Inc.||Data-base independent, scalable, object-oriented architecture and API for managing digital multimedia assets|
|US6665835||Dec 23, 1997||Dec 16, 2003||Verizon Laboratories, Inc.||Real time media journaler with a timing event coordinator|
|US6686970||Oct 5, 1998||Feb 3, 2004||Canon Kabushiki Kaisha||Multi-media editing method and apparatus|
|US6708217||Jan 5, 2000||Mar 16, 2004||International Business Machines Corporation||Method and system for receiving and demultiplexing multi-modal document content|
|US6763175||Sep 1, 2000||Jul 13, 2004||Matrox Electronic Systems, Ltd.||Flexible video editing architecture with software video effect filter components|
|US6803925||Sep 6, 2001||Oct 12, 2004||Microsoft Corporation||Assembling verbal narration for digital display images|
|US6823013||Mar 23, 1998||Nov 23, 2004||International Business Machines Corporation||Multiple encoder architecture for extended search|
|US7073127||Jul 1, 2002||Jul 4, 2006||Arcsoft, Inc.||Video editing GUI with layer view|
|US7240297||Jun 12, 2000||Jul 3, 2007||International Business Machines Corporation||User assistance system|
|US20010040592||Jul 23, 2001||Nov 15, 2001||Foreman Kevin J.||Graphical user interface for a video editing system|
|US20020057348 *||Mar 21, 2001||May 16, 2002||Masaki Miura||Video display control method, video display control system, and apparatus employed in such system|
|US20020065635||Nov 30, 2000||May 30, 2002||Joseph Lei||Virtual reality room|
|US20020109712||Jan 16, 2002||Aug 15, 2002||Yacovone Mark E.||Method of and system for composing, delivering, viewing and managing audio-visual presentations over a communications network|
|US20020118287 *||Feb 19, 2002||Aug 29, 2002||Grosvenor David Arthur||Method of displaying a digital image|
|US20020156702||Jun 22, 2001||Oct 24, 2002||Benjamin Kane||System and method for producing, publishing, managing and interacting with e-content on multiple platforms|
|US20030085913||Aug 21, 2002||May 8, 2003||Yesvideo, Inc.||Creation of slideshow based on characteristic of audio content used to produce accompanying audio display|
|US20030189580 *||Jan 14, 2003||Oct 9, 2003||Kun-Nan Cheng||Scaling method by using dual point cubic-like slope control ( DPCSC)|
|US20040017390||Jul 26, 2002||Jan 29, 2004||Knowlton Ruth Helene||Self instructional authoring software tool for creation of a multi-media presentation|
|US20040017508||Jul 23, 2002||Jan 29, 2004||Mediostream, Inc.||Method and system for direct recording of video information onto a disk medium|
|US20040056882||Sep 29, 2003||Mar 25, 2004||Foreman Kevin J.||Graphical user interface for a motion video planning and editing system for a computer|
|US20040066395||Sep 29, 2003||Apr 8, 2004||Foreman Kevin J.||Graphical user interface for a motion video planning and editing system for a computer|
|US20040071441||Sep 29, 2003||Apr 15, 2004||Foreman Kevin J||Graphical user interface for a motion video planning and editing system for a computer|
|US20040095379||Nov 15, 2002||May 20, 2004||Chirico Chang||Method of creating background music for slideshow-type presentation|
|US20040130566||Jun 6, 2003||Jul 8, 2004||Prashant Banerjee||Method for producing computerized multi-media presentation|
|US20040199866||Mar 31, 2003||Oct 7, 2004||Sharp Laboratories Of America, Inc.||Synchronized musical slideshow language|
|US20050034077||Dec 18, 2003||Feb 10, 2005||Denny Jaeger||System and method for creating, playing and modifying slide shows|
|US20050042591||Oct 28, 2003||Feb 24, 2005||Bloom Phillip Jeffrey||Methods and apparatus for use in sound replacement with automatic synchronization to images|
|US20050132284||May 5, 2004||Jun 16, 2005||Lloyd John J.||System and method for defining specifications for outputting content in multiple formats|
|US20050138559||Dec 19, 2003||Jun 23, 2005||International Business Machines Corporation||Method, system and computer program for providing interactive assistance in a computer application program|
|US20060041632||Aug 23, 2004||Feb 23, 2006||Microsoft Corporation||System and method to associate content types in a portable communication device|
|US20060072017||Oct 6, 2004||Apr 6, 2006||Microsoft Corporation||Creation of image based video using step-images|
|US20060188173||Feb 23, 2005||Aug 24, 2006||Microsoft Corporation||Systems and methods to adjust a source image aspect ratio to match a different target aspect ratio|
|US20060203199||Mar 8, 2005||Sep 14, 2006||Microsoft Corporation||Photostory 3 - automated motion generation|
|US20060204214||Mar 14, 2005||Sep 14, 2006||Microsoft Corporation||Picture line audio augmentation|
|US20060218488||Mar 28, 2005||Sep 28, 2006||Microsoft Corporation||Plug-in architecture for post-authoring activities|
|US20060224778||Apr 4, 2005||Oct 5, 2006||Microsoft Corporation||Linked wizards|
|EP1126721A1||Feb 16, 2001||Aug 22, 2001||Siemens Corporate Research, Inc.||Systems and methods for generating and playback of annotated multimedia presentation|
|EP1443420A2||Jan 9, 2004||Aug 4, 2004||Hitachi Software Engineering Co., Ltd.||Method, program, and system for editing contents of multimedia|
|GB2388241A||Title not available|
|JPH11205786A *||Title not available|
|WO2004062260A2||Dec 22, 2003||Jul 22, 2004||Univ Leland Stanford Junior||Methods and apparatus for interactive point-of-view authoring of digital video content|
|1||A. Schodl et al. Video Textures. Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 489-498, 2000.|
|2||ACD VideoMagic, http://www.acdsystems.com/English/Products/ACDVideoMagic/index.htm, as viewed on Jun. 15, 2005.|
|3||ACDSee, http://www.acdsystems.com/nr/acdsee7/en/homelife/index.html, last viewed May 25, 2005.|
|4||Apple iMovie, http://www.apple.com/ilife/imovie/, last viewed May 25, 2005.|
|5||Blogger. 1 page. Last Viewed on Sep. 9, 2004. http://www.blogger.com.|
|6||Blogging goes Mobile. BBC News. 2 pages. Last Updated Feb. 23, 2003. news.bbc.co.uk/1/hi/technology/2783951.stm.|
|7||C. Chiang, et al. A New Image Morphing Technique for Smooth Vista Transitions in Panoramic Image-based Virtual Environment. Proceedings of the ACM Symposium on Virtual Reality Software and Technology, pp. 81-90, 1998.|
|8||DigiRostrum, http://www.lumidium.com/digirostrum.htm, last viewed May 25, 2005.|
|9||Dissolve Factory, http://www.buena.com/dissolvefactory.shtml, last viewed May 25, 2005.|
|10||Getting Started with Apple Applications, http://developer.apple.com/referencelibrary/GettingStarted/GS<SUB>-</SUB>AppleApplications/, as viewed on Jun. 15, 2005.|
|11||Gultekin Ozsoyoglu, et al., Automating the Assembly of Presentations from Multimedia Databases, Case Western Reserve University, 1996, 20 pages, Cleveland, USA.|
|12||ImageMatics StillMotion, http://www.imagematics.com/Product/benefits, last viewed May 25, 2005.|
|13||Inductive User Interface Guidelines, http://msdn.microsoft.com/library/en-us/dnwui/html/iuiguidelines.asp?frame=true, as viewed on Jun. 14, 2005.|
|14||Jacob Rosenberg, Adobe Premiere Pro 1.5 Studio Techniques, Jun. 28, 2004, Adobe Press, Chapter 1, App. E, App. G.|
|15||Matrox releases WYSIWYG output Plug-in for Combustion, http://www.digitalproducer.com/articles/viewarticle.jsp?id=27959, as viewed on Jun. 15, 2005.|
|16||Minos N. Garofalakis, et al., Resource Scheduling for Composite Multimedia Objects, Proceedings of the 24th VLDB Conference, 1998, 12 pages, New York, USA.|
|17||MobileBlogs. 3 pages. Last Viewed on Sep. 9, 2004. http://www.mobileblogs.info.|
|18||Moving Picture, http://www.stagetools.com/, last viewed May 25, 2005.|
|19||Mozilla Foundation Announces More Open, Scriptable Plugins, http://www.mozilla.org/press/mozilla-2004-06-30.html, as viewed on Jun. 15, 2005.|
|20||Nokia LifeBlog. 1 page. Last Viewed on Sep. 9, 2004. http://www.nokia.com/nokia/0,,55360,00.html.|
|21||Pan & Zoom, http://www.applied-magic.com/pdf/panandzoom.pdf, last viewed May 25, 2005.|
|22||Photo Jam, http://www.shockwave.com/sw/content/photojam, last viewed May 25, 2005.|
|23||Photo Show, http://www.simplestar.com/, last viewed May 25, 2005.|
|24||PhotoStory. 3 pages. Last Viewed on Sep. 9, 2004. http://www.microsoft.com/PhotoStory.|
|25||ProShow, http://www.photodex.com/products/proshow/, last viewed May 25, 2005.|
|26||S. Counts and E. Fellheimer. Supporting a Social Presence Through Lightweight Photo Sharing On and Off the Desktop. Proceedings of the 2004 Conference on Human Factors in Computing Systems, pp. 599-606, 2004.|
|27||Still Life, http://www.grantedsw.com/still<SUB>-</SUB>life/, last viewed May 25, 2005.|
|28||U.S. Appl. No. 10/924,383, filed Aug. 23, 2004, Shah, et al.|
|29||Wizard 97, http://msdn.microsoft.com/library/en-us/wizard/sdkwizv4<SUB>-</SUB>7awn.asp?frame=true, last viewed on Jun. 14, 2005.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7893949 *||Jul 17, 2006||Feb 22, 2011||Brother Kogyo Kabushiki Kaisha||Image display apparatus|
|US8732611 *||Sep 4, 2007||May 20, 2014||Apple Inc.||Graphical user interface for mobile electronic device|
|US8812538||May 21, 2010||Aug 19, 2014||Wendy Muzatko||Story generation methods, story generation apparatuses, and articles of manufacture|
|US8930845||Sep 4, 2007||Jan 6, 2015||Apple Inc.||Multi-pane graphical user interface for mobile electronic device|
|US9235918||Jan 14, 2015||Jan 12, 2016||International Business Machines Corporation||Audio media mood visualization|
|US20070024733 *||Jul 17, 2006||Feb 1, 2007||Brother Kogyo Kabushiki Kaisha||Image display apparatus|
|US20090063972 *||Sep 4, 2007||Mar 5, 2009||Jeffrey Ma||Multi-Pane Graphical User Interface for Mobile Electronic Device|
|US20090064057 *||Sep 4, 2007||Mar 5, 2009||Bull William E||Graphical User Interface for Mobile Electronic Device|
|US20110084982 *||Oct 12, 2009||Apr 14, 2011||Sony Corporation||Apparatus and Method for Displaying Image Data With Memory Reduction|
|US20110191368 *||May 21, 2010||Aug 4, 2011||Wendy Muzatko||Story Generation Methods, Story Generation Apparatuses, And Articles Of Manufacture|
|U.S. Classification||348/240.99, 375/E07.145, 375/E07.146, 375/E07.249, 375/E07.163, 375/E07.182, 375/E07.252, 348/155, 375/E07.168, 375/E07.167, 375/E07.13|
|Cooperative Classification||H04N19/137, H04N19/154, H04N19/132, H04N19/17, H04N19/59, H04N19/103, H04N19/156, H04N19/192|
|European Classification||H04N7/26A8R, H04N7/26A6C4, H04N7/26A6R, H04N7/26A4Z, H04N7/46A, H04N7/26A10T, H04N7/46S, H04N7/26A6Q, H04N7/26A4C|
|Nov 12, 2004||AS||Assignment|
Owner name: MICROSOFT CORPORATION, WASHINGTON
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZHANG, DONGMEI;SHAH, MEHUL Y.;REEL/FRAME:015354/0007
Effective date: 20041005
|Apr 19, 2011||CC||Certificate of correction|
|Sep 21, 2011||FPAY||Fee payment|
Year of fee payment: 4
|Dec 9, 2014||AS||Assignment|
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034541/0477
Effective date: 20141014