|Publication number||US6992707 B2|
|Application number||US 10/090,804|
|Publication date||Jan 31, 2006|
|Filing date||Mar 6, 2002|
|Priority date||Mar 6, 2002|
|Also published as||US20030169278|
|Publication number||090804, 10090804, US 6992707 B2, US 6992707B2, US-B2-6992707, US6992707 B2, US6992707B2|
|Original Assignee||Hewlett-Packard Development Company, L.P.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (20), Referenced by (29), Classifications (26), Legal Events (5)|
|External Links: USPTO, USPTO Assignment, Espacenet|
The technical field relates to video imaging systems, and, in particular, to joint video and still image pipelines.
Digital cameras are widely used to acquire high resolution still image photographs. Digital video cameras are also used to record home videos, television programs, movies, concerts, or sports events on a magnetic disk or optical DVD for storage or transmission through communications channels. Some commercial cameras are able to take both digital video and digital still image photographs. However, most of these cameras required a user to switch between a video recording mode and a digital still image mode. Separate pipelines are generally used for each of the video recording and still image modes. Examples of these cameras include SANYO ID-SHOT® and CANNON POWERSHOT S300®. The SANYO ID-SHOT® uses an optical disk, whereas the CANNON POWERSHOT S300® uses synchronous dynamic random access memory (SDRAM). However, both cameras are still image cameras that have the capability of taking video clips, using separate pipelines.
Other cameras use a single software pipeline to acquire both digital video and low quality still images by taking one of the video frames as is, and storing the particular video frame as a high resolution still image. Examples of such cameras include JVC GR-DVL9800®, which is a digital video camera that allows a user to take a picture at certain point in time. However, the pictures taken generally are of low quality, because a low resolution video pipeline is used to generate the high resolution still image pictures.
When still images are acquired in burst mode, current cameras try to process both pipelines independently. If a single hardware processing pipeline is used, a large frame buffer may be needed to store video frames while the burst mode still images are processed. However, a large frame buffer is costly, and build up delay on the video side may be undesirable.
Other cameras try brute force real time processing, which is costly.
A method and corresponding apparatus for concurrently processing digital video frames and high resolution still images in burst mode include acquiring with high priority video frames and high resolution still images in burst mode from one or more image sensors, and storing with high priority the video frames and the high resolution still images in raw format in a memory during acquisition of the high resolution still images in burst mode. The method and corresponding apparatus further include processing with low priority the video frames stored in the memory using a video pipeline, and processing the high resolution still images acquired during the burst mode using a high resolution still image pipeline. The high resolution still image pipeline runs concurrently with the video pipeline.
In an embodiment, the video frames and the high resolution still images are acquired and stored in real time. In another embodiment, the high resolution still images are filtered and downsampled to be inputted into the video pipeline to make up deficiencies. In yet another embodiment, the video frames and the high resolution still images are processed into a standard format by an image/video transcoding agent.
The preferred embodiments of the method and corresponding apparatus for concurrently processing digital video frames and high resolution still images in burst mode will be described in detail with reference to the following figures, in which like numerals refer to like elements, and wherein:
A digital video camera system may utilize a joint video and still image pipeline that simultaneously acquires, processes, transmits and/or stores digital video and high resolution digital still image photographs. The joint pipeline may include a video pipeline optimized for digital video frames and a high resolution still image pipeline optimized for high resolution digital still images. The digital video camera system may also concurrently acquire and process video frames and high resolution still image in burst mode using delayed encoding technology. The delayed encoding technology acquires video frames and burst mode still images in raw format without processing, and stores the video frames and the high resolution still images acquired during the burst mode into a memory or storage device. The video frames and the high resolution still images may be processed with low priority if extra time and processing power are available. The digital video camera system processes the stored video frames and the stored high resolution still images acquired during the burst mode after the burst mode or video recording stops.
The image sensor 240 typically continuously acquires high resolution video frames 120 at a rate of, for example, 30 fps. Each of the high resolution video frames 120 may be converted into a high resolution still image photograph 110. When a user is not interested in taking a high resolution still image photograph 110, the only pipeline running may be the video pipeline 220, which acquires high resolution video frames 120, and downsamples the frames to medium resolution (for example, 640×480), then processes the medium resolution video frames 120. When the user wants to acquire a high resolution still image frame 110, the image acquired by the high resolution image sensor 240 can be used both in the video pipeline 220 as well as in the high resolution still image pipeline 210 (described in detail later).
The video camera system 200 may include a storage device 250 and a connection with a communications channel/network 260, such as the Internet or other type of computer or telephone networks. The storage device 250 may include a hard disk drive, floppy disk drive, CD-ROM drive, or other types of non-volatile data storage, and may correspond with various databases or other resources. After the video frames 120 and the high resolution still image frames 110 are acquired, the video frames 120 and the high resolution still image frames 110 may be stored in the storage device 250 or transmitted through the communication channel 260. The video camera system 200 may also include an image/video transcoding agent 270 for encoding the video frames 120 and the high resolution still image frames 110 into a standard format, for example, tagged image file format (TIFF) or Joint Photographic Experts Group (JPEG).
In the video pipeline 220, the processor 360 may downsample, demosaic, and color correct the video frames 120. Next, the processor 360 may compress and transmit the video frames 120 through an input/output (I/O) unit 340. Alternatively, the video frames 120 may be stored in the storage device 250.
Both pipelines 210, 220 may be executed concurrently, i.e., acquiring high resolution still image photographs 110 during video recording. A frame buffer 330 may store video frames 120 while the processor 360 is processing the high resolution still image frame 110. The sensor controller 310 may still capture video frames 120 at a rate of, for example, 30 fps, and store the video frames 120 into the memory 320. The processor 360 may downsample the video frames 120 and send the downsampled video frames 120 into the frame buffer 330. The frame buffer 330 may store the downsampled video frames 120 temporarily without further processing. This may incur some delay in the video pipeline 220 if the video is directly transmitted through the communications channel 260. However, this delay may be compensated by a similar buffer on the receiver end. During video frame buffering, the high resolution still image frame 110 may be processed by the processor 360, using complex algorithms. At the same time, the video frames 120 may be continuously stored into the memory 320, downsampled, and sent into the frame buffer 330 to be stored.
Although the video camera system 200 is shown with various components, one skilled in the art will appreciate that the video camera system 200 can contain additional or different components. In addition, although the video frames 120 and the still image frames 110 are described as being stored in memory, one skilled in the art will appreciate that the video frames 120 and the still image frames 110 can also be stored on or read from other types of computer program products or computer-readable media, such as secondary storage devices, including hard disks, floppy disks, or CD-ROM; a carrier wave from the Internet or other network; or other forms of RAM or ROM. The computer-readable media may include instructions for controlling the video camera system 200 to perform a particular method.
After raw pixel video data of video frames 120 are acquired, for example, at 1024×1008 and 30 fps (block 400), the video frames 120 may be downsampled and demosaiced in order to save memory space (block 410). Then, the frame buffer 330 may buffer the video frames 120 while the high resolution still image frame 110 is being acquired, processed, stored, and/or transmitted (block 420). Alternatively, demosaicing may be performed after the video frames 120 are buffered. Thereafter, the video pipeline 220 may start emptying the frame buffer 330 as fast as possible, and performing color correction, compression, storage and/or transmission (blocks 430, 440, 450). Once the frame buffer 330 is emptied, another high resolution still image frame 110 may be acquired.
For high resolution still image frames 110, sophisticated demosaicing may be performed (block 412), followed by high quality color correction (block 432). The high resolution still image frames 110 may optionally be compressed (block 442), and then stored and/or transmitted through similar communications channels 260 (block 452).
Once the high resolution still image frame 110 is demosaiced, the high resolution still image frame 110 may be color corrected depending on the illumination present at the time of the capture (block 432). Complex transformation matrices may be used to restore accurate color to the high resolution still image frames 110, in order to generate an excellent photograph. The color correction algorithms, may be similar to the algorithm used in the HP-PHOTOSMART 618®.
When a user acquires high resolution still images in burst mode, the digital video camera system 200 uses delayed encoding technology to acquire and store video frames and burst mode high resolution still images in raw format into the memory 320 or the storage device 250.
The frame buffer 330 may be used for loss-less compression of the raw high resolution still image frames 110 and intermediate processing of the video frames 120 until one of the high resolution still image frames 110 is acquired. The length of the burst mode and the amount of processing power define the size of the frame buffer 330, which is preferably kept to minimum due to cost. The high resolution still image frames 110 may be used to reset Moving Picture Experts Group (MPEG) encoding process as intraframes (I-frames). I-frames are frames not compressed depending on previous or future frames, i.e., stand alone compressed frames. I-frames do not depend on information from other frames to be compressed. Accordingly, all compression algorithms may start with an I-frame, and all other frames may be compressed based on the I-frame.
After one of the I-frames are acquired, the processor 360 stores the video frames 120 and high resolution still image frames 110 in raw format without any processing into the memory 320 or the storage device 250. If extra time and processing power are available, some stored video frames 120 and high resolution still image frames 110 may be processed. After the user stops video recording or acquiring high resolution still images in burst mode, the processor 360 starts processing the video frames 120 and the high resolution still image frames 110 in parallel.
During the burst mode still image acquisition, a multithread system may be employed.
Block 520 represents real time acquisition, downsampling, and storage of video frames 120 at, for example, (30-B) fps. The high resolution still image frames 110, for example, B frames, are inputted into the video processing pipeline 220. During processing, the high resolution still image frames 110 may be filtered and downsampled to generate lower resolution video frames to be inputted into the video processing pipeline 220 to make up the deficiency. For example, if video frames are sampled at 30 fps, and high resolution still image frames are acquired at 3 fps, then one out of ten frames are sent to the high resolution still image pipeline 210. The frames are later downsampled and inputted into the video pipeline 220. Alternatively, the filtering and downsampling process may be performed in block 530 (described later). The video frames 120 are also stored in raw format in the memory 320 or the storage device 250. This process also has high priority.
In block 530, low priority video processing pipeline 220 processes and compresses buffered video frames 120 and the video frames 120 stored during process 520. Therefore, while processes 510 and 520 have high priority, any extra time and processing power may be used to process and compress the stored video frames 120.
In block 540, low priority still image processing pipeline 210 processes and compresses each of the raw high resolution still image frames 110. Whenever extra time and processing power are available, the processors 360 may process small amount of high resolution still image frames 110.
Processes 530 and 540 remain active with low priority until all the video frames 120 and the high resolution still image frames 110 stored in processes 510 and 520 have been successfully encoded and stored. Therefore, the overall data is stored in real time, and low priority processes process the data in the background with non-real time processing, so as to reduce computational burden. Processes 510, 520, 530 and 540 may be implemented independently with the one or more processors 360.
For example, 90% of time may be spent on processes 510 and 520, and 10% of time on processes 530 and 540. When the user stops the burst mode or video recording, the low priority processes 530 and 540 gain higher share of the total processing power. In the above example, if burst mode is stopped, process 520 is processed at 30 fps, as opposed to (30-B) fps, because no more high resolution still image frames 110 are acquired.
If memory space is available, the video camera system 200 continues to compress video frames 120 and still image frames 110 in the memory 320 or the storage device 250. However, if the memory 320 or the storage device 250 is filled up with no extra space to process and compress new video frames 120 and burst mode high resolution still image frames 110, a flag may be used to signal that image and/or video acquisition needs to stop. Processes 530 and 540 may take advantage of the internal memory 320 and frame buffer 330 to continue processing and compressing the buffered video frames 120 and the raw still images 110, thus freeing up some storage space for more image and/or video acquisition. If this is not achieved, then the video frames 120 and the high resolution still image frames 110 may be encoded at transmission/download time with the image/video transcoding agent 270. In other words, if the video frames or the still image frames are not fully encoded due to lack of memory space, the video frames and the still images frames can be encoded fully at download time by the image/video transcoding agent 270.
Within the video camera system 200, the video frames 120 and the high resolution still image frame 110 may be kept in a nonstandard proprietary format. The image/video transcoding agent 270, which typically runs on the video camera system 200, detects when a video frame 120 or a high resolution still image frame 110 is to be downloaded and transcodes the proprietary loss-less (or near loss-less) raw video frame 120 or high resolution still image frame 110 into a processed video or image, which is then packed into a standard compression format, for example, TIFF or JPEG. Alternatively, the image/video transcoding agent 270 may run on a docking station or on the host personal computer (PC).
Video frames 120 and high resolution still image frames 110 acquired during the burst mode are stored in raw format at the bottom of the memory map 600. For example, S1, S7, S13 are high resolution still image frames #1, #7, and #13, whereas V2, V3, V4, V5, V6, V8, V9, V10, V11, V12, V14–V18 are video frames #2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 14–18. In other words, in this example, three burst mode high resolution still image frames S1, S7, S13 are generated from 18 frames. After the high resolution still image frames 110 are acquired, or if extra time and processing power exist, low priority processes 530, 540 start processing the raw data, i.e., still image frames S1, S7, S13 and the rest of the video frames. The low priority processes also combines the video frames 120 with filtered and downsampled versions of the high resolution still image frames 110 in order to generate a continuous compressed video sequence 120. For example, the processors 360 downsample S1 into V1, S7 into V7, and S13 into V13, so that a continuous video sequence is generated, from V1 to V18.
While the method and apparatus for concurrently processing digital video frames and high resolution still images in burst mode have been described in connection with an exemplary embodiment, those skilled in the art will understand that many modifications in light of these teachings are possible, and this application is intended to cover any variations thereof.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4484349 *||Mar 11, 1982||Nov 20, 1984||Environmental Research Institute Of Michigan||Parallel pipeline image processor|
|US5022090 *||Sep 26, 1988||Jun 4, 1991||Minolta Camera Kabushiki Kaisha||Digital image processing apparatus for correctly addressing image memory|
|US5046190 *||Sep 6, 1988||Sep 3, 1991||Allen-Bradley Company, Inc.||Pipeline image processor|
|US5287416 *||Aug 17, 1992||Feb 15, 1994||Unisys Corporation||Parallel pipelined image processor|
|US5561777||Aug 30, 1993||Oct 1, 1996||Xerox Corporation||Process for sequentially reading a page from an image memory in either of two directions|
|US5790125||Apr 22, 1996||Aug 4, 1998||International Business Machines Corporation||System and method for use in a computerized imaging system to efficiently transfer graphics information to a graphics subsystem employing masked span|
|US5991465 *||Aug 29, 1996||Nov 23, 1999||Apple Computer, Inc.||Modular digital image processing via an image processing chain with modifiable parameter controls|
|US6292218 *||Jul 16, 1997||Sep 18, 2001||Eastman Kodak Company||Electronic camera for initiating capture of still images while previewing motion images|
|US6330400 *||Jan 28, 2000||Dec 11, 2001||Concord Camera-Corp.||Compact through-the-lens digital camera|
|US6563535 *||May 19, 1998||May 13, 2003||Flashpoint Technology, Inc.||Image processing system for high performance digital imaging devices|
|US6720968 *||Dec 11, 1998||Apr 13, 2004||National Instruments Corporation||Video acquisition system including a virtual dual ported memory with adaptive bandwidth allocation|
|US20020054116 *||Oct 9, 2001||May 9, 2002||Flashpoint Technology, Inc.||Method and apparatus for editing heterogeneous media objects in a digital imaging device|
|US20020135683 *||Dec 20, 2000||Sep 26, 2002||Hideo Tamama||Digital still camera system and method|
|US20030021474 *||Apr 30, 2002||Jan 30, 2003||Hewlett Packard Company||Electronic image colour plane reconstruction|
|US20030048279 *||Sep 13, 2001||Mar 13, 2003||Chi Wah Kok||Method and system for interpolation in color images|
|US20030052981 *||Aug 27, 2001||Mar 20, 2003||Ramakrishna Kakarala||Digital image system and method for implementing an adaptive demosaicing method|
|US20030052986 *||Sep 12, 2002||Mar 20, 2003||Canon Kabushiki Kaisha||Image processing apparatus, image processing method, program, and storage medium|
|US20030058355 *||Sep 23, 1998||Mar 27, 2003||Sau C. Wong||Analog buffer memory for high-speed digital image capture|
|US20030112347 *||Dec 13, 2001||Jun 19, 2003||International Business Machines Corporation||Method and apparatus for producing still video images using electronic motion video apparatus|
|US20030147640 *||Feb 6, 2002||Aug 7, 2003||Voss James S.||System and method for capturing and embedding high-resolution still image data into a video data stream|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7319480 *||Jun 26, 2003||Jan 15, 2008||Eastman Kodak Company||Method and apparatus for compressing motion image files to provide an improved image navigation display|
|US7369683 *||Aug 4, 2004||May 6, 2008||Sanyo Electric Co., Ltd.||Imaging device|
|US7388605 *||Nov 12, 2002||Jun 17, 2008||Hewlett-Packard Development Company, L.P.||Still image capturing of user-selected portions of image frames|
|US7528865 *||Nov 21, 2002||May 5, 2009||Fujifilm Corporation||Digital movie camera and method of controlling operations thereof|
|US7548258 *||Nov 22, 2004||Jun 16, 2009||Arecont Vision Llc.||High resolution network video camera with massively parallel implementation of image processing, compression and network server|
|US7548259 *||Aug 12, 2004||Jun 16, 2009||Microsoft Corporation||System and method for producing a higher resolution still image from video information|
|US7619661 *||Nov 17, 2009||Fujifilm Corporation||Camera system|
|US7667741 *||Feb 23, 2010||Samsung Electronics Co., Ltd||Device and method for taking picture while recording moving picture|
|US8432804||Sep 3, 2008||Apr 30, 2013||Hewlett-Packard Development Company, L.P.||Transmitting video streams|
|US8643746||May 18, 2011||Feb 4, 2014||Intellectual Ventures Fund 83 Llc||Video summary including a particular person|
|US8665345||May 18, 2011||Mar 4, 2014||Intellectual Ventures Fund 83 Llc||Video summary including a feature of interest|
|US9247141 *||Feb 21, 2013||Jan 26, 2016||Htc Corporation||Burst image capture method and image capture system thereof|
|US20030095191 *||Nov 21, 2002||May 22, 2003||Fuji Photo Film Co., Ltd.||Digital movie camera and method of controlling operations thereof|
|US20030184658 *||Dec 5, 2002||Oct 2, 2003||Fredlund John R.||System and method for capturing motion video segments and providing still and motion image files|
|US20040090548 *||Nov 12, 2002||May 13, 2004||Pere Obrador||Image capture systems and methods|
|US20040252202 *||Jun 10, 2003||Dec 16, 2004||Silverstein D. Amnon||Image data processing methods, imaging apparatuses, and articles of manufacture|
|US20040264789 *||Jun 26, 2003||Dec 30, 2004||Hideki Akiyama||Method and apparatus for compressing motion image files to provide an improved image navigation display|
|US20050100224 *||Sep 24, 2003||May 12, 2005||Cannon Kabushiki Kaisha||Transcoding of digital data|
|US20050140787 *||Nov 22, 2004||Jun 30, 2005||Michael Kaplinsky||High resolution network video camera with massively parallel implementation of image processing, compression and network server|
|US20060034533 *||Aug 12, 2004||Feb 16, 2006||Microsoft Corporation||System and method for producing a higher resolution still image from video information|
|US20060050154 *||Aug 4, 2004||Mar 9, 2006||Akio Kobayashi||Imaging device|
|US20060216001 *||Mar 24, 2006||Sep 28, 2006||Samsung Electronics Co., Ltd.||Device and method for taking picture while recording moving picture|
|US20060262199 *||May 10, 2006||Nov 23, 2006||Fuji Photo Film Co., Ltd.||Camera system|
|US20090141800 *||Sep 3, 2008||Jun 4, 2009||Larson Arnold W||Transmitting Video Streams|
|US20100135644 *||Nov 25, 2009||Jun 3, 2010||Samsung Digital Imaging Co., Ltd.||Photographing apparatus and method of controlling the same|
|US20100295966 *||Apr 13, 2010||Nov 25, 2010||John Furlan||Digital video camera with high resolution imaging system|
|US20120120083 *||Apr 18, 2011||May 17, 2012||Novatek Microelectronics Corp.||Display apparatus, and display controller and operating method thereof|
|US20130222671 *||Feb 21, 2013||Aug 29, 2013||Htc Corporation||Burst Image Capture Method and Image Capture System thereof|
|CN102467866A *||May 25, 2011||May 23, 2012||联咏科技股份有限公司||Display apparatus, and display controller and operating method thereof|
|U.S. Classification||348/220.1, 348/207.99, 386/E05.072|
|International Classification||H04N9/804, H04N5/765, H04N5/77, H04N5/85, H04N9/79, H04N5/225, H04N5/781, H04N1/40, G06T3/40|
|Cooperative Classification||G06T3/4015, H04N5/23245, H04N9/8047, H04N5/772, H04N1/40, H04N5/765, H04N5/85, H04N9/8042, H04N5/781, H04N9/7921|
|European Classification||H04N5/232R, G06T3/40C, H04N5/77B, H04N1/40|
|Oct 25, 2002||AS||Assignment|
Owner name: HEWLETT-PACKARD COMPANY, COLORADO
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:OBRADOR, PERE;REEL/FRAME:013445/0276
Effective date: 20020304
|Jun 18, 2003||AS||Assignment|
Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P., COLORAD
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HEWLETT-PACKARD COMPANY;REEL/FRAME:013776/0928
Effective date: 20030131
Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P.,COLORADO
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HEWLETT-PACKARD COMPANY;REEL/FRAME:013776/0928
Effective date: 20030131
|Jul 31, 2009||FPAY||Fee payment|
Year of fee payment: 4
|Nov 29, 2012||AS||Assignment|
Owner name: TARSIUM B.V., NETHERLANDS
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P.;REEL/FRAME:029369/0905
Effective date: 20121026
|Mar 7, 2013||FPAY||Fee payment|
Year of fee payment: 8