Search Images Maps Play YouTube News Gmail Drive More »
Sign in
Screen reader users: click this link for accessible mode. Accessible mode has the same essential features but works better with your reader.

Patents

  1. Advanced Patent Search
Publication numberUS20060291751 A1
Publication typeApplication
Application numberUS 11/302,073
Publication dateDec 28, 2006
Filing dateDec 12, 2005
Priority dateDec 16, 2004
Also published asUS7477802, US20070217713
Publication number11302073, 302073, US 2006/0291751 A1, US 2006/291751 A1, US 20060291751 A1, US 20060291751A1, US 2006291751 A1, US 2006291751A1, US-A1-20060291751, US-A1-2006291751, US2006/0291751A1, US2006/291751A1, US20060291751 A1, US20060291751A1, US2006291751 A1, US2006291751A1
InventorsPeyman Milanfar, Sina Farsiu, Michael Elad, Michael Robinson
Original AssigneePeyman Milanfar, Sina Farsiu, Michael Elad, Robinson Michael D
Export CitationBiBTeX, EndNote, RefMan
External Links: USPTO, USPTO Assignment, Espacenet
Robust reconstruction of high resolution grayscale images from a sequence of low-resolution frames (robust gray super-resolution)
US 20060291751 A1
Abstract
A method for computing a high resolution gray-tone image from a sequence of low-resolution images uses an L1 norm minimization. In a preferred embodiment, the technique also uses a robust regularization based on a bilateral prior to deal with different data and noise models. This robust super-resolution technique uses the L1 norm both for the regularization and the data fusion terms. Whereas the former is responsible for edge preservation, the latter seeks robustness with respect to motion error, blur, outliers, and other kinds of errors not explicitly modeled in the fused images. This computationally inexpensive method is resilient against errors in motion and blur estimation, resulting in images with sharp edges. The method also reduces the effects of aliasing, noise and compression artifacts. The method's performance is superior to other super-resolution methods and has fast convergence.
Images(3)
Previous page
Next page
Claims(2)
1. A computer-implemented method for super-resolution, the method comprising:
computing a super-resolved image from a plurality of lower-resolution images using a maximum liklihood estimator based on an L1 norm minimization.
2. The method of claim 1 wherein the computing further comprises using a bilateral total variation regularization term.
Description
    CROSS-REFERENCE TO RELATED APPLICATIONS
  • [0001]
    This application claims priority from U.S. provisional patent application No. 60/637282 filed Dec. 16, 2004, which is incorporated herein by reference.
  • STATEMENT OF GOVERNMENT SPONSORED SUPPORT
  • [0002]
    This invention was supported in part by the National Science Foundation under grant CCR-9984246 and by the US Air Force under contract F49620-03-01-0387. The U.S. Government may have certain rights in the invention.
  • FIELD OF THE INVENTION
  • [0003]
    This invention relates generally to high resolution image restoration and reconstruction. More particularly, it relates to a method for computing a high resolution gray-tone image from a sequence of low-resolution images.
  • BACKGROUND OF THE INVENTION
  • [0004]
    Super-resolution image reconstruction is a kind of digial image processing that increases the resolvable detail in images. The earliest techniques for super-resolution generated a still image of a scene from a collection of similar lower-resolution images of the same scene. For example, several frames of low-resolution video may be combined using super-resolution techniques to produce a single still image whose resolution is significantly higher than that of any single frame of the original video. Because each low-resolution frame is slightly different and contributes some unique information that is absent from the other frames, the reconstructed still image has more information, i.e., higher resolution, than that of any one of the originals alone. Super-resolution techniques have many applications in diverse areas such as medical imaging, remote sensing, surveillance, still photography, and motion pictures.
  • [0005]
    The details of how to reconstruct the best high-resolution image from multiple low-resolution images is a complicated problem that has been an active topic of research for many years, and many different techniques have been proposed. One reason the super-resolution reconstruction problem is so challenging is because the reconstruction process is, in mathematical terms, an under-constrained inverse problem. In the mathematical formulation of the problem, the known low-resolution images are represented as resulting from a transformation of the unknown high-resolution image by effects of image warping due to motion, optical blurring, sampling, and noise. When the model is inverted, the original set of low-resolution images does not, in general, determine a single high-resolution image as a unique solution. Moreover, in cases where a unique solution is determined, it is not stable, i.e., small noise perturbations in the images can result in large differences in the super-resolved image. To address these problems, super-resolution techniques require the introduction of additional assumptions (e.g., assumptions about the nature of the noise, blur, or spatial movement present in the original images). Part of the challenge rests in selecting constraints that sufficiently restrict the solution space without an unacceptable increase in the computational complexity. Another challenge is to select constraints that properly restrict the solution space to good high-resolution images for a wide variety of input image data. For example, constraints that are selected to produce optimal results for a restricted class of image data (e.g., images limited to pure translational movement between frames and common space-invariant blur) may produce significantly degraded results for images that deviate even slightly from the restricted class. In summary, super-resolution techniques should be computationally efficient and produce desired improvements in image quality that are robust to variations in the properties of input image data.
  • SUMMARY OF THE INVENTION
  • [0006]
    One popular approach to super-resolution known in the art is based on a maximum likelihood (ML) estimator that uses the L2 norm (i.e., least-squares). The inventors have discovered that this least-squares-based approach is not robust, and produces images with visually apparent errors in some cases (e.g., images with non-Gaussian noise). Upon further investigation, the inventors discovered a superior ML estimator which is based on the L1 norm instead of the L2 norm (i.e., least-squares). They have demonstrated that this L1 norm has a higher breakpoint value and is a demonstrably more robust estimator than the prior L2 norm. In the case of pure translational motion, the method may be implemented using an extremely efficient pixel-wise “shift-and-add” technique.
  • [0007]
    It is known in the art of super-resolution to introduce a regularization term into the model to help stabilize the solution, remove image artifacts, and improve the rate of convergence. The regularization term compensates for missing measurement information by introducing some general information about the desired super-resolved solution, and is often implemented as a penalty factor in the generalized minimization cost function. A common regularization cost function is the class of Tikhonov cost functions, which is based on the L2 norm and constrains the total image energy or imposes spatial smoothness. This type of regularization term, however, removes sharp edges along with image noise. A regularization term that preseves edges better is the total variation (TV) method which limits the total change in the image as measured by the L1 norm of the magnitude of the gradient. The inventors have discovered that the TV method may be improved by combining it with a bilateral filter to provide a very robust regularization method, which they call bilateral TV. They have shown that bilateral TV not only produces sharp edges and retains point-like details in the super-resolved image but also allows for computationally efficient implementation superior to other regularization methods.
  • [0008]
    Accordingly, in one aspect, the invention provides a method for computing a high resolution gray-tone image from a sequence of low-resolution images using an L1 norm minimization and robust regularization based on a bilateral prior to deal with different data and noise models. This robust super-resolution technique uses the L1 norm both for the regularization and the data fusion terms. Whereas the former is responsible for edge preservation, the latter seeks robustness with respect to motion error, blur, outliers, and other kinds of errors not explicitly modeled in the fused images.
  • [0009]
    This computationally inexpensive method is resilient against errors in motion and blur estimation, resulting in images with sharp edges. The method also reduces the effects of aliasing, noise and compression artifacts. The method's performance is superior to other super-resolution methods and has fast convergence.
  • DETAILED DESCRIPTION
  • [0010]
    Details of various embodiments of the present invention are disclosed in the following appendices:
    • Appendix A: Sina Farsiu, Dirk Robinson, Michael Elad, Peyman Milanfar “Fast and Robust Multiframe Super Resolution” IEEE Trans. Image. Processing, October 2004, Vol. 13, No. 10, pp. 1327-1344.
    • Appendix B: Sina Farsiu, Dirk Robinson, Michael Elad, Peyman Milanfar “Advances and Challenges in Super-Resolution” International Journal of Imaging Systems and Technology, August 2004, Vol. 14, No 2, pp. 47-57.
  • [0013]
    As one of ordinary skill in the art will appreciate, various changes, substitutions, and alterations could be made or otherwise implemented without departing from the principles of the present invention. Accordingly, the examples and drawings disclosed herein including the appendix are for purposes of illustrating the preferred embodiments of the present invention and are not to be construed as limiting the invention.
Patent Citations
Cited PatentFiling datePublication dateApplicantTitle
US6411333 *Apr 2, 1999Jun 25, 2002Teralogic, Inc.Format conversion using patch-based filtering
US6434280 *Nov 9, 1998Aug 13, 2002Gentech CorporationSystem and method for generating super-resolution-enhanced mosaic images
US6650335 *Feb 22, 2001Nov 18, 2003Mitsubishi Electric Research Laboratories, Inc.Method for extracting static and dynamic super-resolution textures form a sequence of images
US6766067 *Apr 20, 2001Jul 20, 2004Mitsubishi Electric Research Laboratories, Inc.One-pass super-resolution images
US7085323 *Apr 3, 2002Aug 1, 2006Stmicroelectronics, Inc.Enhanced resolution video construction method and apparatus
US7218796 *Apr 30, 2003May 15, 2007Microsoft CorporationPatch-based video super-resolution
Referenced by
Citing PatentFiling datePublication dateApplicantTitle
US7702184 *Sep 28, 2004Apr 20, 2010Seiko Epson CorporationGeneration of high-resolution image based on multiple low-resolution images
US7941004 *Apr 30, 2008May 10, 2011Nec Laboratories America, Inc.Super resolution using gaussian regression
US7953297Feb 12, 2010May 31, 2011Seiko Epson CorporationGeneration of high-resolution images based on multiple low-resolution images
US8750647Feb 3, 2011Jun 10, 2014Massachusetts Institute Of TechnologyKinetic super-resolution imaging
US20050163402 *Sep 28, 2004Jul 28, 2005Seiji AisoGeneration of high-resolution image based on multiple low-resolution images
US20090274385 *Apr 30, 2008Nov 5, 2009Nec Laboratories AmericaSuper resolution using gaussian regression
US20100150474 *Feb 12, 2010Jun 17, 2010Seiko Epson CorporationGeneration of high-resolution images based on multiple low-resolution images
CN103854267A *Mar 12, 2014Jun 11, 2014昆明理工大学Image fusion and super-resolution achievement method based on variation and fractional order differential
CN103914815A *Dec 31, 2012Jul 9, 2014诺基亚公司Image fusion method and device
CN104408697A *Oct 23, 2014Mar 11, 2015西安电子科技大学Image super-resolution reconstruction method based on genetic algorithm and regular prior model
WO2014102458A1 *Dec 30, 2013Jul 3, 2014Nokia CorporationMethod and apparatus for image fusion
Classifications
U.S. Classification382/299
International ClassificationG06K9/32
Cooperative ClassificationG06T3/4053
European ClassificationG06T3/40S
Legal Events
DateCodeEventDescription
Dec 5, 2006ASAssignment
Owner name: REGENTS OF THE UNIVERSITY OF CALIFORNIA, SANTA CRU
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MILANFAR, PEYMAN;FARSIU, SINA;ROBINSON, MICHAEL D.;REEL/FRAME:018670/0782
Effective date: 20061120
Dec 22, 2010ASAssignment
Owner name: NATIONAL SCIENCE FOUNDATION, VIRGINIA
Free format text: CONFIRMATORY LICENSE;ASSIGNOR:UNIVERSITY OF CALIFORNIA;REEL/FRAME:025762/0349
Effective date: 20080724
May 25, 2011ASAssignment
Owner name: AIR FORCE, UNITED STATES, VIRGINIA
Free format text: CONFIRMATORY LICENSE;ASSIGNOR:CALIFORNIA, UNIVERSITY OF;REEL/FRAME:026420/0761
Effective date: 20080724