Dynamic reconstruction of high resolution video from low-resolution color-filtered video (video-to-video super-resolution)

Abstract

In one aspect, the present invention provides a dynamic super-resolution technique that is computationally efficient. A recursive computation takes as input a previously computed super-resolved image derived from a sequence of low-resolution input frames. Combining this super-resolved image with a later low-resolution input frame in the sequence, the technique produces a new super-resolved image. By recursive application, a sequence of super-resolved images is produced. In a preferred embodiment, the technique uses a computationally simple and effective method based on adaptive filtering for computing a high resolution image and updating this high resolution image over time to produce an enhanced sequence of images. The method may be implemented as a general super-resolution software tool capable of handing a wide variety of input image data.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. provisional patent application Ser. No. 60/637058 filed 12/16/2004, which is incorporated herein by reference.

STATEMENT OF GOVERNMENT SPONSORED SUPPORT

This invention was supported in part by the National Science Foundation under grant CCR-9984246 and by the U.S. Air Force under contract F49620-03-01-0387. The U.S. Government may have certain rights in the invention.

FIELD OF THE INVENTION

This invention relates generally to a class of digital image processing techniques known as digital image reconstruction. More particularly, it relates to methods for computing a temporal sequence of resolution-enhanced images from an original sequence of lower-resolution images.

BACKGROUND OF THE INVENTION

Super-resolution image reconstruction is a kind of digial image processing that increases the resolvable detail in images. The earliest techniques for super-resolution generated a still image of a scene from a collection of similar lower-resolution images of the same scene. For example, several frames of low-resolution video may be combined using super-resolution techniques to produce a single still image whose resolution is significantly higher than that of any single frame of the original video. Because each low-resolution frame is slightly different and contributes some unique information that is absent from the other frames, the reconstructed still image has more information, i.e., higher resolution, than that of any one of the originals alone. Super-resolution techniques have many applications in diverse areas such as medical imaging, remote sensing, surveillance, still photography, and motion pictures.

The details of how to reconstruct the best high-resolution image from multiple low-resolution images is a complicated problem that has been an active topic of research for many years, and many different techniques have been proposed. One reason the super-resolution reconstruction problem is so challenging is because the reconstruction process is, in mathematical terms, an under-constrained inverse problem. In the mathematical formulation of the problem, the known low-resolution images are represented as resulting from a transformation of the unknown high-resolution image by effects of image warping due to motion, optical blurring, sampling, and noise. When the model is inverted, the original set of low-resolution images does not, in general, determine a single high-resolution image as a unique solution. Moreover, in cases where a unique solution is determined, it is not stable, i.e., small noise perturbations in the images can result in large differences in the super-resolved image. To address these problems, super-resolution techniques require the introduction of additional assumptions (e.g., assumptions about the nature of the noise, blur, or spatial movement present in the original images). Part of the challenge rests in selecting constraints that sufficiently restrict the solution space without an unacceptable increase in the computational complexity. Another challenge is to select constraints that properly restrict the solution space to good high-resolution images for a wide variety of input image data. For example, constraints that are selected to produce optimal results for a restricted class of image data (e.g., images limited to pure translational movement between frames and common space-invariant blur) may produce significantly degraded results for images that deviate even slightly from the restricted class. In summary, super-resolution techniques should be computationally efficient and produce desired improvements in image quality that are robust to variations in the properties of input image data.

More recently, researchers have started to investigate dynamic super-resolution techniques, i.e., the use of super-resolution to produce not just a single still image, but a temporal sequence of resolution-enhanced video frames. While it may appear that this problem is a simple extension of the static SR situation, the memory and computational requirements for the dynamic case are so taxing as to preclude its application without highly efficient algorithms. While it may be obvious to repeatedly apply still image super-resolution techniques to produce a sequence of super-resolved frames, this approach is computationally inefficient. It thus remains a challenge to design a dynamic super-resolution technique that is computationally efficient.

SUMMARY OF THE INVENTION

In one aspect, the present invention provides a dynamic super-resolution technique that is computationally efficient. A recursive computation takes as input a previously computed super-resolved image, combines this super-resolved image with a next low-resolution input frame in a sequence, and produces a new super-resolved image. By recursive application, a sequence of super-resolved images is produced. In a preferred embodiment, the technique uses a computationally simple and effective method based on adaptive filtering for computing a high resolution image and updating this high resolution image over time to produce an enhanced sequence of images. The method may be implemented as a general super-resolution software tool capable of handing a wide variety of input image data.

DETAILED DESCRIPTION

Descriptions and figures of various embodiments of the present invention are disclosed in the following appendices:

• • Appendix A: Sina Farsiu, Dirk Robinson, Michael Elad, Peyman Milanfar “Dynamic Demosaicing and Color Super-Resolution of Video Sequences,” Proceedings, 2004 SPIE Conf. on Image Reconstruction from Incomplete Data, August 2004, Denver, Colo., 10 pages.
  • Appendix B: Sina Farsiu, Dirk Robinson, Michael Elad, Peyman Milanfar “Dynamic Super-Resolution,” 10 pages.
  • Appendix C: Sina Farsiu, Michael Elad, Peyman Milanfar “Dynamic Super-Resolution,” 5 pages.
  • Appendix D: Sina Farsiu, Dirk Robinson, Michael Elad, Peyman Milanfar “Dynamic Demosaicing and Color Super-Resolution of Video Sequences,” 8 pages.
  • Appendix E: Sina Farsiu, Dirk Robinson, Michael Elad, Peyman Milanfar “Fast Dynamic Super-Resolution” Abstract, 1 page.
  • Appendix F: Sina Farsiu, Dirk Robinson, Michael Elad, Peyman Milanfar “Advances and Challenges in Super-Resolution” International Journal of Imaging Systems and Technology, V. 14, No 2, pp. 47-57, August 2004.

As one of ordinary skill in the art will appreciate, various changes, substitutions, and alterations could be made or otherwise implemented without departing from the principles of the present invention. Accordingly, the examples and drawings disclosed herein including the appendix are for purposes of illustrating the preferred embodiments of the present invention and are not to be construed as limiting the invention.

Claims

1. A computer-implemented method for dynamic super-resolution, the method comprising:

computing an initial super-resolved image from at least one low-resolution image; and

computing a next super-resolved image in a super-resolved image sequence from information in a previous super-resolved image in the super-resolved image sequence and a next low-resolution image in a low-resolution image sequence.