Abstract: Methods, apparatus, and computer-readable storage media for simulating artistic effects in images rendered from plenoptic data. An impressionistic-style artistic effect may be generated in output images of a rendering process by an “impressionist” 4D filter applied to the microimages in a flat captured with focused plenoptic camera technology. Individual pixels are randomly selected from blocks of pixels in the microimages, and only the randomly selected pixels are used to render an output image. The randomly selected pixels are rendered to generate the artistic effect, such as an “impressionistic” effect, in the output image. A rendering technique is applied that samples pixel values from microimages using a thin sampling kernel, for example a thin Gaussian kernel, so that pixel values are sampled only from one or a few of the microimages.

Abstract: Methods, apparatus, and computer-readable storage media for rendering focused plenoptic camera data. A depth-based rendering technique is described that estimates depth at each microimage and then applies that depth to determine a position in the input flat from which to read a value to be assigned to a given point in the output image. The techniques may be implemented according to parallel processing technology that renders multiple points of the output image in parallel. In at least some embodiments, the parallel processing technology is graphical processing unit (GPU) technology.

Abstract: Various methods and apparatus for removing artifacts in frequency domain processing of light-field images are described. Methods for the reduction or removal of the artifacts are described that include methods that may be applied during frequency domain processing and a method that may be applied during post-processing of resultant angular views. The methods may be implemented in software as or in a light-field frequency domain processing module. The described methods include an oversampling method to determine the correct centers of slices, a phase multiplication method to determine the correct centers of slices, a method to exclude low-energy slices, and a cosmetic correction method.

Abstract: Methods, apparatus, and computer-readable storage media for rendering focused plenoptic camera data. A rendering with blending technique is described that blends values from positions in multiple microimages and assigns the blended value to a given point in the output image. A rendering technique that combines depth-based rendering and rendering with blending is also described. Depth-based rendering estimates depth at each microimage and then applies that depth to determine a position in the input flat from which to read a value to be assigned to a given point in the output image. The techniques may be implemented according to parallel processing technology that renders multiple points of the output image in parallel. In at least some embodiments, the parallel processing technology is graphical processing unit (GPU) technology.

Abstract: Methods and apparatus for capturing and rendering images with focused plenoptic cameras employing microlenses with different focal lengths. A focused plenoptic camera that includes an array of microlenses with at least two different focal lengths may be used to simultaneously capture microimages from at least two different planes at different distances from the microlens array. Image operations such as refocusing and focus bracketing may be performed on flats captured with the camera. Images may be constructed from subsets of the microimages captured using each type of microlens, thus creating multiple images each focused at a different depth. An array of stacked microlenses including stacks that provide different focal lengths may be used. The lens stacks may be provided by stacking two microlenses arrays on top of each other in the camera.

Abstract: A method and apparatus for the block-based compression of light-field images. Light-field images may be preprocessed by a preprocessing module into a format that is compatible with the blocking scheme of a block-based compression technique, for example JPEG. The compression technique is then used to compress the preprocessed light-field images. The light-field preprocessing module reshapes the angular data in a captured light-field image into shapes compatible with the blocking scheme of the compression technique so that blocking artifacts of block-based compression are not introduced in the final compressed image. Embodiments may produce compressed 2D images for which no specific light-field image viewer is needed to preview the full light-field image. Full light-field information is contained in one compressed 2D image.

Abstract: Method and apparatus for full-resolution light-field capture and rendering. A radiance camera is described in which the microlenses in a microlens array are focused on the image plane of the main lens instead of on the main lens, as in conventional plenoptic cameras. The microlens array may be located at distances greater than f from the photosensor, where f is the focal length of the microlenses. Radiance cameras in which the distance of the microlens array from the photosensor is adjustable, and in which other characteristics of the camera are adjustable, are described. Digital and film embodiments of the radiance camera are described. A full-resolution light-field rendering method may be applied to flats captured by a radiance camera to render higher-resolution output images than are possible with conventional plenoptic cameras and rendering methods.

Abstract: Method and apparatus for full-resolution light-field capture and rendering. A radiance camera is described in which the microlenses in a microlens array are focused on the image plane of the main lens instead of on the main lens, as in conventional plenoptic cameras. The microlens array may be located at distances greater than f from the photosensor, where f is the focal length of the microlenses. Radiance cameras in which the distance of the microlens array from the photosensor is adjustable, and in which other characteristics of the camera are adjustable, are described. Digital and film embodiments of the radiance camera are described. A full-resolution light-field rendering method may be applied to light-fields captured by a radiance camera to render higher-resolution output images than are possible with conventional plenoptic cameras and rendering methods.

Abstract: Methods and apparatus for high-speed digital imaging. Temporal modulation technology, for example digital micromirror device (DMD) technology, may be integrated with focused plenoptic camera technology to produce a compact and relatively inexpensive high-speed digital video camera capable of capturing video clips at frame rates up to and including 32,000 (32K) frames per second, or higher. In embodiments, the temporal modulation technology is used to temporally modulate image data captured in a single exposure; the temporally modulated image data is captured in a single raw image using focused plenoptic camera technology. A rendering method may be applied to the raw image data to extract multiple frames captured at the frame rate, the frames each showing an image of a scene at a different time in a sequence, the duration of which is the exposure time.

Abstract: Methods and apparatus for rich image capture using focused plenoptic camera technology. A radiance camera employs focused plenoptic camera technology and includes sets of modulating elements that may be used to modulate the sampling of different aspects of the range of plenoptic data. The radiance camera, via the modulating elements, may capture a particular property of light, such as luminance, color, polarization, etc., differently in different microimages or in different portions of microimages. With the focused plenoptic camera technology, the microimages are captured at the same time in a single image. Thus, multiple microimages of the same image of a scene may be captured at different exposures, different colors, different polarities, and so on, in a single image at the same time. Captured images may be used, for example, in High Dynamic Range (HDR) imaging, spectral imaging, polarization imaging, 3D imaging, and other imaging applications.

Abstract: Methods and apparatus for super-resolution in focused plenoptic cameras. By examining the geometry of data capture for super-resolution with the focused plenoptic camera, configurations for which super-resolution is realizable at different modes in the focused plenoptic camera are generated. A focused plenoptic camera is described in which infinity is super resolved directly, with registration provided by the camera geometry and the microlens pitch. In an algorithm that may be used to render super-resolved images from flats captured with a focused plenoptic camera, a high-resolution observed image is generated from a flat by interleaving pixels from adjacent microlens images. A deconvolution method may then be applied to the high-resolution observed image to deblur the image.

Abstract: Methods and apparatus for capturing and rendering images with focused plenoptic cameras employing different filtering at different microlenses. In a focused plenoptic camera, the main lens creates an image at the focal plane. That image is re-imaged on the sensor multiple times by an array of microlenses. Different filters that provide different levels and/or types of filtering may be combined with different ones of the microlenses. A flat captured with the camera includes multiple microimages captured according to the different filters. Multiple images may be assembled from the microimages, with each image assembled from microimages captured using a different filter. A final image may be generated by appropriately combining the images assembled from the microimages. Alternatively, a final image, or multiple images, may be assembled from the microimages by first combining the microimages and then assembling the combined microimages to produce one or more output images.

Abstract: Methods and apparatus for light-field imaging. Light-field camera designs are described that produce higher spatial resolution than conventional plenoptic camera designs, while trading-off the light-field's angular sampling density. This lower angular resolution may be compensated for by a light-field image processing method that inserts data synthesized by view interpolation of the measured light-field. In one embodiment, a light-field image processing method that performs three-view morphing may be used to interpolate the missing angular samples of radiance. The light-field camera designs may be implemented in hand-held light-field cameras that may capture a light-field with a single exposure. Some of the light-field camera designs are internal to the camera, while others are external to the camera. One light-field camera design includes a single, relatively large lens and an array of negative lenses that are placed in front of (external to) the main lens of a conventional camera.

Abstract: Method and apparatus for full-resolution light-field capture and rendering. A radiance camera is described in which the microlenses in a microlens array are focused on the image plane of the main lens instead of on the main lens, as in conventional plenoptic cameras. The microlens array may be located at distances greater than f from the photosensor, where f is the focal length of the microlenses. Radiance cameras in which the distance of the microlens array from the photosensor is adjustable, and in which other characteristics of the camera are adjustable, are described. Digital and film embodiments of the radiance camera are described. A full-resolution light-field rendering method may be applied to flats captured by a radiance camera to render higher-resolution output images than are possible with conventional plenoptic cameras and rendering methods.

Abstract: Methods and apparatus for light-field capture with large depth of field. A design methodology is described in which the relationships among various plenoptic camera parameters, including inverse magnification, F-number, focal length, wavelength, and pixel size, may be analyzed to design plenoptic cameras that provide increased depth-of-field when compared to conventional plenoptic cameras. Plenoptic cameras are described, which may be implemented according to the design methodology, and in which both Keplerian telescopic and Galilean telescopic imaging can be realized at the same time while providing a larger depth of field than is realized in conventional plenoptic cameras, thus capturing light-field images that capture “both sides” in which all but a small region of the scene is in focus. In some embodiments, apertures may be added to the microlenses so that depth of field is increased.

Abstract: Various methods and apparatus for removing artifacts in frequency domain processing of light-field images are described. Methods for the reduction or removal of the artifacts are described that include methods that may be applied during frequency domain processing and a method that may be applied during post-processing of resultant angular views. The methods may be implemented in software as or in a light-field frequency domain processing module. The described methods include an oversampling method to determine the correct centers of slices, a phase multiplication method to determine the correct centers of slices, a method to exclude low-energy slices, and a cosmetic correction method.

Abstract: One embodiment of the present invention provides a plenoptic camera which captures information about the direction distribution of light rays entering the camera. Like a conventional camera, this plenoptic camera includes a main lens which receives light from objects in an object field and directs the received light onto an image plane of the camera. It also includes a photodetector array located at the image plane of the camera, which captures the received light to produce an image. However, unlike a conventional camera, the plenoptic camera additionally includes an array of optical elements located between the object field and the main lens. Each optical element in this array receives light from the object field from a different angle than the other optical elements in the array, and consequently directs a different view of the object field into the main lens. In this way, the photodetector array receives a different view of the object field from each optical element in the array.

Abstract: Methods and apparatus for capturing and rendering images with focused plenoptic cameras employing different filtering at different microlenses. In a focused plenoptic camera, the main lens creates an image at the focal plane. That image is re-imaged on the sensor multiple times by an array of microlenses. Different filters that provide different levels and/or types of filtering may be combined with different ones of the microlenses. A flat captured with the camera includes multiple microimages captured according to the different filters. Multiple images may be assembled from the microimages, with each image assembled from microimages captured using a different filter. A final image may be generated by appropriately combining the images assembled from the microimages. Alternatively, a final image, or multiple images, may be assembled from the microimages by first combining the microimages and then assembling the combined microimages to produce one or more output images.

Abstract: A method and apparatus for the block-based compression of light-field images. Light-field images may be preprocessed by a preprocessing module into a format that is compatible with the blocking scheme of a block-based compression technique, for example JPEG. The compression technique is then used to compress the preprocessed light-field images. The light-field preprocessing module reshapes the angular data in a captured light-field image into shapes compatible with the blocking scheme of the compression technique so that blocking artifacts of block-based compression are not introduced in the final compressed image. Embodiments may produce compressed 2D images for which no specific light-field image viewer is needed to preview the full light-field image. Full light-field information is contained in one compressed 2D image.

Abstract: Method and apparatus for full-resolution light-field capture and rendering. A radiance camera is described in which the microlenses in a microlens array are focused on the image plane of the main lens instead of on the main lens, as in conventional plenoptic cameras. The microlens array may be located at distances greater than ƒ from the photosensor, where ƒ is the focal length of the microlenses. Radiance cameras in which the distance of the microlens array from the photosensor is adjustable, and in which other characteristics of the camera are adjustable, are described. Digital and film embodiments of the radiance camera are described. A full-resolution light-field rendering method may be applied to light-fields captured by a radiance camera to render higher-resolution output images than are possible with conventional plenoptic cameras and rendering methods.