Abstract: Apparatuses, methods and systems for processing, rendering and displaying plenoptic images are disclosed. One exemplary embodiment is a method comprising storing a plenoptic image in a non-transitory computer readable memory associated with a processor, receiving at the processor a viewing position information, processing the plenoptic image and the viewing position information to render a visual output based upon information of the plenoptic image and the viewing position information, performing a blending or smoothing function on information of the plenoptic image including weighting each of a plurality of pixels of the plenoptic image based upon a first color value associated with each pixel and a set of second color values associated with a plurality of neighboring pixels, and displaying the visual output on a display device. The visual output displayed on the display device varies as a function of the viewing position information.

Abstract: One embodiment is a method of operating a plenoptic image processing system. The method includes storing plenoptic image data in a non-transitory computer readable memory of the system. The plenoptic image data includes data of a plurality of microimages. The method includes applying a frequency domain transform to the stored plenoptic image data to provide a frequency domain plenoptic image data structure stored in the memory, and processing the data structure using an extended slice technique to select a focal plane from the data structure. The extended slice technique comprises taking multiple slices of the frequency domain transformed plenoptic image data structure and configuring the multiple slices to provide extended slice data. The method further includes applying an inverse frequency domain transform to the extended slice data to provide inverse transform extended slice data, and providing an image output based upon the inverse transformed extended slice data.

Abstract: Methods and apparatus for reducing plenoptic camera artifacts. A first method is based on careful design of the optical system of the focused plenoptic camera to reduce artifacts that result in differences in depth in the microimages. A second method is computational; a focused plenoptic camera rendering algorithm is provided that corrects for artifacts resulting from differences in depth in the microimages. While both the artifact-reducing focused plenoptic camera design and the artifact-reducing rendering algorithm work by themselves to reduce artifacts, the two approaches may be combined.

Abstract: One embodiment is a method of operating a plenoptic image processing system. The method includes storing plenoptic image data in a non-transitory computer readable memory of the system. The plenoptic image data includes data of a plurality of microimages. The method includes applying a frequency domain transform to the stored plenoptic image data to provide a frequency domain plenoptic image data structure stored in the memory, and processing the data structure using an extended slice technique to select a focal plane from the data structure. The extended slice technique comprises taking multiple slices of the frequency domain transformed plenoptic image data structure and configuring the multiple slices to provide extended slice data. The method further includes applying an inverse frequency domain transform to the extended slice data to provide inverse transform extended slice data, and providing an image output based upon the inverse transformed extended slice data.

Abstract: Apparatuses, methods and systems for processing, rendering and displaying plenoptic images are disclosed. One exemplary embodiment is a method comprising storing a plenoptic image in a non-transitory computer readable memory associated with a processor, receiving at the processor a viewing position information, processing the plenoptic image and the viewing position information to render a visual output based upon information of the plenoptic image and the viewing position information, performing a blending or smoothing function on information of the plenoptic image including weighting each of a plurality of pixels of the plenoptic image based upon a first color value associated with each pixel and a set of second color values associated with a plurality of neighboring pixels, and displaying the visual output on a display device. The visual output displayed on the display device varies as a function of the viewing position information.

Abstract: Methods, apparatus, and computer-readable storage media for rendering focused plenoptic camera data. A rendering with blending technique is described that blends values from positions in multiple microimages and assigns the blended value to a given point in the output image. A rendering technique that combines depth-based rendering and rendering with blending is also described. Depth-based rendering estimates depth at each microimage and then applies that depth to determine a position in the input flat from which to read a value to be assigned to a given point in the output image. The techniques may be implemented according to parallel processing technology that renders multiple points of the output image in parallel. In at least some embodiments, the parallel processing technology is graphical processing unit (GPU) technology.

Abstract: Methods, apparatus, and computer-readable storage media for rendering focused plenoptic camera data. A depth-based rendering technique is described that estimates depth at each microimage and then applies that depth to determine a position in the input flat from which to read a value to be assigned to a given point in the output image. The techniques may be implemented according to parallel processing technology that renders multiple points of the output image in parallel. In at least some embodiments, the parallel processing technology is graphical processing unit (GPU) technology.

Abstract: The present invention involves a sparse matrix processing system and method which uses sparse matrices that are compressed to reduce memory traffic and improve performance of computations using sparse matrices.

Abstract: Methods and apparatus for super-resolution in integral photography are described. Several techniques are described that, alone or in combination, may improve the super-resolution process and/or the quality of super-resolved images that may be generated from flats captured with a focused plenoptic camera using a super-resolution algorithm. At least some of these techniques involve modifications to the focused plenoptic camera design. In addition, at least some of these techniques involve modifications to the super-resolution rendering algorithm. The techniques may include techniques for reducing the size of pixels, techniques for shifting pixels relative to each other so that super-resolution is achievable at more or all depths of focus, and techniques for sampling using an appropriate filter or kernel. These techniques may, for example, reduce or eliminate the need to perform deconvolution on a super-resolved image, and may improve super-resolution results and/or increase performance.

Abstract: Methods and apparatus for capturing and rendering images with focused plenoptic cameras employing different filtering at different microlenses. In a focused plenoptic camera, the main lens creates an image at the focal plane. That image is re-imaged on the sensor multiple times by an array of microlenses. Different filters that provide different levels and/or types of filtering may be combined with different ones of the microlenses. A flat captured with the camera includes multiple microimages captured according to the different filters. Multiple images may be assembled from the microimages, with each image assembled from microimages captured using a different filter. A final image may be generated by appropriately combining the images assembled from the microimages. Alternatively, a final image, or multiple images, may be assembled from the microimages by first combining the microimages and then assembling the combined microimages to produce one or more output images.

Abstract: Methods and apparatus for reducing plenoptic camera artifacts. A first method is based on careful design of the optical system of the focused plenoptic camera to reduce artifacts that result in differences in depth in the microimages. A second method is computational; a focused plenoptic camera rendering algorithm is provided that corrects for artifacts resulting from differences in depth in the microimages. While both the artifact-reducing focused plenoptic camera design and the artifact-reducing rendering algorithm work by themselves to reduce artifacts, the two approaches may be combined.

Abstract: Methods and apparatus for super-resolution in integral photography are described. Several techniques are described that, alone or in combination, may improve the super-resolution process and/or the quality of super-resolved images that may be generated from flats captured with a focused plenoptic camera using a super-resolution algorithm. At least some of these techniques involve modifications to the focused plenoptic camera design. In addition, at least some of these techniques involve modifications to the super-resolution rendering algorithm. The techniques may include techniques for reducing the size of pixels, techniques for shifting pixels relative to each other so that super-resolution is achievable at more or all depths of focus, and techniques for sampling using an appropriate filter or kernel. These techniques may, for example, reduce or eliminate the need to perform deconvolution on a super-resolved image, and may improve super-resolution results and/or increase performance.

Abstract: Methods, apparatus, and computer-readable storage media for rendering focused plenoptic camera data. A rendering with blending technique is described that blends values from positions in multiple microimages and assigns the blended value to a given point in the output image. A rendering technique that combines depth-based rendering and rendering with blending is also described. Depth-based rendering estimates depth at each microimage and then applies that depth to determine a position in the input flat from which to read a value to be assigned to a given point in the output image. The techniques may be implemented according to parallel processing technology that renders multiple points of the output image in parallel. In at least some embodiments, the parallel processing technology is graphical processing unit (GPU) technology.

Abstract: Methods, apparatus, and computer-readable storage media for rendering focused plenoptic camera data. A depth-based rendering technique is described that estimates depth at each microimage and then applies that depth to determine a position in the input flat from which to read a value to be assigned to a given point in the output image. The techniques may be implemented according to parallel processing technology that renders multiple points of the output image in parallel. In at least some embodiments, the parallel processing technology is graphical processing unit (GPU) technology.

Abstract: Methods and apparatus for capturing and rendering images with focused plenoptic cameras employing microlenses with different focal lengths. A focused plenoptic camera that includes an array of microlenses with at least two different focal lengths may be used to simultaneously capture microimages from at least two different planes at different distances from the microlens array. Image operations such as refocusing and focus bracketing may be performed on flats captured with the camera. Images may be constructed from subsets of the microimages captured using each type of microlens, thus creating multiple images each focused at a different depth. An array of stacked microlenses including stacks that provide different focal lengths may be used. The lens stacks may be provided by stacking two microlenses arrays on top of each other in the camera.

Abstract: Method and apparatus for full-resolution light-field capture and rendering. A radiance camera is described in which the microlenses in a microlens array are focused on the image plane of the main lens instead of on the main lens, as in conventional plenoptic cameras. The microlens array may be located at distances greater than f from the photosensor, where f is the focal length of the microlenses. Radiance cameras in which the distance of the microlens array from the photosensor is adjustable, and in which other characteristics of the camera are adjustable, are described. Digital and film embodiments of the radiance camera are described. A full-resolution light-field rendering method may be applied to flats captured by a radiance camera to render higher-resolution output images than are possible with conventional plenoptic cameras and rendering methods.

Abstract: Method and apparatus for full-resolution light-field capture and rendering. A radiance camera is described in which the microlenses in a microlens array are focused on the image plane of the main lens instead of on the main lens, as in conventional plenoptic cameras. The microlens array may be located at distances greater than f from the photosensor, where f is the focal length of the microlenses. Radiance cameras in which the distance of the microlens array from the photosensor is adjustable, and in which other characteristics of the camera are adjustable, are described. Digital and film embodiments of the radiance camera are described. A full-resolution light-field rendering method may be applied to light-fields captured by a radiance camera to render higher-resolution output images than are possible with conventional plenoptic cameras and rendering methods.

Abstract: Methods and apparatus for rich image capture using focused plenoptic camera technology. A radiance camera employs focused plenoptic camera technology and includes sets of modulating elements that may be used to modulate the sampling of different aspects of the range of plenoptic data. The radiance camera, via the modulating elements, may capture a particular property of light, such as luminance, color, polarization, etc., differently in different microimages or in different portions of microimages. With the focused plenoptic camera technology, the microimages are captured at the same time in a single image. Thus, multiple microimages of the same image of a scene may be captured at different exposures, different colors, different polarities, and so on, in a single image at the same time. Captured images may be used, for example, in High Dynamic Range (HDR) imaging, spectral imaging, polarization imaging, 3D imaging, and other imaging applications.

Abstract: Methods and apparatus for super-resolution in focused plenoptic cameras. By examining the geometry of data capture for super-resolution with the focused plenoptic camera, configurations for which super-resolution is realizable at different modes in the focused plenoptic camera are generated. A focused plenoptic camera is described in which infinity is super resolved directly, with registration provided by the camera geometry and the microlens pitch. In an algorithm that may be used to render super-resolved images from flats captured with a focused plenoptic camera, a high-resolution observed image is generated from a flat by interleaving pixels from adjacent microlens images. A deconvolution method may then be applied to the high-resolution observed image to deblur the image.

Abstract: Methods and apparatus for capturing and rendering images with focused plenoptic cameras employing different filtering at different microlenses. In a focused plenoptic camera, the main lens creates an image at the focal plane. That image is re-imaged on the sensor multiple times by an array of microlenses. Different filters that provide different levels and/or types of filtering may be combined with different ones of the microlenses. A flat captured with the camera includes multiple microimages captured according to the different filters. Multiple images may be assembled from the microimages, with each image assembled from microimages captured using a different filter. A final image may be generated by appropriately combining the images assembled from the microimages. Alternatively, a final image, or multiple images, may be assembled from the microimages by first combining the microimages and then assembling the combined microimages to produce one or more output images.