Abstract: Techniques to capture and fuse short- and long-exposure images of a scene from a stabilized image capture device are disclosed. More particularly, the disclosed techniques use not only individual pixel differences between co-captured short- and long-exposure images, but also the spatial structure of occluded regions in the long-exposure images (e.g., areas of the long-exposure image(s) exhibiting blur due to scene object motion). A novel device used to represent this feature of the long-exposure image is a “spatial difference map.” Spatial difference maps may be used to identify pixels in the short- and long-exposure images for fusion and, in one embodiment, may be used to identify pixels from the short-exposure image(s) to filter post-fusion so as to reduce visual discontinuities in the output image.

Abstract: Systems, methods, and computer readable media to capture and process high dynamic range (HDR) images when appropriate for a scene are disclosed. When appropriate, multiple images at a single—slightly underexposed—exposure value are captured (making a constant bracket HDR capture sequence) and local tone mapping (LTM) applied to each image. Local tone map and histogram information can be used to generate a noise-amplification mask which can be used during fusion operations. Images obtained and fused in the disclosed manner provide high dynamic range with improved noise and de-ghosting characteristics.

Abstract: Systems, methods, and computer readable media to improve image stabilization operations are described. Novel approaches for fusing non-reference images with a pre-selected reference frame in a set of commonly captured images are disclosed. The fusing approach may use a soft transition by using a weighted average for ghost/non-ghost pixels to avoid sudden transition between neighborhood and almost similar pixels. Additionally, the ghost/non-ghost decision can be made based on a set of neighboring pixels rather than independently for each pixel. An alternative approach may involve performing a multi-resolution decomposition of all the captured images, using temporal fusion, spatio-temporal fusion, or combinations thereof, at each level and combining the different levels to generate an output image.

Abstract: Lens flare mitigation techniques determine which pixels in images of a sequence of images are likely to be pixels affected by lens flare. Once the lens flare areas of the images are determined, unwanted lens flare effects may be mitigated by various approaches, including reducing border artifacts along a seam between successive images, discarding entire images of the sequence that contain lens flare areas, and using tone-mapping to reduce the visibility of lens flare.

Abstract: Techniques to capture and fuse short- and long-exposure images of a scene from a stabilized image capture device are disclosed. More particularly, the disclosed techniques use not only individual pixel differences between co-captured short- and long-exposure images, but also the spatial structure of occluded regions in the long-exposure images (e.g., areas of the long-exposure image(s) exhibiting blur due to scene object motion). A novel device used to represent this feature of the long-exposure image is a “spatial difference map.” Spatial difference maps may be used to identify pixels in the short- and long-exposure images for fusion and, in one embodiment, may be used to identify pixels from the short-exposure image(s) to filter post-fusion so as to reduce visual discontinuities in the output image.

Abstract: Techniques to capture and fuse short- and long-exposure images of a scene from a stabilized image capture device are disclosed. More particularly, the disclosed techniques use not only individual pixel differences between co-captured short- and long-exposure images, but also the spatial structure of occluded regions in the long-exposure images (e.g., areas of the long-exposure image(s) exhibiting blur due to scene object motion). A novel device used to represent this feature of the long-exposure image is a “spatial difference map.” Spatial difference maps may be used to identify pixels in the short- and long-exposure images for fusion and, in one embodiment, may be used to identify pixels from the short-exposure image(s) to filter post-fusion so as to reduce visual discontinuities in the output image.

Abstract: Systems, methods, and computer readable media to capture and process high dynamic range (HDR) images when appropriate for a scene are disclosed. When appropriate, multiple images at a single—slightly underexposed—exposure value are captured (making a constant bracket HDR capture sequence) and local tone mapping (LTM) applied to each image. Local tone map and histogram information can be used to generate a noise-amplification mask which can be used during fusion operations. Images obtained and fused in the disclosed manner provide high dynamic range with improved noise and de-ghosting characteristics.

Abstract: Special blend operations (referred to as “image seam-matching”) are described that keep the pixel values in the two images being blended the same along their transition border or seam, and smoothly increases/decreases pixel values on either side of the seam through the images' transition band (an area around the seam in which image blend operations are constrained). Image seam-matching provides many of the benefits of gradient blending (e.g., the avoidance of ghosting), without the associated computational overhead. This makes image seam-matching a particularly useful approach for real-time image processing such as during the real-time generation of wide area-of-view images. In situations where image seam-matching may be inappropriate, such as when the images being blended include long objects that span an entire overlap region(s), a mechanism is described which allows the selection of either, or both, seam-matching and cross-fading blend operations in a graceful or smooth manner.

Abstract: Techniques for registering images based on an identified region of interest (ROI) are described. In general, the disclosed techniques identify a region of ROI within an image and assign areas within the image corresponding to those regions more importance during the registration process. More particularly, the disclosed techniques may employ user-input or image content information to identify the ROI. Once identified, features within the ROI may be given more weight or significance during registration operations than other areas of the image having high-feature content but which are not as important to the individual capturing the image.

Abstract: Motion sensor data may be used to register a sequence of standard dynamic range images for producing a high dynamic range (HDR) image, reducing use of computational resources over software visual feature mapping techniques. A rotational motion sensor may produce information about orientation changes in the imaging device between images in the sequence of images sufficient to allow registration of the images, instead of using registration based on analysis of visual features of the images. If the imaging device has been moved laterally, then the motion sensor data may not be useful and visual feature mapping techniques may be employed to produce the HDR image.

Abstract: Systems, methods, and computer readable media to improve image stabilization operations are described. A novel combination of image quality and commonality metrics are used to identify a reference frame from a set of commonly captured images which, when the set's other images are combined with it, results in a quality stabilized image. The disclosed image quality and commonality metrics may also be used to optimize the use of a limited amount of image buffer memory during image capture sequences that return more images that the memory may accommodate at one time. Image quality and commonality metrics may also be used to effect the combination of multiple relatively long-exposure images which, when combined with a one or more final (relatively) short-exposure images, yields images exhibiting motion-induced blurring in interesting and visually pleasing ways.

Abstract: Special blend operations for wide area-of-view image generation utilizing a “floating auto exposure” scheme are described. Pixel values in the two images being stitched together are blended within a transition band around a “seam.” identified in the overlap region between the images after changes in exposure and/or color saturation are accounted for. In some embodiments, changes in exposure and/or color saturation are accounted for through the use of one or more exposure mapping curves, the selection and use of which are based, at least in part, on a determined “Exposure Ratio” value, i.e., the amount that the camera's exposure settings have deviated from their initial capture settings. In other embodiments, the Exposure Ratio value is also used to determine regions along the seam where either: alpha blending, Poisson blending—or a combination of the two—should be used to blend in the transitional areas on each side of the seam.

Abstract: Systems, methods, and computer readable media to improve image stabilization operations are described. A novel approach to pixel-based registration of non-reference images to a reference frame in a set of commonly captured images is disclosed which makes use of pyramid decomposition to more efficiently detect corners. The disclosed pixel-based registration operation may also be combined with motion sensor data-based registration approaches to register non-reference images with respect to the reference frame. When the registered non-reference images are combined with the pre-selected reference image, the resulting image is a quality stabilized image.

Abstract: Systems, methods, and computer readable media to improve image stabilization operations are described. Novel approaches for fusing non-reference images with a pre-selected reference frame in a set of commonly captured images are disclosed. The fusing approach may use a soft transition by using a weighted average for ghost/non-ghost pixels to avoid sudden transition between neighborhood and almost similar pixels. Additionally, the ghost/non-ghost decision can be made based on a set of neighboring pixels rather than independently for each pixel. An alternative approach may involve performing a multi-resolution decomposition of all the captured images, using temporal fusion, spatio-temporal fusion, or combinations thereof, at each level and combining the different levels to generate an output image.

Abstract: Procedures are described for blending images in real-time that avoid ghosting artifacts (attributable to moving objects), maintain the proper appearance of contiguous edges in the final image, and permits the use of fast (real-time) blending operations. A “guard-band” may be defined around an initially identified seam that perturbs the path of the initial seam so that both the seam and the guard-band's edges avoid moving objects by at least a specified amount. Rapid blend operations may then be performed in the region demarcated by the guard-band. The seam may be further adjusted to bias its position toward a specified trajectory within the overlap region when there is no moving object present. If visual registration techniques are not able to provide a properly aligned overlap region, motion sensor data for the image capture device, may be used instead to facilitate blending operations.

Abstract: Techniques are disclosed for selectively capturing, retaining, and combining multiple sub-exposure images or brackets to yield a final image having diminished motion-induced blur and good noise characteristics. More specifically, after or during the capture of N brackets, the M best may be identified for combining into a single output image, (N>M). As used here, the term “best” means those brackets that exhibit the least amount of relative motion with respect to one another—with one caveat: integer pixel shifts may be preferred over sub-pixel shifts.

Abstract: Techniques for registering images based on an identified region of interest (ROI) are described. In general, the disclosed techniques identify a region of ROI within an image and assign areas within the image corresponding to those regions more importance during the registration process. More particularly, the disclosed techniques may employ user-input or image content information to identify the ROI. Once identified, features within the ROI may be given more weight or significance during registration operations than other areas of the image having high-feature content but which are not as important to the individual capturing the image.

Abstract: Motion sensor data may be used to register a sequence of standard dynamic range images for producing a high dynamic range (HDR) image, reducing use of computational resources over software visual feature mapping techniques. A rotational motion sensor may produce information about orientation changes in the imaging device between images in the sequence of images sufficient to allow registration of the images, instead of using registration based on analysis of visual features of the images. If the imaging device has been moved laterally, then the motion sensor data may not be useful and visual feature mapping techniques may be employed to produce the HDR image.

Abstract: Special blend operations (referred to as “image seam-matching”) are described that keep the pixel values in the two images being blended the same along their transition border or seam, and smoothly increases/decreases pixel values on either side of the seam through the images' transition band (an area around the seam in which image blend operations are constrained). Image seam-matching provides many of the benefits of gradient blending (e.g., the avoidance of ghosting), without the associated computational overhead. This makes image seam-matching a particularly useful approach for real-time image processing such as during the real-time generation of wide area-of-view images.

Abstract: Lens flare mitigation techniques determine which pixels in images of a sequence of images are likely to be pixels affected by lens flare. Once the lens flare areas of the images are determined, unwanted lens flare effects may be mitigated by various approaches, including reducing border artifacts along a seam between successive images, discarding entire images of the sequence that contain lens flare areas, and using tone-mapping to reduce the visibility of lens flare.