Abstract: An input rescale module that performs cross-color correlated downscaling of sensor data in the horizontal and vertical dimensions. The module may perform a first-pass demosaic of sensor data, apply horizontal and vertical scalers to resample and downsize the data in the horizontal and vertical dimensions, and then remosaic the data to provide horizontally and vertically downscaled sensor data as output for additional image processing. The module may, for example, act as a front end scaler for an image signal processor (ISP). The demosaic performed by the module may be a relatively simple demosaic, for example a demosaic function that works on 3×3 blocks of pixels. The front end of module may receive and process sensor data at two pixels per clock (ppc); the horizontal filter component reduces the sensor data down to one ppc for downstream components of the input rescale module and for the ISP pipeline.

Abstract: An image processing pipeline may perform noise filtering and image sharpening utilizing common spatial support. A noise filter may perform a spatial noise filtering technique to determine a filtered value of a given pixel based on spatial support obtained from line buffers. Sharpening may also be performed to generate a sharpened value of the given pixel based on spatial support obtained from the same line buffers. A filtered and sharpened version of the pixel may be generated by combining the filtered value of the given pixel with the sharpened value of the given pixel. In at least some embodiments, the noise filter performs spatial noise filtering and image sharpening on a luminance value of the given pixel, when the given pixel is received in a luminance-chrominance encoding.

Abstract: An image processing pipeline may account for clipped pixels in auto focus statistics. Generating auto focus statistics may include evaluating a neighborhood of pixels with respect to a given pixel in a stream of pixels for an image. If a clipped pixel is identified within the neighborhood of pixels then the evaluation of the given pixel may be excluded from an auto focus statistic. The image processing pipeline may also provide auto focus statistics that do not exclude clipped pixels. A luminance edge detection value may, in some embodiments, be generated by applying an IIR filter to the given pixel in a stream of pixels to band-pass filter the given pixel before including the band-pass filtered pixel in the generation of the luminance edge detection value.

Abstract: An image signal processor may include a pixel defect correction component that tracks defect history for frames captured by an image sensor and applies the history when identifying and correcting defective pixels in a frame. The component maintains a defect pixel location table that includes a defect confidence value for pixels of the image sensor. The component identifies defective pixels in a frame, for example by comparing each pixel's value to the values of its neighbor pixels. If a pixel is detected as defective, its defect confidence value may be incremented. Otherwise, the value may be decremented. If a pixel's defect confidence value is over a defect confidence threshold, the pixel is considered defective and thus may be corrected. If a pixel's defect confidence value is under the threshold, the pixel is considered not defective and thus may not be corrected even if the pixel was detected as defective.

Abstract: Systems and methods for automatic lens flare compensation may include a non-uniformity detector configured to operate on pixel data for an image in an image sensor color pattern. The non-uniformity detector may detect a non-uniformity in the pixel data in a color channel of the image sensor color pattern. The non-uniformity detector may generate output including location and magnitude values of the non-uniformity. A lens flare detector may determine, based at least on the location and magnitude values, whether the output of the non-uniformity detector corresponds to a lens flare in the image. In some embodiments, the lens flare detector may generate, in response to determining that the output corresponds to the lens flare, a representative map of the lens flare. A lens flare corrector may determine one or more pixel data correction values corresponding to the lens flare and apply the pixel data correction values to the pixel data.

Abstract: Methods and systems for detecting keypoints in image data may include an image sensor interface receiving pixel data from an image sensor. A front-end pixel data processing circuit may receive pixel data and convert the pixel data to a different color space format. A back-end pixel data processing circuit may perform one or more operations on the pixel data. An output circuit may receive pixel data and output the pixel data to a system memory. A keypoint detection circuit may receive pixel data from the image sensor interface in the image sensor pixel data format or receive pixel data after processing by the front-end or the back-end pixel data processing circuits. The keypoint detection circuit may perform a keypoint detection operation on the pixel data to detect one or more keypoints in the image frame and output to the system memory a description of the one or more keypoints.

Abstract: An image processing pipeline may account for clipped pixels in auto focus statistics. Generating auto focus statistics may include evaluating a neighborhood of pixels with respect to a given pixel in a stream of pixels for an image. If a clipped pixel is identified within the neighborhood of pixels then the evaluation of the given pixel may be excluded from an auto focus statistic. The image processing pipeline may also provide auto focus statistics that do not exclude clipped pixels. A luminance edge detection value may, in some embodiments, be generated by applying an IIR filter to the given pixel in a stream of pixels to band-pass filter the given pixel before including the band-pass filtered pixel in the generation of the luminance edge detection value.

Abstract: An image processing pipeline may perform noise filtering and image sharpening utilizing common spatial support. A noise filter may perform a spatial noise filtering technique to determine a filtered value of a given pixel based on spatial support obtained from line buffers. Sharpening may also be performed to generate a sharpened value of the given pixel based on spatial support obtained from the same line buffers. A filtered and sharpened version of the pixel may be generated by combining the filtered value of the given pixel with the sharpened value of the given pixel. In at least some embodiments, the noise filter performs spatial noise filtering and image sharpening on a luminance value of the given pixel, when the given pixel is received in a luminance-chrominance encoding.

Abstract: An image processing pipeline may process image data at multiple rates. A stream of raw pixel data collected from an image sensor for an image frame may be processed through one or more pipeline stages of an image signal processor. The stream of raw pixel data may then be converted into a full-color domain and scaled to a data size that is less than an initial data size for the image frame. The converted pixel data may be processed through one or more other pipelines stages and output for storage, further processing, or display. In some embodiments, a back-end interface may be implemented as part of the image signal processor via which image data collected from sources other than the image sensor may be received and processed through various pipeline stages at the image signal processor.

Abstract: An image signal processor of a device, apparatus, or computing system that includes a camera capable of capturing image data may apply piecewise perspective transformations to image data received from the camera's image sensor. A scaling unit of an Image Signal Processor (ISP) may perform piecewise perspective transformations on a captured image to correct for rolling shutter artifacts and to provide video image stabilization. Image data may be divided into a series of horizontal slices and perspective transformations may be applied to each slice. The transformations may be based on motion data determined in any of various manners, such as by using gyroscopic data and/or optical-flow calculations. The piecewise perspective transforms may be encoded as Digital Difference Analyzer (DDA) steppers and may be implemented using separable scalar operations. The image signal processor may not write the received image data to system memory until after the transformations have been performed.

Abstract: An input rescale module that performs cross-color correlated downscaling of sensor data in the horizontal and vertical dimensions. The module may perform a first-pass demosaic of sensor data, apply horizontal and vertical scalers to resample and downsize the data in the horizontal and vertical dimensions, and then remosaic the data to provide horizontally and vertically downscaled sensor data as output for additional image processing. The module may, for example, act as a front end scaler for an image signal processor (ISP). The demosaic performed by the module may be a relatively simple demosaic, for example a demosaic function that works on 3×3 blocks of pixels. The front end of module may receive and process sensor data at two pixels per clock (ppc); the horizontal filter component reduces the sensor data down to one ppc for downstream components of the input rescale module and for the ISP pipeline.

Abstract: A pedestal level for an image sensor can be dynamically adjusted based on one or more parameters. The parameters include one or more operating conditions associated with the image sensor, pre-determined image sensor characterization data, the number of unused digital codes, and/or the number of clipped pixel signals. The operating conditions can include the temperature of the image sensor, the gain of at least one amplifier included in processing circuitry operably connected to at least one pixel, and/or the length of the integration period for at least one pixel in the image sensor. Based on the one or more of the parameters, the pedestal level is adjusted to reduce a number of unused digital codes in a distribution of dark current. Additionally or alternatively, the variance of the pixel signals can be reduced to permit the use of a lower pedestal level.

Abstract: An image signal processor of a device, apparatus, or computing system that includes a camera capable of capturing image data may apply piecewise perspective transformations to image data received from the camera's image sensor. A scaling unit of an Image Signal Processor (ISP) may perform piecewise perspective transformations on a captured image to correct for rolling shutter artifacts and to provide video image stabilization. Image data may be divided into a series of horizontal slices and perspective transformations may be applied to each slice. The transformations may be based on motion data determined in any of various manners, such as by using gyroscopic data and/or optical-flow calculations. The piecewise perspective transforms may be encoded as Digital Difference Analyzer (DDA) steppers and may be implemented using separable scalar operations. The image signal processor may not write the received image data to system memory until after the transformations have been performed.

Abstract: An electronic device may have a camera module. The camera module may capture images having an initial depth of field. The electronic device may receive user input selecting a focal plane and an effective f-stop for use in producing a modified image with a reduced depth of field. The electronic device may include image processing circuitry that selectively blurs various regions of a captured image, with each region being blurred to an amount that varies with distance to the user selected focal plane and in response to the user selected effective f-stop (e.g., a user selected level of depth of field).

Abstract: An input rescale module that performs cross-color correlated downscaling of sensor data in the horizontal and vertical dimensions. The module may perform a first-pass demosaic of sensor data, apply horizontal and vertical scalers to resample and downsize the data in the horizontal and vertical dimensions, and then remosaic the data to provide horizontally and vertically downscaled sensor data as output for additional image processing. The module may, for example, act as a front end scaler for an image signal processor (ISP). The demosaic performed by the module may be a relatively simple demosaic, for example a demosaic function that works on 3×3 blocks of pixels. The front end of module may receive and process sensor data at two pixels per clock (ppc); the horizontal filter component reduces the sensor data down to one ppc for downstream components of the input rescale module and for the ISP pipeline.

Abstract: A system of stereo imagers, including image processing units and methods of blurring an image, is presented. The image is received from an image sensor. For each pixel of the image, a depth filter component is determined based on a focal area of the image and a depth map associated with the image. For each pixel of the image, a trilateral filter is generated that includes a spatial filter component, a range filter component and the depth filter component. The respective trilateral filter is applied to corresponding pixels of the image to blur the image outside of the focal area. A refocus area or position may be determined by imaging geometry or may be selected manually via a user interface.

Abstract: Imaging systems with image sensors and image processing circuitry are provided. The image processing circuitry may identify motion and perform autofocus (e.g., continuous autofocus) using images captured by an image sensor. Auto exposure metrics such as average luminance values and autofocus statistics such as sharpness scores may be calculated for each image. The auto exposure metrics may be used to calculate motion scores and identify directional motion between a series of captured images. The motion scores may be used with the sharpness scores to determine when to perform autofocus functions such as when to refocus a lens for a continuous autofocus application. For example, the motion scores may be monitored to identify motion that exceeds a given magnitude and duration. After identification of motion, motion scores and sharpness scores may be used to determine when a given scene has stabilized and when the lens should be refocused.

Abstract: A method of scanning a scene using an image sensor includes (a) dividing the scene into multiple first portions; and scanning a first portion for presence of objects in an object class. The method further includes continuing the scanning of the multiple first portions for presence of other objects in the scene. The method also selects a second portion of the scene, in response to detecting an object in the first portion; and then tracking the object in the selected second portion. The second portion of the scene is selected based on estimating motion of the object detected in the first portion, so that it may still be located in the second portion.

Abstract: Systems and methods are provided for focus recovery of multi-channel images. Control circuitry of an imaging system can restore an image by removing image blurring introduced by the lens, sensor noise introduced by the sensor, and a signal offset between multiple channels of the image. In some embodiments, the control circuitry can calculate one or more estimates of a signal offset of multiple observed signals. Using statistics generated from offset-removed signals, the control circuitry can generate one or more recovery kernels which can be applied to offset-removed signals to generate recovered signals. In other embodiments, instead of explicitly removing a signal offset from each observed signal, the control circuitry can implicitly remove the signal offset when calculating the first and second order statistics of one or more observed signals.

Abstract: A system of stereo imagers, including image processing units and methods of blurring an image, is presented. The image is received from an image sensor. For each pixel of the image, a depth filter component is determined based on a focal area of the image and a depth map associated with the image. For each pixel of the image, a trilateral filter is generated that includes a spatial filter component, a range filter component and the depth filter component. The respective trilateral filter is applied to corresponding pixels of the image to blur the image outside of the focal area. A refocus area or position may be determined by imaging geometry or may be selected manually via a user interface.