Abstract: The present disclosure provides techniques relates to the implementation of a raw pixel processing unit using a set of line buffers. In one embodiment, the set of line buffers may include a first subset and second subset. Various logical units of the raw pixel processing unit may be implemented using the first and second subsets of line buffers in a shared manner. For instance, in one embodiment, defective pixel correction and detection logic may be implemented using the first subset of line buffers. The second subset of line buffers may be used to implement lens shading correction logic, gain, offset, and clamping logic, and demosaicing logic. Further, noise reduction may also be implemented using at least a portion of each of the first and second subsets of line buffers.

Abstract: Certain embodiments of the present disclosure provide a flexible memory input/output controller that is configured to the storing and reading of multiple types of pixels and pixel memory formats. For instance, the memory I/O controller may support the storing and reading of raw image pixels at various bits of precision, such as 8-bit, 10-bit, 12-bit, 14-bit, and 16-bit. Pixel formats that are unaligned with memory bytes (e.g., not being a multiple of 8-bits) may be stored in a packed manner. The memory I/O controller may also support various formats of RGB pixel sets and YCC pixel sets.

Abstract: Various techniques are provided for processing image data acquired using a digital image sensor. In accordance with aspects of the present disclosure, one such technique may relate to the processing of image data in a system that supports multiple image sensors. In one embodiment, the image processing system may include control circuitry configured to determine whether a device is operating in a single sensor mode (one active sensor) or a dual sensor mode (two active sensors). When operating in the single sensor mode, data may be provided directly to a front-end pixel processing unit from the sensor interface of the active sensor. When operating in a dual sensor mode, the image frames from the first and second sensors are provided to the front-end pixel processing unit in an interleaved manner. For instance, in one embodiment, the image frames from the first and second sensors are written to a memory, and then read out to the front-end pixel processing unit in an interleaved manner.

Abstract: Certain aspects of this disclosure relate to an image signal processing system that includes a flash controller that is configured to activate a flash device prior to the start of a target image frame by using a sensor timing signal. In one embodiment, the flash controller receives a delayed sensor timing signal and determines a flash activation start time by using the delayed sensor timing signal to identify a time corresponding to the end of the previous frame, increasing that time by a vertical blanking time, and then subtracting a first offset to compensate for delay between the sensor timing signal and the delayed sensor timing signal. Then, the flash controller subtracts a second offset to determine the flash activation time, thus ensuring that the flash is activated prior to receiving the first pixel of the target frame.

Abstract: Various techniques for lens shading correction are provided. In one embodiment, the location of a current pixel is determined relative to a gain grid having a plurality of grid points distributed in horizontal and vertical directions. If the location of the current pixel corresponds to a grid point, a lens shading gain associated with that grid point is applied to the current pixel. If the location of the current pixel is between four grid points, bi-linear interpolation is applied to the four grid points to determine an interpolated lens shading gain. In another embodiment, a radial gain grid may be provided, and lens shading gains may be interpolated based upon grid points neighboring a current pixel in the radial and angular directions. In a further embodiment, a radial lens shading gain is determined by determining a radial distance from the center of the image to the current pixel and multiplying the radial distance by a global gain parameter based upon the color of the current pixel.

Abstract: In an embodiment, a compression unit is provided which may perform compression of images with low latency and relatively little hardware. Similarly, a decompression unit may be provided which may decompress the images with low latency and hardware. In an embodiment, the transmission of compressed coefficients may be performed using less than two passes through the list of coefficients. During the first pass, the most significant coefficients may be transmitted and other significance groups may be identified as linked lists. The linked lists may then be traverse to send the other significance groups. In an embodiment, a color space conversion may be made to permit filtering of fewer color components than might be possible in the source color space.

Abstract: Various techniques are provided herein for processing raw image data acquired using a digital image sensor in an image processing pipeline of an image signal processing system. In one embodiment, the image processing pipeline may first process the raw image data (e.g., Bayer image data) for the detection and correction of defective pixels. Next, the image processing pipeline may process the raw image data to reduce noise. Thereafter, the image processing pipeline may correct lens shading distortion in the raw image data and, subsequently, apply a demosaicing algorithm to convert the raw image data into full color image data (e.g., RGB image data). The color image data may be further processed by the image processing pipeline to correct color and gamma properties prior to being converted into a luma and chroma color space (e.g., YCbCr color space).

Abstract: Various techniques are provided herein for processing raw image data in front-end processing logic of an image signal processing system. In one embodiment, the front-end processing logic includes a statistics processing unit configured to process raw image data acquired by an image sensor to obtain one or more sets of statistics. The statistics processing unit may first correct defective pixels in the raw image data and then correct lens shading errors in the raw image data prior to extracting the statistics information. In certain embodiments, black level compensation may be applied between the defective pixel correction and lens shading correction steps, and inverse black level compensation may be applied between the lens shading correction step and the extraction of the statistics information. The acquired statistics information may be utilized by an image signal processing pipeline for converting the raw image data into a color (e.g., RGB) and/or luma (e.g., YCbCr) image.

Abstract: Various techniques relating to image sharpening are provided. In one embodiment, a luminance image is obtained based upon image data acquired by an image sensor. A multi-scale unsharp mask, which may include at least two Gaussian filters of difference radii, is applied to the luminance image to determine a plurality of unsharp values. Each of the unsharp values may be compared to a corresponding threshold and, for the unsharp values that exceed their respective thresholds, the unsharp value is multiplied by a corresponding gain and added to a base image, which may be selected as the luminance image or the output of one of the Gaussian filters. Each gained unsharp value may be summed with the base image to produce a final sharpened output. In some embodiments, an attenuated gain may be applied to unsharp values that do not exceed their respective thresholds.

Abstract: Various techniques are provided for the detection and correction of defective pixels in an image sensor. In accordance with one embodiment, a static defect table storing the locations of known static defects is provided, and the location of a current pixel is compared to the static defect table. If the location of the current pixel is found in the static defect table, the current pixel is identified as a static defect and is corrected using the value of the previous pixel of the same color. If the current pixel is not identified as a static defect, a dynamic defect detection process includes comparing pixel-to-pixel gradients between the current pixel a set of neighboring pixels against a dynamic defect threshold. If a dynamic defect is detected, a replacement value for correcting the dynamic defect may be determined by interpolating the value of two neighboring pixels on opposite sides of the current pixel in a direction exhibiting the smallest gradient.

Abstract: Certain aspects of this disclosure relate to an image signal processing system that includes a flash controller that is configured to activate a flash device prior to the start of a target image frame by using a sensor timing signal. In one embodiment, the flash controller receives a delayed sensor timing signal and determines a flash activation start time by using the delayed sensor timing signal to identify a time corresponding to the end of the previous frame, increasing that time by a vertical blanking time, and then subtracting a first offset to compensate for delay between the sensor timing signal and the delayed sensor timing signal. Then, the flash controller subtracts a second offset to determine the flash activation time, thus ensuring that the flash is activated prior to receiving the first pixel of the target frame.

Abstract: The present disclosure provides techniques relates to the implementation of a raw pixel processing unit using a set of line buffers. In one embodiment, the set of line buffers may include a first subset and second subset. Various logical units of the raw pixel processing unit may be implemented using the first and second subsets of line buffers in a shared manner. For instance, in one embodiment, defective pixel correction and detection logic may be implemented using the first subset of line buffers. The second subset of line buffers may be used to implement lens shading correction logic, gain, offset, and clamping logic, and demosaicing logic. Further, noise reduction may also be implemented using at least a portion of each of the first and second subsets of line buffers.

Abstract: Certain embodiments of the present disclosure provide a flexible memory input/output controller that is configured to the storing and reading of multiple types of pixels and pixel memory formats. For instance, the memory I/O controller may support the storing and reading of raw image pixels at various bits of precision, such as 8-bit, 10-bit, 12-bit, 14-bit, and 16-bit. Pixel formats that are unaligned with memory bytes (e.g., not being a multiple of 8-bits) may be stored in a packed manner. The memory I/O controller may also support various formats of RGB pixel sets and YCC pixel sets.

Abstract: The present disclosure provides techniques for performing audio-video synchronization using an image signal processing system. In one embodiment, a time code register provides a current time stamp when sampled. The value of the time code register may be incremented at regular intervals based on a clock of the image signal processing system. At the start of a current frame acquired by an image sensor, the time code register is sampled, and a timestamp is stored into a timestamp register associated with the image sensor. The timestamp is then read from the time stamp register and written to a set of metadata associated with the current frame. The timestamp stored in the frame metadata may then be used to synchronize the current frame with a corresponding set of audio data.

Abstract: Disclosed embodiments provide for a an image signal processing system that includes back-end pixel processing unit that receives pixel data after being processed by at least one of a front-end pixel processing unit and a pixel processing pipeline. In certain embodiments, the back-end processing unit receives luma/chroma image data and may be configured to apply face detection operations, local tone mapping, bright, contrast, color adjustments, as well as scaling. Further, the back-end processing unit may also include a back-end statistics unit that may collect frequency statistics. The frequency statistics may be provided to an encoder and may be used to determine quantization parameters that are to be applied to an image frame.

Abstract: Certain embodiments disclosed herein relate to an image signal processing system includes overflow control logic that detects an overflow condition when a destination unit when a sensor input queue and/or front-end processing unit receives back pressure from a downstream destination unit. In one embodiment, pixels of a current frame are dropped when an overflow condition occurs. The number of dropped pixels may be tracked using a counter. Upon recovery of the overflow condition, the remaining pixels of the frame are received and each dropped pixel may be replaced using a replacement pixel value.

Abstract: Various techniques are disclosed for collecting and processing auto-focus statistics data in an image signal processor (ISP). In one embodiment, a statistics collection engine in an ISP front-end processing unit may be configured to collect coarse (based on decimated raw data) and fine auto-focus statistics. Coarse auto-focus statistics may be collected on decimated Bayer RGB data and/or on linear camera luma values. Fine auto-focus statistics may be collected on raw Bayer RGB using a combination of a horizontal filter and edge detector, or may be collected on BayerY data (by applying a 3×1 transform to the raw Bayer RGB data). Edge sums may be accumulated using the filter outputs to determine auto-focus statistics.

Abstract: Techniques are provided for determining an optimal focal position using auto-focus statistics. In one embodiment, such techniques may include generating coarse and fine auto-focus scores for determining an optimal focal length at which to position a lens associated with the image sensor. For instance, the statistics logic may determine a coarse position that indicates an optimal focus area which, in one embodiment, may be determined by searching for the first coarse position in which a coarse auto-focus score decreases with respect to a coarse auto-focus score at a previous position. Using this position as a starting point for fine score searching, the optimal focal position may be determined by searching for a peak in fine auto-focus scores. In another embodiment, auto-focus statistics may also be determined based on each color of the Bayer RGB, such that, even in the presence of chromatic aberrations, relative auto-focus scores for each color may be used to determine the direction of focus.

Abstract: Various techniques are disclosed for processing statistics data in an image signal processor (ISP). In one embodiment, a statistics collection engine may be implemented in a front-end processing unit of the ISP, such that statistics are collected prior to processing by an ISP pipeline downstream from the front-end processing unit. In one embodiment, the statistics collection engine may be configured to acquire statistics relating to auto white-balance, auto-exposure, and auto-focus, as well as flicker detection. Collected statistics may be output to a memory and used by the ISP to process acquired image data.

Abstract: Various techniques are disclosed for processing statistics data in an image signal processor (ISP). In one embodiment, a statistics collection engine may be configured to acquire statistics relating to auto white-balance. The statistics collection engine may receive raw Bayer RGB data acquired by an image sensor and may be configured to perform one or more color space conversions to obtain pixel data in other color spaces. A set of pixel filters may be configured to accumulate sums of the pixel data conditionally based upon YC1C2 characteristics, as defined by a pixel condition per pixel filter. Depending on a selected color space, the pixel filters may generate color sums, which may be used to match a current illuminant against a set of reference illuminants with which the image sensor has been previously calibrated.