Abstract: Methods and systems for detecting keypoints in image data may include an image sensor interface receiving pixel data from an image sensor. A front-end pixel data processing circuit may receive pixel data and convert the pixel data to a different color space format. A back-end pixel data processing circuit may perform one or more operations on the pixel data. An output circuit may receive pixel data and output the pixel data to a system memory. A keypoint detection circuit may receive pixel data from the image sensor interface in the image sensor pixel data format or receive pixel data after processing by the front-end or the back-end pixel data processing circuits. The keypoint detection circuit may perform a keypoint detection operation on the pixel data to detect one or more keypoints in the image frame and output to the system memory a description of the one or more keypoints.

Abstract: A foveated down sampling and correction (FDS-C) circuit for combined down sampling and correction of chromatic aberrations in images. The FDS-C circuit performs down sampling and interpolation of pixel values of a first subset of pixels of a color in a raw image using down sampling scale factors and first interpolation coefficients to generate first corrected pixel values for pixels of the color in a first corrected version of the raw image. The FDS-C circuit further performs interpolation of pixel values of a second subset of the pixels in the first corrected version using second interpolation coefficients to generate second corrected pixel values for pixels of the color in a second corrected version of the raw image. Pixels in the first subset are arranged in a first direction, pixels in the second subset are arranged in a second direction, and the down sampling scale factors vary along the first direction.

Abstract: Devices, methods, and non-transitory program storage devices are disclosed herein to perform predictive image sensor cropping operations to improve the performance of video image stabilization operations, especially for high resolution image sensors. According to some embodiments, the techniques include, for each of one or more respective images of a first plurality of images: obtaining image information corresponding to one or more images in the first plurality of images captured prior to the respective image; predicting, for the respective image, an image sensor cropping region to be read out from the first image sensor; and then reading out, into a memory, a first cropped version of the respective image comprising only the predicted image sensor cropping region for the respective image. Then, a first video may be produced based, at least in part, on the first cropped versions of the one or more respective images of the first plurality of images.

Abstract: Embodiments relate to a front-end scaler circuit configured to receive and process demosaiced image data in different modes depending on if the demosaiced image data was demosaiced from Bayer or Quad Bayer raw image data. The front-end scaler circuit shares memory with a demosaicing circuit, and is configured to perform different operations that use different amounts of the shared memory based on the original image format of the demosaiced image data being processed, to compensate for additional memory utilized by the demosaicing circuit when demosaicing certain types of image data. For example, when processing image data demosaiced from Quad Bayer image data, the front-end scaler circuit discards a portion of the chrominance component data for the received image data before performing chromatic suppression, compared to when processing image data demosaiced from Bayer image data.

Abstract: Embodiments relate to image signal processors (ISP) that include binner circuits that down-sample an input image. An input image may include a plurality of pixels. The output image of the binner circuit may include a reduced number of pixels. The binner circuit may include a plurality of different operation modes. In a bin mode, the binner circuit may blend a subset of input pixel values to generate an output pixel quad. In a skip mode, the binner circuit may select one of the input pixel values as the output pixel pixel. The selection may be performed randomly to avoid aliasing. In a luminance mode, the binner circuit may take a weighted average of a subset of pixel values having different colors. In a color value mode, the binner circuit may select one of the colors in a subset of pixel values as an output pixel value.

Abstract: An image processing circuit for multi-illumination white balance with thumbnail processing. The image processing circuit determines a set of initial weights for a source pixel in a thumbnail image by determining component values for multiple color channels of the source pixel. The image processing circuit determines a set of weights for the source pixel in a weight map for the thumbnail image. Each weight in the set of weights is determined based on corresponding initial weights from the set of initial weights. Each weight in the set of weights represents an intensity level of a respective chrominance class of multiple chrominance classes for the source pixel. The image processing circuit applies the set of weights to values of the color channels of the source pixel to generate color component values of the color channels of a target pixel in a target thumbnail image.

Abstract: In one embodiment, a system includes a first device rendering image data, a second device storing the image data, and a display panel that displays the image data stored in the memory. The first device renders multiple frames of the image data, compresses the multiple frames into a single superframe, and transports the single superframe. The second device receives the single superframe, decompresses the single superframe into the multiple frames of image data, and stores the image data on a memory of the second device.

Abstract: Embodiments relate to image signal processors (ISP) that include binner circuits that down-sample an input image. An input image may include a plurality of pixels. The output image of the binner circuit may include a reduced number of pixels. The binner circuit may include a plurality of different operation modes. In a bin mode, the binner circuit may blend a subset of input pixel values to generate an output pixel quad. In a skip mode, the binner circuit may select one of the input pixel values as the output pixel pixel. The selection may be performed randomly to avoid aliasing. In a luminance mode, the binner circuit may take a weighted average of a subset of pixel values having different colors. In a color value mode, the binner circuit may select one of the colors in a subset of pixel values as an output pixel value.

Abstract: An image processing circuit for multi-illumination white balance with thumbnail processing. The image processing circuit determines a set of initial weights for a source pixel in a thumbnail image by determining component values for multiple color channels of the source pixel. The image processing circuit determines a set of weights for the source pixel in a weight map for the thumbnail image. Each weight in the set of weights is determined based on corresponding initial weights from the set of initial weights. Each weight in the set of weights represents an intensity level of a respective chrominance class of multiple chrominance classes for the source pixel. The image processing circuit applies the set of weights to values of the color channels of the source pixel to generate color component values of the color channels of a target pixel in a target thumbnail image.

Abstract: An image processing circuit for performing local tone mapping (LTM) after image warping. The image processing circuit includes a warping circuit that warps an input image to generate a warped image, and a LTM circuit coupled to the warping circuit. The LTM circuit determines an input color component value for a color component of a pixel in a version of the warped image, determines an output color component value for the color component of the pixel, based on mapping of coordinates of pixels in the warped image to coordinates of pixels in the input image, determines a gain value for the pixel as a ratio of the output color component value relative to the input color component value, and adjusts color component values for color components of the pixel using the gain value to generate adjusted color component values for the color components of the pixel in an output image.

Abstract: Embodiments relate to image signal processors (ISP) that include binner circuits that down-sample an input image. An input image may include a plurality of pixels. The output image of the binner circuit may include a reduced number of pixels. The binner circuit may include a plurality of different operation modes. In a bin mode, the binner circuit may blend a subset of input pixel values to generate an output pixel quad. In a skip mode, the binner circuit may select one of the input pixel values as the output pixel. The selection may be performed randomly to avoid aliasing. In a luminance mode, the binner circuit may take a weighted average of a subset of pixel values having different colors. In a color value mode, the binner circuit may select one of the colors in a subset of pixel values as an output pixel value.

Abstract: Embodiments relate to extracting features from images, such as by identifying keypoints and generating keypoint descriptors of the keypoints. An apparatus includes a pyramid image generator circuit, a keypoint descriptor generator circuit, and a pyramid image buffer. The pyramid image generator circuit generates an image pyramid from an input image. The keypoint descriptor generator circuit processes the pyramid images for keypoint descriptor generation. The pyramid image buffer stores different portions of the pyramid images generated by the pyramid image generator circuit at different times and provides the stored portions of the pyramid images to the keypoint descriptor generator circuit for keypoint descriptor generation. When first portions of the pyramid images are no longer needed for the keypoint descriptor generation, the first portions are removed from the pyramid image buffer to provide space for second portions of the pyramid images that are needed for the keypoint descriptor generation.

Abstract: Embodiments ralte to a multi-mode demosaicing circuit able to receive and demosaic image data in a different raw image formats, such as Bayer raw image format and Quad Bayer raw image format. The multi-mode demosaicing circuit demosaics Quad Bayer image data by interpolating a green channel of the image data along each of a plurality of directions, generating a gradient of the image data along each of the plurality of directions, modifying the interpolated green channels based on respective gradients to generate full-resolution green channel image data, which is combined with red and blue image data to generate the demosaiced image data. Interpolation is performed for non-green pixels based on neighboring green pixels along a specified direction, modified by a residual value based upon valued of one or more nearby same-color pixels and a correlation between values of the same color pixels and neighboring green pixels.

Abstract: Embodiments relate to a multi-mode demosaicing circuit able to receive and demosaic image data in a different raw image formats, such as Bayer raw image format and Quad Bayer raw image format. The multi-mode demosaicing circuit demosaics Quad Bayer image data by interpolating a green channel of the image data along each of a plurality of directions, generating a gradient of the image data along each of the plurality of directions, modifying the interpolated green channels based on respective gradients to generate full-resolution green channel image data, which is combined with red and blue image data to generate the demosaiced image data. Interpolation is performed for non-green pixels based on neighboring green pixels along a specified direction, modified by a residual value based upon values of one or more nearby same-color pixels and a correlation between values of the same-color pixels and neighboring green pixels.

Abstract: Embodiments relate to extracting features from images, such as by identifying keypoints and generating keypoint descriptors of the keypoints. An apparatus includes a pyramid image generator circuit, a keypoint descriptor generator circuit, and a pyramid image buffer. The pyramid image generator circuit generates an image pyramid from an input image. The keypoint descriptor generator circuit processes the pyramid images for keypoint descriptor generation. The pyramid image buffer stores different portions of the pyramid images generated by the pyramid image generator circuit at different times and provides the stored portions of the pyramid images to the keypoint descriptor generator circuit for keypoint descriptor generation. When first portions of the pyramid images are no longer needed for the keypoint descriptor generation, the first portions are removed from the pyramid image buffer to provide space for second portions of the pyramid images that are needed for the keypoint descriptor generation.

Abstract: Embodiments relate to an image processing circuit able to perform image fusion on received images in at least a first mode for fusing demosaiced and downscaled image data, and a second mode for fusing raw image data. Raw image data is received from an image sensor in Bayer RGB format. In the first mode, the raw image data is demosaiced and resampled prior to undergoing image fusion. On the other hand, in the second raw image mode, the image processing circuit performs image fusion on the raw Bayer image data, and demosaics and resamples the generated fused raw Bayer image. This may ensure a cleaner image signal for image fusion, but consumes more memory. The image processing circuit is configured to support both modes of operation, allowing for fused images to be generated to satisfy the requirements of different applications.

Abstract: Devices, methods, and non-transitory program storage devices are disclosed herein to perform predictive image sensor cropping operations to improve the performance of video image stabilization operations, especially for high resolution image sensors. According to some embodiments, the techniques include, for each of one or more respective images of a first plurality of images: obtaining image information corresponding to one or more images in the first plurality of images captured prior to the respective image; predicting, for the respective image, an image sensor cropping region to be read out from the first image sensor; and then reading out, into a memory, a first cropped version of the respective image comprising only the predicted image sensor cropping region for the respective image. Then, a first video may be produced based, at least in part, on the first cropped versions of the one or more respective images of the first plurality of images.

Abstract: Embodiments relate to a multi-mode demosaicing circuit able to receive and demosaic image data in a different raw image formats, such as Bayer raw image format and Quad Bayer raw image format. The multi-mode demosaicing circuit comprises different circuitry for demosaicing different image formats that access a shared working memory. In addition, the multi-mode demosaicing circuit shares memory with a post-processing and scaling circuit configured to perform subsequent post-processing and/or scaling of the demosaiced image data, in which the operations of the post-processing and scaling circuit are modified based on the original raw image format of the demosaiced image data to use different amounts of the shared memory, to compensate for additional memory utilized by the multi-mode demosaicing circuit when demosaicing certain types of image data.

Abstract: A foveated down sampling (FDS) circuit for down sampling of pixels in images. The FDS circuit down samples a first subset of pixels of a same color in an image using first scaling factors to generate first down sampled pixels in a first down sampled version of the image. The FDS circuit further down samples a second subset of the first down sampled pixels of the same color using second scaling factors to generate second down sampled pixels of the same color in a second down sampled version of the image. Pixels from the first subset are arranged in a first direction, and pixels from the second subset are arranged in a second direction.

Abstract: A foveated down sampling and correction (FDS-C) circuit for combined down sampling and correction of chromatic aberrations in images. The FDS-C circuit performs down sampling and interpolation of pixel values of a first subset of pixels of a color in a raw image using down sampling scale factors and first interpolation coefficients to generate first corrected pixel values for pixels of the color in a first corrected version of the raw image. The FDS-C circuit further performs interpolation of pixel values of a second subset of the pixels in the first corrected version using second interpolation coefficients to generate second corrected pixel values for pixels of the color in a second corrected version of the raw image. Pixels in the first subset are arranged in a first direction, pixels in the second subset are arranged in a second direction, and the down sampling scale factors vary along the first direction.