Abstract: Embodiments relate to a multi-mode demosaicing circuit able to receive and demosaic image data in a different raw image formats, such as Bayer raw image format and Quad Bayer raw image format. The multi-mode demosaicing circuit demosaics Quad Bayer image data by interpolating a green channel of the image data along each of a plurality of directions, generating a gradient of the image data along each of the plurality of directions, modifying the interpolated green channels based on respective gradients to generate full-resolution green channel image data, which is combined with red and blue image data to generate the demosaiced image data. Interpolation is performed for non-green pixels based on neighboring green pixels along a specified direction, modified by a residual value based upon values of one or more nearby same-color pixels and a correlation between values of the same-color pixels and neighboring green pixels.

Abstract: Embodiments relate to extracting features from images, such as by identifying keypoints and generating keypoint descriptors of the keypoints. An apparatus includes a pyramid image generator circuit, a keypoint descriptor generator circuit, and a pyramid image buffer. The pyramid image generator circuit generates an image pyramid from an input image. The keypoint descriptor generator circuit processes the pyramid images for keypoint descriptor generation. The pyramid image buffer stores different portions of the pyramid images generated by the pyramid image generator circuit at different times and provides the stored portions of the pyramid images to the keypoint descriptor generator circuit for keypoint descriptor generation. When first portions of the pyramid images are no longer needed for the keypoint descriptor generation, the first portions are removed from the pyramid image buffer to provide space for second portions of the pyramid images that are needed for the keypoint descriptor generation.

Abstract: Embodiments relate to an image processing circuit able to perform image fusion on received images in at least a first mode for fusing demosaiced and downscaled image data, and a second mode for fusing raw image data. Raw image data is received from an image sensor in Bayer RGB format. In the first mode, the raw image data is demosaiced and resampled prior to undergoing image fusion. On the other hand, in the second raw image mode, the image processing circuit performs image fusion on the raw Bayer image data, and demosaics and resamples the generated fused raw Bayer image. This may ensure a cleaner image signal for image fusion, but consumes more memory. The image processing circuit is configured to support both modes of operation, allowing for fused images to be generated to satisfy the requirements of different applications.

Abstract: Devices, methods, and non-transitory program storage devices are disclosed herein to perform predictive image sensor cropping operations to improve the performance of video image stabilization operations, especially for high resolution image sensors. According to some embodiments, the techniques include, for each of one or more respective images of a first plurality of images: obtaining image information corresponding to one or more images in the first plurality of images captured prior to the respective image; predicting, for the respective image, an image sensor cropping region to be read out from the first image sensor; and then reading out, into a memory, a first cropped version of the respective image comprising only the predicted image sensor cropping region for the respective image. Then, a first video may be produced based, at least in part, on the first cropped versions of the one or more respective images of the first plurality of images.

Abstract: Embodiments relate to a multi-mode demosaicing circuit able to receive and demosaic image data in a different raw image formats, such as Bayer raw image format and Quad Bayer raw image format. The multi-mode demosaicing circuit comprises different circuitry for demosaicing different image formats that access a shared working memory. In addition, the multi-mode demosaicing circuit shares memory with a post-processing and scaling circuit configured to perform subsequent post-processing and/or scaling of the demosaiced image data, in which the operations of the post-processing and scaling circuit are modified based on the original raw image format of the demosaiced image data to use different amounts of the shared memory, to compensate for additional memory utilized by the multi-mode demosaicing circuit when demosaicing certain types of image data.

Abstract: Methods and systems for detecting keypoints in image data may include an image sensor interface receiving pixel data from an image sensor. A front-end pixel data processing circuit may receive pixel data and convert the pixel data to a different color space format. A back-end pixel data processing circuit may perform one or more operations on the pixel data. An output circuit may receive pixel data and output the pixel data to a system memory. A keypoint detection circuit may receive pixel data from the image sensor interface in the image sensor pixel data format or receive pixel data after processing by the front-end or the back-end pixel data processing circuits. The keypoint detection circuit may perform a keypoint detection operation on the pixel data to detect one or more keypoints in the image frame and output to the system memory a description of the one or more keypoints.

Abstract: A foveated down sampling (FDS) circuit for down sampling of pixels in images. The FDS circuit down samples a first subset of pixels of a same color in an image using first scaling factors to generate first down sampled pixels in a first down sampled version of the image. The FDS circuit further down samples a second subset of the first down sampled pixels of the same color using second scaling factors to generate second down sampled pixels of the same color in a second down sampled version of the image. Pixels from the first subset are arranged in a first direction, and pixels from the second subset are arranged in a second direction.

Abstract: A foveated down sampling and correction (FDS-C) circuit for combined down sampling and correction of chromatic aberrations in images. The FDS-C circuit performs down sampling and interpolation of pixel values of a first subset of pixels of a color in a raw image using down sampling scale factors and first interpolation coefficients to generate first corrected pixel values for pixels of the color in a first corrected version of the raw image. The FDS-C circuit further performs interpolation of pixel values of a second subset of the pixels in the first corrected version using second interpolation coefficients to generate second corrected pixel values for pixels of the color in a second corrected version of the raw image. Pixels in the first subset are arranged in a first direction, pixels in the second subset are arranged in a second direction, and the down sampling scale factors vary along the first direction.

Abstract: Embodiments relate to lateral chromatic aberration (LCA) recovery of raw image data generated by image sensors. A chromatic aberration recovery circuit performs chromatic aberration recovery on the raw image data to correct the resulting LCA in the full color images using pre-calculated offset values of a subset of colors of pixels.

Abstract: Embodiments relate to an image processing circuit able to perform image fusion on received images in at least a first mode for fusing demosaiced and downscaled image data, and a second mode for fusing raw image data. Raw image data is received from an image sensor in Bayer RGB format. In the first mode, the raw image data is demosaiced and resampled prior to undergoing image fusion. On the other hand, in the second raw image mode, the image processing circuit performs image fusion on the raw Bayer image data, and demosaics and resamples the generated fused raw Bayer image. This may ensure a cleaner image signal for image fusion, but consumes more memory. The image processing circuit is configured to support both modes of operation, allowing for fused images to be generated to satisfy the requirements of different applications.

Abstract: In one embodiment, a system includes a first device rendering image data, a second device storing the image data, and a display panel that displays the image data stored in the memory. The first device renders multiple frames of the image data, compresses the multiple frames into a single superframe, and transports the single superframe. The second device receives the single superframe, decompresses the single superframe into the multiple frames of image data, and stores the image data on a memory of the second device.

Abstract: Embodiments relate to circuitry for temporal processing and image fusion. An image fusion circuit receives captured images, and generates corresponding image pyramids. The generated image pyramids are raster or tiled processed, and stored in memory. A fusion module receives a first and second image pyramids from the memory, and warps and fuses image pyramids to generate a fused image pyramid, which may be used for further processing, and may also be stored back into the memory. The image fusion circuitry is configurable to operate in a plurality of different configuration modes corresponding to different image fusion applications for fusing image pyramids of received images, including two-frame fusion, temporal filtering, infinite impulse response (IIR) temporal processing, and/or finite impulse response (FIR) temporal processing.

Abstract: Embodiments relate to circuitry for temporal processing and image fusion. An image fusion circuit receives captured images, and generates corresponding image pyramids. The generated image pyramids are raster or tiled processed, and stored in memory. A fusion module receives a first and second image pyramids from the memory, warps the first image pyramid based upon the second image pyramid, and fuses the warped first image pyramid with the second image pyramid to generate a fused image pyramid, which may be used for further processing, and may also be stored back into the memory. Because pyramid generation occurs prior to warping and fusion, and by allowing fused image pyramids to be stored back into memory, the image fusion circuitry is configurable to implement a variety of temporal processing functions involving different image fusion combinations.

Abstract: A foveated down sampling and correction (FDS-C) circuit for combined down sampling and correction of chromatic aberrations in images. The FDS-C circuit performs down sampling and interpolation of pixel values of a first subset of pixels of a color in a raw image using down sampling scale factors and first interpolation coefficients to generate first corrected pixel values for pixels of the color in a first corrected version of the raw image. The FDS-C circuit further performs interpolation of pixel values of a second subset of the pixels in the first corrected version using second interpolation coefficients to generate second corrected pixel values for pixels of the color in a second corrected version of the raw image. Pixels in the first subset are arranged in a first direction, pixels in the second subset are arranged in a second direction, and the down sampling scale factors vary along the first direction.

Abstract: Embodiments relate to an image processing circuit that performs machine learning (ML) based noise reduction on image data. The image processing circuit includes a ML based noise reduction circuit that includes a hybrid kernel calculation circuit and a noise filtering circuit coupled to the hybrid kernel calculation circuit. The hybrid kernel calculation circuit generates, for each pixel of an image, a hybrid kernel by combining a ML kernel of each pixel of the image and a bilateral kernel of each pixel of the image. The noise filtering circuit performs, for each pixel of the image, noise filtering of the image using the hybrid kernel for each pixel of the image to generate a de-noised version of the image.

Abstract: Methods and systems for detecting keypoints in image data may include an image sensor interface receiving pixel data from an image sensor. A front-end pixel data processing circuit may receive pixel data and convert the pixel data to a different color space format. A back-end pixel data processing circuit may perform one or more operations on the pixel data. An output circuit may receive pixel data and output the pixel data to a system memory. A keypoint detection circuit may receive pixel data from the image sensor interface in the image sensor pixel data format or receive pixel data after processing by the front-end or the back-end pixel data processing circuits. The keypoint detection circuit may perform a keypoint detection operation on the pixel data to detect one or more keypoints in the image frame and output to the system memory a description of the one or more keypoints.

Abstract: In one embodiment, a system includes a first device rendering image data, a second device storing the image data, and a display panel that displays the image data stored in the memory. The first device renders multiple frames of the image data, compresses the multiple frames into a single superframe, and transports the single superframe. The second device receives the single superframe, decompresses the single superframe into the multiple frames of image data, and stores the image data on a memory of the second device.

Abstract: Embodiments relate to a multi-mode demosaicing circuit able to receive and demosaic image data in a different raw image formats, such as Bayer raw image format and Quad Bayer raw image format. The multi-mode demosaicing circuit demosaics Quad Bayer image data by interpolating a green channel of the image data along each of a plurality of directions, generating a gradient of the image data along each of the plurality of directions, modifying the interpolated green channels based on respective gradients to generate full-resolution green channel image data, which is combined with red and blue image data to generate the demosaiced image data. Interpolation is performed for non-green pixels based on neighboring green pixels along a specified direction, modified by a residual value based upon values of one or more nearby same-color pixels and a correlation between values of the same-color pixels and neighboring green pixels.

Abstract: Embodiments relate to a multi-mode demosaicing circuit able to receive and demosaic image data in a different raw image formats, such as Bayer raw image format and Quad Bayer raw image format. The multi-mode demosaicing circuit comprises different circuitry for demosaicing different image formats that access a shared working memory. In addition, the multi-mode demosaicing circuit shares memory with a post-processing and scaling circuit configured to perform subsequent post-processing and/or scaling of the demosaiced image data, in which the operations of the post-processing and scaling circuit are modified based on the original raw image format of the demosaiced image data to use different amounts of the shared memory, to compensate for additional memory utilized by the multi-mode demosaicing circuit when demosaicing certain types of image data.

Abstract: Embodiments relate to a front-end scaler circuit configured to receive and process demosaiced image data in different modes depending on if the demosaiced image data was demosaiced from Bayer or Quad Bayer raw image data. The front-end scaler circuit shares memory with a demosaicing circuit, and is configured to perform different operations that use different amounts of the shared memory based on the original image format of the demosaiced image data being processed, to compensate for additional memory utilized by the demosaicing circuit when demosaicing certain types of image data. For example, when processing image data demosaiced from Quad Bayer image data, the front-end scaler circuit discards a portion of the chrominance component data for the received image data before performing chromatic suppression, compared to when processing image data demosaiced from Bayer image data.