Abstract: In some embodiments, an image sensor structure receives light from a color filter array that includes multiple regions. A high resolution region includes a first bayer pattern of color filters that filter light for individual sensors of a sensor array while a lower resolution region includes second bayer pattern of color filters that filter light for groups of sensors of the sensor array. Images may be read from the image sensor in multiple resolutions, with images read in a highest resolution mode preserving maximum detail while images read in lower resolution modes may requires less hardware and computational resources and may exhibit better noise performance, in some embodiments. Various regions of the image sensor structure may correspond to different modes of operation or differing resolving capabilities of a corresponding lens assembly for the image sensor structure.

Abstract: An image sensing device includes a semiconductor substrate and a first array of photodetectors disposed on the substrate. Readout circuits, which are disposed on the substrate, are coupled to respective pairs of the photodetectors, and are configured to output signals from the respective pairs in at least a first mode, in which the signals are read out individually from each of the photodetectors, and a second mode, in which the signals are binned together pairwise. A color filter layer includes a matrix of red, green, and blue tiles overlying respective pairs of the photodetectors. A second of array microlenses, overlying the color filter layer, includes first microlenses overlying the respective pairs of the photodetectors that are overlain by the red and blue tiles and second microlenses individually overlying each of the photodetectors in the respective pairs that are overlain by at least some of the green tiles.

Abstract: An image sensing device includes a semiconductor substrate on which a first array of photodetectors is disposed. Readout circuits are disposed on the substrate and coupled to respective pairs of the photodetectors, and output signals from the respective pairs in a first mode, in which the signals are read out individually from each of the photodetectors, and in a second mode, in which the signals are binned together pairwise. A color filter layer disposed over the photodetectors includes a matrix of red, green, and blue tiles, such that in a first set of the pairs of the photodetectors, both the photodetectors are overlain by respective green tiles, while in a second set of the pairs of the photodetectors, one of the photodetectors is overlain by a first tile of a first color, while the other one of the photodetectors is overlain by a second tile of a different, second color.

Abstract: An image sensing device includes a semiconductor substrate and a first array of photodetectors disposed on the substrate at a predefined pitch. A color filter layer is disposed over the photodetectors and includes a matrix of red, green, and blue regions, each region overlying a respective group of the photodetectors. A second array of microlenses is disposed over the color filter layer and includes first microlenses having a first transverse dimension no greater than the pitch disposed respectively over all the photodetectors within the green regions, and second microlenses having a second transverse dimension greater than the pitch disposed over at least some of the photodetectors within the red and blue regions. Each second microlens is configured to focus light onto two or more of the photodetectors in the respective group.

Abstract: An image capture device is described. The image capture device includes an array of pixels, each pixel including a photodetector. A Bayer pattern color filter is disposed over a 4×4 subset of pixels in the array of pixels. The Bayer pattern color filter defines a first 2×2 subset of pixels sensitive to red light; a second 2×2 subset of pixels sensitive to green light; a third 2×2 subset of pixels sensitive to green light; and a fourth 2×2 subset of pixels sensitive to blue light. A set of 1×1 on-chip lenses (OCLs) includes a different 1×1 OCL disposed over each pixel in the second 2×2 subset of pixels and the third 2×2 subset of pixels. A set of 2×1 OCLs or 2×2 OCLs includes a 2×1 OCL or a 2×2 OCL disposed over each pixel in the first 2×2 subset of pixels and the fourth 2×2 subset of pixels.

Abstract: Embodiments are disclosed for a single RGBIR camera module that is capable of imaging at both the visible and IR wavelengths. In some embodiments, a camera module comprises: an image sensor comprising: a microlens array; a color filter array (CFA) comprising a red filter, a blue filter, a green filter and at least one infrared (IR) filter; and a pixel array comprising pixels to convert light received through the color filter array into electrical signals; and an image signal processor (ISP) configured to: initiate capture of a first frame by reading signal pixels from the pixel array; initiate capture of a second frame by reading IR pixels from the pixel array; align the first and second frames; and extract the second frame from the first frame to generate a third enhanced frame.

Abstract: An apparatus to facilitate partially-frozen neural networks for efficient computer vision systems is disclosed. The apparatus includes a frozen core to store fixed weights of a machine learning model, one or more trainable cores coupled to the frozen core, the one or more trainable cores comprising multipliers for trainable weights of the machine learning model, and wherein the alpha blending layer includes a trainable alpha blending parameter, and wherein the trainable alpha blending parameter is a function of a trainable parameter, a sigmoid function, and outputs of frozen and trainable blocks in a preceding layer of the machine learning model.

Abstract: An apparatus to facilitate partially-frozen neural networks for efficient computer vision systems is disclosed. The apparatus includes a frozen core to store fixed weights of a machine learning model, one or more trainable cores coupled to the frozen core, the one or more trainable cores comprising multipliers for trainable weights of the machine learning model, and wherein the alpha blending layer includes a trainable alpha blending parameter, and wherein the trainable alpha blending parameter is a function of a trainable parameter, a sigmoid function, and outputs of frozen and trainable blocks in a preceding layer of the machine learning model.

Abstract: An example apparatus for adjusting eye gaze in images one or more processors to execute instructions to bidirectionally train a neural network; access a target angle and an input image, the input image including an eye in a first position; generate a vector field with the neural network; and generate a gaze-adjusted image based on the vector field, the gaze-adjusted image including the eye in a second position.

Abstract: An example apparatus for adjusting eye gaze in images one or more processors to execute instructions to bidirectionally train a neural network; access a target angle and an input image, the input image including an eye in a first position; generate a vector field with the neural network; and generate a gaze-adjusted image based on the vector field, the gaze-adjusted image including the eye in a second position.

Abstract: An apparatus, method, and a computer readable medium for attenuating visual artifacts in processed images. An annotated dataset of images to be processed by an image processing system is created. An adversarial control network is trained to operate as an image quality expert in classifying images. After the adversarial control network has been trained, the adversarial control network is used to supervise the image processing system on-the-fly.

Abstract: An image capture device is described. The image capture device includes an array of pixels, each pixel including a photodetector. A Bayer pattern color filter is disposed over a 4×4 subset of pixels in the array of pixels. The Bayer pattern color filter defines a first 2×2 subset of pixels sensitive to red light; a second 2×2 subset of pixels sensitive to green light; a third 2×2 subset of pixels sensitive to green light; and a fourth 2×2 subset of pixels sensitive to blue light. A set of 1×1 on-chip lenses (OCLs) includes a different 1×1 OCL disposed over each pixel in the second 2×2 subset of pixels and the third 2×2 subset of pixels. A set of 2×1 OCLs or 2×2 OCLs includes a 2×1 OCL or a 2×2 OCL disposed over each pixel in the first 2×2 subset of pixels and the fourth 2×2 subset of pixels.

Abstract: An apparatus to facilitate partially-frozen neural networks for efficient computer vision systems is disclosed. The apparatus includes a frozen core to store fixed weights of a machine learning model, one or more trainable cores coupled to the frozen core, the one or more trainable cores comprising multipliers for trainable weights of the machine learning model, and wherein the alpha blending layer includes a trainable alpha blending parameter, and wherein the trainable alpha blending parameter is a function of a trainable parameter, a sigmoid function, and outputs of frozen and trainable blocks in a preceding layer of the machine learning model.

Abstract: An example apparatus for processing images includes a trainable vision scaler to receive an image. The trainable vision scaler is to generate output including a feature map or an enhanced image based on the image. The trainable vision scaler is to transmit the output to a computer vision network. The computer vision network is trained to perform a computer vision task using the output.

Abstract: In an example method for training image signal processors, a reconstructed image is generated via an image signal processor based on a sensor image. An intermediate loss function is generated based on a comparison of an output of one or more corresponding layers of a computer vision network and a copy of the computer vision network. The output of the computer vision network is based on the reconstructed image. An image signal processor is trained based on the intermediate loss function.

Abstract: An example apparatus for processing images includes a hybrid infinite impulse response-finite impulse response (IIR-FIR) convolution block to receive an image and generate processed image information. The hybrid IIR-FIR convolution block includes a vertical infinite impulse response (IIR) component to approximate a vertical convolution when processing the image.

Abstract: An example apparatus for adjusting eye gaze in images one or more processors to execute instructions to bidirectionally train a neural network; access a target angle and an input image, the input image including an eye in a first position; generate a vector field with the neural network; and generate a gaze-adjusted image based on the vector field, the gaze-adjusted image including the eye in a second position.

Abstract: An example apparatus for correcting gaze in images includes an image receiver to receive an image comprising an eye and a target angle set to a center. The apparatus also includes a bidirectionally trained convolutional neural network (CNN) to receive the image and the target angle from the image receiver and generate a vector field and a brightness map based on the image and the target angle. The apparatus further includes an image corrector to generate a gaze corrected image based on the vector field and the brightness map.

Abstract: In an example method for training image signal processors, a reconstructed image is generated via an image signal processor based on a sensor image. An intermediate loss function is generated based on a comparison of an output of one or more corresponding layers of a computer vision network and a copy of the computer vision network. The output of the computer vision network is based on the reconstructed image. An image signal processor is trained based on the intermediate loss function.

Abstract: Techniques related to computer vision tasks are discussed. Such techniques include applying a pretrained non-linear transform and pretrained details boosting factor to generate an enhanced image from an input image and reducing the bit depth of the enhanced image prior to applying a pretrained computer vision network to perform the computer vision task.