Abstract: A image sensor includes a first integrated circuit layer including pixel sensors that are grouped based on position into pixel sensor groups, a second integrated circuit layer in electrical communication with the first integrated circuit layer, the second integrated circuit layer including image processing circuitry groups that are configured to each receive pixel information from a corresponding pixel sensor group, the image processing circuitry groups further configured to perform image processing operations on the pixel information to provide processed pixel information during operation of the image sensor, a third integrated circuit layer in electrical communication with the second integrated circuit layer, and the third integrated circuit layer including neural network circuitry groups that are configured to each receive the processed pixel information from a corresponding image processing circuitry group and perform analysis for object detection on the processed pixel information during operation of the

Abstract: Systems, apparatus, and methods are presented for rendering color images. One method may include receiving first color values of a first image from a first image capture device. The first color values may represent colors in a first color space. The method also may include applying first color transformation information to the first color values to generate first transformed color values of the first image in a particular color space and applying desaturation information to the first transformed color values to generate first desaturated color values of the first image. Further, the method may include providing the first desaturated color values of the first image for rendering on a display device.

Abstract: The technology relates to determining whether a camera is occluded. For instance, an image may be captured using a camera having red, green, and blue pixels each including a photosensor. Output values for the photosensors of each of the red pixels, green pixels, and blue pixels may be determined for the images. The output values of the green pixels may be compared to one or more of the output values of the red pixels or the output values of the blue pixels. That the camera is occluded is determined based on the comparison.

Abstract: The disclosure relates to assessing operation of two or more cameras. These cameras may be a group of cameras of a perception system of a vehicle having an autonomous driving mode. A first image captured by a first camera and a second image captured by a second camera may be received. A first feature vector for the first image and a second feature vector for the second image may be generated. A similarity score may be determined using the first feature vector and the second feature vector. This similarity score may be used to assess the operation of the two cameras and an appropriate action may be taken.

Abstract: The technology relates to determining whether a camera is occluded. For instance, an image may be captured using a camera having red, green, and blue pixels each including a photosensor. Output values for the photosensors of each of the red pixels, green pixels, and blue pixels may be determined for the images. The output values of the green pixels may be compared to one or more of the output values of the red pixels or the output values of the blue pixels. That the camera is occluded is determined based on the comparison.

Abstract: The disclosure relates to assessing operation of a camera. In one instance, a volume of space corresponding to a first vehicle in an environment of a second vehicle may be identified using sensor data generated by a LIDAR system of the second vehicle. An image captured by a camera of the second vehicle may be identified. The camera may have an overlapping field of view of the LIDAR system at a time when the sensor data was generated. An area of the image corresponding to the volume of space may be identified and processed in order to identify a vehicle light. The operation of the camera may be assessed based on the processing.

Abstract: A image sensor includes a first integrated circuit layer including pixel sensors that are grouped based on position into pixel sensor groups, a second integrated circuit layer in electrical communication with the first integrated circuit layer, the second integrated circuit layer including image processing circuitry groups that are configured to each receive pixel information from a corresponding pixel sensor group, the image processing circuitry groups further configured to perform image processing operations on the pixel information to provide processed pixel information during operation of the image sensor, a third integrated circuit layer in electrical communication with the second integrated circuit layer, and the third integrated circuit layer including neural network circuitry groups that are configured to each receive the processed pixel information from a corresponding image processing circuitry group and perform analysis for object detection on the processed pixel information during operation of the

Abstract: A image sensor includes a first integrated circuit layer including pixel sensors that are grouped based on position into pixel sensor groups, a second integrated circuit layer in electrical communication with the first integrated circuit layer, the second integrated circuit layer including image processing circuitry groups that are configured to each receive pixel information from a corresponding pixel sensor group, the image processing circuitry groups further configured to perform image processing operations on the pixel information to provide processed pixel information during operation of the image sensor, a third integrated circuit layer in electrical communication with the second integrated circuit layer, and the third integrated circuit layer including neural network circuitry groups that are configured to each receive the processed pixel information from a corresponding image processing circuitry group and perform analysis for object detection on the processed pixel information during operation of the

Abstract: The technology relates to determining whether a camera is occluded. For instance, an image may be captured using a camera having red, green, and blue pixels each including a photosensor. Output values for the photosensors of each of the red pixels, green pixels, and blue pixels may be determined for the images. The output values of the green pixels may be compared to one or more of the output values of the red pixels or the output values of the blue pixels. That the camera is occluded is determined based on the comparison.

Abstract: The technology relates to determining whether a camera is occluded. For instance, an image may be captured using a camera having red, green, and blue pixels each including a photosensor. Output values for the photosensors of each of the red pixels, green pixels, and blue pixels may be determined for the images. The output values of the green pixels may be compared to one or more of the output values of the red pixels or the output values of the blue pixels. That the camera is occluded is determined based on the comparison.

Abstract: A image sensor includes a first integrated circuit layer including pixel sensors that are grouped based on position into pixel sensor groups, a second integrated circuit layer in electrical communication with the first integrated circuit layer, the second integrated circuit layer including image processing circuitry groups that are configured to each receive pixel information from a corresponding pixel sensor group, the image processing circuitry groups further configured to perform image processing operations on the pixel information to provide processed pixel information during operation of the image sensor, a third integrated circuit layer in electrical communication with the second integrated circuit layer, and the third integrated circuit layer including neural network circuitry groups that are configured to each receive the processed pixel information from a corresponding image processing circuitry group and perform analysis for object detection on the processed pixel information during operation of the

Abstract: The technology relates to determining whether a camera is occluded. For instance, an image may be captured using a camera having red, green, and blue pixels each including a photosensor. Output values for the photosensors of each of the red pixels, green pixels, and blue pixels may be determined for the images. The output values of the green pixels may be compared to one or more of the output values of the red pixels or the output values of the blue pixels. That the camera is occluded is determined based on the comparison.

Abstract: The disclosure relates to assessing operation of two or more cameras. These cameras may be a group of cameras of a perception system of a vehicle having an autonomous driving mode. A first image captured by a first camera and a second image captured by a second camera may be received. A first feature vector for the first image and a second feature vector for the second image may be generated. A similarity score may be determined using the first feature vector and the second feature vector. This similarity score may be used to assess the operation of the two cameras and an appropriate action may be taken.

Abstract: RAW images and/or light field images may be compressed through the use of specialized techniques. The color depth of a light field image may be reduced through the use of a bit reduction algorithm such as a K-means algorithm. The image may then be retiled to group pixels of similar intensities and/or colors. The retiled image may be padded with extra pixel rows and/or pixel columns as needed, and compressed through the use of an image compression algorithm. The compressed image may be assembled with metadata pertinent to the manner in which compression was done to form a compressed image file. The compressed image file may be decompressed by following the compression method in reverse.

Abstract: According to various embodiments of the present invention, the optical systems of light field capture devices are optimized so as to improve captured light field image data. Optimizing optical systems of light field capture devices can result in captured light field image data (both still and video) that is cheaper and/or easier to process. Optical systems can be optimized to yield improved quality or resolution when using cheaper processing approaches whose computational costs fit within various processing and/or resource constraints. As such, the optical systems of light field cameras can be optimized to reduce size and/or cost and/or increase the quality of such optical systems.

Abstract: According to various embodiments of the present invention, the optical systems of light field capture devices are optimized so as to improve captured light field image data. Optimizing optical systems of light field capture devices can result in captured light field image data (both still and video) that is cheaper and/or easier to process. Optical systems can be optimized to yield improved quality or resolution when using cheaper processing approaches whose computational costs fit within various processing and/or resource constraints. As such, the optical systems of light field cameras can be optimized to reduce size and/or cost and/or increase the quality of such optical systems.

Abstract: According to various embodiments of the present invention, the optical systems of light field capture devices are optimized so as to improve captured light field image data. Optimizing optical systems of light field capture devices can result in captured light field image data (both still and video) that is cheaper and/or easier to process. Optical systems can be optimized to yield improved quality or resolution when using cheaper processing approaches whose computational costs fit within various processing and/or resource constraints. As such, the optical systems of light field cameras can be optimized to reduce size and/or cost and/or increase the quality of such optical systems.

Abstract: According to various embodiments, the system and method disclosed herein serve to at least partially compensate for departures of an actual main lens of a light-field camera from the properties of an ideal main lens. Light-field data may be captured and processed through the use of product calibration data and unit calibration data. The product calibration data may be descriptive of departure of a main lens design of the light-field camera from an ideal main lens design. The unit calibration data may be descriptive of departure of the actual main lens of the light-field camera from the main lens design. Corrected light-field data may be generated as a result of the processing, and may be used to generate a light-field image.

Abstract: RAW images and/or light field images may be compressed through the use of specialized techniques. The color depth of a light field image may be reduced through the use of a bit reduction algorithm such as a K-means algorithm. The image may then be retiled to group pixels of similar intensities and/or colors. The retiled image may be padded with extra pixel rows and/or pixel columns as needed, and compressed through the use of an image compression algorithm. The compressed image may be assembled with metadata pertinent to the manner in which compression was done to form a compressed image file. The compressed image file may be decompressed by following the compression method in reverse.

Abstract: RAW images and/or light field images may be compressed through the use of specialized techniques. The color depth of a light field image may be reduced through the use of a bit reduction algorithm such as a K-means algorithm. The image may then be retiled to group pixels of similar intensities and/or colors. The retiled image may be padded with extra pixel rows and/or pixel columns as needed, and compressed through the use of an image compression algorithm. The compressed image may be assembled with metadata pertinent to the manner in which compression was done to form a compressed image file. The compressed image file may be decompressed by following the compression method in reverse.