Abstract: An auto-exposure control is proposed for high dynamic range images, along with a neural network for exposure selection that is trained jointly, end-to-end with an object detector and an image signal processing (ISP) pipeline. Corresponding method and system for high dynamic range object detection are also provided.

Abstract: A nanophotonic neural network, wherein the nanophotonic neural network comprises a large-kernel spatially-varying convolutional neural network. The large-kernel spatially-varying convolutional neural network is learned via a low-dimensional re-parameterization technique. The large-kernel spatially-varying convolutional neural network comprises a flat meta-optical system that encompasses an array of nanophotonic structures designed to induce angle-dependent responses. The large-kernel spatially-varying convolutional neural network comprises an extremely lightweight electronic backend with approximately 2K parameters configured to reach a 73.80% blind test classification accuracy on CIFAR-10 dataset.

Abstract: An auto-exposure control is proposed for high dynamic range images, along with a neural network for exposure selection that is trained jointly, end-to-end with an object detector and an image signal processing (ISP) pipeline. Corresponding method and system for high dynamic range object detection are also provided.

Abstract: An autonomy computing system and a method of an autonomous vehicle for rectifying stereo images includes a memory storing computer executable instructions and a processor coupled to the memory, the processor, upon execution of the computer executable instructions, configured to: receive an image pair captured using respective cameras in the stereo camera pair; predict a rotation matrix between the first image and the second image by: extracting a first feature map and a second feature map; applying positional feature enhancement on the feature maps to derive a pair of enhanced feature maps; computing a correlation volume across the enhanced feature maps; determining a set of likely matches between the enhanced feature maps; computing a predicted relative pose; and computing the rotation matrix. The system and method further include calibrating the stereo camera pair to rectify the first image and the second image based on the rotation matrix.

Abstract: An auto-exposure control is proposed for high dynamic range images, along with a neural network for exposure selection that is trained jointly, end-to-end with an object detector and an image signal processing (ISP) pipeline. Corresponding method and system for high dynamic range object detection are also provided.

Abstract: An autonomy computing system of an autonomous vehicle for object detection in adverse environmental conditions is provided. The at least one processor of the autonomy computing system is programmed to receive sensor data from one or more sensors of a plurality of modalities, the second sensor data being in a bird's eye view (BEV). The at least one processor is further programmed to extract first features and second features in the environment, and to fuse, in the BEV, the first features and the second features into first enriched features and second enriched features. The at least one processor is also programmed to detect object proposals based on the first enriched features and the second enriched features, predict objects in the environment based on the object proposals, and control operation of the autonomous vehicle based on predicted objects.

Abstract: Methods and systems are described for analyzing images. One or more machine learning models may be trained based on a plurality of images. The one or more machine learning models may comprise a model representing a feature in a scene. The one or more machine learning models may be trained to map input image coordinates to vectors of spline control points. Images may be reconstructed removing the feature from the scene.

Abstract: An autonomy computing system and a method of an autonomous vehicle for rectifying stereo images includes a memory storing computer executable instructions and a processor coupled to the memory, the processor, upon execution of the computer executable instructions, configured to: receive an image pair captured using respective cameras in the stereo camera pair; predict a rotation matrix between the first image and the second image by: extracting a first feature map and a second feature map; applying positional feature enhancement on the feature maps to derive a pair of enhanced feature maps; computing a correlation volume across the enhanced feature maps; determining a set of likely matches between the enhanced feature maps; computing a predicted relative pose; and computing the rotation matrix. The system and method further include calibrating the stereo camera pair to rectify the first image and the second image based on the rotation matrix.

Abstract: The application generally relates to a polarimetric wavefront light detection and ranging (PolLidar) sensor. The PolLidar sensor includes an emitter module having an optical emitter aperture, a receiver module having an optical receiver aperture, and a mirror for scene scanning. The receiver module is separate from the emitter module.

Abstract: The system related to a gated camera stereo system, a red-clear-clear-blue (RCCB) camera, at least one memory, and at least one processor communicatively coupled with the at least one memory. The processor is configured to: control the gated camera stereo system to emit a pulse of light at a predetermined wavelength, control a detector of the gated camera stereo system to capture light reflected from a scene after a preset time delay, The processor embeds depth data into two-dimensional gated images using a gate function based upon the captured light by the detector of the gated camera stereo system, control the RCCB camera to capture passive high dynamic range (HDR) images, extract features corresponding to the gated images and the passive HDR images, and fuse the extracted features corresponding to the gated images and the passive HDR images by cross-spectral matching for depth estimation.

Abstract: The application generally relates to a computing system including at least one processor. The at least one processor is configured to execute instructions stored in at least one memory to: initiate emission of a light pulse by an illuminator, after a predetermined delay from emission of the light pulse, initiate capturing of a plurality of pixels in a scene using a plurality of sensors based on the plurality of captured pixels, for a point in the scene, compute a respective value for volumetric density, normal, reflectance and ambient light using a corresponding neural field. The processor further, based upon the emitted light pulse, computes a shadow component corresponding to an origin of the illuminator and a direction of the emitted light pulse and using the computed respective value for volumetric density, normal, reflectance and ambient light and the computed shadow component, constructs a gated image through a volume rendering formulation.

Abstract: Metasurfaces and systems including metasurfaces for imaging and methods of imaging are described. In one embodiment, a method for acquiring images by an imaging system comprising a metalens includes: illuminating the metalens; acquiring light passing through the metalens as a first image by an image sensor; and processing the first image into a second image that is a deconvolved version of the first image by a post-processing engine. The metalens includes a plurality of nanoposts carried by a substrate.

Abstract: An auto-exposure control is proposed for high dynamic range images, along with a neural network for exposure selection that is trained jointly, end-to-end with an object detector and an image signal processing (ISP) pipeline. Corresponding method and system for high dynamic range object detection are also provided.

Abstract: An autonomy computing system and a method of an autonomous vehicle for rectifying stereo images includes a memory storing computer executable instructions and a processor coupled to the memory, the processor, upon execution of the computer executable instructions, configured to: receive an image pair captured using respective cameras in the stereo camera pair; predict a rotation matrix between the first image and the second image by: extracting a first feature map and a second feature map; applying positional feature enhancement on the feature maps to derive a pair of enhanced feature maps; computing a correlation volume across the enhanced feature maps; determining a set of likely matches between the enhanced feature maps; computing a predicted relative pose; and computing the rotation matrix. The system and method further include calibrating the stereo camera pair to rectify the first image and the second image based on the rotation matrix.

Abstract: An autonomous vehicle including a microphone array of a plurality of microphones, a visual sensor network configured to receive visual signals, at least one processor, and at least one memory storing instructions is disclosed. The instructions, when executed by the at least one processor, cause the at least one processor to: (i) generate spatial beamforming maps locating a sound source using a beamforming model corresponding to acoustic signals received at the plurality of microphones of the microphone array; (ii) apply a synthetic aperture expansion to the acoustic signals to increase resolution of the spatial beamforming maps; and (iii) generate a future visual frame based at least partially upon temporal information extracted from the spatial beamforming maps and visualization maps generated based on the visual signals received by the visual sensor network.

Abstract: A flat nanophotonic computational camera, which employs an array of skewed lenslets (meta-optics) and a learned reconstruction approach is disclosed herein. The optical array is embedded on a metasurface that with a height of approximately one micron, is flat and sits on the sensor cover glass at approximately 2.5 mm focal distance from the sensor. A differentiable optimization method continuously samples over the visible spectrum and factorizes the optical modulation for different incident fields into individual lenses. A megapizel image is reconstructed from a flat imager with a learned probabilistic reconstruction method that employs a generative diffusion model to sample an implicit prior. A method for acquiring paired captured training data in varying illumination conditions is proposed. The proposed flat camera design is assessed in simulation and with an experimental prototype, validating that the method is capable of recovering images from diverse scenes in broadband with a single nanophotonic layer.

Abstract: The present disclosure relates generally to image processing, and more particularly, toward techniques for structured illumination and reconstruction of three-dimensional (3D) images. Disclosed herein is a method to jointly learn structured illumination and reconstruction, parameterized by a diffractive optical element and a neural network in an end-to-end fashion. The disclosed approach has a differentiable image formation model for active stereo, relying on both wave and geometric optics, and a trinocular reconstruction network. The jointly optimized pattern, dubbed “Polka Lines,” together with the reconstruction network, makes accurate active-stereo depth estimates across imaging conditions. The disclosed method is validated in simulation and used with an experimental prototype, and several variants of the Polka Lines patterns specialized to the illumination conditions are demonstrated.

Abstract: A system including at least one memory and at least one processor configured to: (i) identify a set of hyperparameters affecting a wavefront and a pipeline processing a signal corresponding to a pulse received at a detector of a light detection and ranging (LiDAR) sensor; (ii) identify a set of 3-dimensional (3D) objects for detection using a neural network with the set of hyperparameters optimized based at least in part on a Covariance Matrix Adaptation-Evolution Strategy (CMA-ES) and a square root of covariance matrix scale factor; (iii) detect the set of 3D objects from a plurality of LiDAR point clouds using the neural network with the optimized set of hyperparameters and using a manually tuned set of hyperparameters; and (iv) validate the neural network optimized set of hyperparameters and the manually tuned set of hyperparameters using an average precision based upon the detected set of 3D objects, is disclosed.

Abstract: A system including at least one memory storing instructions, and at least one processor in communication with the at least one memory is disclosed. The at least one processor is configured to execute the stored instructions to: (i) control a light detection and ranging (LiDAR) sensor to emit a pulse into an environment of the LiDAR sensor; (ii) generate temporal histograms corresponding to a signal detected by a detector of the LiDAR sensor for the pulse emitted by the LiDAR sensor; (iii) denoise a temporal waveform generated based on the temporal histograms; (iv) estimate ambient light; (v) determine a noise threshold corresponding to the ambient light; (vi) determine a peak of a plurality of peaks that has a maximum intensity; and (vii) add the peak to a point cloud.

Abstract: An autonomous vehicle including a microphone array of a plurality of microphones, a visual sensor network configured to receive visual signals, at least one processor, and at least one memory storing instructions is disclosed. The instructions, when executed by the at least one processor, cause the at least one processor to: (i) generate spatial beamforming maps locating a sound source using a beamforming model corresponding to acoustic signals received at the plurality of microphones of the microphone array; (ii) apply a synthetic aperture expansion to the acoustic signals to increase resolution of the spatial beamforming maps; and (iii) generate feature maps for an application in autonomous vehicle driving by combining the improved spatial beamforming maps with visualization maps generated based on the visual signals received by the visual sensor network.