Abstract: A flat nanophotonic computational camera, which employs an array of skewed lenslets (meta-optics) and a learned reconstruction approach is disclosed herein. The optical array is embedded on a metasurface that with a height of approximately one micron, is flat and sits on the sensor cover glass at approximately 2.5 mm focal distance from the sensor. A differentiable optimization method continuously samples over the visible spectrum and factorizes the optical modulation for different incident fields into individual lenses. A megapizel image is reconstructed from a flat imager with a learned probabilistic reconstruction method that employs a generative diffusion model to sample an implicit prior. A method for acquiring paired captured training data in varying illumination conditions is proposed. The proposed flat camera design is assessed in simulation and with an experimental prototype, validating that the method is capable of recovering images from diverse scenes in broadband with a single nanophotonic layer.

Abstract: The present disclosure relates generally to image processing, and more particularly, toward techniques for structured illumination and reconstruction of three-dimensional (3D) images. Disclosed herein is a method to jointly learn structured illumination and reconstruction, parameterized by a diffractive optical element and a neural network in an end-to-end fashion. The disclosed approach has a differentiable image formation model for active stereo, relying on both wave and geometric optics, and a trinocular reconstruction network. The jointly optimized pattern, dubbed “Polka Lines,” together with the reconstruction network, makes accurate active-stereo depth estimates across imaging conditions. The disclosed method is validated in simulation and used with an experimental prototype, and several variants of the Polka Lines patterns specialized to the illumination conditions are demonstrated.

Abstract: A system including at least one memory storing instructions, and at least one processor in communication with the at least one memory is disclosed. The at least one processor is configured to execute the stored instructions to: (i) initiate optimization of a pulse emitted by a light detection and ranging (LiDAR) sensor into an environment of the LiDAR sensor using a respective channel of a plurality of channels of the LiDAR sensor; (ii) initiate optimization of a pipeline processing a signal corresponding to the emitted pulse received at a detector of the LiDAR sensor; (iii) construct a max-rank loss scalarization for the signal using the optimized pipeline; (iv) compute transients using centroid weights based upon the max-rank loss scalarization; and (v) replace a centroid based upon a covariance matrix adaptation-evolution strategy (CMA-ES) upon determining a new centroid corresponding to the computed transients.

Abstract: System and method for end-to-end differentiable joint image refinement and perception are provided. A learning machine employs an image acquisition device for acquiring a set of training raw images. A processor determines a representation of a raw image, initializes a set of image representation parameters, defines a set of analysis parameters of an image analysis network configured to process the image's representation, and jointly trains the set of representation parameters and the set of analysis parameters to optimize a combined objective function. A module for transforming pixel-values of the raw image to produce a transformed image comprising pixels of variance-stabilized values, a module for successively performing processes of soft camera projection and image projection, and a module for inverse transforming the transformed pixels are disclosed. The image projection performs multi-level spatial convolution, pooling, subsampling, and interpolation.

Abstract: A perception system including at least one memory, and at least one processor configured to: (i) compute, in a stereo branch, disparity from a pair of stereo images including a left image and a right image; (ii) based on the computed disparity from the pair of stereo images, output, by the stereo branch, a depth for the left image and a depth for the right image; (iii) compute an absolute depth for the left image in a first monocular branch and an absolute depth for the right image in a second monocular branch; (iv) compute, in a first fusion branch, a depth map for the left image; (v) compute, in a second fusion branch, a depth map for the right image; and (vi) generate a single fused depth map based on the depth map for the left image and the depth map for the right image, is disclosed.

Abstract: An autonomous vehicle including a microphone array of a plurality of microphones, a visual sensor network configured to receive visual signals, at least one processor, and at least one memory storing instructions is disclosed. The instructions, when executed by the at least one processor, cause the at least one processor to: (i) generate spatial beamforming maps locating a sound source using a beamforming model corresponding to acoustic signals received at the plurality of microphones of the microphone array; (ii) apply a synthetic aperture expansion to the acoustic signals to increase resolution of the spatial beamforming maps; and (iii) generate feature maps for an application in autonomous vehicle driving by combining the improved spatial beamforming maps with visualization maps generated based on the visual signals received by the visual sensor network.

Abstract: An autonomous vehicle including a microphone array of a plurality of microphones, a visual sensor network configured to receive visual signals, at least one processor, and at least one memory storing instructions is disclosed. The instructions, when executed by the at least one processor, cause the at least one processor to: (i) generate spatial beamforming maps locating a sound source using a beamforming model corresponding to acoustic signals received at the plurality of microphones of the microphone array; (ii) apply a synthetic aperture expansion to the acoustic signals to increase resolution of the spatial beamforming maps; and (iii) generate a future visual frame based at least partially upon temporal information extracted from the spatial beamforming maps and visualization maps generated based on the visual signals received by the visual sensor network.

Abstract: System and method for end-to-end differentiable joint image refinement and perception are provided. A learning machine employs an image acquisition device for acquiring a set of training raw images. A processor determines a representation of a raw image, initializes a set of image representation parameters, defines a set of analysis parameters of an image analysis network configured to process the image's representation, and jointly trains the set of representation parameters and the set of analysis parameters to optimize a combined objective function. A module for transforming pixel-values of the raw image to produce a transformed image comprising pixels of variance-stabilized values, a module for successively performing processes of soft camera projection and image projection, and a module for inverse transforming the transformed pixels are disclosed. The image projection performs multi-level spatial convolution, pooling, subsampling, and interpolation.

Abstract: A system including at least one memory storing instructions, and at least one processor in communication with the at least one memory is disclosed. The at least one processor is configured to execute the stored instructions to: (i) control a light detection and ranging (LiDAR) sensor to emit a pulse into an environment of the LiDAR sensor; (ii) generate temporal histograms corresponding to a signal detected by a detector of the LiDAR sensor for the pulse emitted by the LiDAR sensor; (iii) denoise a temporal waveform generated based on the temporal histograms; (iv) estimate ambient light; (v) determine a noise threshold corresponding to the ambient light; (vi) determine a peak of a plurality of peaks that has a maximum intensity; and (vii) add the peak to a point cloud.

Abstract: System and method for end-to-end differentiable joint image refinement and perception are provided. A learning machine employs an image acquisition device for acquiring a set of training raw images. A processor determines a representation of a raw image, initializes a set of image representation parameters, defines a set of analysis parameters of an image analysis network configured to process the image's representation, and jointly trains the set of representation parameters and the set of analysis parameters to optimize a combined objective function. A module for transforming pixel-values of the raw image to produce a transformed image comprising pixels of variance-stabilized values, a module for successively performing processes of soft camera projection and image projection, and a module for inverse transforming the transformed pixels are disclosed. The image projection performs multi-level spatial convolution, pooling, subsampling, and interpolation.

Abstract: An autonomous vehicle including a network of sensors including a plurality of acoustic sensors and a plurality of visual sensors, at least one processor, and at least one memory storing instructions is disclosed. The instructions, when executed by the at least one processor, cause the at least one processor to: (i) generate spatial beamforming maps locating a sound source based upon acoustic signals received at the plurality of acoustic sensors; (ii) identify a type of an object generating the acoustic signals received at the plurality of acoustic sensors based upon comparison of the acoustic signals with a plurality of acoustic signals and respective objects stored in a dataset; and (iii) generate feature maps for an application in an autonomous vehicle driving by enhancing visualization maps generated based upon visual signals received by the plurality of visual sensors.

Abstract: A system including at least one memory and at least one processor configured to: (i) identify a set of hyperparameters affecting a wavefront and a pipeline processing a signal corresponding to a pulse received at a detector of a light detection and ranging (LiDAR) sensor; (ii) identify a set of 3-dimensional (3D) objects for detection using a neural network with the set of hyperparameters optimized based at least in part on a Covariance Matrix Adaptation-Evolution Strategy (CMA-ES) and a square root of covariance matrix scale factor; (iii) detect the set of 3D objects from a plurality of LiDAR point clouds using the neural network with the optimized set of hyperparameters and using a manually tuned set of hyperparameters; and (iv) validate the neural network optimized set of hyperparameters and the manually tuned set of hyperparameters using an average precision based upon the detected set of 3D objects, is disclosed.

Abstract: System and method for end-to-end differentiable joint image refinement and perception are provided. A learning machine employs an image acquisition device for acquiring a set of training raw images. A processor determines a representation of a raw image, initializes a set of image representation parameters, defines a set of analysis parameters of an image analysis network configured to process the image's representation, and jointly trains the set of representation parameters and the set of analysis parameters to optimize a combined objective function. A module for transforming pixel-values of the raw image to produce a transformed image comprising pixels of variance-stabilized values, a module for successively performing processes of soft camera projection and image projection, and a module for inverse transforming the transformed pixels are disclosed. The image projection performs multi-level spatial convolution, pooling, subsampling, and interpolation.

Abstract: System and method for end-to-end differentiable joint image refinement and perception are provided. A learning machine employs an image acquisition device for acquiring a set of training raw images. A processor determines a representation of a raw image, initializes a set of image representation parameters, defines a set of analysis parameters of an image analysis network configured to process the image's representation, and jointly trains the set of representation parameters and the set of analysis parameters to optimize a combined objective function. A module for transforming pixel-values of the raw image to produce a transformed image comprising pixels of variance-stabilized values, a module for successively performing processes of soft camera projection and image projection, and a module for inverse transforming the transformed pixels are disclosed. The image projection performs multi-level spatial convolution, pooling, subsampling, and interpolation.

Abstract: Metasurfaces and systems including metasurfaces for imaging and methods of imaging are described. In one embodiment, a method for acquiring images by an imaging system comprising a metalens includes: illuminating the metalens; acquiring light passing through the metalens as a first image by an image sensor; and processing the first image into a second image that is a deconvolved version of the first image by a post-processing engine. The metalens includes a plurality of nanoposts carried by a substrate.

Abstract: A computer-vision pipeline is organized as a closed loop of a sensor-processing phase, an image-processing phase, and an object-detection phase, each comprising a respective phase processor coupled to a master processor. The sensor-processing phase creates multiple exposure images, and derives multi-exposure multi-scale zonal illumination-distributions, to be processed independently in the image-processing phase. In a first implementation of the object-detection phase, extracted exposure-specific features are pooled prior to overall object detection. In a second implementation, exposure-specific objects, detected from the exposure-specific features, are fused to produce the sought objects of a scene under consideration. The two implementations enable detecting fine details of a scene under diverse illumination conditions. The master processor performs loss-function computations to derive updated training parameters of the processing phases.

Abstract: A system for testing control units via simulation includes: a simulator; a host computer; and at least one connection for a communication system. At least one communication tool is stored on the system. Real control units are connectable to the system via the communication system. At least one controller is provided on the system for the connection to the communication system. Driver software for the at least one controller is stored on the system. The at least one communication tool is configured to generate communication code for communication between simulated control units and/or the real control units, wherein the communication code is configured to interact with the driver software and to relay signals and/or messages from the real and simulated control units to the driver software and to receive the signals and/or messages from the driver software. A loop mode is provided for the driver software.

Abstract: Departing from conventional HIDR image fusion approach, a learned task-driven fusion in the feature domain is disclosed. Instead of using a single companded image, the disclosed method exploits semantic features from all exposures learned in an end-to-end fashion with supervision from downstream detection losses. The method outperforms all tested conventional HDR exposure fusion and auto-exposure methods in challenging automotive HIDR scenarios.

Abstract: A computer-vision pipeline is organized as a closed loop of a sensor-processing phase, an image-processing phase, and an object-detection phase, each comprising a respective phase processor coupled to a master processor. The sensor-processing phase creates multiple exposure images, and derives multi-exposure multi-scale zonal illumination-distributions, to be processed independently in the image-processing phase. In a first implementation of the object-detection phase, extracted exposure-specific features are pooled prior to overall object detection. In a second implementation, exposure-specific objects, detected from the exposure-specific features, are fused to produce the sought objects of a scene under consideration. The two implementations enable detecting fine details of a scene under diverse illumination conditions. The master processor performs loss-function computations to derive updated training parameters of the processing phases.

Abstract: System and method for end-to-end differentiable joint image refinement and perception are provided. A learning machine employs an image acquisition device for acquiring a set of training raw images. A processor determines a representation of a raw image, initializes a set of image representation parameters, defines a set of analysis parameters of an image analysis network configured to process the image's representation, and jointly trains the set of representation parameters and the set of analysis parameters to optimize a combined objective function. A module for transforming pixel-values of the raw image to produce a transformed image comprising pixels of variance-stabilized values, a module for successively performing processes of soft camera projection and image projection, and a module for inverse transforming the transformed pixels are disclosed. The image projection performs multi-level spatial convolution, pooling, subsampling, and interpolation.