Abstract: Systems and methods described herein relate to reconstructing a scene in three dimensions from a two-dimensional image. One embodiment processes an image using a detection transformer to detect an object in the scene and to generate a NOCS map of the object and a background depth map; uses MLPs to relate the object to a differentiable database of object priors (PriorDB); recovers, from the NOCS map, a partial 3D object shape; estimates an initial object pose; fits a PriorDB object prior to align in geometry and appearance with the partial 3D shape to produce a complete shape and refines the initial pose estimate; generates an editable and re-renderable 3D scene reconstruction based, at least in part, on the complete shape, the refined pose estimate, and the depth map; and controls the operation of a robot based, at least in part, on the editable and re-renderable 3D scene reconstruction.

Abstract: Systems and methods for training an adapter network that adapts a model pre-trained on synthetic images to real-world data are disclosed herein. A system may include a processor and a memory in communication with the processor and having machine-readable that cause the processor to output, using a neural network, a predicted scene that includes a three-dimensional bounding box having pose information of an object, generate a rendered map of the object that includes a rendered shape of the object and a rendered surface normal of the object, and train the adapter network, which adapts the predicted scene to adjust for a deformation of the input image by comparing the rendered map to the output map acting as a ground truth.

Abstract: An apparatus for estimating distance according to the present disclosure extracts feature maps from respective images including a reference image and a source image; projects a source feature map extracted from the source image onto hypothetical planes to generate a cost volume; sets sampling points on a ray extending from the viewpoint of the reference image in a direction corresponding to a target pixel in the reference image; interpolates features of the respective sampling points, using features associated with nearby coordinates; inputs the features corresponding to the respective sampling points into a classifier to calculate occupancy probabilities corresponding to the respective sampling points; and adds up products of the occupancy probabilities of the respective sampling points and the distances from the viewpoint of the reference image to the corresponding sampling points to estimate the distance from the viewpoint of the reference image to a surface of the object.

Abstract: Systems and methods described herein relate to reconstructing a scene in three dimensions from a two-dimensional image. One embodiment processes an image using a detection transformer to detect an object in the scene and to generate a NOCS map of the object and a background depth map; uses MLPs to relate the object to a differentiable database of object priors (PriorDB); recovers, from the NOCS map, a partial 3D object shape; estimates an initial object pose; fits a PriorDB object prior to align in geometry and appearance with the partial 3D shape to produce a complete shape and refines the initial pose estimate; generates an editable and re-renderable 3D scene reconstruction based, at least in part, on the complete shape, the refined pose estimate, and the depth map; and controls the operation of a robot based, at least in part, on the editable and re-renderable 3D scene reconstruction.

Abstract: A system for self-calibrating sensors includes an electronic control unit, a first image sensor and a second image sensor communicatively coupled to the electronic control unit. The electronic control unit is configured to obtain a first image and a second image, where the first image and the second image contain an overlapping portion, determine an identity of an object present within the overlapping portion, obtain parameters of the identified object, determine a miscalibration of the first image sensor or the second image sensor based on a comparison of the identified object in the overlapping portions and the parameters of the identified object, in response to determining a miscalibration of the first image sensor or the second image sensor, calibrate the first image sensor or the second image sensor based on the parameters of the identified object and the second image or the first image, respectively.

Abstract: A method for 3D auto-labeling of objects with predetermined structural and physical constraints includes identifying initial object-seeds for all frames from a given frame sequence of a scene. The method also includes refining each of the initial object-seeds over the 2D/3D data, while complying with the predetermined structural and physical constraints to auto-label 3D object vehicles within the scene. The method further includes linking the auto-label 3D object vehicles over time into trajectories while respecting the predetermined structural and physical constraints.

Abstract: Systems and methods for three-dimensional object detection are disclosed herein. One embodiment inputs, to a neural network, a two-dimensional label associated with an object to produce a Normalized-Object-Coordinate-Space (NOCS) image and a shape vector, the shape vector mapping to a continuously traversable coordinate shape space (CSS); decodes the NOCS image and the shape vector to an object model in the CSS; back-projects, in a frustum, the NOCS image to a LIDAR point cloud; identifies correspondences between the LIDAR point cloud and the object model to estimate an affine transformation between the LIDAR point cloud and the object model; iteratively refines the affine transformation using a differentiable SDF renderer; extracts automatically a three-dimensional label for the object based, at least in part, on the iteratively refined affine transformation; and performs three-dimensional object detection of the object based, at least in part, on the extracted three-dimensional label for the object.

Abstract: Systems and methods for training an adapter network that adapts a model pre-trained on synthetic images to real-world data are disclosed herein. A system may include a processor and a memory in communication with the processor and having machine-readable that cause the processor to output, using a neural network, a predicted scene that includes a three-dimensional bounding box having pose information of an object, generate a rendered map of the object that includes a rendered shape of the object and a rendered surface normal of the object, and train the adapter network, which adapts the predicted scene to adjust for a deformation of the input image by comparing the rendered map to the output map acting as a ground truth.

Abstract: A system and a method for processing an image include inputting the image to a neural network configured to: obtain a plurality of feature maps, each feature map having a respective resolution and a respective depth, perform a classification on each feature map to deliver, for each feature map: the type of at least one object visible on the image, the position and shape in the image of at least one two-dimensional bounding box surrounding the at least one object, at least one possible viewpoint for the at least one object, at least one possible in-plane rotation for the at least one object.

Abstract: A method for monocular 3D object perception is described. The method includes sampling multiple, stochastic latent variables from a learned latent feature distribution of an RGB image for a 2D object detected in the RGB image. The method also includes lifting a 3D proposal for each stochastic latent variable sampled for the detected 2D object. The method further includes selecting a 3D proposal for the detected 2D object using a proposal selection algorithm to reduce 3D proposal lifting overlap. The method also includes planning a trajectory of an ego vehicle according to a 3D location and pose of the 2D object according to the selected 3D proposal.

Abstract: A method for inferring a location of an object includes extracting features from sensor data obtained from a number of sensors of an autonomous vehicle and encoding the features to a number of sensor space representations. The method also reshapes the number of sensor space representations to a feature space representation corresponding to a feature space of a spatial area. The method further identifies the object based on a mapping of the features to the feature space representation. The method still further projects a representation of the identified object to a location of the feature space and controls an action of the autonomous vehicle based on projecting the representation.

Abstract: A system and a method for processing an image include inputting the image to a neural network configured to: obtain a plurality of feature maps, each feature map having a respective resolution and a respective depth, perform a classification on each feature map to deliver, for each feature map: the type of at least one object visible on the image, the position and shape in the image of at least one two-dimensional bounding box surrounding the at least one object, at least one possible viewpoint for the at least one object, at least one possible in-plane rotation for the at least one object.

Abstract: A system for self-calibrating sensors includes an electronic control unit, a first image sensor and a second image sensor communicatively coupled to the electronic control unit. The electronic control unit is configured to obtain a first image and a second image, where the first image and the second image contain an overlapping portion, determine an identity of an object present within the overlapping portion, obtain parameters of the identified object, determine a miscalibration of the first image sensor or the second image sensor based on a comparison of the identified object in the overlapping portions and the parameters of the identified object, in response to determining a miscalibration of the first image sensor or the second image sensor, calibrate the first image sensor or the second image sensor based on the parameters of the identified object and the second image or the first image, respectively.

Abstract: Systems and methods for three-dimensional object detection are disclosed herein. One embodiment inputs, to a neural network, a two-dimensional label associated with an object to produce a Normalized-Object-Coordinate-Space (NOCS) image and a shape vector, the shape vector mapping to a continuously traversable coordinate shape space (CSS); decodes the NOCS image and the shape vector to an object model in the CSS; back-projects, in a frustum, the NOCS image to a LIDAR point cloud; identifies correspondences between the LIDAR point cloud and the object model to estimate an affine transformation between the LIDAR point cloud and the object model; iteratively refines the affine transformation using a differentiable SDF renderer; extracts automatically a three-dimensional label for the object based, at least in part, on the iteratively refined affine transformation; and performs three-dimensional object detection of the object based, at least in part, on the extracted three-dimensional label for the object.

Abstract: A method for 3D auto-labeling of objects with predetermined structural and physical constraints includes identifying initial object-seeds for all frames from a given frame sequence of a scene. The method also includes refining each of the initial object-seeds over the 2D/3D data, while complying with the predetermined structural and physical constraints to auto-label 3D object vehicles within the scene. The method further includes linking the auto-label 3D object vehicles over time into trajectories while respecting the predetermined structural and physical constraints.

Abstract: In one embodiment, example systems and methods relate to a manner of generating 3D representations from monocular 2D images. A monocular 2D image is captured by a camera. The 2D image is processed to create one or more feature maps. The features may include depth features, or object labels, for example. Based on the image and the feature map, regions-of-interest corresponding to vehicles in the image are determined. For each region-of-interest a lifting function is applied to the region-of-interest to determine values such as height and width, camera distance, and rotation. The determined values are used to create an eight-point box that is a 3D representation of the vehicle depicted by the region-of-interest. The 3D representation can be used for a variety of purposes such as route planning, object avoidance, or as training data, for example.

Abstract: A method for monocular 3D object perception is described. The method includes sampling multiple, stochastic latent variables from a learned latent feature distribution of an RGB image for a 2D object detected in the RGB image. The method also includes lifting a 3D proposal for each stochastic latent variable sampled for the detected 2D object. The method further includes selecting a 3D proposal for the detected 2D object using a proposal selection algorithm to reduce 3D proposal lifting overlap. The method also includes planning a trajectory of an ego vehicle according to a 3D location and pose of the 2D object according to the selected 3D proposal.

Abstract: In one embodiment, example systems and methods relate to a manner of generating training data for a classifier or a regression function using labeled synthetic images and a mapping that accounts for the differences between synthetic images and real images. The mapping may be a neural network that was trained using image pairs that each include an image of an object and a synthetic image that is generated from the image of the object by overlaying a rendering of the object into the image. The mapping may recognize the differences between features of the object in the real image and features of the rendering of the object in the synthetic image such as color, contrast, sensor noise, etc. Later, a set of labeled synthetic images is received, and the mapping is used to generate training data from the labeled synthetic images.

Abstract: A method for inferring a location of an object includes extracting features from sensor data obtained from a number of sensors of an autonomous vehicle and encoding the features to a number of sensor space representations. The method also reshapes the number of sensor space representations to a feature space representation corresponding to a feature space of a spatial area. The method further identifies the object based on a mapping of the features to the feature space representation. The method still further projects a representation of the identified object to a location of the feature space and controls an action of the autonomous vehicle based on projecting the representation.

Abstract: A method for a 3D scene reconstruction of autonomous agent operation sequences includes iteratively parsing sequence segmentation parts of agent operation sequence images into dynamic sequence segmentation parts and static sequence segmentation parts. The method also includes fitting 3D points over the static sequence segmentation parts to construct a 3D model of the static sequence segmentation parts over multiple frames of the agent operation sequence images. The method further includes removing the 3D model of the static sequence segmentation parts from the agent operation sequence images. The method also includes processing the dynamic sequence segmentation parts from each of the agent operation sequence images over the multiple frames of the agent operation sequence images to determine trajectories of the dynamic sequence segmentation parts.