Abstract: System, methods, and other embodiments described herein relate to performing depth estimation and object detection using a common network architecture. In one embodiment, a method includes generating, using a backbone of a combined network, a feature map at multiple scales from an input image. The method includes decoding, using a top-down pathway of the combined network, the feature map to provide features at the multiple scales. The method includes generating, using a head of the combined network, a depth map from the features for a scene depicted in the input image, and bounding boxes identifying objects in the input image.

Abstract: Described are systems and methods for self-learned label refinement of a training set. In on example, a system includes a processor and a memory having a training set generation module that causes the processor to train a model using an image as an input to the model and 2D bounding based on 3D bounding boxes as ground truths, select a first subset from predicted 2D bounding boxes previously outputted by the model, retrain the model using the image as the input and the first subset as ground truths, select a second set of predicted 2D bounding boxes previously outputted by the model, and generate the training set by selecting the 3D bounding boxes from a master set of 3D bounding boxes that have corresponding 2D bounding boxes that form the second subset.

Abstract: A method for multi-camera monocular depth estimation using pose averaging is described. The method includes determining a multi-camera photometric loss associated with a multi-camera rig of an ego vehicle. The method also includes determining a multi-camera pose consistency constraint (PCC) loss associated with the multi-camera rig of the ego vehicle. The method further includes adjusting the multi-camera photometric loss according to the multi-camera PCC loss to form a multi-camera PCC photometric loss. The method also includes training a multi-camera depth estimation model and an ego-motion estimation model according to the multi-camera PCC photometric loss. The method further includes predicting a 360° point cloud of a scene surrounding the ego vehicle according to the trained multi-camera depth estimation model and the ego-motion estimation model.

Abstract: System, methods, and other embodiments described herein relate to a manner of training a depth prediction system using bounding boxes. In one embodiment, a method includes segmenting an image to mask areas beyond bounding boxes and identify unmasked areas within the bounding boxes. The method also includes training a depth model using depth losses from comparing weighted points associated with pixels of the image within the unmasked areas to ground-truth depth. The method also includes providing the depth model for object detection.

Abstract: A method for multi-camera self-supervised depth evaluation is described. The method includes training a self-supervised depth estimation model and an ego-motion estimation model according to a multi-camera photometric loss associated with a multi-camera rig of an ego vehicle. The method also includes generating a single-scale correction factor according to a depth map of each camera of the multi-camera rig during a time-step. The method further includes predicting a 360° point cloud of a scene surrounding the ego vehicle according to the self-supervised depth estimation model and the ego-motion estimation model. The method also includes scaling the 360° point cloud according to the single-scale correction factor to form an aligned 360° point cloud.

Abstract: A system and related method for predicting movement of a plurality of pedestrians may include one or more processors and a memory. The memory includes an initial trajectory module, an exit point prediction module, a path planning module, and an adjustment module. The modules include instructions that when executed by the one or more processors cause the one or more processors to obtain trajectories of the plurality of pedestrians, predict future exit points for the plurality of pedestrians from a scene based on the trajectories of the plurality of pedestrians, determine trajectory paths of the plurality of pedestrians based on the future exit points and at least one scene element of a map, and adjust the trajectory paths based on at least one predicted interaction between at least two of the plurality of pedestrians.

Abstract: A method for navigating a vehicle through an environment includes assigning a first weight to each pixel associated with a dynamic object and assigning a second weight to each pixel associated with a static object. The method also includes generating a dynamic object depth estimate for the dynamic object and generating a static object depth estimate for the static object, an accuracy of the dynamic object depth estimate being greater than an accuracy of the static object depth estimate. The method still further includes generating a 3D estimate of the environment based on the dynamic object depth estimate and the static object depth estimate. The method also includes controlling an action of the vehicle based on the 3D estimate of the environment.

Abstract: A neural architecture search system for generating a neural network includes one or more processors and a memory. The memory includes a generator module, a self-supervised training module, and an output module. The modules cause the one or more processors to generate a candidate neural network by a controller neural network, obtain training data, generate an output by the candidate neural network performing a specific task using the training data as an input, determine a loss value using a loss function that considers the output of the candidate neural network and at least a portion of the training data, adjust the one or more model weights of the controller neural network based on the loss value, and output the candidate neural network. The candidate neural network may be derived from the controller neural network and one or more model weights of the controller neural network.

Abstract: System, methods, and other embodiments described herein relate to semi-supervised training of a depth model using a neural camera model that is independent of a camera type. In one embodiment, a method includes acquiring training data including at least a pair of training images and depth data associated with the training images. The method includes training the depth model using the training data to generate a self-supervised loss from the pair of training images and a supervised loss from the depth data. Training the depth model includes learning the camera type by generating, using a ray surface model, a ray surface that approximates an image character of the training images as produced by a camera having the camera type. The method includes providing the depth model to infer depths from monocular images in a device.

Abstract: A bird's eye view feature map, augmented with semantic information, can be used to detect an object in an environment. A point cloud data set augmented with the semantic information that is associated with identities of classes of objects can be obtained. Features can be extracted from the point cloud data set. Based on the features, an initial bird's eye view feature map can be produced. Because operations performed on the point cloud data set to extract the features or to produce the initial bird's eye view feature map can have an effect of diminishing an ability to distinguish the semantic information in the initial bird's eye view feature map, the initial bird's eye view feature map can be augmented with the semantic information to produce an augmented bird's eye view feature map. Based on the augmented bird's eye view feature map, the object in the environment can be detected.

Abstract: Systems and methods determining velocity of an object associated with a three-dimensional (3D) scene may include: a LIDAR system generating two sets of 3D point cloud data of the scene from two consecutive point cloud sweeps; a pillar feature network encoding data of the point cloud data to extract two-dimensional (2D) bird's-eye-view embeddings for each of the point cloud data sets in the form of pseudo images, wherein the 2D bird's-eye-view embeddings for a first of the two point cloud data sets comprises pillar features for the first point cloud data set and the 2D bird's-eye-view embeddings for a second of the two point cloud data sets comprises pillar features for the second point cloud data set; and a feature pyramid network encoding the pillar features and performing a 2D optical flow estimation to estimate the velocity of the object.

Abstract: A method includes capturing a two-dimensional (2D) image of an environment adjacent to an ego vehicle, the environment includes at least a dynamic object and a static object. The method also includes generating, via a depth estimation network, a depth map of the environment based on the 2D image, an accuracy of a depth estimate for the dynamic object in the depth map is greater than an accuracy of a depth estimate for the static object in the depth map. The method further includes generating a three-dimensional (3D) estimate of the environment based on the depth map and identifying a location of the dynamic object in the 3D estimate. The method additionally includes controlling an action of the ego vehicle based on the identified location.

Abstract: Systems and methods for generating depth models and depth maps from images obtained from an imaging system are presented. A self-supervised neural network may be capable of regularizing depth information from surface normals. Rather than rely on separate depth and surface normal networks, surface normal information is extracted from the depth information and a smoothness function is applied to the surface normals instead of a depth gradient. Smoothing the surface normal may provide improved representation of environmental structures by both smoothing texture-less areas while preserving sharp boundaries between structures.

Abstract: System, methods, and other embodiments described herein relate to semi-supervised training of a depth model for monocular depth estimation. In one embodiment, a method includes training the depth model according to a first stage that is self-supervised and that includes using first training data that comprises pairs of training images. Respective ones of the pairs including separate frames depicting a scene of a monocular video. The method includes training the depth model according to a second stage that is weakly supervised and that includes using second training data to produce depth maps according to the depth model. The second training data comprising individual images with corresponding sparse depth data. The second training data providing for updating the depth model according to second stage loss values that are based, at least in part, on the depth maps and the depth data.

Abstract: A bird's eye view feature map, augmented with semantic information, can be used to detect an object in an environment. A point cloud data set augmented with the semantic information that is associated with identities of classes of objects can be obtained. Features can be extracted from the point cloud data set. Based on the features, an initial bird's eye view feature map can be produced. Because operations performed on the point cloud data set to extract the features or to produce the initial bird's eye view feature map can have an effect of diminishing an ability to distinguish the semantic information in the initial bird's eye view feature map, the initial bird's eye view feature map can be augmented with the semantic information to produce an augmented bird's eye view feature map. Based on the augmented bird's eye view feature map, the object in the environment can be detected.

Abstract: System, methods, and other embodiments described herein relate to improving controls in a device according to risk. In one embodiment, a method includes, in response to receiving sensor data about a surrounding environment of the device, identifying objects from the sensor data that are present in the surrounding environment. The method includes generating a control sequence for controlling the device according to a risk-sensitivity parameter to navigate toward a destination while considering risk associated with encountering the objects defined by the risk-sensitivity parameter. The method includes controlling the device according to the control sequence.

Abstract: System, methods, and other embodiments described herein relate to improving trajectory forecasting in a device. In one embodiment, a method includes, in response to receiving sensor data about a surrounding environment of the device, identifying an object from the sensor data that is present in the surrounding environment. The method includes determining category probabilities for the object, the category probabilities indicating semantic classes for classifying the object and probabilities that the object belongs to the semantic classes. The method includes forecasting trajectories for the object based, at least in part, on the category probabilities and the sensor data. The method includes controlling the device according to the trajectories.

Abstract: A method is presented. The method includes estimating an ego-motion of an agent based on a current image from a sequence of images and at least one previous image from the sequence of images. Each image in the sequence of images may be a two-dimensional (2D) image. The method also includes estimating a depth of the current image based the at least one previous image. The estimated depth accounts for a depth uncertainty measurement in the current image and the at least one previous image. The method further includes generating a three-dimensional (3D) reconstruction of the current image based on the estimated ego-motion and the estimated depth. The method still further includes controlling an action of the agent based on the three-dimensional reconstruction.

Abstract: System, methods, and other embodiments described herein relate to training a multi-task network using real and virtual data. In one embodiment, a method includes acquiring training data that includes real data and virtual data for training a multi-task network that performs at least depth prediction and semantic segmentation. The method includes generating a first output from the multi-task network using the real data and second output from the multi-task network using the virtual data. The method includes generating a mixed loss by analyzing the first output to produce a real loss and the second output to produce a virtual loss. The method includes updating the multi-task network using the mixed loss.

Abstract: Systems and methods described herein relate to training a machine-learning-based monocular depth estimator.