Abstract: Systems, methods, and other embodiments described herein relate to improving controls in a device according to risk. In one embodiment, a method includes, in response to receiving sensor data about a surrounding environment of the device, identifying objects from the sensor data that are present in the surrounding environment. The method includes generating a control sequence for controlling the device according to a risk-sensitivity parameter to navigate toward a destination while considering risk associated with encountering the objects defined by the risk-sensitivity parameter. The method includes controlling the device according to the control sequence.

Abstract: A method for depth estimation performed by a depth estimation system associated with an agent includes determining a first depth of a first image and a second depth of a second image, the first image and the second image being captured by a sensor associated with the agent. The method also includes generating a first 3D image of the first image based on the first depth, a first pose associated with the sensor, and the second image. The method further includes generating a warped depth image based on transforming the first depth in accordance with the first pose. The method also includes updating the first pose based on a second pose associated with the warped depth image and the second depth, and updating the first 3D image based on the updated first pose. The method further includes controlling an action of the agent based on the updated first 3D image.

Abstract: Described are systems and methods for self-learned label refinement of a training set. In on example, a system includes a processor and a memory having a training set generation module that causes the processor to train a model using an image as an input to the model and 2D bounding based on 3D bounding boxes as ground truths, select a first subset from predicted 2D bounding boxes previously outputted by the model, retrain the model using the image as the input and the first subset as ground truths, select a second set of predicted 2D bounding boxes previously outputted by the model, and generate the training set by selecting the 3D bounding boxes from a master set of 3D bounding boxes that have corresponding 2D bounding boxes that form the second subset.

Abstract: System, methods, and other embodiments described herein relate to training a depth model for monocular depth estimation using photometric loss masks derived from motion estimates of dynamic objects. In one embodiment, a method includes generating depth maps from images of an environment. The method includes determining motion of points within the depth maps. The method includes associating the points between the depth maps to identify an object according to a correlation of the motion for a first cluster of the points with a second cluster of the points. The method includes providing the depth maps and the object as an electronic output.

Abstract: A method for self-supervised learning is described. The method includes generating a plurality of augmented data from unlabeled image data. The method also includes generating a population augmentation graph for a class determined from the plurality of augmented data. The method further includes minimizing a contrastive loss based on a spectral decomposition of the population augmentation graph to learn representations of the unlabeled image data. The method also includes classifying the learned representations of the unlabeled image data to recover ground-truth labels of the unlabeled image data.

Abstract: A method for controlling an agent to navigate through an environment includes generating a depth map associated with a monocular image of the environment. The method also includes generating a group of surface normal. Each surface normal of the group of surface normals is associated with a respective polygon of a group of polygons associated with the depth map. The method further includes identifying one or more ground planes in the depth map based on the group of surface normal. The method further includes controlling the agent to navigate through the environment based on identifying the one or more ground planes.

Abstract: A method for three-dimensional (3D) scene reconstruction by an agent includes estimating an ego-motion of the agent based on a current image from a sequence of images and a previous image from the sequence of images. Each image in the sequence of images may be a two-dimensional (2D) image. The method also includes estimating a depth of the current image via a depth estimation model comprising a group of encoder layers and a group of decoder layers. The method further includes generating a 3D reconstruction of the current image based on the estimated ego-motion and the estimated depth. The method still further includes controlling an action of the agent based on the 3D reconstruction.

Abstract: A representation of a spatial structure of objects in an image can be determined. A mode of a neural network can be set, in response to a receipt of the image and a receipt of a facing direction of a camera that produced the image. The mode can account for the facing direction. The facing direction can include one or more of a first facing direction of a first camera disposed on a vehicle or a second facing direction of a second camera disposed on the vehicle. The neural network can be executed, in response to the mode having been set, to determine the representation of the spatial structure of the objects in the image. The representation of the spatial structure of the objects in the image can be transmitted to an automotive navigation system to determine a distance between the vehicle and a specific object in the image.

Abstract: A method for navigating a vehicle through an environment includes assigning a first weight to each pixel associated with a dynamic object and assigning a second weight to each pixel associated with a static object. The method also includes generating a dynamic object depth estimate for the dynamic object and generating a static object depth estimate for the static object, an accuracy of the dynamic object depth estimate being greater than an accuracy of the static object depth estimate. The method still further includes generating a 3D estimate of the environment based on the dynamic object depth estimate and the static object depth estimate. The method also includes controlling an action of the vehicle based on the 3D estimate of the environment.

Abstract: A method for using an artificial neural network associated with an agent to estimate depth, includes receiving, at the artificial neural network, an input image captured via a sensor associated with the agent. The method also includes upsampling, at each decoding layer of a plurality of decoding layers of the artificial neural network, decoded features associated with the input image to a resolution associated with a final output of the artificial neural network. The method further includes concatenating, at each decoding layer, the upsampled decoded features with features obtained at a convolution layer associated with a respective decoding layer. The method still further includes estimating, at a recurrent module of the artificial neural network, a depth of the input image based on receiving the concatenated upsampled decoded features from each decoding layer. The method also includes controlling an action of an agent based on the depth estimate.

Abstract: A method for keypoint matching includes determining a first set of keypoints corresponding to a current environment of the agent. The method further includes determining a second set of keypoints from a pre-built map of the current environment. The method still further includes identifying matching pairs of keypoints from the first set of keypoints and the second set of keypoints based on geometrical similarities between respective keypoints of the first set of keypoints and the second set of keypoints. The method also includes determining a current location of the agent based on the identified matching pairs of keypoints. The method further includes controlling an action of the agent based on the current location.

Abstract: Systems and methods for extracting ground plane information directly from monocular images using self-supervised depth networks are disclosed. Self-supervised depth networks are used to generate a three-dimensional reconstruction of observed structures. From this reconstruction the system may generate surface normals. The surface normals can be calculated directly from depth maps in a way that is much less computationally expensive and accurate than surface normals extraction from standard LiDAR data. Surface normals facing substantially the same direction and facing upwards may be determined to reflect a ground plane.

Abstract: A method for multi-camera monocular depth estimation using pose averaging is described. The method includes determining a multi-camera photometric loss associated with a multi-camera rig of an ego vehicle. The method also includes determining a multi-camera pose consistency constraint (PCC) loss associated with the multi-camera rig of the ego vehicle. The method further includes adjusting the multi-camera photometric loss according to the multi-camera PCC loss to form a multi-camera PCC photometric loss. The method also includes training a multi-camera depth estimation model and an ego-motion estimation model according to the multi-camera PCC photometric loss. The method further includes predicting a 360° point cloud of a scene surrounding the ego vehicle according to the trained multi-camera depth estimation model and the ego-motion estimation model.

Abstract: A method for depth estimation performed by a depth estimation system of an autonomous agent includes determining a first pose of a sensor based on a first image captured by the sensor and a second image captured by the sensor. The method also includes determining a first depth of the first image and a second depth of the second image. The method further includes generating a warped depth image based on at least the first depth and the first pose. The method still further includes determining a second pose based on the warped depth image and the second depth image. The method also includes updating the first pose based on the second pose and updating a first warped image based on the updated first pose.

Abstract: A method of compositional feature representation learning for video understanding is described. The method includes individually processing a sequence of video frames received as an input of a feature map network to generate a plurality of feature maps. The method also includes binding the plurality of feature maps to a fixed set of slot variables using an attention model according to a motion segmentation signal. The method further includes combining slot states corresponding to the fixed set of slot variables into a combined feature map. The method also includes decoding the combined feature map to form a reconstructed sequence of video frames, in which objects discovered in the reconstructed sequence of video frames are identified.

Abstract: A method for monocular depth/pose estimation in a camera agnostic network is described. The method includes projecting lifted 3D points onto an image plane according to a predicted ray vector based on a monocular depth model, a monocular pose model, and a camera center of a camera agnostic network. The method also includes predicting a warped target image from a predicted depth map of the monocular depth model, a ray surface of the predicted ray vector, and a projection of the lifted 3D points according to the camera agnostic network.

Abstract: A method for multi-camera self-supervised depth evaluation is described. The method includes training a self-supervised depth estimation model and an ego-motion estimation model according to a multi-camera photometric loss associated with a multi-camera rig of an ego vehicle. The method also includes generating a single-scale correction factor according to a depth map of each camera of the multi-camera rig during a time-step. The method further includes predicting a 360° point cloud of a scene surrounding the ego vehicle according to the self-supervised depth estimation model and the ego-motion estimation model. The method also includes scaling the 360° point cloud according to the single-scale correction factor to form an aligned 360° point cloud.

Abstract: One or more embodiments of the present disclosure include systems and methods that use neural architecture fusion to learn how to combine multiple separate pre-trained networks by fusing their architectures into a single network for better computational efficiency and higher accuracy. For example, a computer implemented method of the disclosure includes obtaining multiple trained networks. Each of the trained networks may be associated with a respective task and has a respective architecture. The method further includes generating a directed acyclic graph that represents at least a partial union of the architectures of the trained networks. The method additionally includes defining a joint objective for the directed acyclic graph that combines a performance term and a distillation term. The method also includes optimizing the joint objective over the directed acyclic graph.

Abstract: A method for 3D object detection is described. The method includes predicting, using a trained monocular depth network, an estimated monocular input depth map of a monocular image of a video stream and an estimated depth uncertainty map associated with the estimated monocular input depth map. The method also includes feeding back a depth uncertainty regression loss associated with the estimated monocular input depth map during training of the trained monocular depth network to update the estimated monocular input depth map. The method further includes detecting 3D objects from a 3D point cloud computed from the estimated monocular input depth map based on seed positions selected from the 3D point cloud and the estimated depth uncertainty map. The method also includes selecting 3D bounding boxes of the 3D objects detected from the 3D point cloud based on the seed positions and an aggregated depth uncertainty.

Abstract: A method for 3D object detection is described. The method includes concurrently training a monocular depth network and a 3D object detection network. The method also includes predicting, using a trained monocular depth network, a monocular depth map of a monocular image of a video stream. The method further includes inferring a 3D point cloud of a 3D object within the monocular image according to the predicted monocular depth map. The method also includes predicting 3D bounding boxes from a selection of 3D points from the 3D point cloud of the 3D object based on a selection regression loss.