Abstract: A method for keypoint matching includes determining a first set of keypoints corresponding to a current environment of the agent. The method further includes determining a second set of keypoints from a pre-built map of the current environment. The method still further includes identifying matching pairs of keypoints from the first set of keypoints and the second set of keypoints based on geometrical similarities between respective keypoints of the first set of keypoints and the second set of keypoints. The method also includes determining a current location of the agent based on the identified matching pairs of keypoints. The method further includes controlling an action of the agent based on the current location.

Abstract: A method for localization performed by an agent includes receiving a query image of a current environment of the agent captured by a sensor integrated with the agent. The method also includes receiving a target image comprising a first set of keypoints matching a second set of keypoints of the query image. The first set of keypoints may be generated based on a task specified for the agent. The method still further includes determining a current location based on the target image.

Abstract: A method for keypoint matching includes receiving an input image obtained by a sensor of an agent. The method also includes identifying a set of keypoints of the received image. The method further includes augmenting the descriptor of each of the keypoints with semantic information of the input image. The method also includes identifying a target image based on one or more semantically augmented descriptors of the target image matching one or more semantically augmented descriptors of the input image. The method further includes controlling an action of the agent in response to identifying the target.

Abstract: One or more embodiments of the disclosure include systems and methods that use meta-learning to learn how to optimally find a new neural network architecture for a task using past architectures that were optimized for other tasks, including for example tasks associated with autonomous, semi-autonomous, assisted, or other driving applications. A computer implemented method of the disclosure includes configuring a search space lattice comprising nodes representing operator choices, edges, and a maximum depth. The method includes defining an objective function. The method further includes configuring a graph network over the search space lattice to predict edge weights over the search space lattice.

Abstract: A method for depth estimation performed by a depth estimation system of an autonomous agent includes determining a first pose of a sensor based on a first image captured by the sensor and a second image captured by the sensor. The method also includes determining a first depth of the first image and a second depth of the second image. The method further includes generating a warped depth image based on at least the first depth and the first pose. The method still further includes determining a second pose based on the warped depth image and the second depth image. The method also includes updating the first pose based on the second pose and updating a first warped image based on the updated first pose.

Abstract: System, methods, and other embodiments described herein relate to estimating ego-motion. In one embodiment, a method for estimating ego-motion based on a plurality of input images in a self-supervised system includes receiving a source image and a target image, determining a depth estimation Dt based on the target image, determining a depth estimation Ds based on a source image, and determining an ego-motion estimation in a form of a six degrees-of-freedom (6 DOF) transformation between the target image and the source image by inputting the depth estimations (Dt, Ds), the target image, and the source image into a two-stream network architecture trained to output the 6 DOF transformation based at least in part on the depth estimations (Dt, Ds), the target image, and the source image.

Abstract: System, methods, and other embodiments described herein relate to generating depth estimates of an environment depicted in a monocular image. In one embodiment, a method includes, in response to receiving the monocular image, processing the monocular image according to a depth model to generate a depth map. Processing the monocular images includes encoding the monocular image according to encoding layers of the depth model including iteratively encoding features of the monocular image to generate feature maps at successively refined representations using packing blocks within the encoding layers. Processing the monocular image further includes decoding the feature maps according to decoding layers of the depth model including iteratively decoding the features maps associated with separate ones of the packing blocks using unpacking blocks of the decoding layers to generate the depth map. The method includes providing the depth map as the depth estimates of objects represented in the monocular image.

Abstract: System, methods, and other embodiments described herein relate to training a depth model for monocular depth estimation. In one embodiment, a method includes generating, as part of training the depth model according to a supervised training stage, a depth map from a first image of a pair of training images using the depth model. The pair of training images are separate frames depicting a scene from a monocular video. The method includes generating a transformation from the first image and a second image of the pair using a pose model. The method includes computing a supervised loss based, at least in part, on reprojecting the depth map and training depth data onto an image space of the second image according to at least the transformation. The method includes updating the depth model and the pose model according to at least the supervised loss.

Abstract: A model can be trained to detect interactions of other drivers through a window of their vehicle. A human driver behind a window (e.g., front windshield) of a vehicle can be detected in a real-world driving data. The human driver can be tracked over time through the window. The real-world driving data can be augmented by replacing at least a portion of the human driver with at least a portion of a virtual driver performing a target driver interaction to generate an augmented real-world driving dataset. The target driver interaction can be a gesture or a gaze. Using the augmented real-world driving data set, a machine learning model can be trained to detect the target driver interactions. Thus, simulation can be leveraged to provide a large set of useful training data without having to acquire real-world data of drivers performing target driver interactions as viewed from outside the vehicle.

Abstract: A method for scene perception using video captioning based on a spatio-temporal graph model is described. The method includes decomposing the spatio-temporal graph model of a scene in input video into a spatial graph and a temporal graph. The method also includes modeling a two branch framework having an object branch and a scene branch according to the spatial graph and the temporal graph to learn object interactions between the object branch and the scene branch. The method further includes transferring the learned object interactions from the object branch to the scene branch as privileged information. The method also includes captioning the scene by aligning language logits from the object branch and the scene branch according to the learned object interactions.

Abstract: A system for trajectory prediction using a predicted endpoint conditioned network includes one or more processors and a memory that includes a sensor input module, an endpoint distribution module, and a future trajectory module. The modules cause the one or more processors to the one or more processors to obtain sensor data of a scene having a plurality of pedestrians, determine endpoint distributions of the plurality of pedestrians within the scene, the endpoint distributions representing desired end destinations of the plurality of pedestrians from the scene, and determine future trajectory points for at least one of the plurality of pedestrians based on prior trajectory points of the plurality of pedestrians and the endpoint distributions of the plurality of pedestrians. The future trajectory points may be conditioned not only on the pedestrian and their immediate neighbors' histories (observed trajectories) but also on all the other pedestrian's estimated endpoints.

Abstract: System, methods, and other embodiments described herein relate to improving depth estimates for monocular images using a neural camera model that is independent of a camera type. In one embodiment, a method includes receiving a monocular image from a pair of training images derived from a monocular video. The method includes generating, using a ray surface network, a ray surface that approximates an image character of the monocular image as produced by a camera having the camera type. The method includes creating a synthesized image according to at least the ray surface and a depth map associated with the monocular image.

Abstract: System, methods, and other embodiments described herein relate to generating depth estimates from a monocular image. In one embodiment, a method includes, in response to receiving the monocular image, flipping, by a disparity model, the monocular image to generate a flipped image. The disparity model is a machine learning algorithm. The method includes analyzing, using the disparity model, the monocular image and the flipped image to generate disparity maps including a monocular disparity map corresponding to the monocular image and a flipped disparity map corresponding with the flipped image. The method includes generating, in the disparity model, a fused disparity map from the monocular disparity map and the flipped disparity map. The method includes providing the fused disparity map as the depth estimates of objects represented in the monocular image.

Abstract: A system and related method for predicting movement of a plurality of pedestrians may include one or more processors and a memory. The memory includes an initial trajectory module, an exit point prediction module, a path planning module, and an adjustment module. The modules include instructions that when executed by the one or more processors cause the one or more processors to obtain trajectories of the plurality of pedestrians, predict future exit points for the plurality of pedestrians from a scene based on the trajectories of the plurality of pedestrians, determine trajectory paths of the plurality of pedestrians based on the future exit points and at least one scene element of a map, and adjust the trajectory paths based on at least one predicted interaction between at least two of the plurality of pedestrians.

Abstract: A method for learning depth-aware keypoints and associated descriptors from monocular video for monocular visual odometry is described. The method includes training a keypoint network and a depth network to learn depth-aware keypoints and the associated descriptors. The training is based on a target image and a context image from successive images of the monocular video. The method also includes lifting 2D keypoints from the target image to learn 3D keypoints based on a learned depth map from the depth network. The method further includes estimating a trajectory of an ego-vehicle based on the learned 3D keypoints.

Abstract: A method for learning depth-aware keypoints and associated descriptors from monocular video for ego-motion estimation is described. The method includes training a keypoint network and a depth network to learn depth-aware keypoints and the associated descriptors. The training is based on a target image and a context image from successive images of the monocular video. The method also includes lifting 2D keypoints from the target image to learn 3D keypoints based on a learned depth map from the depth network. The method further includes estimating ego-motion from the target image to the context image based on the learned 3D keypoints.

Abstract: Systems and methods for panoptic image segmentation are disclosed herein. One embodiment performs semantic segmentation and object detection on an input image, wherein the object detection generates a plurality of bounding boxes associated with an object in the input image; selects a query bounding box from among the plurality of bounding boxes; maps at least one of the bounding boxes in the plurality of bounding boxes other than the query bounding box to the query bounding box based on similarity between the at least one of the bounding boxes and the query bounding box to generate a mask assignment for the object, the mask assignment defining a contour of the object; compares the mask assignment with results of the semantic segmentation to produce a refined mask assignment for the object; and outputs a panoptic segmentation of the input image that includes the refined mask assignment for the object.

Abstract: A method for 3D auto-labeling of objects with predetermined structural and physical constraints includes identifying initial object-seeds for all frames from a given frame sequence of a scene. The method also includes refining each of the initial object-seeds over the 2D/3D data, while complying with the predetermined structural and physical constraints to auto-label 3D object vehicles within the scene. The method further includes linking the auto-label 3D object vehicles over time into trajectories while respecting the predetermined structural and physical constraints.

Abstract: System, methods, and other embodiments described herein relate to self-supervised training of a depth model for monocular depth estimation. In one embodiment, a method includes processing a first image of a pair according to the depth model to generate a depth map. The method includes processing the first image and a second image of the pair according to a pose model to generate a transformation that defines a relationship between the pair. The pair of images are separate frames depicting a scene of a monocular video. The method includes generating a monocular loss and a pose loss, the pose loss including at least a velocity component that accounts for motion of a camera between the training images. The method includes updating the pose model according to the pose loss and the depth model according to the monocular loss to improve scale awareness of the depth model in producing depth estimates.

Abstract: System, methods, and other embodiments described herein relate to generating depth estimates of an environment depicted in a monocular image. In one embodiment, a method includes, in response to receiving the monocular image, processing the monocular image according to a depth model to generate a depth map. Processing the monocular images includes encoding the monocular image according to encoding layers of the depth model including iteratively encoding features of the monocular image to generate feature maps at successively refined representations using packing blocks within the encoding layers. Processing the monocular image further includes decoding the feature maps according to decoding layers of the depth model including iteratively decoding the features maps associated with separate ones of the packing blocks using unpacking blocks of the decoding layers to generate the depth map. The method includes providing the depth map as the depth estimates of objects represented in the monocular image.