Abstract: System, methods, and other embodiments described herein relate to determining depths of a scene from a monocular image. In one embodiment, a method includes generating depth features from sensor data according to whether the sensor data includes sparse depth data. The method includes selectively injecting the depth features into a depth model. The method includes generating a depth map from at least a monocular image using the depth model that is guided by the depth features when injected. The method includes providing the depth map as depth estimates of objects represented in the monocular image.

Abstract: Systems and methods described herein relate to controlling a robot. One embodiment receives an initial state of the robot, an initial nominal control trajectory of the robot, and a Kullback-Leibler (KL) divergence bound between a modeled probability distribution for a stochastic disturbance and an unknown actual probability distribution for the stochastic disturbance; solves a bilevel optimization problem subject to the modeled probability distribution and the KL divergence bound using an iterative Linear-Exponential-Quadratic-Gaussian (iLEQG) algorithm and a cross-entropy process, the iLEQG algorithm outputting an updated nominal control trajectory, the cross-entropy process outputting a risk-sensitivity parameter; and controls operation of the robot based, at least in part, on the updated nominal control trajectory and the risk-sensitivity parameter.

Abstract: System, methods, and other embodiments described herein relate to determining depths of a scene from a monocular image. In one embodiment, a method includes generating depth features from depth data using a sparse auxiliary network (SAN) by i) sparsifying the depth data, ii) applying sparse residual blocks of the SAN to the depth data, and iii) densifying the depth features. The method includes generating a depth map from the depth features and a monocular image that corresponds with the depth data according to a depth model that includes the SAN. The method includes providing the depth map as depth estimates of objects represented in the monocular image.

Abstract: A system for producing a depth map can include a processor and a memory. The memory can store a neural network. The neural network can include an encoding portion module, a multi-frame feature matching portion module, and a decoding portion module. The encoding portion module can include instructions that, when executed by the processor, cause the processor to encode an image to produce single-frame features. The multi-frame feature matching portion module can include instructions that, when executed by the processor, cause the processor to process the single-frame features to produce information. The decoding portion module can include instructions that, when executed by the processor, cause the processor to decode the information to produce the depth map. A first training dataset, used to train the multi-frame feature matching portion module, can be different from a second training dataset used to train the encoding portion module and the decoding portion module.

Abstract: A system for producing a depth map can include a processor and a memory. The memory can store a candidate depth production module and a depth map production module. The candidate depth production module can include instructions that cause the processor to: (1) identify, in a first image, an epipolar line associated with a pixel in a second image and (2) sample, from a first image feature set, a set of candidate depths for pixels along the epipolar line. The depth map production module can include instructions that cause the processor to: (1) determine a similarity measure between a feature, from a second image feature set, and a member of the set and (2) produce, from the second image, the depth map with a depth for the pixel being a depth associated with a member, of the set, having a greatest similarity measure.

Abstract: System, methods, and other embodiments described herein relate to improving depth estimates for monocular images using a neural camera model that is independent of a camera type. In one embodiment, a method includes receiving a monocular image from a pair of training images derived from a monocular video. The method includes generating, using a ray surface network, a ray surface that approximates an image character of the monocular image as produced by a camera having the camera type. The method includes creating a synthesized image according to at least the ray surface and a depth map associated with the monocular image.

Abstract: A method for risk-aware game-theoretic trajectory planning is described. The method includes modeling an ego vehicle and at least one other vehicle as risk-aware agents in a game-theoretic driving environment. The method also includes ranking upcoming planned trajectories according to a risk-aware cost function of the ego vehicle and a risk-sensitivity of the other vehicle associated with each of the upcoming planned trajectories. The method further includes selecting a vehicle trajectory according to the ranking of the upcoming planned trajectories based on the risk-aware cost function and the risk-sensitivity of the other vehicle associated with each of the upcoming planned trajectories to reach a target destination according to a mission plan.

Abstract: A method for behavior cloned vehicle trajectory planning is described. The method includes perceiving vehicles proximate an ego vehicle in a driving environment, including a scalar confidence value of each perceived vehicle. The method also includes generating a bird's-eye-view (BEV) grid showing the ego vehicle and each perceived vehicle based on each of the scalar confidence values. The method further includes ignoring at least one of the perceived vehicles when the scalar confidence value of the at least one of the perceived vehicles is less than a predetermined value. The method also includes selecting an ego vehicle trajectory based on a cloned expert vehicle behavior policy according to remaining perceived vehicles.

Abstract: Systems and methods determining velocity of an object associated with a three-dimensional (3D) scene may include: a LIDAR system generating two sets of 3D point cloud data of the scene from two consecutive point cloud sweeps; a pillar feature network encoding data of the point cloud data to extract two-dimensional (2D) bird's-eye-view embeddings for each of the point cloud data sets in the form of pseudo images, wherein the 2D bird's-eye-view embeddings for a first of the two point cloud data sets comprises pillar features for the first point cloud data set and the 2D bird's-eye-view embeddings for a second of the two point cloud data sets comprises pillar features for the second point cloud data set; and a feature pyramid network encoding the pillar features and performing a 2D optical flow estimation to estimate the velocity of the object.

Abstract: A method for tracking an object performed by an object tracking system includes encoding locations of visible objects in an environment captured in a current frame of a sequence of frames. The method also includes generating a representation of a current state of the environment based on an aggregation of the encoded locations and an encoded location of each object visible in one or more frames of the sequence of frames occurring prior to the current frame. The method further includes predicting a location of an object occluded in the current frame based on a comparison of object centers decoded from the representation of the current state to object centers saved from each prior representation associated with a different respective frame of the sequence of frames occurring prior to the current frame. The method still further includes adjusting a behavior of an autonomous agent in response to identifying the location of the occluded object.

Abstract: Systems and methods described herein relate to training a machine-learning-based monocular depth estimator. One embodiment selects a virtual image in a virtual dataset, the virtual dataset including a plurality of computer-generated virtual images; generates, from the virtual image in accordance with virtual-camera intrinsics, a point cloud in three-dimensional space based on ground-truth depth information associated with the virtual image; reprojects the point cloud back to two-dimensional image space in accordance with real-world camera intrinsics to generate a transformed virtual image; and trains the machine-learning-based monocular depth estimator, at least in part, using the transformed virtual image.

Abstract: Systems and methods described herein relate to dynamics-aware comparison of reward functions. One embodiment generates a reference reward function; computes a dynamics-aware transformation of the reference reward function based on a transition model of an environment of a robot; computes a dynamics-aware transformation of a first candidate reward function based on the transition model; computes a dynamics-aware transformation of a second candidate reward function based on the transition model; selects, as a final reward function, the first or second candidate reward function based on which is closer to the reference reward function as measured by pseudometrics computed between their respective dynamics-aware transformations and the dynamics-aware transformation of the reference reward function; and optimizes the final reward function to control, at least in part, operation of the robot.

Abstract: Systems and methods for panoptic segmentation of an image of a scene, comprising: receiving a synthetic data set as simulation data set in a simulation domain, the simulation data set comprising a plurality of synthetic data objects; disentangling the synthetic data objects by class for a plurality of object classes; training each class of the plurality of classes separately by applying a Generative Adversarial Network (GAN) to each class from the data set in the simulation domain to create a generated instance for each class; combining the generated instances for each class with labels for the objects in each class to obtain a fake instance of an object; fusing the fake instances to create a fused image; and applying a GAN to the fused image and a corresponding real data set in a real-world domain to obtain an updated data set. The process can be repeated across multiple iterations.

Abstract: Systems and methods described herein relate to self-supervised learning of camera intrinsic parameters from a sequence of images. One embodiment produces a depth map from a current image frame captured by a camera; generates a point cloud from the depth map using a differentiable unprojection operation; produces a camera pose estimate from the current image frame and a context image frame; produces a warped point cloud based on the camera pose estimate; generates a warped image frame from the warped point cloud using a differentiable projection operation; compares the warped image frame with the context image frame to produce a self-supervised photometric loss; updates a set of estimated camera intrinsic parameters on a per-image-sequence basis using one or more gradients from the self-supervised photometric loss; and generates, based on a converged set of learned camera intrinsic parameters, a rectified image frame from an image frame captured by the camera.

Abstract: A method for scene perception using video captioning based on a spatio-temporal graph model is described. The method includes decomposing the spatio-temporal graph model of a scene in input video into a spatial graph and a temporal graph. The method also includes modeling a two branch framework having an object branch and a scene branch according to the spatial graph and the temporal graph to learn object interactions between the object branch and the scene branch. The method further includes transferring the learned object interactions from the object branch to the scene branch as privileged information. The method also includes captioning the scene by aligning language logits from the object branch and the scene branch according to the learned object interactions.

Abstract: A model can be trained to detect interactions of other drivers through a window of their vehicle. A human driver behind a window (e.g., front windshield) of a vehicle can be detected in a real-world driving data. The human driver can be tracked over time through the window. The real-world driving data can be augmented by replacing at least a portion of the human driver with at least a portion of a virtual driver performing a target driver interaction to generate an augmented real-world driving dataset. The target driver interaction can be a gesture or a gaze. Using the augmented real-world driving data set, a machine learning model can be trained to detect the target driver interactions. Thus, simulation can be leveraged to provide a large set of useful training data without having to acquire real-world data of drivers performing target driver interactions as viewed from outside the vehicle.

Abstract: One or more embodiments of the present disclosure include systems and methods that use neural architecture fusion to learn how to combine multiple separate pre-trained networks by fusing their architectures into a single network for better computational efficiency and higher accuracy. For example, a computer implemented method of the disclosure includes obtaining multiple trained networks. Each of the trained networks may be associated with a respective task and has a respective architecture. The method further includes generating a directed acyclic graph that represents at least a partial union of the architectures of the trained networks. The method additionally includes defining a joint objective for the directed acyclic graph that combines a performance term and a distillation term. The method also includes optimizing the joint objective over the directed acyclic graph.

Abstract: System, methods, and other embodiments described herein relate to training a depth model for joint depth completion and prediction. In one arrangement, a method includes generating depth features from sparse depth data according to a sparse auxiliary network (SAN) of a depth model. The method includes generating a first depth map from a monocular image and a second depth map from the monocular image and the depth features using the depth model. The method includes generating a depth loss from the second depth map and the sparse depth data and an image loss from the first depth map and the sparse depth data. The method includes updating the depth model including the SAN using the depth loss and the image loss.

Abstract: A method includes generating a first warped image based on a pose and a depth estimated from a current image and a previous image in a sequence of images captured by a camera of the agent. The method also includes estimating a motion of dynamic object between the previous image and the target image. The method further includes generating a second warped image from the first warped image based on the estimated motion. The method still further includes controlling an action of an agent based on the second warped image.

Abstract: A method of generating an output trajectory of an ego vehicle includes recording trajectory data of the ego vehicle and pedestrian agents from a scene of a training environment of the ego vehicle. The method includes identifying at least one pedestrian agent from the pedestrian agents within the scene of the training environment of the ego vehicle causing a prediction-discrepancy by the ego vehicle greater than the pedestrian agents within the scene. The method includes updating parameters of a motion prediction model of the ego vehicle based on a magnitude of the prediction-discrepancy caused by the at least one pedestrian agent on the ego vehicle to form a trained, control-aware prediction objective model. The method includes selecting a vehicle control action of the ego vehicle in response to a predicted motion from the trained, control-aware prediction objective model regarding detected pedestrian agents within a traffic environment of the ego vehicle.