Abstract: A method comprises receiving training data comprising a plurality of images containing one or more objects, a plurality of depth maps associated with the plurality of images, and ground truth data associated with the plurality of images, the ground truth data comprising shapes and grasp poses associated with the one or more objects in the plurality of images, and training a machine learning model, using the training data, to receive a first image containing one or more first objects and a first depth map associated with the first image, and output first shapes of the one or more first objects and first grasp poses for the one or more first objects. The machine learning model comprises a conditional variational autoencoder, a multi-object encoder to encode multi-object reasoning associated with an object, and 3D occlusion fields determined by ray casting.

Abstract: Described herein are embodiments for machine-learning by generating pseudo-labels for unlabeled training data using multiple temporal directions. Examples capturing a video of an environment surrounding a vehicle, the video comprising a sequence of image frames, detecting a first set of objects in the environment by applying the sequence of image frames to a 3-dimensional (3D) object detector in a first temporal direction, and detecting a second set of objects in the environment by applying the sequence of image frames to the 3D detector in a second temporal direction that differs from the first temporal direction. Examples also include generating a pseudo-label for the video based on the first and second set of objects and training the 3D object detector based on the generated pseudo-label.

Abstract: Described herein are embodiments for continual learning (CL), and more particularly data sampling techniques for CL. Examples include obtaining first data samples, transforming the first data samples to generate second data samples, and creating a plurality of candidates that comprise a plurality of subsets of the first and second data samples. A plurality of pseudo-updated models can be generated from the plurality of candidates by applying the plurality of subsets of the first and second data samples to a CL model. A candidate of the plurality of candidates can be selected based on the plurality of pseudo-updated models. A subset of the first and second data samples corresponding to the selected candidate to a data store can be stored and the CL model can be trained by sampling the subset of the first and second data samples from the data store.

Abstract: A method and system for generating virtual pedestrian-vehicle interaction data includes generating a virtual reality environment in virtual reality device, generating a scenario in the virtual reality environment, the scenario comprising virtual vehicle movements, displaying the scenario in a virtual reality device, storing virtual reality movements relative to the scenario, the virtual reality movements comprising at least a yaw movement, communicating the virtual vehicle movements to a simulator controller, communicating the virtual vehicle movements to the simulator controller, associating the virtual reality movements, the virtual vehicle movements and the scenario to form pedestrian-vehicle data, and training an autonomous vehicle system using the pedestrian-vehicle data.

Abstract: A method may include receiving training data comprising a plurality of RGB-D images of an object at a plurality of time steps, and a plurality of robot actions associated with the object at the plurality of time steps; and optimizing, using the training data, a dynamics function to predict a future state of the object based on a current state of the object and a robot action. A state of the object is estimated as a plurality of particles comprising 3D Gaussians using particle filtering.

Abstract: Systems, methods, and other embodiments described herein relate to improving radar data. In one embodiment, a method includes, responsive to acquiring radar data from a radar sensor, transforming the radar data into improved data according to a super-resolution model. The method includes converting the improved data into a range-azimuth-Doppler map. The method includes generating a high-resolution map from the range-azimuth-Doppler map by applying the super-resolution model to the range-azimuth-Doppler map. The method includes providing the high-resolution map.

Abstract: A method includes generating a radar-based intensity map and a lidar-based intensity map and performing one or more augmentation routines on the radar-based intensity map and the lidar-based intensity map to generate a radar input and a lidar input. The method includes generating a plurality of teacher-based bounding boxes and a plurality of student-based bounding boxes based on the radar input and the lidar input. The method includes determining a loss value of the plurality of student-based bounding boxes based on the plurality of teacher-based bounding boxes and a plurality of ground truth bounding boxes, updating one or more weights of the student neural network based on the loss value, and updating one or more weights of the teacher neural network based on a moving average associated with the one or more weights of the student neural network.

Abstract: A method includes generating a plurality of lidar inputs based on the lidar data, where each lidar input from among the plurality of lidar inputs comprises an image-based portion and a geometric-based portion, and where each lidar input from among the plurality of lidar inputs defines a position coordinate of the one or more objects. The method includes performing, for each lidar input from among the plurality of lidar inputs, a convolutional neural network (CNN) routine based on the image-based portion to generate one or more image-based outputs and assigning the plurality of lidar inputs to a plurality of echo groups based on the geometric-based portion. The method includes concatenating the one or more image-based outputs and the plurality of echo groups to generate a plurality of fused outputs and identifying the one or more objects based on the plurality of fused outputs.

Abstract: Systems, methods, and other embodiments described herein relate improve the estimation of poses within an environment including multiple people and uncalibrated cameras. In one embodiment, a method includes acquiring sensor data including images with depth information of a surrounding environment that includes multiple people. The method includes determining 2D poses and 3D features for the people according to the sensor data. The method includes generating camera poses using at least the depth information and the features for cameras that generated the images. The method includes generating 3D poses for the people according to the camera poses and the 3D features. The method includes providing the 3D poses of the people.

Abstract: Systems, methods, and other embodiments described herein relate to a deep learning approach for depth completion according to variable depth inputs. In one embodiment, a method includes acquiring sensor data including at least an image of a surrounding environment. The method includes encoding the sensor data into features using an encoder of a depth model. The method includes decoding the features into a depth map using a decoder of the depth model according to an affinity-based shift correction embedded with the decoder. The method includes providing the depth map that indicates depths within the surrounding environment.

Abstract: A method of generating virtual sensor data of a virtual single-photon avalanche diode (SPAD) lidar sensor includes generating a two-dimensional (2D) lidar array having a plurality of cells. The method further includes interpolating image data from a virtual camera with the 2D lidar array to define auxiliary image data, generating a virtual ambient image based on a red-channel (R-channel) data of the auxiliary image data, identifying a plurality of virtual echoes of the virtual SPAD lidar sensor based on the R-channel data and a defined photon threshold, defining a virtual point cloud indicative of virtual photon measurements of the virtual SPAD lidar sensor based on the plurality of virtual echoes, and outputting data indicative of the virtual ambient image, the virtual photon measurements, the virtual point cloud, or a combination thereof, as the virtual sensor data of the virtual SPAD lidar sensor.

Abstract: A method includes generating a plurality of lidar inputs based on the lidar data, where each lidar input from among the plurality of lidar inputs comprises an image-based portion and a geometric-based portion, and where each lidar input from among the plurality of lidar inputs defines a position coordinate of the one or more objects. The method includes performing, for each lidar input from among the plurality of lidar inputs, a convolutional neural network (CNN) routine based on the image-based portion to generate one or more image-based outputs and assigning the plurality of lidar inputs to a plurality of echo groups based on the geometric-based portion. The method includes concatenating the one or more image-based outputs and the plurality of echo groups to generate a plurality of fused outputs and identifying the one or more objects based on the plurality of fused outputs.

Abstract: A method of generating virtual sensor data of a virtual single-photon avalanche diode (SPAD) lidar sensor includes generating a two-dimensional (2D) lidar array having a plurality of cells. The method further includes interpolating image data from a virtual camera with the 2D lidar array to define auxiliary image data, generating a virtual ambient image based on a red-channel (R-channel) data of the auxiliary image data, identifying a plurality of virtual echoes of the virtual SPAD lidar sensor based on the R-channel data and a defined photon threshold, defining a virtual point cloud indicative of virtual photon measurements of the virtual SPAD lidar sensor based on the plurality of virtual echoes, and outputting data indicative of the virtual ambient image, the virtual photon measurements, the virtual point cloud, or a combination thereof, as the virtual sensor data of the virtual SPAD lidar sensor.

Abstract: In one embodiment, a method includes receiving sensor data. The sensor data is based on information from a first set of echo points and a second set of echo points. At least one echo point from the first set of echo points and one echo point from the second set of echo points originate from a single beam. The method includes generating a first set of feature maps based on the first set of echo points and a second set of feature maps based on the second set of echo points. The method includes predicting a bounding box for the object based on the first set of feature maps and the second set of feature maps.

Abstract: Systems, methods, and other embodiments described herein relate to generating bounding box proposals. In one embodiment, a method includes generating blended 2-dimensional (2D) data based on 2D data and 3-dimensional (3D) data, and generating blended 3D data based on the 2D data and the 3D data. The method includes generating 2D features based on the 2D data and the blended 2D data, generating 3D features based on the 3D data and the blended 3D data, and generating the bounding box proposals based on the 2D features and the 3D features.