Abstract: A method for controlling an ego agent includes capturing a two-dimensional (2D) image of an environment adjacent to the ego agent. The method also includes generating a semantically segmented image of the environment based on the 2D image. The method further includes generating a depth map of the environment based on the semantically segmented image. The method additionally includes generating a three-dimensional (3D) estimate of the environment based on the depth map. The method also includes controlling an action of the ego agent based on the identified location.

Abstract: Systems and methods are provided for developing/updating training datasets for traffic light detection/perception models. V2I-based information may indicate a particular traffic light state/state of transition. This information can be compared to a traffic light perception prediction. When the prediction is inconsistent with the V2I-based information, data regarding the condition(s)/traffic light(s)/etc. can be saved and uploaded to a training database to update/refine the training dataset(s) maintained therein. In this way, an existing traffic light perception model can be updated/improved and/or a better traffic light perception model can be developed.

Abstract: A method for 3D object tracking is described. The method includes inferring first 2D semantic keypoints of a 3D object within a sparsely annotated video stream. The method also includes matching the first 2D semantic keypoints of a current frame with second 2D semantic keypoints in a next frame of the sparsely annotated video stream using embedded descriptors within the current frame and the next frame. The method further includes warping the first 2D semantic keypoints to the second 2D semantic keypoints to form warped 2D semantic keypoints in the next frame. The method also includes labeling a 3D bounding box in the next frame according to the warped 2D semantic keypoints in the next frame.

Abstract: Systems and methods to improve machine learning by explicitly over-fitting environmental data obtained by an imaging system, such as a monocular camera are disclosed. The system includes training self-supervised depth and pose networks in monocular visual data collected from a certain area over multiple passes. Pose and depth networks may be trained by extracting data from multiple images of a single environment or trajectory, allowing the system to overfit the image data.

Abstract: A method for learning depth-aware keypoints and associated descriptors from monocular video for ego-motion estimation is described. The method includes training a keypoint network and a depth network to learn depth-aware keypoints and the associated descriptors. The training is based on a target image and a context image from successive images of the monocular video. The method also includes lifting 2D keypoints from the target image to learn 3D keypoints based on a learned depth map from the depth network. The method further includes estimating ego-motion from the target image to the context image based on the learned 3D keypoints.

Abstract: Information that identifies a location can be received. In response to a receipt of the information that identifies the location, a file can be retrieved. The file can be for the location. The file can include image data and a set of node data. The set of node data can include information that identifies nodes in a neural network, information that identifies inputs of the nodes, and values of weights to be applied to the inputs. In response to a retrieval of the file, the weights can be applied to the inputs of the nodes and the image data can be received for the neural network. In response to an application of the weights and a receipt of the image data, the neural network can be executed to produce a digital map for the location. The digital map for the location can be transmitted to an automotive navigation system.

Abstract: A method for 3D object detection is described. The method includes detecting semantic keypoints from monocular images of a video stream capturing a 3D object. The method also includes inferring a 3D bounding box of the 3D object corresponding to the detected semantic vehicle keypoints. The method further includes scoring the inferred 3D bounding box of the 3D object. The method also includes detecting the 3D object according to a final 3D bounding box generated based on the scoring of the inferred 3D bounding box.

Abstract: Described herein are systems and methods for determining if a vehicle is parked. In one example, a system includes a processor, a sensor system, and a memory. Both the sensor system and the memory are in communication with the processor. The memory includes a parking determination module having instructions that, when executed by the processor, cause the processor to determine, using a random forest model, when the vehicle is parked based on vehicle estimated features, vehicle learned features, and vehicle taillight features of the vehicle that are based on sensor data from the sensor system.

Abstract: A representation of a spatial structure of objects in an image can be determined. A mode of a neural network can be set, in response to a receipt of the image and a receipt of a facing direction of a camera that produced the image. The mode can account for the facing direction. The facing direction can include one or more of a first facing direction of a first camera disposed on a vehicle or a second facing direction of a second camera disposed on the vehicle. The neural network can be executed, in response to the mode having been set, to determine the representation of the spatial structure of the objects in the image. The representation of the spatial structure of the objects in the image can be transmitted to an automotive navigation system to determine a distance between the vehicle and a specific object in the image.

Abstract: A traffic light classification system for a vehicle includes an image capture device to capture an image of a scene that includes a traffic light with multiple light signals, a processor, and a memory communicably coupled to the processor and storing a first neural network module including instructions that when executed by the processor cause the processor to determine, based on inputting the image into a neural network, a semantic keypoint for each light signal in the traffic light, and determine, based on each semantic keypoint, a classification state of each light signal.

Abstract: A method for keypoint matching includes determining a first set of keypoints corresponding to a current environment of the agent. The method further includes determining a second set of keypoints from a pre-built map of the current environment. The method still further includes identifying matching pairs of keypoints from the first set of keypoints and the second set of keypoints based on geometrical similarities between respective keypoints of the first set of keypoints and the second set of keypoints. The method also includes determining a current location of the agent based on the identified matching pairs of keypoints. The method further includes controlling an action of the agent based on the current location.

Abstract: A method for 3D object modeling includes linking 2D semantic keypoints of an object within a video stream into a 2D structured object geometry. The method includes inputting, to a neural network, the object to generate a 2D NOCS image and a shape vector, the shape vector being mapped to a continuously traversable coordinate shape. The method includes applying a differentiable shape renderer to the SDF shape and the 2D NOCS image to render a shape of the object corresponding to a 3D object model in the continuously traversable coordinate shape space. The method includes lifting the linked, 2D semantic keypoints of the 2D structured object geometry to a 3D structured object geometry. The method includes geometrically and projectively aligning the 3D object model, the 3D structured object geometry, and the rendered shape to form a rendered object. The method includes generating 3D bounding boxes from the rendered object.

Abstract: A method for traffic light auto-labeling includes aggregating vehicle-to-infrastructure (V2I) traffic light signals at an intersection to determine transition states of each driving lane at the intersection during operation of an ego vehicle. The method also includes automatically labeling image training data to form auto-labeled image training data for a traffic light recognition model within the ego vehicle according to the determined transition states of each driving lane at the intersection. The method further includes planning a trajectory of the ego vehicle to comply with a right-of-way according to the determined transition states of each driving lane at the intersection according to a trained traffic light detection model. A federated learning module may train the traffic light recognition model using the auto-labeled image training data during the operation of the ego vehicle.

Abstract: A method for monocular depth/pose estimation in a camera agnostic network is described. The method includes projecting lifted 3D points onto an image plane according to a predicted ray vector based on a monocular depth model, a monocular pose model, and a camera center of a camera agnostic network. The method also includes predicting a warped target image from a predicted depth map of the monocular depth model, a ray surface of the predicted ray vector, and a projection of the lifted 3D points according to the camera agnostic network.

Abstract: System, methods, and other embodiments described herein relate to improving depth estimates for monocular images using a neural camera model that is independent of a camera type. In one embodiment, a method includes receiving a monocular image from a pair of training images derived from a monocular video. The method includes generating, using a ray surface network, a ray surface that approximates an image character of the monocular image as produced by a camera having the camera type. The method includes creating a synthesized image according to at least the ray surface and a depth map associated with the monocular image.

Abstract: Systems and methods for dynamically selecting idle or underutilized resources to complete tasks in a queue are disclosed. The systems and methods include maintaining a plurality of processing resources operable to process one or more tasks. Each resource is scalable to increase or decrease a number of nodes available to perform the one or more tasks. The systems and methods include maintaining a queue for tasks to be processed, receiving a first task requiring a processing resource, and accessing at least a portion of the plurality of processing resources. A first processing resource of the plurality of processing resources is identified that is operating below a predetermined processing threshold. The first processing resource is assigned to the first task and scaled up according according to a processing requirement of the first task.

Abstract: Systems and methods determining velocity of an object associated with a three-dimensional (3D) scene may include: a LIDAR system generating two sets of 3D point cloud data of the scene from two consecutive point cloud sweeps; a pillar feature network encoding data of the point cloud data to extract two-dimensional (2D) bird's-eye-view embeddings for each of the point cloud data sets in the form of pseudo images, wherein the 2D bird's-eye-view embeddings for a first of the two point cloud data sets comprises pillar features for the first point cloud data set and the 2D bird's-eye-view embeddings for a second of the two point cloud data sets comprises pillar features for the second point cloud data set; and a feature pyramid network encoding the pillar features and performing a 2D optical flow estimation to estimate the velocity of the object.

Abstract: Systems and methods for map construction using a video sequence captured on a camera of a vehicle in an environment, comprising: receiving a video sequence from the camera, the video sequence including a plurality of image frames capturing a scene of the environment of the vehicle; using a neural camera model to predict a depth map and a ray surface for the plurality of image frames in the received video sequence; and constructing a map of the scene of the environment based on image data captured in the plurality of frames and depth information in the predicted depth maps.

Abstract: Systems, methods, computer-readable media, techniques, and methodologies are disclosed for performing end-to-end, learning-based keypoint detection and association. A scene graph of a signalized intersection is constructed from an input image of the intersection. The scene graph includes detected keypoints and linkages identified between the keypoints. The scene graph can be used along with a vehicle's localization information to identify which keypoint that represents a traffic signal is associated with the vehicle's current travel lane. An appropriate vehicle action may then be determined based on a transition state of the traffic signal keypoint and trajectory information for the vehicle. A control signal indicative of this vehicle action may then be output to cause an autonomous vehicle, for example, to implement the appropriate vehicle action.

Abstract: A method for 2D semantic keypoint detection and tracking is described. The method includes learning embedded descriptors of salient object keypoints detected in previous images according to a descriptor embedding space model. The method also includes predicting, using a shared image encoder backbone, salient object keypoints within a current image of a video stream. The method further includes inferring an object represented by the predicted, salient object keypoints within the current image of the video stream. The method also includes tracking the inferred object by matching embedded descriptors of the predicted, salient object keypoints representing the inferred object within the previous images of the video stream based on the descriptor embedding space model.