Abstract: Systems and methods for predicting vehicle behavior includes capturing images of a vehicle in traffic using an imaging device. Future behavior of the vehicle is stochastically modeled using a processing device including an energy-based model stored in a memory of the processing device. The energy-based model includes generating a distribution of possible future trajectories of the vehicle using a generator, sampling the distribution of possible future trajectories according to an energy value of each trajectory in the distribution of possible future trajectories an energy model to determine probable future trajectories, and optimizing parameters of each of the generator and the energy model using an optimizer. A user is audibly alerted with a speaker upon an alert system recognizing hazardous trajectories of the probable future trajectories.

Abstract: Systems and methods for predicting vehicle behavior includes capturing images of a vehicle in traffic using an imaging device. Future behavior of the vehicle is stochastically modeled using a processing device including an energy-based model stored in a memory of the processing device. The energy-based model includes generating a distribution of possible future trajectories of the vehicle using a generator, sampling the distribution of possible future trajectories according to an energy value of each trajectory in the distribution of possible future trajectories an energy model to determine probable future trajectories, and optimizing parameters of each of the generator and the energy model using an optimizer. A user is audibly alerted with a speaker upon an alert system recognizing hazardous trajectories of the probable future trajectories.

Abstract: Systems and methods for generating an occlusion-aware bird's eye view map of a road scene include identifying foreground objects and background objects in an input image to extract foreground features and background features corresponding to the foreground objects and the background objects, respectively. The foreground objects are masked from the input image with a mask. Occluded objects and depths of the occluded objects are inferred by predicting semantic features and depths in masked areas of the masked image according to contextual information related to the background features visible in the masked image. The foreground objects and the background objects are mapped to a three-dimensional space according to locations of each of the foreground objects, the background objects and occluded objects using the inferred depths. A bird's eye view is generated from the three-dimensional space and displayed with a display device.

Abstract: Systems and methods for generating an occlusion-aware bird's eye view map of a road scene include identifying foreground objects and background objects in an input image to extract foreground features and background features corresponding to the foreground objects and the background objects, respectively. The foreground objects are masked from the input image with a mask. Occluded objects and depths of the occluded objects are inferred by predicting semantic features and depths in masked areas of the masked image according to contextual information related to the background features visible in the masked image. The foreground objects and the background objects are mapped to a three-dimensional space according to locations of each of the foreground objects, the background objects and occluded objects using the inferred depths. A bird's eye view is generated from the three-dimensional space and displayed with a display device.

Abstract: Methods and systems for data segmentation include determining initial segmentation scores for each unit of an input data set using a neural network, with each unit being assigned an initial segmentation score for each of multiple segmentation classes. Final segmentation scores are determined for each unit of the input data set by enforcing a smoothness criterion. The input data set is segmented in accordance with the final segmentation scores.

Abstract: Systems and methods for correspondence estimation and flexible ground modeling include communicating two-dimensional (2D) images of an environment to a correspondence estimation module, including a first image and a second image captured by an image capturing device. First features, including geometric features and semantic features, are hierarchically extract from the first image with a first convolutional neural network (CNN) according to activation map weights, and second features, including geometric features and semantic features, are hierarchically extracted from the second image with a second CNN according to the activation map weights. Correspondences between the first features and the second features are estimated, including hierarchical fusing of geometric correspondences and semantic correspondences. A 3-dimensional (3D) model of a terrain is estimated using the estimated correspondences belonging to the terrain surface.

Abstract: Systems and methods for detecting traffic scenarios include an image capturing device which captures two or more images of an area of a traffic environment with each image having a different view of vehicles and a road in the traffic environment. A hierarchical feature extractor concurrently extracts features at multiple neural network layers from each of the images, with the features including geometric features and semantic features, and for estimating correspondences between semantic features for each of the images and refining the estimated correspondences with correspondences between the geometric features of each of the images to generate refined correspondence estimates. A traffic localization module uses the refined correspondence estimates to determine locations of vehicles in the environment in three dimensions to automatically determine a traffic scenario according to the locations of vehicles. A notification device generates a notification of the traffic scenario.

Abstract: A surveillance system and method are provided. The surveillance system includes at least one camera configured to capture a set of images of a given target area that includes a set of objects to be tracked. The surveillance system includes a memory storing a learning model configured to perform multi-object tracking by jointly learning arbitrarily parameterized and differentiable cost functions for all variables in a linear program that associates object detections with bounding boxes to form trajectories. The surveillance system includes a processor configured to perform surveillance of the target area to (i) detect the objects and track locations of the objects by applying the learning model to the images in a surveillance task that uses the multi-object tracking, and (ii), provide a listing of the objects and their locations for surveillance task. A bi-level optimization is used to minimize a loss defined on a solution of the linear program.

Abstract: A multi-object tracking system and method are provided. The multi-object tracking system includes at least one camera configured to capture a set of input images of a set of objects to be tracked. The multi-object tracking system further includes a memory storing a learning model configured to perform multi-object tracking by jointly learning arbitrarily parameterized and differentiable cost functions for all variables in a linear program that associates object detections with bounding boxes to form trajectories. The multi-object tracking system also includes a processor configured to (i) detect the objects and track locations of the objects by applying the learning model to the set of input images in a multi-object tracking task, and (ii), provide a listing of the objects and the locations of the objects for the multi-object tracking task. A bi-level optimization is used to minimize a loss defined on a solution of the linear program.

Abstract: Systems and methods for training semantic segmentation. Embodiments of the present invention include predicting semantic labeling of each pixel in each of at least one training image using a semantic segmentation model. Further included is predicting semantic boundaries at boundary pixels of objects in the at least one training image using a semantic boundary model concurrently with predicting the semantic labeling. Also included is propagating sparse labels to every pixel in the at least one training image using the predicted semantic boundaries. Additionally, the embodiments include optimizing a loss function according the predicted semantic labeling and the propagated sparse labels to concurrently train the semantic segmentation model and the semantic boundary model to accurately and efficiently generate a learned semantic segmentation model from sparsely annotated training images.

Abstract: Methods and systems for predicting a trajectory include determining prediction samples for agents in a scene based on a past trajectory. The prediction samples are ranked according to a likelihood score that incorporates interactions between agents and semantic scene context. The prediction samples are iteratively refined using a regression function that accumulates scene context and agent interactions across iterations. A response activity is triggered when the prediction samples satisfy a predetermined condition.

Abstract: Methods and systems for data segmentation include determining initial segmentation scores for each unit of an input data set using a neural network, with each unit being assigned an initial segmentation score for each of multiple segmentation classes. Final segmentation scores are determined for each unit of the input data set by enforcing a smoothness criterion. The input data set is segmented in accordance with the final segmentation scores.

Abstract: Methods and systems for monitoring a video feed include capturing input data representing a monitored area using one or more cameras to produce an input data set comprising individual images. Initial segmentation scores are determined for each image of the input data set using a neural network, with each image being assigned an initial segmentation score for each of multiple segmentation classes. Final segmentation scores are determined for each image of the input data set by enforcing a smoothness criterion. The input data set is segmented in accordance with the final segmentation scores. It is determined whether an alert condition is met based on segmented data set. An alert is generated if the alert condition is met.