Abstract: Systems and methods are provided for generating a statistical parameter representing a state of a surgical procedure from sensor data. Sensor data representing a time period. is received from a sensor. Numerical features representing the time period are generated from the sensor data. Each of a plurality of long short term memory units are updated according to the plurality of numerical features via a message passing process. The long short term memory units are connected to form a graph, with a first set of the long short term memory units representing a plurality of nodes of the graph and a second set of the long short term memory units representing a plurality of hyperedges of the graph. A statistical parameter representing a state of the surgical procedure for the time period is derived from an output of one of the long short term memory units and provided to a user.

Abstract: A method for indicating occlusion information at an ego agent includes observing a spatial area from a first viewpoint of one or more first sensors associated with the ego agent. The method also includes identifying the spatial area as an occluded area in accordance with observing the spatial area from a second viewpoint of the one or more first sensors after observing the spatial area from the first viewpoint. The method further includes receiving, from a target agent, a message indicating the spatial area is occluded from one or more second sensors associated with the target agent. The method still further includes transmitting, to the target agent in accordance with receiving the message, the occlusion information indicating information associated the spatial area based on identifying the spatial area as the occluded area.

Abstract: Systems and methods of model or prediction algorithm selection are provided. An autonomous control system may include a perception component that, based on environmental inputs regarding an object(s), a vehicle's operating characteristics, etc., outputs a current state of the vehicle's surrounding environment. This in turn, is used as input to a prediction component comprising a plurality of prediction algorithms. The prediction component outputs a set of predictions regarding the trajectory of the object(s). Accordingly, for each object, a set of trajectories at specific timesteps may be generated by the different prediction algorithms which are input to a planner component. These trajectories may then be analyzed, compared, or otherwise processed to determine which trajectory regarding the object is most accurate. The prediction algorithm or model that produced the most accurate predicted trajectory may then be used for subsequent predictions/timesteps.

Abstract: In one embodiment, a computer-implemented method for driver training using zone of proximal learning (ZPL) includes receiving, by one or more processors, driving data with respect to a driver operating a vehicle, estimating, using a personal behavior model, a driver profile based on the driving data, estimating one or more zone of proximal development (ZPD) states based at least in part on the driver profile, and performing one or more vehicle actions to place the driver into the one or more ZPD states.

Abstract: A system for training an operator of a vehicle to perform a maneuver of the vehicle can include a processor and a memory. The memory can store an automated motion module, an instruction module, and a communications module. The automated motion module can cause the processor to cause the vehicle to perform, in an automated manner, a first iteration of the maneuver. The instruction module can cause the processor to cause an instruction to be provided to the operator during a second iteration of the maneuver. The communications module can cause the processor to receive, during the second iteration, a query from the operator about a performance of the second iteration. The communications module can cause the processor to communicate a response to the query to train the operator to perform the second iteration or a third iteration of the maneuver in a manner that mimics the first iteration.

Abstract: Systems, methods, and other embodiments described herein relate to integrating human decision-making into a model-based system. In one embodiment, a method includes acquiring sensor data, including driver data about a driver of a vehicle and driving data about the vehicle and a surrounding environment of the vehicle. The method includes encoding, using a world encoder, the sensor data into a latent representation. The method includes determining human decision- making characteristics according to the latent representation. The method includes generating a control signal for providing shared control of the vehicle according to the human decision-making characteristics and the latent representation.

Abstract: A teaching curriculum method for generating teaching actions for drivers, includes obtaining driving data from a plurality of driving scenarios, the driving data comprises vehicle trajectory information and corresponding scene context information, the driving scenarios comprising instructed driving events and uninstructed driving events, encoding, with a behavior model, the driving data, wherein the encoded driving data comprises an indication that a corresponding one of the driving scenarios comprises one of the instructed driving event or the uninstructed driving event, determining, with a trajectory estimator processing the encoded driving data, one or more driving skill transitions based on a presence or an absence of the indication, and generating, with a teacher action model, a teaching action for one of the plurality of driving scenarios.

Abstract: Systems, methods, and other embodiments described herein relate to predicting future trajectories of ado vehicles and an ego vehicle based on the awareness of the driver of the ego vehicle towards the ado vehicles. In one embodiment, a method includes determining an awareness of a driver of an ego vehicle to ado vehicles in the vicinity of the ego vehicle. The method also includes altering track data of ado vehicles based on a lack of awareness of the driver towards the ado vehicles. The method also includes transmitting altered track data of the ado vehicles to a prediction module. The prediction module predicts future trajectories of the ado vehicles and the ego vehicle based on the altered track data and an ego vehicle track data.

Abstract: A method for a driver prediction system is described. The method includes training a neural network to a learn a set of polynomial basis functions. The method also includes selecting, using a trained neural network, a learned polynomial basis function to perform a prediction of an action of an autonomous dynamic object (ADO) agent. The method further includes computing projection coefficients of the learned polynomial basis function to weigh the learned polynomial basis function. The method also includes using the projection coefficients of the learned polynomial basis function to weigh the learned polynomial basis function to provide a distribution regarding a likelihood of the prediction of the action of the ADO agent.

Abstract: Systems, methods, and other embodiments described herein relate to stylizing messages within a vehicle according to an occupant and a current context. In one embodiment, a method includes determining a style for presenting messages associated with an occupant of a vehicle according to a context defined in relation to an occupant and an environment of the vehicle. The method includes generating a message according to the style for the occupant. The method includes providing the message to the occupant.

Abstract: Systems and methods are provided for determining intoxication in a driver. The system can receive data of a driver's face over a time interval and for each frame of the data, determine one or more parameters associated with eye movements and characteristics of the driver. Based on the one or more parameters for each frame, the frames can be featurized into one or more vectors, where each of the one or more vectors corresponds to a parameter of the one or more parameters. A weight can be applied to each of the one or more vectors and based on the weight of each of the one or more vectors, the system can predict whether the driver surpassed an intoxication threshold. If the driver surpassed the intoxication threshold, the system can alter an operating characteristic of a vehicle of the driver.

Abstract: Systems and methods are provided for determining driver surprise. The system can receive image data of a driver's pupils and video data of the driver's face over a time interval and determine a diameter of the driver's pupils over the time interval based on the image data. This diameter can be used to generate a pupil confidence value based on the diameter over the time interval. The system can extract facial features from the video data and generate a facial confidence value based on the facial features. A first weight can be applied to the pupil confidence value and a second weight can be applied to the facial confidence value to determine whether the driver is expressing surprise.

Abstract: A vehicle system includes one or more sensors configured to capture aspects of an environment and a computing device. The computing device is configured to receive information about the environment captured by the one or more sensors, determine one or more structures within the environment based on the received information, select a kernel that is parameterized for predicting a vehicle trajectory based on the one or more structures determined within the environment, and perform a convolution of the selected kernel and an array defining the environment, wherein the convolution predicts a future trajectory of a vehicle within the environment.

Abstract: System, methods, and other embodiments described herein relate to coordinating and optimizing prediction by a model using intermediate data and vehicle-to-vehicle (V2V) communications that improves bandwidth utilization. In one embodiment, a method includes processing acquired data, by a subject vehicle using a prediction model, to execute a vehicle task associated with navigating a driving scene. The method also includes extracting processed data according to the acquired data from intermediate activations of layers within the prediction model according to a cooperation model. The method also includes quantifying relevancy of the processed data, for a target vehicle, to select a subset using the cooperation model. The method also includes communicating the subset to a selected vehicle having the prediction model for additional prediction according to the relevancy and available resources to transmit the subset.

Abstract: Systems and methods for predicting a trajectory of a moving object are disclosed herein. One embodiment downloads, to a robot, a probabilistic hybrid discrete-continuous automaton (PHA) model learned as a deep neural network; uses the deep neural network to infer a sequence of high-level discrete modes and a set of associated low-level samples, wherein the high-level discrete modes correspond to candidate maneuvers for the moving object and the low-level samples are candidate trajectories; uses the sequence of high-level discrete modes and the set of associated low-level samples, via a learned proposal distribution in the deep neural network, to adaptively sample the sequence of high-level discrete modes to produce a reduced set of low-level samples; applies a sample selection technique to the reduced set of low-level samples to select a predicted trajectory for the moving object; and controls operation of the robot based, at least in part, on the predicted trajectory.

Abstract: Systems and methods are provided for an advanced driver-assistance system (ADAS) that obtains data from a plurality of sensors. In some embodiments, the system can retrieve data regarding a user's past interactions and analyze the data with the sensor data to determine the user's behavior. In some embodiments, the ADAS can determine whether a user is unaware of an ADAS feature based on this behavior and a prompt that recommends the ADAS feature. The user's response to this prompt may be incorporated into the user's behavior for future recommendations.

Abstract: Systems and methods are provided trajectory prediction that leverages game-theory to improve coverage of multi-modal predictions. Examples of the systems and methods include obtaining training data including first trajectories for a first plurality of agent devices and first map information of a first environment for a past time horizon and applying the training data to a game-theoretic mode-finding algorithm to generate a mode-finding model for each agent device that predicts modes of the first trajectories. A trajectory prediction model can be trained on the predicted modes as a coverage loss term between predicted modes. Future trajectories can be predicted for a second plurality of agent devices based on applying observed data to the trajectory prediction model. A control signal can then be generated to effectuate an autonomous driving command on an agent device of the second plurality of agent devices based on the predicted future trajectories.

Abstract: A method for triggering capture of diverse driving data from captions is described. The method includes training a discriminator network to identify similarities between a received text description and a received scene description. The method also includes feeding a trained discriminator network with real scene information along with text/sentence descriptions to verify whether the real scene information matches the text/sentence description. The method further includes generating a dataset of diverse driving scenarios retrieved from a dataset of vehicle driving log data in response to a text/sentence query.

Abstract: Systems and methods for learning and managing robot user interfaces are disclosed herein. One embodiment generates, based on input data including information about past interactions of a particular user with a robot and with existing HMIs of the robot, a latent space using one or more encoder neural networks, wherein the latent space is a reduced-dimensionality representation of learned behavior and characteristics of the particular user, and uses the latent space as input to train a decoder neural network associated with (1) a new HMI distinct from the existing HMIs or (2) a particular HMI among the existing HMIs to alter operation of the particular HMI. The trained first decoder neural network is deployed in the robot to control, at least in part, operation of the new HMI or the particular HMI in accordance with the learned behavior and characteristics of the particular user.

Abstract: A mobile robot can be caused to move according to a planned trajectory. The mobile robot can be a vehicle. Information about agents in an environment of the mobile robot can be received from sensors. At a first time, a spatiotemporal graph can be produced. The spatiotemporal graph can represent relationships among the agents in the environment. The mobile robot can be one of the agents in the environment. Information from the spatiotemporal graph can be input to neural networks to produce information for a mixture of affine time-varying systems. The mixture of affine time-varying systems can represent an evolution of agent states of the agents. Using the mixture of affine time-varying systems and information associated with the first time, a prediction of the agent states at a second time can be calculated. The mobile robot can be caused to move according to the planned trajectory determined from the prediction.