Abstract: Techniques are described herein for determining one or more actions for an autonomous vehicle to perform, based on simulation of at least one possible scenario. A possible scenario may involve, for example, the autonomous vehicle interacting with an object in the environment. The possible scenario may be simulated by modifying a first internal map containing information about the autonomous vehicle and the environment. As part of the simulation, one or more parameters of the first internal map can be modified in order to, for example, determine the state of the object at a particular point in the future. Based on the modification of the one or more parameters, a second internal map representing a possible scenario is generated from the first internal map. Both the first internal map and the second internal map can be evaluated to decide which action to take.

Abstract: The present disclosure relates generally to autonomous machines (AMs) and more particularly to techniques for intelligently planning, managing and performing various tasks using AMs. A control system (referred to as a fleet management system or FMS) is disclosed for managing a set of resources at a site, which may include AMs. The FMS is configured to control and manage the AMs at the site such that tasks are performed autonomously by the AMs. An AM may directly communicate with another AM located on the site to complete a task without requiring to be in constant communication with the FMS during the performance of the task. The FMS is configured to use various optimization techniques to allocate resources (e.g., AMs) for performing tasks at the site. The resource allocation is performed so as to maximize the use of available AMs while ensuring that the tasks get performed in a timely manner.

Abstract: Techniques are described herein for determining one or more actions for an autonomous vehicle to perform, based on simulation of at least one possible scenario. A possible scenario may involve, for example, the autonomous vehicle interacting with an object in the environment. The possible scenario may be simulated by modifying a first internal map containing information about the autonomous vehicle and the environment. As part of the simulation, one or more parameters of the first internal map can be modified in order to, for example, determine the state of the object at a particular point in the future. Based on the modification of the one or more parameters, a second internal map representing a possible scenario is generated from the first internal map. Both the first internal map and the second internal map can be evaluated to decide which action to take.

Abstract: The present disclosure relates techniques for autonomously controlling heavy equipment and vehicles using task hierarchies. Particularly, aspects of the present disclosure are directed to obtaining a task to be performed by an autonomous vehicle, determining subtasks to be performed to perform the task, obtaining sensor data providing a representation of operation of the autonomous vehicle in a worksite environment and situational context of the worksite environment, determining a task context for a subtask based on the sensor data, identifying a predictive model from a library of predictive models based on the task context, estimating, by the predictive model, a set of output data based on sensor data, and controlling operations of the autonomous vehicle in the worksite environment to perform the subtask using a set of input data derived from the sensor data and the set of output data.

Abstract: Systems and methods enable an autonomous vehicle to perform an iterative task of transferring material from a source location to a destination location, such as moving dirt from a pile, in a more efficient manner, using a combination of reinforcement learning techniques to select a motion path for a particular iteration and visual servo control to guide the motion of the vehicle along the selected path. Lifting, carrying, and depositing of material by the autonomous vehicle can also be managed using similar techniques.

Abstract: The present disclosure relates generally to identification of drivable surfaces in connection with autonomously performing various tasks at industrial work sites and, more particularly, to techniques for distinguishing drivable surfaces from non-drivable surfaces based on sensor data. A framework for the identification of drivable surfaces is provided for an autonomous machine to facilitate it to autonomously detect the presence of a drivable surface and to estimate, based on sensor data, attributes of the drivable surface such as road condition, road curvature, degree of inclination or declination, and the like. In certain embodiments, at least one camera image is processed to extract a set features from which surfaces and objects in a physical environment are identified, and to generate additional images for further processing. The additional images are combined with a 3D representation, derived from LIDAR or radar data, to generate an output representation indicating a drivable surface.

Abstract: Techniques are described herein for providing information regarding one or more actions an autonomous vehicle management system is planning to perform. The autonomous vehicle management system can also provide information indicative of one or more reasons for a planned action. This information can be provided through a user interface that displays an indication of the action together with an indication of the reason for the action. The information provided through the user interface can improve the user's trust in the safety of the autonomous vehicle, provide the user with context for evaluating the decisions made by the autonomous vehicle management system, and allow the user to decide for himself or herself whether the actions planned by the autonomous vehicle make sense.

Abstract: Systems and methods for object detection in a dusty environment can enhance the ability of autonomous machines to distinguish dust clouds from solid obstacles and proceed appropriately. A library of dust classifiers can be provided, where each dust classifier is separately trained to distinguish airborne dust from objects in the environment. Different dust classifiers can correspond to different categories of dusty environments. Based on current conditions, control logic in an autonomous machine can categorize its environment and select a corresponding dust classifier. The dust classifier output can be used to alter a behavior of the autonomous machine, including a behavior of the control logic. For instance, the control logic can apply a consistency check to the output of the dust classifier and an output of an AI-based object classifier to detect instances where the object classifier misidentifies dust as an object.

Abstract: An infrastructure is provided for improving the safety of autonomous systems. An autonomous vehicle management system (AVMS) controls one or more autonomous functions or operations performed by a vehicle or machine such that the autonomous operations are performed in a safe manner. The AVMS is capable of dynamically controlling the behavior of sensors associated with a vehicle. For example, for a sensor, the AVMS can dynamically change and control what sensor data is captured by the sensor and/or communicated from the sensor to the AVMS (e.g., granularity/resolution, field of view, control zoom), when the data is captured by the sensor and/or communicated by the sensor to the AVMS (e.g., on-demand, according to a schedule), and how the data is captured by the sensor and/or communicated from the sensor to the AVMS (e.g., communication format, communication protocol, rate of data communication).

Abstract: The present disclosure relates generally to techniques for the kinematic estimation and dynamic behavior estimation of autonomous heavy equipment or vehicles to improve navigation, digging and material carrying tasks at various industrial work sites. Particularly, aspects of the present disclosure are directed to obtaining a set of sensor data providing a representation of operation of an autonomous vehicle in a worksite environment, estimating, by a trained model comprising a Gaussian process, a set of output data based on the set of sensor data, controlling an operation of the autonomous vehicle in the worksite environment using input data derived from the set of sensor data and the set of output data, obtaining actual output data from the operation of the autonomous vehicle in the worksite environment, and updating the trained model with the input data and the actual output data.

Abstract: The present disclosure relates generally to the operation of autonomous machinery for performing various tasks at various industrial work sites, and more particularly to the volumetric estimation and dimensional estimation of a pile of material or other object, and the use of multiple sensors for the volumetric estimation and dimensional estimation of a pile of material or other object at such work sites. An application and a framework is disclosed for volumetric estimation and dimensional estimation of a pile of material or other object using at least one sensor, preferably a plurality of sensors, on an autonomous machine (e.g., robotic machines or autonomous vehicles) in various work-site environments applicable to various industries such as, construction, mining, manufacturing, warehousing, logistics, sorting, packaging, agriculture, etc.

Abstract: The present disclosure relates techniques for autonomously controlling heavy equipment and vehicles using task hierarchies. Particularly, aspects of the present disclosure are directed to obtaining a task to be performed by an autonomous vehicle, determining subtasks to be performed to perform the task, obtaining sensor data providing a representation of operation of the autonomous vehicle in a worksite environment and situational context of the worksite environment, determining a task context for a subtask based on the sensor data, identifying a predictive model from a library of predictive models based on the task context, estimating, by the predictive model, a set of output data based on sensor data, and controlling operations of the autonomous vehicle in the worksite environment to perform the subtask using a set of input data derived from the sensor data and the set of output data.

Abstract: An infrastructure is provided for improving the safety of autonomous systems. An autonomous vehicle management system (AVMS) controls one or more autonomous functions or operations performed by a vehicle or machine such that the autonomous operations are performed in a safe manner. The AVMS uses various artificial intelligence (AI) based techniques (e.g., neural networks, reinforcement learning (RL) techniques, etc.) and models as part of its processing. For an inferring data point, for which a prediction is made by AVMS using an AI model, the AVMS checks how statistically similar (or dissimilar) the inferring data point is to the distribution of the training dataset. A score (confidence score) is generated indicative of how similar or dissimilar the inferring data point is to the training dataset. The AVMS uses this confidence score to decide how the prediction made by the AI model is to be used.

Abstract: A multi-processor architecture for automated driving systems can be used to improve performance and provide design flexibility. For example, a multi-processor architecture can be used to implement command generation and safety functionality in different processors. The command generation processor can be a high performing processor compared with the safety processor. The safety processor can verify the safety of commands output from the command generation processor and provide additional I/O channels that are typically absent on high performing processors. Additionally, processing of some sensor data can be moved to expansion modules with additional processors to reduce bottlenecks and provide design flexibility for systems with different sensing requirements.

Abstract: In one aspect, In one aspect, a method for data transfer and processing communications is provided. The method includes the step of providing a machine-to-everything (M2X) application layer on each machine of the plurality of machines. The method includes the step of providing a plurality of communication nodes on each machine for communication between the plurality of machines with every other machine, the plurality of machines and any infrastructure at a work site, and a plurality of communication nodes communicating using the at least one application layer. The method includes the step of providing a communication processing system for receiving a data transfer and processing communications. The communication processing system includes a plurality of processing stations, one or more multiple data management protocols, a plurality of network protocols, a plurality of databases and plurality of data processing network architectures.

Abstract: Example user identification systems and methods are described. In one implementation, a method determines a first step pulse associated with a user carrying a key fob and detects a user device proximate the key fob. The method further identifies a second step pulse measured by the user device and determines whether the first and second step pulses match. If the step pulses match, the user carrying the key fob is identified as the owner of the user device.

Abstract: Techniques are described herein for determining one or more actions for an autonomous vehicle to perform, based on simulation of at least one possible scenario. A possible scenario may involve, for example, the autonomous vehicle interacting with an object in the environment. The possible scenario may be simulated by modifying a first internal map containing information about the autonomous vehicle and the environment. As part of the simulation, one or more parameters of the first internal map can be modified in order to, for example, determine the state of the object at a particular point in the future. Based on the modification of the one or more parameters, a second internal map representing a possible scenario is generated from the first internal map. Both the first internal map and the second internal map can be evaluated to decide which action to take.

Abstract: The present disclosure relates generally to identification of drivable surfaces in connection with autonomously performing various tasks at industrial work sites and, more particularly, to techniques for distinguishing drivable surfaces from non-drivable surfaces based on sensor data. A framework for the identification of drivable surfaces is provided for an autonomous machine to facilitate it to autonomously detect the presence of a drivable surface and to estimate, based on sensor data, attributes of the drivable surface such as road condition, road curvature, degree of inclination or declination, and the like. In certain embodiments, at least one camera image is processed to extract a set features from which surfaces and objects in a physical environment are identified, and to generate additional images for further processing. The additional images are combined with a 3D representation, derived from LIDAR or radar data, to generate an output representation indicating a drivable surface.

Abstract: Techniques are described herein for determining one or more actions for an autonomous vehicle to perform, based on simulation of at least one possible scenario. A possible scenario may involve, for example, the autonomous vehicle interacting with an object in the environment. The possible scenario may be simulated by modifying a first internal map containing information about the autonomous vehicle and the environment. As part of the simulation, one or more parameters of the first internal map can be modified in order to, for example, determine the state of the object at a particular point in the future. Based on the modification of the one or more parameters, a second internal map representing a possible scenario is generated from the first internal map. Both the first internal map and the second internal map can be evaluated to decide which action to take.

Abstract: Systems and methods for object detection in a dusty environment can enhance the ability of autonomous machines to distinguish dust clouds from solid obstacles and proceed appropriately. A library of dust classifiers can be provided, where each dust classifier is separately trained to distinguish airborne dust from objects in the environment. Different dust classifiers can correspond to different categories of dusty environments. Based on current conditions, control logic in an autonomous machine can categorize its environment and select a corresponding dust classifier. The dust classifier output can be used to alter a behavior of the autonomous machine, including a behavior of the control logic. For instance, the control logic can apply a consistency check to the output of the dust classifier and an output of an AI-based object classifier to detect instances where the object classifier misidentifies dust as an object.