Abstract: Systems and methods for performing real-time multi-robot collaboration in dynamic environments are provided. A system may obtain sensor data of an environment of the robot, and generate tokenized sensor data from the sensor data. The system may input the tokenized sensor data into a robotics foundational model (RFM) associated with the robot causing the RFM to generate insight data used for making decisions associated with performing the mission. Generating the insight data includes generating one or more beliefs about the environment and generating one or more risk-reward maps indicating potential risks and/or a. The system implement a token sharing policy causing the robot to generate tokenized insight data and transmit the tokenized insight data to recipient robots of the robot fleet.

Abstract: A method of and system for priority-based instruction handling in a human-robot interactive system can include receiving a plurality of instructions from a plurality of users, processing the instructions through a priority-based instruction manager, generating prioritized instruction embeddings, inputting the prioritized instruction embeddings into a sequence decision model, and generating at least one robot command. The priority-based instruction manager can be in communication with a database, where the database can include context information about members of the user set. The prioritized instruction embeddings can include the output of a scalar weighting function calculated by the priority-based instruction manager.

Abstract: Systems and methods for generating virtual test environments (VTEs) to optimize performance of robots are provided. A system may generate, at a simulation platform via a language model, a VTE configured for testing performance of a robotics foundational model (RFM) during a virtual mission. The system may test the performance of the RFM in the VTE including: (a) causing the RFM to perform the virtual mission; (b) obtaining virtual operational data associated with the performance of the RFM during the virtual mission; and (c) analyzing the virtual operational data to determine virtual operational characteristic for further testing. Based upon determining virtual operational characteristic for further testing, the system may provide the virtual operational data to the language model as an input causing the language model to reconfigure the VTE for further testing the virtual operational characteristic, and repeat steps (a)-(c).

Abstract: In order to construct dynamic environment data with an automated annotation, a 3D model of an environment is first constructed using sensors located on a machine moving through the environment. The sensors include a first sensor to obtain point cloud data of the environment, a second sensor to obtain 2D images of an object or feature in the environment from different perspectives, and a third sensor to monitor positions and orientations of the machine. Once the 3D model is constructed, an annotated one of the 2D images and a non-annotated one of the 2D images are projected onto the 3D model and aligned with one another. The annotation is then transferred from the annotated 2D image to the non-annotated 2D image to convert the non-annotated 2D image into a second annotated 2D image. The second annotated 2D image is re-projected onto a 2D plane.

Abstract: A mapping system for an autonomous mobile robot includes a 3D convolutional encoder network that generates 3D feature maps from 3D point cloud data. The network sequentially compresses the feature dimension of the 3D input data to reduce the computational complexity and enable feature extraction to be performed in substantially real-time. Skip connections connect the outputs of the encoder layers of the convolutional encoder network to counterpart decoder layers of a 2D convolutional decoder network. An attention-based 3D to 2D projection layer receives the 3D feature maps generated by the encoder layers via the skip connections and projects the 3D feature maps onto 2D BEV feature maps which are provided to the counterpart decoder layers as input. The projection layer automatically estimates ground level of 3D feature maps and filters out overhanging objects that are irrelevant to ground-level navigation.

Abstract: Systems and methods for determining a status of a built environment using a graph neural network are provided. An example system may obtain build data including a building information model (BIM) of the built environment, scan data depicting the built environment, and scheduling data indicating tasks for constructing the built environment. The system may perform a registration of the scan data and build data, and generate BIM-based features indicating hierarchical relationships of elements of the BIM, scan-based features indicating characteristics of scans of the built environment, and scheduling-based features indicating characteristics of the built tasks. The system may provide the build data, the BIM-based features, the scan data, the scan-based features, the scheduling data, and the scheduling-based features to a temporal graph neural network to generate a graph, and the status of the built environment based upon the graph. The system may provide the built status to a computing device.

Abstract: Systems and methods for real-time localization and pose correction of a robot are provided. An example system may obtain sensor data of a localized environment, odometry data indicating the robot's movement in the localized environment, and environmental data including a global two-dimensional model depicting a global environment. The system may generate a localized three-dimensional model depicting the localized environment, and generate a localized two-dimensional model of the localized environment based upon transforming the localized three-dimensional model. The system may obtain an indication of an estimated pose of the robot in the localized environment, perform a registration of the localized two-dimensional model with the global two-dimensional model based upon the estimated pose, and generate corrected pose data indicating a corrected pose of the robot. The system may configure the robot using the corrected pose data to identify the corrected pose of the robot within the global environment.

Abstract: A method of and system for generating a three-dimensional map of an environment can include obtaining a first visual data set, generating a depth prior based on the first visual data set, refining a depth prediction model based on the depth prior, generating a layout based on a refined depth prediction model, and constructing a continuous three-dimensional map of the environment based on the layout and aggregated depth measurements. The visual data set can include visual imagery data and depth data.

Abstract: A method of and system for navigation and manipulation for a robot can include obtaining, by at least one camera and at least one depth sensor, a first visual data set and translating the first visual data set into a continuous three-dimensional map. The three-dimensional map can include semantic information and geometric information. The method and system may further include receiving instruction data and converting the instruction data into at least one task for the robot within the continuous three-dimensional map.

Abstract: An object-goal navigation framework for an autonomous mobile robot uses a dynamic Scene Graph (DSG) and a Relational Semantic Network (RSN) for semantic-guided object-goal navigation. The DSG is a hierarchical world representation generated from a prior spatial configuration of the environment. The RSN encodes relational semantic knowledge between objects and the regions or rooms in the environment. The object-goal navigation problem is then solved using a probabilistic planning framework with relational semantic knowledge. Using DSG and RSN, the global planning problem is formulated as a Markov decision process (MDP). The computed global planning policy directs the robot to visit a room or perform local searches. A finite state local controller is then used to execute the global planning policy and search for the target object.

Abstract: Systems and methods for real-time terrain classification via an autonomous robot are provided. An example method may obtain environmental data indicating one or more characteristics of a physical environment including terrain, and classify the terrain based upon the environmental data. Based upon a first classification of the terrain, the method may determine a terrain assessment task associated with reclassifying at least the portion of the terrain and determine whether performing the terrain assessment task will exceed a performance threshold indicating an adverse effect on performing a mission task. Based upon determining whether performing the terrain assessment task will exceed the performance threshold, the method may generate terrain assessment task configuration data for configuring the autonomous robot and transmit the terrain assessment task configuration data to the autonomous robot causing configuration of the autonomous robot associated with the terrain assessment task.

Abstract: Sparse centralized real-time mapping for efficient belief state representations of complex geometry in unstructured environments is provided by a space mapping system which collects, from sensors, point cloud data having auxiliary identifying data, performs real-time voxelization and probabilistic modeling using a probability theory to account for uncertainty in the point cloud data, based on the auxiliary identifying data, to create a sparse 3D belief space representation map of the point cloud data, generates a 2D map from the sparse 3D belief space representation map, wherein the space mapping system includes at least one graphic processing unit (GPU) and at least one central processing unit (CPU) to perform the probabilistic modeling, and dynamically allocates processing operations of at least one of the voxelization and the probabilistic modeling of the point cloud data between the GPU and the CPU during operations to create the sparse 3D belief space representation map.