Abstract: Disclosed are systems and methods to control a robotic device using natural language commands. A natural language command may be received by a system. The system may convert the command into low-level machine controls and logic for implementation by a robotic device to achieve a desired action. In some instances, an API module may include mapping data to associate high-level commands with low-level machine controls. A language model may process the natural language command (input), high-level commands, and/or other information, such as system state, sensor observation data, parameters, etc., to determine one or more commands to execute by a robotic device and possibly logic for execution by the robotic device. The robotic device may receive the low-level machine controls and logic to cause the robotic device to perform the requested actions.

Abstract: A process can include obtaining a plurality of data points each associated with a respective edge device of a fleet of edge devices, each respective edge device associated with an edge site location or edge device asset group. The plurality of data points are stored to a fleet map data catalog and a filtering selection for viewing a filtered subset of the fleet map data catalog is received, indicating a selected geographic view area and selected edge device types from a plurality of edge device types. Data points corresponding to the filtered subset are obtained from the fleet map data catalog using the filtering selection. A fleet map GUI view is generated using the data points corresponding to the filtered subset, the fleet map GUI view comprising a converged geographic map of the selected geographic view area, with data points are rendered at corresponding locations within the converged geographic map.

Abstract: Disclosed are systems and methods to track an object in a scene even when the object is partially or wholly obscured by an artifact, such as another object. As the tracked object moves through the scene, frames of video are processed and used to refine and tune a diffusion model that predicts an appearance of the tracked object in future frames of the video as well as the appearance of artifacts in the frames of the video. Frames of the video may then be enhanced to illustrate the tracked object as if the tracked object were visible through the artifact.

Abstract: Disclosed are systems and methods that convert digital video data, such as two-dimensional digital video data, into a natural language text description describing the subject matter represented in the video. For example, the disclosed implementations may process video data in real-time, near real-time, or after the video data is created and generate a text-based video narrative describing the subject matter of the video. In addition, the disclosed implementations may also support a question and answer session in which a user may submit queries about the subject matter of one or more videos and the disclosed implementations will present natural language responses based on the subject matter of the video and any corresponding context.

Abstract: Disclosed implementations include systems, methods, and apparatus that process multiple, disparate streams of data, determine correlations and relationships between the data and provide natural language responses that provide insights for events or activities that have occurred and foresights for events or activities that are forecasted to occur. The disclosed implementations include a model that understands data statistics and provides both insights and foresights that are backed with statistical support that can be presented to and understood by operators. Still further, the disclosed implementations are capable of operating at edge locations that may be frequently or permanently disconnected from conventional or cloud based systems.

Abstract: Computing units provided at local sites or edge locations are programmed to execute conversational tools that generate pertinent, domain-specific responses to queries received from workers at such sites or locations. The conversational tools are large language models that are trained with domain-specific knowledge documents. Data representing queries are received from workers at such sites or locations and provided as inputs to the conversational tools along with text representing nearest knowledge documents from a knowledge base associated with the domain, as well as contextual data. Responses identified based on outputs received from the conversational tools in response to the inputs are provided to the workers that generated the queries.

Abstract: A process can include obtaining a plurality of data points each associated with a respective edge device of a fleet of edge devices, each respective edge device associated with an edge site location or edge device asset group. The plurality of data points are stored to a fleet map data catalog and a filtering selection for viewing a filtered subset of the fleet map data catalog is received, indicating a selected geographic view area and selected edge device types from a plurality of edge device types. Data points corresponding to the filtered subset are obtained from the fleet map data catalog using the filtering selection. A fleet map GUI view is generated using the data points corresponding to the filtered subset, the fleet map GUI view comprising a converged geographic map of the selected geographic view area, with data points are rendered at corresponding locations within the converged geographic map.

Abstract: Systems and techniques are provided for processing streaming data. For example, a process can include obtaining, by an edge device, a plurality of data streams corresponding to objects, activities, or events registered in an edge environment associated with the edge device. One or more machine learning networks implemented on the edge device can generate a plurality of features and/or embeddings corresponding to each respective data stream of the plurality of data streams. The edge device can determine a subset of salient content from the plurality of data streams, based on analyzing the plurality of features and/or embeddings generated for each respective data stream. The edge device can generate index information corresponding to the determined subset of salient content. The subset of salient content and the generated index information can be stored locally at the edge device for search and retrieval based on local queries received at the edge device.

Abstract: Disclosed are systems and methods that convert digital video data, such as two-dimensional digital video data, into a natural language text description describing the subject matter represented in the video. For example, the disclosed implementations may process video data in real-time, near real-time, or after the video data is created and generate a text-based video narrative describing the subject matter of the video. In addition, the disclosed implementations may also support a question and answer session in which a user may submit queries about the subject matter of one or more videos and the disclosed implementations will present natural language responses based on the subject matter of the video and any corresponding context.

Abstract: A process can include transmitting a pre-trained machine learning model to an edge compute unit associated with a request. The edge compute unit can perform inference using the pre-trained machine learning model and one or more sensor data streams obtained at an edge location. One or more batch uploads of information can be received, associated with inference performed by the edge compute unit and using the pre-trained machine learning model. One or more updated machine learning models can be generated, based on using the batch uploads of information from the edge compute unit to retrain or finetune the pre-trained machine learning model. The one or more updated machine learning models can be transmitted to the edge compute unit, wherein transmission of the updated machine learning model is responsive to receiving the one or more batch uploads of information.

Abstract: An apparatus can be configured to receive monitoring information associated with a machine learning (ML) or artificial intelligence (AI) workload implemented by an edge compute unit of a plurality of edge compute units. Status information corresponding to a plurality of connected edge assets can be received, the plurality of edge compute units and connected edge assets included in a fleet of edge devices. A remote fleet management graphical user interface (GUI) can display a portion of the monitoring or status information for a subset of the fleet of edge devices, based on a user selection input, and can receive a user configuration input indicative of an updated configuration for at least one workload corresponding to a pre-trained ML or AI model deployed on the at least one edge compute unit. A cloud computing environment can transmit control information corresponding to the updated configuration to the at least one edge compute unit.

Abstract: Computing units provided at local sites or edge locations are programmed to execute conversational tools that generate pertinent, domain-specific responses to queries received from workers at such sites or locations. The conversational tools are large language models that are trained with domain-specific knowledge documents. Data representing queries are received from workers at such sites or locations and provided as inputs to the conversational tools along with text representing nearest knowledge documents from a knowledge base associated with the domain, as well as contextual data. Responses identified based on outputs received from the conversational tools in response to the inputs are provided to the workers that generated the queries.

Abstract: Disclosed implementations include systems, methods, and apparatus that process multiple, disparate streams of data, determine correlations and relationships between the data and provide natural language responses that provide insights for events or activities that have occurred and foresights for events or activities that are forecasted to occur. The disclosed implementations include a model that understands data statistics and provides both insights and foresights that are backed with statistical support that can be presented to and understood by operators. Still further, the disclosed implementations are capable of operating at edge locations that may be frequently or permanently disconnected from conventional or cloud based systems.

Abstract: Disclosed implementations include systems, methods, and apparatus that process multiple, disparate streams of data, determine correlations and relationships between the data and provide natural language responses that provide insights for events or activities that have occurred and foresights for events or activities that are forecasted to occur. The disclosed implementations include a model that understands data statistics and provides both insights and foresights that are backed with statistical support that can be presented to and understood by operators. Still further, the disclosed implementations are capable of operating at edge locations that may be frequently or permanently disconnected from conventional or cloud based systems.

Abstract: A process can include receiving monitoring information associated with a machine learning (ML) or artificial intelligence (AI) workload implemented by an edge compute unit of a plurality of edge compute units. Status information corresponding to a plurality of connected edge assets can be received, the plurality of edge compute units and connected edge assets included in a fleet of edge devices. A remote fleet management graphical user interface (GUI) can display a portion of the monitoring or status information for a subset of the fleet of edge devices, based on a user selection input, and can receive a user configuration input indicative of an updated configuration for at least one workload corresponding to a pre-trained ML or AI model deployed on the at least one edge compute unit. A cloud computing environment can transmit control information corresponding to the updated configuration to the at least one edge compute unit.

Abstract: A process can include receiving, by an edge compute unit, a pre-trained machine learning model from a cloud management platform, wherein the edge compute unit is deployed to an edge location and configured to obtain one or more sensor data streams at the edge location. The edge compute unit can transmit one or more batch uploads of information associated with inference performed by the edge compute unit using the pre-trained machine learning model and the one or more sensor data streams. The edge compute unit can receive one or more updated machine learning models generated by the cloud management platform responsive to the one or more batch uploads of information, wherein the one or more updated machine learning models are based on retraining or finetuning of the pre-trained machine learning model with the one or more batch uploads of information.

Abstract: Systems and techniques are provided for processing streaming data. For example, a process can include obtaining, by an edge device, a plurality of data streams corresponding to objects, activities, or events registered in an edge environment associated with the edge device. One or more machine learning networks implemented on the edge device can generate a plurality of features and/or embeddings corresponding to each respective data stream of the plurality of data streams. The edge device can determine a subset of salient content from the plurality of data streams, based on analyzing the plurality of features and/or embeddings generated for each respective data stream. The edge device can generate index information corresponding to the determined subset of salient content. The subset of salient content and the generated index information can be stored locally at the edge device for search and retrieval based on local queries received at the edge device.

Abstract: An apparatus can be configured to receive monitoring information associated with a machine learning (ML) or artificial intelligence (AI) workload implemented by an edge compute unit of a plurality of edge compute units. Status information corresponding to a plurality of connected edge assets can be received, the plurality of edge compute units and connected edge assets included in a fleet of edge devices. A remote fleet management graphical user interface (GUI) can display a portion of the monitoring or status information for a subset of the fleet of edge devices, based on a user selection input, and can receive a user configuration input indicative of an updated configuration associated with at least one edge compute unit of the fleet. A cloud computing environment can transmit control information corresponding to the updated configuration to the at least one edge compute unit.

Abstract: A method can include receiving monitoring information associated with a machine learning (ML) or artificial intelligence (AI) workload implemented by an edge compute unit of a plurality of edge compute units. Status information corresponding to a plurality of connected edge assets can be received, the plurality of edge compute units and connected edge assets included in a fleet of edge devices. A remote fleet management graphical user interface (GUI) can display a portion of the monitoring or status information for a subset of the fleet of edge devices, based on a user selection input, and can receive a user configuration input indicative of an updated configuration associated with at least one edge compute unit of the fleet. A cloud computing environment can transmit control information corresponding to the updated configuration to the at least one edge compute unit.

Abstract: Computing units provided at local sites or edge locations are programmed to execute conversational tools that generate pertinent, domain-specific responses to queries received from workers at such sites or locations. The conversational tools are large language models that are trained with domain-specific knowledge documents. Data representing queries are received from workers at such sites or locations and provided as inputs to the conversational tools along with text representing nearest knowledge documents from a knowledge base associated with the domain, as well as contextual data. Responses identified based on outputs received from the conversational tools in response to the inputs are provided to the workers that generated the queries.