Abstract: A system includes a controller configured to receive one or more fixed text prompts and one or more images, wherein the fixed text prompts are associated with the one or more images. The controller is further configured to, in response to utilizing the fixed text prompt and the one or more images at a foundation model associated with a machine-learning network, output an intermediate representation from generating a series of objects and a task, decode the intermediate representation utilizing a decoder associated with the foundation model to generate a matrix associated with the task associated with the fixed text prompt and the image, and in response to identifying a highest probability associated with the matrix utilizing label selection, output a final label associated with a visual based prediction task.

Abstract: A computer-implemented method and system relate to a multimodal classifier. A video includes a digital recording of an anomalous event. A data pair includes video frames of the anomalous event and corresponding text data describing the anomalous event. A scene graph is generated to include (i) nodes that represent selected objects of the video frames and (ii) edges that define spatial relationships between pairs of the selected objects. A scene graph encoder generates scene graph embeddings using the scene graph. A video encoder generates image embeddings using the video frames. A text encoder generates text embeddings using the text data. The multimodal classifier is trained to generate class data that classifies the anomalous event based on a concatenation of the scene graph embeddings, the image embeddings, and the text embeddings. As an example, the anomalous event is a traffic accident.

Abstract: A method of generating audio to obtain manipulated audio data includes receiving textual descriptions of audio associated with operation of a device, receiving audio data associated with the operation of the device, generating, based on the textual descriptions, descriptive text inputs of audio features associated with the operation of the device, generating the manipulated audio data based on the descriptive text inputs and the audio data, the manipulated audio data including the one or more audio features indicative of faults associated with the descriptive text inputs, training a machine learning (ML) model to diagnose the faults using the manipulated audio data, the ML model being trained to generate an output indicative of the faults based on audio data obtained during the operation of the device, and, based on convergence during the training, outputting a trained ML model configured to generate the output indicative of the faults.

Abstract: A method includes receiving input dialog including a text string corresponding to at least one question and extracting at least one keyword from the text string. The method also includes generating at least one action prediction and providing one or more sub-questions associated with the at least one question. The method also includes receiving one or more answers to the one or more sub-questions, generating at least one sub-goal based on the one or more answers, and traversing an environment based on the at least one sub-goal. The method also includes receiving one or more images associated with the environment, predicting, using the one or more images, an answer to the at least one question, and providing, at an output mechanism, the answer to the at least one question.

Abstract: A method of controlling navigation of a device in an environment using machine learning (ML) models includes receiving visual and audio observation data of the environment as sensed by the device, determining classification scores for objects and regions in the environment based on the visual and audio observation data, encoding visual information based on the classification scores, determining audio-semantic feature embeddings based at least in part on the classification scores, the audio-semantic feature embeddings indicating spatial relationships between objects in the environment, between regions in the environment, and between objects and regions in the environment, and determining and outputting, based on the encoded visual information and the audio-semantic feature embeddings, a state representation corresponding to a state of the device within the environment.

Abstract: A computer-implemented system and method relates to natural language processing. The computer-implemented system and method are configured to obtain a current data structure from a global knowledge graph, which comprises various knowledge graphs. The current data structure includes a current head element, a current relationship element, and a current tail element. A sentence is obtained based on the current data structure. A question is generated by removing the current tail element from the sentence. A correct answer is generated for the question. The correct answer includes the current tail element. A pool of data structures is extracted from the global knowledge graph based on a set of distractor criteria. The set of distractor criteria ensures that each extracted data structure includes the current relationship element. Tail elements from the pool of data structures are extracted to create a pool of distractor candidates. A set of distractors are selected from the pool of distractor candidates.

Abstract: A method of generating audio to obtain manipulated audio data includes receiving textual descriptions of audio associated with operation of a device, receiving audio data associated with the operation of the device, generating, based on the textual descriptions, descriptive text inputs of audio features associated with the operation of the device, generating the manipulated audio data based on the descriptive text inputs and the audio data, the manipulated audio data including the one or more audio features indicative of faults associated with the descriptive text inputs, training a machine learning (ML) model to diagnose the faults using the manipulated audio data, the ML model being trained to generate an output indicative of the faults based on audio data obtained during the operation of the device, and, based on convergence during the training, outputting a trained ML model configured to generate the output indicative of the faults.

Abstract: A method includes generating, using a machine learning model and at a first time interval, a first current vehicle position prediction and generating, using the machine learning model, at a second time interval, a first historical vehicle trajectory prediction based on at least the first current vehicle position prediction and previous spatial information. The method also includes generating, using the machine learning model, at a third time interval, a first future vehicle position prediction based on the first current vehicle position prediction and the first historical vehicle trajectory predication. The method also includes receiving, at the first time interval, sensor data and a sequence of waypoints, and controlling, at the first time interval, at least one vehicle operation of the vehicle using the first current vehicle position prediction, the first historical vehicle trajectory prediction, the first future vehicle position prediction, the sensor data, and the sequence of waypoints.

Abstract: A computer-implemented system and method relate to operating a mobile robot with respect to a reference location. First state data is generated using sensor data obtained from a first set of sensors of a first sensor modality. Second state data is generated using second obtained from a second set of sensors. The second set of sensors provide wireless sensing. The second state data is generated from wireless features of the second sensor data. A first distribution of the first state data is generated. A second distribution of the second state data is generated. A posterior distribution is computed by fusing the first distribution and the second distribution. Optimal state data and associated uncertainty data is generated using the posterior distribution. The optimal state data including a position estimate of the mobile robot. The mobile robot is controlled using at least the optimal state data.

Abstract: A system includes a controller configured to receive one or more fixed text prompts and one or more images, wherein the fixed text prompts are associated with the one or more images. The controller is further configured to, in response to utilizing the fixed text prompt and the one or more images at a foundation model associated with a machine-learning network, output an intermediate representation from generating a series of objects and a task, decode the intermediate representation utilizing a decoder associated with the foundation model to generate a matrix associated with the task associated with the fixed text prompt and the image, and in response to identifying a highest probability associated with the matrix utilizing label selection, output a final label associated with a visual based prediction task.

Abstract: A mobile robot includes a microphone array with a set of microphones. The microphone array is at least partially disposed on the mobile robot. The mobile robot receives audio signals from the microphone array. Audio feature data of acoustic activity is extracted from the audio signals. Direction of arrival (DOA) data of the acoustic activity is generated based on the audio signals. A machine learning model is configured to generate audio event data using the audio feature data. The audio event data identifies at least one sound source of the audio feature data. A knowledge graph is queried using the audio event data to obtain entity data. The entity data has a predetermined relation with the audio event data. Semantic audio scene data is generated using the audio event data, the DOA data, and the entity data. The mobile robot performs an action based on the semantic audio scene data.

Abstract: The systems and methods described herein may include one or more processors configured to receive a command from a user related to a subject; access a representation space associated with the command; receive a first dataset related to the command, a second dataset related to the subject, and a third dataset which includes subjects related to the command; update the representation space based on at least one of the first, second, and third dataset; generate a goal representation based on the representation space; receive, from a plurality of sensors, a sensor data of a current environment; generate a first and a second series of steps based on the goal representation and the current environment; annotate the sensor data based on performance of the first series of steps to generate an annotated senor data; and update the second series of steps based on the annotated sensor data.

Abstract: A method of controlling navigation of a device in an environment using machine learning (ML) models includes receiving visual and audio observation data of the environment as sensed by the device, determining classification scores for objects and regions in the environment based on the visual and audio observation data, encoding visual information based on the classification scores, determining audio-semantic feature embeddings based at least in part on the classification scores, the audio-semantic feature embeddings indicating spatial relationships between objects in the environment, between regions in the environment, and between objects and regions in the environment, and determining and outputting, based on the encoded visual information and the audio-semantic feature embeddings, a state representation corresponding to a state of the device within the environment.

Abstract: A computer implemented method for controlling a load aggregator for a grid includes receiving a predicted power demand over a horizon of time steps associated with one of at least two buildings, aggregating the predicted power demand at each time step to obtain an aggregate power demand, applying a learnable convolutional filter on the aggregate power demand to obtain a target load, computing a difference between the predicted power demand of the one building with the target load to obtain a power shift associated with the one building over the horizon of time steps, apportioning the power shift according to a learnable weighted vector to obtain an apportioned power shift, optimizing the learnable weighted vector and the learnable convolutional filter via an evolutionary strategy based update to obtain an optimized apportioned power shift, and transmitting the optimized apportioned power shift to a building level controller associated with the one building.

Abstract: A method includes generating, using a machine learning model and at a first time interval, a first current vehicle position prediction and generating, using the machine learning model, at a second time interval, a first historical vehicle trajectory prediction based on at least the first current vehicle position prediction and previous spatial information. The method also includes generating, using the machine learning model, at a third time interval, a first future vehicle position prediction based on the first current vehicle position prediction and the first historical vehicle trajectory predication. The method also includes receiving, at the first time interval, sensor data and a sequence of waypoints, and controlling, at the first time interval, at least one vehicle operation of the vehicle using the first current vehicle position prediction, the first historical vehicle trajectory prediction, the first future vehicle position prediction, the sensor data, and the sequence of waypoints.

Abstract: A method includes receiving input dialog including a text string corresponding to at least one question and extracting at least one keyword from the text string. The method also includes generating at least one action prediction and providing one or more sub-questions associated with the at least one question. The method also includes receiving one or more answers to the one or more sub-questions, generating at least one sub-goal based on the one or more answers, and traversing an environment based on the at least one sub-goal. The method also includes receiving one or more images associated with the environment, predicting, using the one or more images, an answer to the at least one question, and providing, at an output mechanism, the answer to the at least one question.

Abstract: Occupant thermal comfort may be inferred and improved using body shape information. Height, weight, and shoulder circumference of an occupant of a room may be obtained using a depth sensor. A model may be utilized that is trained on a dataset including information reflecting of occupant comfort within the room versus temperature, the model receiving, as inputs, the height, the weight, and the shoulder circumference of the occupant and environmental information and outputting a comfort class. A temperature set-point for is identified which the room occupant is identified by the model as having the comfort class being indicative of user comfort. Heating, ventilation, and air conditioning (HVAC) controls are adjusted for the room to the identified temperature set-point.

Abstract: A computer implemented method for controlling a load aggregator for a grid includes receiving a predicted power demand over a horizon of time steps associated with one of at least two buildings, aggregating the predicted power demand at each time step to obtain an aggregate power demand, applying a learnable convolutional filter on the aggregate power demand to obtain a target load, computing a difference between the predicted power demand of the one building with the target load to obtain a power shift associated with the one building over the horizon of time steps, apportioning the power shift according to a learnable weighted vector to obtain an apportioned power shift, optimizing the learnable weighted vector and the learnable convolutional filter via an evolutionary strategy based update to obtain an optimized apportioned power shift, and transmitting the optimized apportioned power shift to a building level controller associated with the one building.

Abstract: A system for image processing includes a first sensor configured to capture at least one or more images, a second sensor configured to capture sound information, a processor in communication with the first sensor and second sensor, wherein the processor is programmed to receive the one or more images and sound information, extract one or more data features associated with the images and sound information utilizing an encoder, output metadata via a decoder to a spatiotemporal reasoning engine, wherein the metadata is derived utilizing the decoder and the one or more data features, determine one or more scenes utilizing the spatiotemporal reasoning engine and the metadata, and output a control command in response to the one or more scenes.

Abstract: A computer-implemented system and method relates to natural language processing. The computer-implemented system and method are configured to obtain a current data structure from a global knowledge graph, which comprises various knowledge graphs. The current data structure includes a current head element, a current relationship element, and a current tail element. A sentence is obtained based on the current data structure. A question is generated by removing the current tail element from the sentence. A correct answer is generated for the question. The correct answer includes the current tail element. A pool of data structures is extracted from the global knowledge graph based on a set of distractor criteria. The set of distractor criteria ensures that each extracted data structure includes the current relationship element. Tail elements from the pool of data structures are extracted to create a pool of distractor candidates. A set of distractors are selected from the pool of distractor candidates.