Abstract: A method of controlling navigation of a device in an environment using machine learning (ML) models includes receiving visual and audio observation data of the environment as sensed by the device, determining classification scores for objects and regions in the environment based on the visual and audio observation data, encoding visual information based on the classification scores, determining audio-semantic feature embeddings based at least in part on the classification scores, the audio-semantic feature embeddings indicating spatial relationships between objects in the environment, between regions in the environment, and between objects and regions in the environment, and determining and outputting, based on the encoded visual information and the audio-semantic feature embeddings, a state representation corresponding to a state of the device within the environment.

Abstract: In some implementations, the device may include receiving a first and second audio dataset. In addition, the device may generate a first, a second, a third, and a fourth audio sample. Moreover, the device may include determining a level of similarity between the first and second audio samples. Also, the device may include combining the first and second audio samples into an audio pair. Further, the device may include training a machine learning model to map audio samples to a latent space visualization in view of time and the similarities between the first and second audio samples to yield a trained machine learning model. In addition, the device may include mapping, by the machine learning model, in the latent space visualization, the third and fourth audio samples, where placement of the third and fourth audio samples depends on the level of similarity between the third and fourth audio samples.

Abstract: A method of training a machine learning (ML) model includes obtaining a dataset that includes first training data obtained using two or more ground truth sensing systems and second training data obtained using a prediction sensing system configured to implement the ML model, determining a loss function based on the first training data, the loss function defining a region of zero loss based on a minimum and a maximum of the first training data, calculating, using the ML model, a prediction output based on the second training data, calculating, using the loss function, a loss of the ML model based on the prediction output, and updating the ML model based on the calculated loss.

Abstract: A method includes receiving audio stream data associated with a data capture environment, and receiving sensor data associated with the data capture environment. The method also includes identifying at least some events in the sensor data, and calculating at least one offset value for at least a portion of the audio stream data that corresponds to at least one event of the sensor data. The method also includes synchronizing at least a portion of the sensor data associated with the portion of the audio stream data that corresponds to the at least one event of the sensor data, and labeling at least the portion of the audio stream data that corresponds to the at least one event of the sensor data. The method also includes generating training data using at least some of the labeled portion of the audio stream data, and training a machine learning model using the training data.

Abstract: A system for automatically selecting a sound recognition model for an environment based on audio data and image data associated with the environment. The system includes a camera, a microphone, a memory including a plurality of sound recognition models, and an electronic processor. The electronic processor is configured to receive the audio data associated with the environment from the microphone, receive the image data associated with the environment from the camera, and determine one or more characteristics of the environment based on the audio data and the image data. The electronic processor is also configured to select the sound recognition model from the plurality of sound recognition models based on the one or more characteristics of the environment, receive additional audio data associated with the environment from the microphone, and analyze the additional audio data using the sound recognition model to perform a sound recognition task.

Abstract: A method data augmentation includes receiving audio stream data associated with at least one impulse event, receiving a label associated with the audio stream data, and detecting, using an onset detector, at least one peak of the at least one impulse event. The method also includes extracting at least one positive sample of the audio stream data associated with the at least one impulse event. The method also includes applying, to the at least one positive sample, the label associated with the audio stream data and extracting at least one negative sample of the audio stream data associated with the at least one impulse event. The method also includes augmenting training data based on the at least one positive sample and the at least one negative sample and training at least one machine-learning model using the augmented training data.

Abstract: A method includes receiving audio stream data associated with a data capture environment, and receiving sensor data associated with the data capture environment. The method also includes identifying at least some events in the sensor data, and calculating at least one offset value for at least a portion of the audio stream data that corresponds to at least one event of the sensor data. The method also includes synchronizing at least a portion of the sensor data associated with the portion of the audio stream data that corresponds to the at least one event of the sensor data, and labeling at least the portion of the audio stream data that corresponds to the at least one event of the sensor data. The method also includes generating training data using at least some of the labeled portion of the audio stream data, and training a machine learning model using the training data.

Abstract: A method for labeling audio data includes receiving video stream data and audio stream data that corresponds to at least a portion of the video stream data. The method also includes labeling, at least some objects of the video stream data. The method also includes calculating at least one offset value for at least a portion of the audio stream data that corresponds to at least one labeled object of the video stream data. The method also includes synchronizing at least a portion of the video stream data with the portion of the audio stream data. The method also includes labeling at least the portion of the audio stream data that corresponds to the at least one labeled object of the video stream data and generating training data using at least some of the labeled portion of the audio stream data.

Abstract: A computer-implemented system and method includes obtaining a plurality of tasks from a first domain. A machine learning system is trained to perform a first task. A first set of prototypes is generated. The first set of prototypes is associated with a first set of classes of the first task. The machine learning system is updated based on a first loss output. The first loss output includes a first task loss, which takes into account the first set of prototypes. The machine learning system is trained to perform a second task. A second set of prototypes is generated. The second set of prototypes is associated with a second set of classes of the second task. The machine learning system is updated based on a second loss output. The second loss output includes a second task loss, which takes into account the second set of prototypes. The machine learning system is updated based on the second loss output. The machine learning system is fine-tuned with a new task from a second domain.