Abstract: Methods and systems of processing audio data with a multi-stage audio front end model is provided. A one-dimensional audio waveform is received as input and processed using a multi-stage audio frontend model to convert the one-dimensional waveform into a two-dimensional matrix representing features of the audio waveform. The multi-stage learnable audio frontend model is configured to apply a first filterbank to the audio waveform to generate a first time-frequency representation of the audio waveform; apply a first decimation filter to the audio waveform to generate a first decimated audio input; apply a second filterbank to the first decimated audio input to generate a second time-frequency representation of the audio waveform; and stack the first time-frequency representation and the second time-frequency representation together to generate the two-dimensional matrix.

Abstract: A method includes defining, by a text encoder, a set of text embeddings for a text prompt indicative of a search query for video content having audio data indicative of a sound feature that is defined as a search parameter of the search query, and ranking a plurality of audio embeddings indicative of a plurality of audio signals and provided in a vector database using the set of text embeddings of the search query. The method further includes detecting a relevant audio record associated with an identified audio embedding from among the ranked audio embeddings, and outputting a relevant video content associated with the relevant audio record to have a computing device play the video content, the relevant video content being obtained from among a plurality of video content.

Abstract: Methods and systems for predicting aggressive behavior associated with a surveillance scene. Images are generated from a camera of a surveillance scene, and audio is also generated. A local computing system executes an object classification model on the images to predict one or more classes of objects in the scene. The local computing system also executes a sound-event detection model on the audio to predict one or more classes of events occurring in the scene. Metadata is generated associated with the image-based classes and audio-based classes. The metadata is transferred to a remote computing system that executes a knowledge graph on the metadata to implement knowledge graph-based reasoning to predict aggressive behavior occurring in the surveillance scene based on the metadata. The metadata associated with the predicted aggressive behavior is labeled as such, and control commands are output accordingly.

Abstract: A method for real-time sound event detection on an embedded device includes pretraining a contrastive language-audio pretraining model as an audio foundation model and preparing offline multimodal query prototypes for sound events of interest. The pretrained model and query prototypes are deployed on an embedded device. The device receives an input audio stream and extracts audio embeddings using the pretrained model. Similarity scores are calculated between the extracted audio embeddings and the prepared query prototypes. The presence of a sound event is determined based on the calculated similarity scores, and a real-time sound event detection result is output. The system includes a memory storing the pretrained model and query prototypes, an audio input interface, and a processor configured to perform the extraction, calculation, determination, and output operations. A non-transitory computer-readable medium stores instructions that, when executed, cause a processor to perform the method.

Abstract: A method includes defining, by a text encoder, a set of text embeddings for a text prompt indicative of a search query for video content having audio data indicative of a sound feature that is defined as a search parameter of the search query, and ranking a plurality of audio embeddings indicative of a plurality of audio signals and provided in a vector database using the set of text embeddings of the search query. The method further includes detecting a relevant audio record associated with an identified audio embedding from among the ranked audio embeddings, and outputting a relevant video content associated with the relevant audio record to have a computing device play the video content, the relevant video content being obtained from among a plurality of video content.

Abstract: Systems and methods for anomaly detection in embedded systems. One example provides an anomaly detection system including a device configured to be connected to a server. The device includes an electronic processor and a memory. The memory stores a first application. The electronic processor is configured to transmit a request for an anomaly detection application. The anomaly detection application includes a set of instructions for detecting anomalies of the device. The electronic processor is configured to receive, from the server, the anomaly detection application, and replace the first application with the anomaly detection application within the memory.

Abstract: Systems and methods for audio event detection. One example provides an event detection system comprising a plurality of audio devices. Each of the plurality of audio devices is configured to be communicatively coupled to a server and includes an electronic processor. The electronic processor is configured to detect, via a microphone, audio, and determine an audio event within the audio. The electronic processor is configured to receive an image from a camera, associate the image data with the audio event to generate event metadata, and transmit the event metadata, the audio, and the image to the server.

Abstract: Methods and systems for predicting aggressive behavior associated with a surveillance scene. Images are generated from a camera of a surveillance scene, and audio is also generated. A local computing system executes an object classification model on the images to predict one or more classes of objects in the scene. The local computing system also executes a sound-event detection model on the audio to predict one or more classes of events occurring in the scene. Metadata is generated associated with the image-based classes and audio-based classes. The metadata is transferred to a remote computing system that executes a knowledge graph on the metadata to implement knowledge graph-based reasoning to predict aggressive behavior occurring in the surveillance scene based on the metadata. The metadata associated with the predicted aggressive behavior is labeled as such, and control commands are output accordingly.

Abstract: A system for automatically selecting a sound recognition model for an environment based on audio data and image data associated with the environment. The system includes a camera, a microphone, a memory including a plurality of sound recognition models, and an electronic processor. The electronic processor is configured to receive the audio data associated with the environment from the microphone, receive the image data associated with the environment from the camera, and determine one or more characteristics of the environment based on the audio data and the image data. The electronic processor is also configured to select the sound recognition model from the plurality of sound recognition models based on the one or more characteristics of the environment, receive additional audio data associated with the environment from the microphone, and analyze the additional audio data using the sound recognition model to perform a sound recognition task.

Abstract: Methods and systems of processing audio data with a multi-stage audio front end model is provided. A one-dimensional audio waveform is received as input and processed using a multi-stage audio frontend model to convert the one-dimensional waveform into a two-dimensional matrix representing features of the audio waveform. The multi-stage learnable audio frontend model is configured to apply a first filterbank to the audio waveform to generate a first time-frequency representation of the audio waveform; apply a first decimation filter to the audio waveform to generate a first decimated audio input; apply a second filterbank to the first decimated audio input to generate a second time-frequency representation of the audio waveform; and stack the first time-frequency representation and the second time-frequency representation together to generate the two-dimensional matrix.

Abstract: Systems and methods for anomaly detection in embedded systems. One example provides an anomaly detection system including a device configured to be connected to a server. The device includes an electronic processor and a memory. The memory stores a first application. The electronic processor is configured to transmit a request for an anomaly detection application. The anomaly detection application includes a set of instructions for detecting anomalies of the device. The electronic processor is configured to receive, from the server, the anomaly detection application, and replace the first application with the anomaly detection application within the memory.

Abstract: Systems and methods for audio event detection. One example provides an event detection system comprising a plurality of audio devices. Each of the plurality of audio devices is configured to be communicatively coupled to a server and includes an electronic processor. The electronic processor is configured to detect, via a microphone, audio, and determine an audio event within the audio. The electronic processor is configured to receive an image from a camera, associate the image data with the audio event to generate event metadata, and transmit the event metadata, the audio, and the image to the server.

Abstract: A system for automatically selecting a sound recognition model for an environment based on audio data and image data associated with the environment. The system includes a camera, a microphone, a memory including a plurality of sound recognition models, and an electronic processor. The electronic processor is configured to receive the audio data associated with the environment from the microphone, receive the image data associated with the environment from the camera, and determine one or more characteristics of the environment based on the audio data and the image data. The electronic processor is also configured to select the sound recognition model from the plurality of sound recognition models based on the one or more characteristics of the environment, receive additional audio data associated with the environment from the microphone, and analyze the additional audio data using the sound recognition model to perform a sound recognition task.