Abstract: Techniques for audio understanding using fixed language models are provided. In one aspect, a system for performing audio understanding tasks includes: a fixed text embedder for, on receipt of a prompt sequence having (e.g., from 0-10) demonstrations of an audio understanding task followed by a new question, converting the prompt sequence into text embeddings; a pretrained audio encoder for converting the prompt sequence into audio embeddings; and a fixed autoregressive language model for answering the new question using the text embeddings and the audio embeddings. A method for performing audio understanding tasks is also provided.

Abstract: A bi-directional spatial-temporal transformer neural network (BDSTT) is trained to predict original coordinates of a skeletal joint in a specific frame through relative relationships of the skeletal joint to other joints and to the state of the skeletal joint in other frames. Obtain a plurality of frames comprising coordinates of the skeletal joint and coordinates of other joints. Produce a spatially masked frame by masking the original coordinates of the skeletal joint. Provide the specific frame, the spatially masked frame, and at least one more frame to a coordinate prediction head of the BDSTT. Obtain, from the coordinate prediction head, a prediction of coordinates for the skeletal joint. Adjust parameters of the BDSTT until a mean-squared error, between the prediction of coordinates for the skeletal joint and the original coordinates of the skeletal joint, converges.

Abstract: One or more computer processors obtain an initial subnetwork at a target sparsity and an initial pruning mask from a pre-trained self-supervised learning (SSL) speech model. The one or more computer processors finetune the initial subnetwork, comprising: the one or more computer processors zero out one or more masked weights in the initial subnetwork specified by the initial pruning mask; the one or more computer processors train a new subnetwork from the zeroed out subnetwork; the one or more computer processors prune one or more weights of lowest magnitude in the new subnetwork regardless of network structure to satisfy the target sparsity. The one or more computer processors classify an audio segment with the finetuned subnetwork.

Abstract: A computer-implemented method is provided of using a machine learning model for disentanglement of prosody in spoken natural language. The method includes encoding, by a computing device, the spoken natural language to produce content code. The method further includes resampling, by the computing device without text transcriptions, the content code to obscure the prosody by applying an unsupervised technique to the machine learning model to generate prosody-obscured content code. The method additionally includes decoding, by the computing device, the prosody-obscured content code to synthesize speech indirectly based upon the content code.

Abstract: A bi-directional spatial-temporal transformer neural network (BDSTT) is trained to predict original coordinates of a skeletal joint in a specific frame through relative relationships of the skeletal joint to other joints and to the state of the skeletal joint in other frames. Obtain a plurality of frames comprising coordinates of the skeletal joint and coordinates of other joints. Produce a spatially masked frame by masking the original coordinates of the skeletal joint. Provide the specific frame, the spatially masked frame, and at least one more frame to a coordinate prediction head of the BDSTT. Obtain, from the coordinate prediction head, a prediction of coordinates for the skeletal joint. Adjust parameters of the BDSTT until a mean-squared error, between the prediction of coordinates for the skeletal joint and the original coordinates of the skeletal joint, converges.

Abstract: A method, a structure, and a computer system for decomposing speech. The exemplary embodiments may include one or more encoders for generating one or more encodings of a speech input comprising rhythm information, pitch information, timbre information, and content information, and a decoder for decoding the one or more encodings.

Abstract: A method, a structure, and a computer system for decomposing speech. The exemplary embodiments may include one or more encoders for generating one or more encodings of a speech input comprising rhythm information, pitch information, timbre information, and content information, and a decoder for decoding the one or more encodings.

Abstract: A system that detects heartbeats includes a sensor or a transducer and algorithms based on deep learning. The algorithms employ techniques of artificial intelligence that enable the system to extract heartbeat features under low signal-to-noise-ratio (SNR) conditions when a user is exercising. The algorithms can be applied to various technologies for heart rate monitoring such as ultrasound Doppler, photoplethysmogram (PPG), electrocardiogram (EKG), acoustic, pressure/force sensing and laser/RF Doppler, among other types of sensing methods.

Abstract: A system that detects heartbeats includes a sensor or a transducer and algorithms based on deep learning. The algorithms employ techniques of artificial intelligence that enable the system to extract heartbeat features under low signal-to-noise-ratio (SNR) conditions when a user is exercising. The algorithms can be applied to various technologies for heart rate monitoring such as ultrasound Doppler, photoplethysmogram (PPG), electrocardiogram (EKG), acoustic, pressure/force sensing and laser/RF Doppler, among other types of sensing methods.