Abstract: System, method and/or computer readable medium for audio visual sound source Separation that applies cross-modal meta consistency learning. Inputs include an audio spectrogram that represents first sounds second sounds, a first video of a first sound producing object, and a second video of a second sound producing object. Audio features and audio tokens are generated by applying audio encoders to the audio spectrogram. Visual tokens are generated by applying a visual encoder to the first video and the second video. Respective audio-visual features are obtained based on combining the audio token with the respective visual tokens. Based on the respective audio-visual features, first and second separated audio masks are generated by applying a decoder.

Abstract: Methods, devices and computer-readable media for processing a compressed video to perform an inference task are disclosed. Processing the compressed video may include selecting a subset of frame encodings of the compressed video, or zero or more modalities (RGB, motion vectors, residuals) of a frame encoding, for further processing to perform the inference task. Pre-existing motion vector and/or residual information in frame encodings of the compressed video are leveraged to adaptively and efficiently perform the inference task. In some embodiments, the inference task is an action recognition task, such as a human action recognition task.