Abstract: A method for training a model for classifying videos as real or fake can include generating image tiles and audio data segments from an input video, generating a sequence of image embeddings based on the image tiles using a visual encoder and a sequence of audio embeddings based on the audio data segments using an audio encoder, transforming, using a V2A network, a first subset of the sequence of image embeddings into synthetic audio embeddings, transforming, using an A2V network, a first subset of the sequence of audio embeddings into synthetic image embeddings, updating the sequence of image embeddings by using the synthetic image embeddings, updating the sequence of audio embeddings using the synthetic audio embeddings, training the encoders and the networks using the updated sequences of image embeddings and audio embeddings, and training a classifier using the trained encoders and the trained networks.

Abstract: Audio deepfake detection (ADD) is crucial to combat the potential misuse of synthesized speech from generative AI models. Existing ADD models suffer from generalization issues, with a large performance discrepancy seen between in-domain and out-of-domain data. Also, the black-box nature of the existing models limits their use in real-world scenarios where interpretation capabilities are required. Described is a new ADD training framework that explicitly uses the Style and LInguistics Mismatch (SLIM) in the fake class to separate it from the real class. The style-linguistics dependency is learned through a self-supervised pretraining stage, where only real samples are needed. Using frozen frontend encoders, SLIM outperforms benchmark methods on out-of-domain datasets while providing competitive results on in-domain datasets. The features learned by SLIM can be directly used to quantify the style-linguistics mismatch of deepfake samples, hence facilitating explainability.

Abstract: An exemplary method for detecting deepfake images and providing customized analysis comprises: receiving, from a user, a textual user inquiry regarding an image; inputting the textual inquiry and the image into a deepfake detection model, wherein the deepfake detection model comprises: an image encoder for generating a plurality of image embeddings based on the image; a text encoder for generating a plurality of textual embeddings based on the textual inquiry; one or more layers for generating a plurality of answer embeddings; and a language model for generating a textual analysis based on the plurality of answer embeddings; and outputting the textual analysis, wherein the textual analysis includes a classification result of whether the image is fake and further includes one or more visual features in the image and one or more attributes of the one or more visual features that contribute to the classification result.

Abstract: An exemplary method for detecting fake audios comprises: converting audio data into an image representation of the audio data; providing the image representation of the audio data to a trained machine-learning model, the machine learning model: generating, using a trained self-attention branch, one or more representation embeddings corresponding to the image representation of the audio data; and receiving, using a trained classifier component, the one or more representation embeddings and outputting a classification result. The machine-learning model is trained by: in a first stage, training one or more self- and cross-attention components via contrastive learning, each self- and cross-attention component comprises a first self-attention branch, a second self-attention branch, and a cross-attention branch; and in a second stage, training the classifier component; and providing the classification result.

Abstract: A method for training a model for classifying videos as real or fake can include generating image tiles and audio data segments from an input video, generating a sequence of image embeddings based on the image tiles using a visual encoder and a sequence of audio embeddings based on the audio data segments using an audio encoder, transforming, using a V2A network, a first subset of the sequence of image embeddings into synthetic audio embeddings, transforming, using an A2V network, a first subset of the sequence of audio embeddings into synthetic image embeddings, updating the sequence of image embeddings by using the synthetic image embeddings, updating the sequence of audio embeddings using the synthetic audio embeddings, training the encoders and the networks using the updated sequences of image embeddings and audio embeddings, and training a classifier using the trained encoders and the trained networks.

Abstract: An exemplary method for detecting fake audios comprises: converting audio data into an image representation of the audio data; providing the image representation of the audio data to a trained machine-learning model, the machine learning model: generating, using a trained self-attention branch, one or more representation embeddings corresponding to the image representation of the audio data; and receiving, using a trained classifier component, the one or more representation embeddings and outputting a classification result. The machine-learning model is trained by: in a first stage, training one or more self- and cross-attention components via contrastive learning, each self- and cross-attention component comprises a first self-attention branch, a second self-attention branch, and a cross-attention branch; and in a second stage, training the classifier component; and providing the classification result.

Abstract: An exemplary method for detecting deepfake images and providing customized analysis comprises: receiving, from a user, a textual user inquiry regarding an image; inputting the textual inquiry and the image into a deepfake detection model, wherein the deepfake detection model comprises: an image encoder for generating a plurality of image embeddings based on the image; a text encoder for generating a plurality of textual embeddings based on the textual inquiry; one or more layers for generating a plurality of answer embeddings; and a language model for generating a textual analysis based on the plurality of answer embeddings; and outputting the textual analysis, wherein the textual analysis includes a classification result of whether the image is fake and further includes one or more visual features in the image and one or more attributes of the one or more visual features that contribute to the classification result.

Abstract: An exemplary method for reducing bias in a training image dataset for training a machine-learning model comprises: receiving a plurality of text strings comprising at least one text string describing each image in the training image dataset; generating a plurality of embeddings based on the plurality of text strings; identifying, based on the plurality of embeddings, a plurality of visual features in the training image dataset; identifying one or more correlations between the plurality of visual features in the training image dataset; receiving a user input identifying at least one biased correlation from the one or more correlations; and training the machine-learning model at least partially by adjusting one or more data sampling weights associated with one or more training images in the training image dataset based on the user input.

Abstract: A method for training a model for classifying videos as real or fake can include generating image tiles and audio data segments from an input video, generating a sequence of image embeddings based on the image tiles using a visual encoder and a sequence of audio embeddings based on the audio data segments using an audio encoder, transforming, using a V2A network, a first subset of the sequence of image embeddings into synthetic audio embeddings, transforming, using an A2V network, a first subset of the sequence of audio embeddings into synthetic image embeddings, updating the sequence of image embeddings by using the synthetic image embeddings, updating the sequence of audio embeddings using the synthetic audio embeddings, training the encoders and the networks using the updated sequences of image embeddings and audio embeddings, and training a classifier using the trained encoders and the trained networks.

Abstract: An exemplary method for detecting deepfake images and providing customized analysis comprises: receiving, from a user, a textual user inquiry regarding an image; inputting the textual inquiry and the image into a deepfake detection model, wherein the deepfake detection model comprises: an image encoder for generating a plurality of image embeddings based on the image; a text encoder for generating a plurality of textual embeddings based on the textual inquiry; one or more layers for generating a plurality of answer embeddings; and a language model for generating a textual analysis based on the plurality of answer embeddings; and outputting the textual analysis, wherein the textual analysis includes a classification result of whether the image is fake and further includes one or more visual features in the image and one or more attributes of the one or more visual features that contribute to the classification result.

Abstract: An exemplary method for reducing bias in a training image dataset for training a machine-learning model comprises: receiving a plurality of text strings comprising at least one text string describing each image in the training image dataset; generating a plurality of embeddings based on the plurality of text strings; identifying, based on the plurality of embeddings, a plurality of visual features in the training image dataset; identifying one or more correlations between the plurality of visual features in the training image dataset; receiving a user input identifying at least one biased correlation from the one or more correlations; and training the machine-learning model at least partially by adjusting one or more data sampling weights associated with one or more training images in the training image dataset based on the user input.

Abstract: An exemplary method for detecting fake audios comprises: converting audio data into an image representation of the audio data; providing the image representation of the audio data to a trained machine-learning model, the machine learning model: generating, using a trained self-attention branch, one or more representation embeddings corresponding to the image representation of the audio data; and receiving, using a trained classifier component, the one or more representation embeddings and outputting a classification result. The machine-learning model is trained by: in a first stage, training one or more self- and cross-attention components via contrastive learning, each self- and cross-attention component comprises a first self-attention branch, a second self-attention branch, and a cross-attention branch; and in a second stage, training the classifier component; and providing the classification result.

Abstract: Power-management techniques are described. In the power-management techniques, a computer system (such as a cloud-based computer system or a local computer system) dynamically controls an energy source that provides electrical power to a load at a location. Notably, the computer system selectively transitions between use of a (typically external) energy source (such as a power grid associated with a utility or power supplier, hydroelectric, a generator or a solar array) and an energy storage device (such as a battery or a battery array, thermal power storage, a mechanical flywheel, or another type of energy storage device) at the location to provide electrical power to at least a dominant (e.g., highest power-consuming) load at the location. Note that the selective transitions may be based at least in part on a price of electricity and/or carbon intensity of electricity.