Abstract: Embodiments of the present disclosure provide a solution for image encoding learning and application. A method for image encoding learning comprises: extracting an image feature representation of a sample image using an image encoder to be trained; extracting a text feature representation of a sample text sequence using a text encoder, the sample text sequency being associated with the sample image; generating, using the text encoder, a predicted text sequence based on the text feature representation and the image feature representation; and training the image encoder at least based on a text error between the predicted text sequence and the sample text sequence.

Abstract: Embodiments of the disclosure provides technologies for image segmentation. The method includes: extracting an image feature representation of a target image using a trained image encoder; for each of a plurality of classes, generating, using a trained text encoder, a text feature representation corresponding to a name of the class, and determining a candidate segmentation map for the target image and a class confidence of the class based on the image feature representation and the text feature representation; selecting, from the plurality of classes, at least one class related to the target image based on a plurality of class confidences determined respectively for the plurality of classes; and determining a target segmentation map for the target image based on the at least one candidate segmentation map and the at least one class confidence determined for the at least one selected class.

Abstract: Embodiments of the disclosure provide a method, apparatus, device and storage medium for feature aggregation. The method comprises: extracting, with an image encoder, an image feature representation of an input image; for each image feature element set of a plurality of image feature element sets divided along a predetermined dimension of the plurality of dimensions in the image feature representation, selecting a first number of image feature elements from the image feature element set based on a ranking of corresponding image feature elements in the image feature element set, and determining an aggregated image feature element by aggregating the selected first number of image feature elements; and determining an aggregated image feature representation of the input image based on a plurality of aggregated image feature elements determined for the plurality of image feature element sets, respectively.

Abstract: Embodiments of the present disclosure provide a solution for image encoding learning and application. A method for image encoding learning comprises: extracting an image feature representation of a sample image using an image encoder to be trained; extracting a text feature representation of a sample text sequence using a text encoder, the sample text sequency being associated with the sample image; generating, using the text encoder, a predicted text sequence based on the text feature representation and the image feature representation; and training the image encoder at least based on a text error between the predicted text sequence and the sample text sequence.

Abstract: There are provided method, apparatus, device, and medium for determining update gradient for contrastive learning model. In the method, a gradient factor of a first type for the contrastive learning model is determined based on a first group of training data and a second group of training data for training the contrastive learning model. The gradient factor of the first type is not used for backpropagation during a training process. In a first stage of the training process, a gradient factor of a second type associated with the first group of training data is determined based on the contrastive learning model. The gradient factor of the second type is used for backpropagation during the training process. Gradient is obtained for updating the contrastive learning model based on the gradient factor of the first type and the gradient factor of the second type associated with the first group of training data.

Abstract: A method of training and applying contrastive learning model. The method includes obtaining a sample set and label information for training contrastive learning model, the sample set including a plurality of first samples of a first modality and a plurality of second samples of a second modality, the label information indicating a correlation between samples of the plurality of first samples and samples of the plurality of second samples; determining whether sample mixing is to be performed on the first modality or the second modality; in accordance with a determination that sample mixing is to be performed on the first modality, generating at least one first mixed sample of the first modality by mixing at least one pair of first samples among the plurality of first samples; and training the contrastive learning model at least based on the at least one first mixed sample and first mixed label information.

Abstract: Methods, apparatuses, a device, and a medium for training a contrastive learning model are provided. In a method, a plurality of sample sets for training the contrastive learning model are obtained, and the plurality of sample sets comprises a first sample set and a second sample set. A first target sample set is selected from the first sample set and the second sample set according to a predetermined rule. A first set of samples are determined based on the first target sample set according to a predefined batch size. The contrastive learning model is trained using the first set of samples. In this way, on the one hand, performance degradation of the contrastive learning model due to sample set bias may be avoided; on the other hand, a forgetting problem in the training process may be alleviated.

Abstract: A method of contrastive learning comprises: determining, based on a model construction criterion, a first encoder for a first modality and a second encoder for a second modality; constructing a first contrastive learning model, the first contrastive learning model comprising the first encoder and a third encoder for the second modality, and a model capacity of the third encoder being greater than a model capacity of the second encoder; performing pre-training of the first contrastive learning model based on a first training dataset for the first modality and the second modality; and providing the pre-trained first encoder in the pre-trained first contrastive learning model for a downstream task. Because only the model capacity of one encoder is increased in the pre-training stage, model performance may be improved without increasing model training overhead during downstream task fine-tuning and model running overhead during model application.