Abstract: A method and apparatus for generating a captioning device, and a method and apparatus for outputting a caption. The method for generating a captioning device comprises: acquiring a sample image set; inputting the sample image set into an image encoder of a sentence generator, so as to output an object set; grouping the object set into a first object set and a second object set, wherein the first object set is an object set that is included within a preset object set, and the second object set is an object set that is excluded from the preset object set; inputting, into a sentence decoder of the sentence generator, the object set output by the image encoder, and performing a beam search in a decoding step by taking the first object set and the second object set as constraint conditions, so as to generate a pseudo-image sentence pair set; and training the sentence generator by taking the pseudo-image sentence pair set as a sample set, so as to obtain a captioning device.

Abstract: The present disclosure provides a multi-modal pre-training method and apparatus. The method includes: sampling a video in a video-text pair to obtain a first video frame sequence; performing word segmentation processing on a text in the video-text pair to obtain a first word segmentation sequence; masking on the first video frame sequence to obtain a second video frame sequence; masking on the first word segmentation sequence to obtain a second word segmentation sequence; encoding the first video frame sequence to obtain a first video feature, and encoding the first word segmentation sequence to obtain a first word segmentation feature; encoding the second video frame sequence to obtain a second video feature, and encoding the second word segmentation sequence to obtain a second word segmentation feature; performing multi-modal pre-training by using the first video feature, the first word segmentation feature, the second video feature and the second word segmentation feature.

Abstract: An image paragraph description generating method and apparatus, a medium and an electronic device. The method comprises: obtaining image features of an image (S101); determining the topic of the image according to the image features by using a convolutional automatic coding method (S102); and determining image description information of the image according to the topic by using a long short-term memory (LSTM)-based paragraph coding method (S103), wherein the LSTM comprises a sentence-level LSTM and a paragraph-level LSTM.

Abstract: Disclosed in the present application are an action recognition method and apparatus. The method comprises: acquiring a video clip, and determining at least two target objects in the video clip; for each of the at least two target objects, connecting positions of the target object in various video frames of the video clip, so as to construct a spatiotemporal graph of the target object; dividing at least two spatiotemporal graphs, which are constructed for the at least two target objects, into a plurality of spatiotemporal graph subsets, and determining a finally selected subset from the plurality of spatiotemporal graph subsets; and determining an action category of the action between the target objects that is indicated by a relationship between the spatiotemporal graphs included in the finally selected subset as the action category of an action included in the video clip.

Abstract: The present disclosure relates to the technical field of image processing, and in particular to an image description generation method, apparatus and system, and a medium and an electronic device.

Abstract: The present disclosure relates to the technical field of image processing, and in particular to an image description generation method, apparatus and system, and a medium and an electronic device.

Abstract: An image paragraph description generating method and apparatus, a medium and an electronic device. The method comprises: obtaining image features of an image (S101); determining the topic of the image according to the image features by using a convolutional automatic coding method (S102); and determining image description information of the image according to the topic by using a long short-term memory (LSTM)-based paragraph coding method (S103), wherein the LSTM comprises a sentence-level LSTM and a paragraph-level LSTM.