Abstract: A system for activity localization in videos is described, comprising a visual concept detection module, which produces a plurality of first visual concept vectors each representing a probability of containing visual concepts for one of a plurality of sampled frames sampled from an input video; wherein each of the plurality of first visual concept vectors dot-product with a second visual concept vector extracted from a given query sentence, resulting a visual-semantic correlation score; a semantic activity proposal generation module, which generates semantic activity proposals by temporally grouping frames with a high visual-semantic correlation score; and a proposal evaluation and refinement module, which takes the semantic activity proposals, the visual concept vectors and the query sentence as input, and outputs alignment scores and refined boundaries for the proposals. The disclosure also relates to methods thereof.

Abstract: For video captioning, with an encoding module and a decoding module, the encoding module comprises a plurality of encoding units each receiving a set of video frames, wherein the sets of video frames received by two neighboring encoding units are in chronological order; and the encoding units each producing a spatially attended feature, so that the plurality of encoding units produce a spatially attended feature sequence; and the decoding module comprises a decoding unit chronologically receiving a temporally attended feature obtained from the spatially attended features sequence. Also disclosed is a method thereof.