Abstract: An electronic device and method for using neural language models for long-term action anticipation from videos is provided. The electronic device receives a video that includes one or more objects performing a physical task and generates, based on the video, a first set of tags that corresponds to a first sequence of actions associated with the physical task. The electronic device generates a first prompt for a neural language model based on the first set of tags and predicts, by application of the neural language model on the first prompt, a second set of tags that corresponds to a second sequence of actions associated with the physical task. The second sequence of actions succeeds the first sequence of actions. The electronic device controls a display device to display first prediction information based on the second set of tags.

Abstract: An electronic device and method for object-centric video representation for action prediction is provided. The electronic device extracts a first sequence of video segments from video content associated with a domain and detects a set of objects in the first sequence of video segments. The electronic device generates a set of embeddings based on the first sequence of video segments and the set of objects. The electronic device applies a PTE model on the set of embeddings. The electronic device predicts, based on the application, a set of object-action pairs associated with a second sequence of video segments of the video content. Each object-action pair includes an action to be executed using an object of the set of objects in a video segment of the second sequence of video segments. The second sequence of video segments succeeds the first sequence of video segments in a timeline of the video content.