Abstract: In one implementation of subject-aware background video generation, a processing device generates mask data and foreground feature data from frames of a subject video. The mask data separates a subject depicted in the subject video from an environment therein. The foreground feature data describes the features of the subject. The processing device receives a condition frame that depicts a different environment. A machine-learning model generates a composite video by aligning the subject's movement with the different environment from inputs of the foreground feature data, the mask data, and the condition frame, which conditions the generation of the different environment for the composite video. The processing device then presents the composite video via a user interface.

Abstract: A method for scene perception using video captioning based on a spatio-temporal graph model is described. The method includes decomposing the spatio-temporal graph model of a scene in input video into a spatial graph and a temporal graph. The method also includes modeling a two branch framework having an object branch and a scene branch according to the spatial graph and the temporal graph to learn object interactions between the object branch and the scene branch. The method further includes transferring the learned object interactions from the object branch to the scene branch as privileged information. The method also includes captioning the scene by aligning language logits from the object branch and the scene branch according to the learned object interactions.

Abstract: A method for scene perception using video captioning based on a spatio-temporal graph model is described. The method includes decomposing the spatio-temporal graph model of a scene in input video into a spatial graph and a temporal graph. The method also includes modeling a two branch framework having an object branch and a scene branch according to the spatial graph and the temporal graph to learn object interactions between the object branch and the scene branch. The method further includes transferring the learned object interactions from the object branch to the scene branch as privileged information. The method also includes captioning the scene by aligning language logits from the object branch and the scene branch according to the learned object interactions.