Abstract: A method includes obtaining a video and a relational space-time query and identifying at least one type of the relational space-time query. The at least one identified type of the relational space-time query represents at least one of: an activity type, an object type, or a time type. The method also includes learning correlations among activities, objects, and time in the video using, one or more cross-attention models. The method further includes obtaining one or more predictions generated using one or more outputs of the one or more cross-attention models based on the at least one identified type of the relational space-time query. In addition, the method includes generating a response to the relational space-time query based on the one or more predictions.

Abstract: A method includes obtaining a video and a relational space-time query and identifying at least one type of the relational space-time query. The at least one identified type of the relational space-time query represents at least one of: an activity type, an object type, or a time type. The method also includes learning correlations among activities, objects, and time in the video using, one or more cross-attention models. The method further includes obtaining one or more predictions generated using one or more outputs of the one or more cross-attention models based on the at least one identified type of the relational space-time query. In addition, the method includes generating a response to the relational space-time query based on the one or more predictions.

Abstract: A method to perform hiearchical video segmentation includes: defining voxels over a spatio-temporal video; grouping into segments contiguous voxels that display similar characteristics including similar appearance or motion; determining a trajectory-based feature that complements color and optical flow cues, wherein trajectory cues are probabilistically meaningful histograms combinable for use in a graph-based framework; and applying a max-margin module for cue combination that learns a supervised distance metric for region dissimilarity that combines color, flow and trajectory features.

Abstract: A method to perform hiearchical video segmentation includes: defining voxels over a spatio-temporal video; grouping into segments contiguous voxels that display similar characteristics including similar appearance or motion; determining a trajectory-based feature that complements color and optical flow cues, wherein trajectory cues are probabilistically meaningful histograms combinable for use in a graph-based framework; and applying a max-margin module for cue combination that learns a supervised distance metric for region dissimilarity that combines color, flow and trajectory features.