Abstract: The present disclosure provides methods, apparatuses, and computer-readable media for step discovery and localization in an instructional video. In some embodiments, the method includes extracting, from the instructional video using a transformer model, a plurality of step slots corresponding to a plurality of procedure steps depicted in the instructional video, matching, using an order-aware sequence-to-sequence alignment model, a plurality of video segments of the instructional video to the plurality of step slots, generating a temporally-ordered plurality of video segments from the plurality of video segments, receiving a user query requesting a procedure step, selecting, from the plurality of video segments of the instructional video, a corresponding video segment corresponding to the requested procedure step, and providing, in response to the user query, the corresponding video segment and the matching textual step description of the corresponding video segment.

Abstract: A first graph which is a flow graph is converted into a second graph using a first algorithm. The second graph, combined with a second algorithm using a dynamic programming recursion, is compared with a video and a best matching thread, with respect to the video, is found through the second graph. The best matching thread may then be used for a user-assistance function. The dynamic programming recursion reduces computational effort needed by the computer performing the matching.

Abstract: The present disclosure provides methods and apparatuses for probabilistic procedure planning for generating a plan based on a goal relating to an end state. In some embodiments, a method includes receiving a request from a user to generate an action plan comprising T intermediate actions between a start state and the end state. The method further includes constructing an input query matrix based on T, the start state, the end state, positional encodings, and pseudo-random noise information. The method further includes generating, using a machine learning transformer decoder, the action plan based on the input query matrix and a plurality of learnable vectors. The method further includes providing the action plan to the user. The action plan indicates a probability distribution of a plurality of distinct action sequences, to be performed by the user, that transform the start state to the end state.

Abstract: A method is provided. The method includes receiving a first signal and a second signal, generating a first sequence based on the first signal by embedding at least one feature in the first signal, generating a second sequence based on the second signal by embedding at least one feature in the second signal, determining a minimum cost path of aligning the first sequence with the second sequence based on the embedded at least one feature in the first sequence and the embedded at least one feature in the second sequence, and aligning the first sequence with second sequence based on the determined minimum cost path.