Abstract: A token engine may receive a data file that describes a three-dimensional (3D) virtual environment using tags for attributes in the 3D virtual environment. The token engine generates a set of one-hot token vectors from the tags in the data file and a set of metadata vectors from metadata in the data file, where one or more metadata vectors in the set of metadata vectors correspond to one or more one-hot token vectors in the set of one-hot token vectors. The token engine combines the set of one-hot token vectors and the set of metadata vectors. The token engine provides a combined set of one-hot token vectors and metadata vectors as input to a deep-learning model.

Abstract: Real-time extraction of human poses from video data for animation of avatars. In some implementations, a computer-implemented method includes obtaining an input video including a plurality of video frames in a sequence that depict movement of a person based on a plurality of poses of the person in the input video. Keypoints of the person are detected in the video frames of the input video data, and a sequence of 3D body poses are determined that correspond to the plurality of poses of the person in the video frames of the input video. Determining the 3D body poses includes using a spatial-temporal transformer to determine joint angles of the keypoints, where the spatial-temporal transformer separately encodes inputs in spatial dimensions within each video frame and a temporal dimension across the video frames.