Abstract: In some implementations, a robot host may receive a video associated with assembly using a plurality of sub-objects. The robot host may determine spatio-temporal features based on the video and may identify a plurality of actions represented in the video based on the spatio-temporal features. The robot host may map the plurality of actions to the plurality of sub-objects to generate an assembly plan and may combine output from a point cloud model and output from a color embedding model to generate a plurality of sets of coordinates corresponding to the plurality of sub-objects. The robot host may perform object segmentation to estimate a plurality of grip points and a plurality of widths corresponding to the plurality of sub-objects. Accordingly, the robot host may generate instructions, for robotic machines, based on the assembly plan, the plurality of sets of coordinates, the plurality of grip points, and the plurality of widths.