Abstract: Various systems may benefit from computer learning. For example, robotics systems may benefit from learning actions, such as manipulation actions, from unconstrained videos. A method can include processing a set of video images to obtain a collection of semantic entities. The method can also include processing the semantic entities to obtain at least one visual sentence from the set of video images. The method can further include deriving an action plan for a robot from the at least one visual sentence. The method can additionally include implementing the action plan by the robot. The processing the set of video images, the processing semantic entities, and the deriving the action plan can be computer implemented.