Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling agents. In particular, an interactive agent can be controlled based on multi-modal inputs that include both an observation image and a natural language text sequence.

Abstract: A system is provided that allows a spectator to spectate video gameplay over a network. Recorded information associated with a selected recording of a player playing a video game that includes a game engine is received over the network. The recorded information includes game commands that were input to the game engine as the player was playing the game, and inputs the player made to the game to control it. The recorded information is replayed to the game at the timing in which this information was originally recorded, where this replay includes replaying the game commands to the game engine. The replay generates a playback of the selected recording which is displayed on the display screen of the spectator's computer. Upon receiving a request from the spectator to view the inputs the player made to the game they are highlighted on the screen during the playback of the selected recording.