Abstract: Systems and methods are provided for teaching and shaping the behavior of artificial intelligence agents using human input. A system may be provided that comprises a natural language processing neural network (NLP NN) and a relational reasoning neural network (RR NN). The NLP NN may receive a human observation and environment observations, and output encodings of labels for entities, actions, and/or policies, wherein the environment observations are indicative of states of the environment, and the human observation represents an observation of the environment made by a human. The RR NN may generate cross-modal embeddings from the environment observations and the encodings of labels generated by the NLP NN. The agent neural network may generate actions and/or policies from the environment observation and the cross-modal embeddings.