Abstract: Methods and artificial intelligence agents are provided to train or guide an artificial intelligence agent. Visual data and/or text data are received from the artificial intelligence agent and/or an environment of the artificial intelligence agent. A text prompt is generated based on the visual information and/or the text data. The text prompt is provided to an ultra-large language model. Text output of the ultra-large language model is received in response to the text prompt. The artificial intelligence agent is supplied with the text output of the ultra-large language model and/or the text output converted into an alternative format. The artificial intelligence agent is configured to select an action, a series of actions, and/or the policy based on the state of an environment of the artificial intelligence agent and on the text output of the ultra-large language model and/or the text output converted into the alternative format.

Abstract: Systems and methods are provided that may generate, based on an agent neural network, actions and/or policies for an environment, the environment comprising an apparatus and/or a software component. The actions and/or the policies may be enacted in the environment. A human observation may be received (“hijacked”) from a voice network module. A natural language processing neural network may output encodings of labels for entities, actions, and/or policies, when the human observation and environment observations are supplied as input to the natural language processing neural network. The environment observations are indicative of states of the environment. A relational reasoning neural network may generate cross-modal embeddings from the environment observations and the encodings of labels for entities, actions, and/or policies. The agent neural network may generate the actions and/or the policies from the environment observation and the cross-modal embeddings.

Abstract: Systems and methods are provided for teaching and shaping the behavior of artificial intelligence agents using human input. A system may be provided that comprises a natural language processing neural network (NLP NN) and a relational reasoning neural network (RR NN). The NLP NN may receive a human observation and environment observations, and output encodings of labels for entities, actions, and/or policies, wherein the environment observations are indicative of states of the environment, and the human observation represents an observation of the environment made by a human. The RR NN may generate cross-modal embeddings from the environment observations and the encodings of labels generated by the NLP NN. The agent neural network may generate actions and/or policies from the environment observation and the cross-modal embeddings.