Abstract: In examples, a developer may define a set of computer-controlled agent attributes, which may be processed by a generative multimodal machine learning model in conjunction with background information associated with a virtual environment (e.g., “lore”) and other agent information to generate multimodal model output with which to control the behavior of the computer-controlled agent. Thus, a player may interact with the computer-controlled agent, such that user input from the player is processed using the ML model to generate model output to affect the behavior of the computer-controlled agent, thereby enabling the user and the computer-controlled agent to interact. As compared to manual dialogue authoring, use of agent information to define the behavior of a computer-controlled agent may result in reduced effort on the part of a creator while also offering increased depth and variety for computer-controlled agents of a virtual environment.