Abstract: Aspects of the present disclosure provide systems and methods which utilizes machine learning techniques to provide enhanced accessibility features to a game. An accessibility service is provided which is capable of instantiating one or more machine learning models which can process current gameplay states and generate commands to assist users during gameplay. The accessibility commands may be provided to a game and used to supplement or modify user provided inputs in order to compensate for specific user needs. In further aspects, an accessibility user interface is provided which allows a user to dynamically enable or disable accessibility features during gameplay. The user interface is operable to receive accessibility selections and provide the selection data to an accessibility service during gameplay.

Abstract: Aspects of the present disclosure relate to a personalized agent service that generates and evolves customized agents that can be instantiated in-game to play with users. Machine learning models are trained to control the agent's interactions with the game environment and the user during gameplay. As the user continues to play with the agent, the one or more machine learning models develop gameplay styles for the agent that complement the user's preferred playstyle, incorporate the user's preferred strategies, and is generally customized for interaction with the user. The agent personalization data generated during gameplay is stored by the service. An application programming interface is provided by the personalized agent service. Using the API, games can import agent personalization data in order to customize in-game non-player characters (NPCs), thereby customizing the in-game NPCs in accordance with the user's preferences.

Abstract: Aspects of the present disclosure relate to a personalized agent service that generates and evolves customized agents that can be instantiated in-game to play with users. Machine learning models are trained to control the agent's interactions with the game environment and the user during gameplay. A user may request that a personalized agent join the user's gameplay session. The user device sends a request for the personalized agent to a game platform. The game platform determines whether the user has a license to execute a second instance of the game. When the user has a license to execute a second instance of the game, the second instance of the game may be executed on the user device. Information received from a personalized agent service is used to instantiate a personalized agent in the second instance of the game.

Abstract: Aspects of the present disclosure relate to grounded multimodal agent interactions, where a user input is processed using a multimodal machine learning model to generate model output. The model output may then be processed to affect the behavior of an application, for example to enable a user to control the application and/or to facilitate user interactions with a conversational agent, among other examples. In some instances, at least a part of the model output may be executed or parsed, for example to call an application programming interface or function of the application. Thus, use of a multimodal machine learning model according to aspects described herein may enable the use of user-provided natural language input to affect the behavior of an application accordingly.

Abstract: Aspects of the present disclosure relate to grounded multimodal agent interactions, where a user input is processed using a multimodal machine learning model to generate model output. The model output may then be processed to affect the behavior of an application, for example to enable a user to control the application and/or to facilitate user interactions with a conversational agent, among other examples. In some instances, at least a part of the model output may be executed or parsed, for example to call an application programming interface or function of the application. Thus, use of a multimodal machine learning model according to aspects described herein may enable the use of user-provided natural language input to affect the behavior of an application accordingly.

Abstract: Aspects of the present disclosure relate to grounded multimodal agent interactions, where a natural language user input is processed using a multimodal machine learning model to generate model output. The model output may then be processed to affect the behavior of an application, for example to enable a user to control the application and/or to facilitate user interactions with a conversational agent, among other examples. In some instances, at least a part of the model output may be executed or parsed, for example to generate three dimensional objects within a coding application. Thus, use of a multimodal machine learning model according to aspects described herein may enable the use of user-provided natural language input to affect the behavior of an application accordingly.

Abstract: In examples, a developer may define a set of computer-controlled agent attributes, which may be processed by a generative multimodal machine learning model in conjunction with background information associated with a virtual environment (e.g., “lore”) and other agent information to generate multimodal model output with which to control the behavior of the computer-controlled agent. Thus, a player may interact with the computer-controlled agent, such that user input from the player is processed using the ML model to generate model output to affect the behavior of the computer-controlled agent, thereby enabling the user and the computer-controlled agent to interact. As compared to manual dialogue authoring, use of agent information to define the behavior of a computer-controlled agent may result in reduced effort on the part of a creator while also offering increased depth and variety for computer-controlled agents of a virtual environment.

Abstract: Aspects of the present disclosure relate to a machine learning-based dialogue authoring environment. In examples, a developer or creator of a virtual environment may use a generative multimodal machine learning (ML) model to create or otherwise update aspects of a dialogue tree for one or more computer-controlled agents and/or players of the virtual environment. For example, the developer may provide an indication of context associated with the dialogue for use by the ML model, such that the ML model may generate a set of candidate interactions accordingly. The developer may select a subset of the candidate interactions for inclusion in the dialogue tree, which may then be used to generate associated nodes within the tree accordingly. Thus, nodes in the dialogue tree may be iteratively defined based on model output of the ML model, thereby assisting the developer with dialogue authoring for the virtual environment.