Abstract: Aspects of the present disclosure provide systems and methods which utilizes machine learning techniques to provide enhanced accessibility features to a game. An accessibility service is provided which is capable of instantiating one or more machine learning models which can process current gameplay states and generate commands to assist users during gameplay. The accessibility commands may be provided to a game and used to supplement or modify user provided inputs in order to compensate for specific user needs. In further aspects, an accessibility user interface is provided which allows a user to dynamically enable or disable accessibility features during gameplay. The user interface is operable to receive accessibility selections and provide the selection data to an accessibility service during gameplay.

Abstract: Aspects of the present disclosure relate to a personalized agent service that generates and evolves customized agents that can be instantiated in-game to play with users. Machine learning models are trained to control the agent's interactions with the game environment and the user during gameplay. As the user continues to play with the agent, the one or more machine learning models develop gameplay styles for the agent that complement the user's preferred playstyle, incorporate the user's preferred strategies, and is generally customized for interaction with the user. The agent personalization data generated during gameplay is stored by the service. An application programming interface is provided by the personalized agent service. Using the API, games can import agent personalization data in order to customize in-game non-player characters (NPCs), thereby customizing the in-game NPCs in accordance with the user's preferences.

Abstract: Aspects of the present disclosure relate to a personalized agent service that generates and evolves customized agents that can be instantiated in-game to play with users. Machine learning models are trained to control the agent's interactions with the game environment and the user during gameplay. A user may request that a personalized agent join the user's gameplay session. The user device sends a request for the personalized agent to a game platform. The game platform determines whether the user has a license to execute a second instance of the game. When the user has a license to execute a second instance of the game, the second instance of the game may be executed on the user device. Information received from a personalized agent service is used to instantiate a personalized agent in the second instance of the game.

Abstract: Aspects of the present disclosure relate to multidirectional generative editing. In examples, content of a source document is used to produce generated content for a target document. A subpart of the source document may be associated with a subpart of the target document that includes the generated content. As a result of the association, if the subpart of the target document is modified (e.g., to add, remove, or edit natural language content or formatting), the subpart of the target document is used to produce generated content with which to update the source document accordingly. Thus, changes to generated content may be propagated back to a source document from which the generated content was produced.

Abstract: Aspects of the present disclosure relate to grounded multimodal agent interactions, where a user input is processed using a multimodal machine learning model to generate model output. The model output may then be processed to affect the behavior of an application, for example to enable a user to control the application and/or to facilitate user interactions with a conversational agent, among other examples. In some instances, at least a part of the model output may be executed or parsed, for example to call an application programming interface or function of the application. Thus, use of a multimodal machine learning model according to aspects described herein may enable the use of user-provided natural language input to affect the behavior of an application accordingly.

Abstract: Aspects of the present disclosure relate to grounded multimodal agent interactions, where a user input is processed using a multimodal machine learning model to generate model output. The model output may then be processed to affect the behavior of an application, for example to enable a user to control the application and/or to facilitate user interactions with a conversational agent, among other examples. In some instances, at least a part of the model output may be executed or parsed, for example to call an application programming interface or function of the application. Thus, use of a multimodal machine learning model according to aspects described herein may enable the use of user-provided natural language input to affect the behavior of an application accordingly.

Abstract: In examples, a developer may define a set of computer-controlled agent attributes, which may be processed by a generative multimodal machine learning model in conjunction with background information associated with a virtual environment (e.g., “lore”) and other agent information to generate multimodal model output with which to control the behavior of the computer-controlled agent. Thus, a player may interact with the computer-controlled agent, such that user input from the player is processed using the ML model to generate model output to affect the behavior of the computer-controlled agent, thereby enabling the user and the computer-controlled agent to interact. As compared to manual dialogue authoring, use of agent information to define the behavior of a computer-controlled agent may result in reduced effort on the part of a creator while also offering increased depth and variety for computer-controlled agents of a virtual environment.

Abstract: Aspects of the present disclosure relate to a machine learning-based dialogue authoring environment. In examples, a developer or creator of a virtual environment may use a generative multimodal machine learning (ML) model to create or otherwise update aspects of a dialogue tree for one or more computer-controlled agents and/or players of the virtual environment. For example, the developer may provide an indication of context associated with the dialogue for use by the ML model, such that the ML model may generate a set of candidate interactions accordingly. The developer may select a subset of the candidate interactions for inclusion in the dialogue tree, which may then be used to generate associated nodes within the tree accordingly. Thus, nodes in the dialogue tree may be iteratively defined based on model output of the ML model, thereby assisting the developer with dialogue authoring for the virtual environment.

Abstract: Aspects of the present disclosure relate to techniques for interactive content generation. In examples, processed content may be produced by a generative model based on a content seed, such as a sentence or paragraph. User input associated with the processed content may be received, for example to revise the processed content or provide additional input with respect to a subpart of the processed content that is associated with a low confidence score. A generative model may produce updated processed content based at least in part on the previously processed content, the user input, and/or, in some examples, additional content, as may be indicated by a user. Thus, a user may iterate on processed content that is produced by such generative models through successive interactions, thereby enabling the user to provide input to the generative model as part of the content generation process.

Abstract: A method and system for providing replacement text segments for a given text segment may include receiving a request to provide the replacement text segment for the text segment in the document, examining a content characteristic of the document, and examining at least one of user-specific information, organization-specific information, or non-linguistic features of the document, before identifying at least one replacement text segment for the text segment, via a machine translation system, based on the content characteristic of the document and at least one of the user-specific information, the organization-specific information, or the non-linguistic features of the document. The method and system may include providing the identified replacement text segment for display to a user, receiving an input indicating a user's selection of the identified replacement text segment, and upon receiving the input, replacing the text segment in the document with the identified replacement text segment.

Abstract: Systems and method directed to assistive document generation are described. More specifically, similar documents share large portions of reusable text structures that can be used to generate an initial document thereby saving a user time. To generate the document, an indication to create the document may be received and based on the indication, a plurality of example documents and grounding content may be identified. Example documents may be existing documents that are similar to a target document of the writer. Grounding information may refer to content that is relevant, timely, and accurate when applied to the target document. The plurality of example documents and the grounding content may be received, and a document sketch based on the example documents and the grounding content may be generated and contains a plurality of predicted text sequences based on the example documents and the grounding content.

Abstract: A method for providing targeted rewrites can include receiving a selection of text in a file; generating a set of target rewrites of the selection of text, the set of target rewrites comprising: at least one phrase or sentence having semantic similarity to a phrase or sentence of the selection of text; and a style that corresponds to a particular target style, wherein a target style is a representative style for a genre, profession, or environment; and providing for selection one or more of the target rewrites of the set of target rewrites.

Abstract: A method and system for providing replacement text segments for a given text segment may include receiving a request to provide the replacement text segment for the text segment in the document, examining a content characteristic of the document, and examining at least one of user-specific information, organization-specific information, or non-linguistic features of the document, before identifying at least one replacement text segment for the text segment, via a machine translation system, based on the content characteristic of the document and at least one of the user-specific information, the organization-specific information, or the non-linguistic features of the document. The method and system may include providing the identified replacement text segment for display to a user, receiving an input indicating a user's selection of the identified replacement text segment, and upon receiving the input, replacing the text segment in the document with the identified replacement text segment.

Abstract: A method for providing targeted rewrites can include receiving a selection of text in a file; generating a set of target rewrites of the selection of text, the set of target rewrites comprising: at least one phrase or sentence having semantic similarity to a phrase or sentence of the selection of text; and a style that corresponds to a particular target style, wherein a target style is a representative style for a genre, profession, or environment; and providing for selection one or more of the target rewrites of the set of target rewrites.

Abstract: Examples are generally directed towards context-sensitive generation of conversational responses. Context-message-response n-tuples are extracted from at least one source of conversational data to generate a set of training context-message-response n-tuples. A response generation engine is trained on the set of training context-message-response n-tuples. The trained response generation engine automatically generates a context-sensitive response based on a user generated input message and conversational context data. A digital assistant utilizes the trained response generation engine to generate context-sensitive, natural language responses that are pertinent to user queries.

Abstract: Conversations can be generated automatically based on any given persona. The conversations can be produced by a language generation model that automatically generates persona-based language in response to a message, wherein a persona identifies a type of person with role specific characteristics. After a language generation model is acquired, for example by identifying a predefined model or generating a new model, the language generation model can be provisioned for use by a conversational agent, such as a chatbot, to enhance the functionality of the conversational agent.

Abstract: Examples are generally directed towards context-sensitive generation of conversational responses. Context-message-response n-tuples are extracted from at least one source of conversational data to generate a set of training context-message-response n-tuples. A response generation engine is trained on the set of training context-message-response n-tuples. The trained response generation engine automatically generates a context-sensitive response based on a user generated input message and conversational context data. A digital assistant utilizes the trained response generation engine to generate context-sensitive, natural language responses that are pertinent to user queries.

Abstract: Examples are generally directed towards context-sensitive generation of conversational responses. Context-message-response n-tuples are extracted from at least one source of conversational data to generate a set of training context-message-response n-tuples. A response generation engine is trained on the set of training context-message-response n-tuples. The trained response generation engine automatically generates a context-sensitive response based on a user generated input message and conversational context data. A digital assistant utilizes the trained response generation engine to generate context-sensitive, natural language responses that are pertinent to user queries.

Abstract: Examples are generally directed towards automatic assessment of machine generated conversational responses. Context-message-response n-tuples are extracted from at least one source of conversational data to generate a set of multi-reference responses. A response in the set of multi-reference responses includes it context-message data pair and rating. The rating indicates a quality of the response relative to the context-message data pair. A response assessment engine generates a metric score for a machine-generated response based on an assessment metric and the set of multi-reference responses. The metric score indicates a quality of the machine-generated conversational response relative to a user-generated message and a context of the user-generated message. A response generation system of a computing device, such as a digital assistant, is optimized and adjusted based on the metric score to improve the accuracy, quality, and relevance of responses output to the user.

Abstract: Different advantageous embodiments provide a crowdsourcing method for modeling user intent in conversational interfaces. One or more stimuli are presented to a plurality of describers. One or more sets of describer data are captured from the plurality of describers using a data collection mechanism. The one or more sets of describer data are processed to generate one or more models. Each of the one or more models is associated with a specific stimulus from the one or more stimuli.