Abstract: In one embodiment, a method includes receiving, at a client system, an audio input, where the audio input comprises a coreference to a target object, accessing visual data from one or more camera associated with the client system, where the visual data comprises images portraying one or more objects, resolving the coreference to the target object from among the one or more objects, resoling the target object to a specific entity, and providing, at the client system, a response to the audio input, where the response comprises information about the specific entity.

Abstract: In one embodiment, a method includes receiving a user request from a client system associated with a user, determining that the user request corresponds to a first suspended task, retrieving a first dialog state of the first suspended task from a dialog history associated with the user, generating a summary of the first suspended task based on the first dialog state using a natural-language generating (NLG) module, and sending instructions to the client system for providing the summary of the first suspended task to the user.

Abstract: In one embodiment, a method includes maintaining contextual information from a first dialog session, wherein the contextual information comprises identifiers of entities and slots, receiving a user request during the first dialog session, determining a context carryover is required for an agent to execute a task associated with the user request, determining a first subset of the identifiers of the entities and a second subset of the slots of the contextual information from the first dialog session are shareable with the agent based on the agent and first resources of the identifiers of the entities and second resources of the slots, and executing the task by the agent based on the first subset of the identifiers of the entities and the second subset of the slots of the shareable contextual information.

Abstract: In one embodiment, a method includes receiving a user request from a first user from a client system associated with a first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user, determining an intent corresponding to the user request based on the gesture-input by a personalized gesture-classification model associated with the first user, executing one or more tasks based on the determined intent and the speech-input, and sending instructions for presenting execution results of the one or more tasks to the client system responsive the user request.

Abstract: In one embodiment, a method includes receiving a user request from a first user from a client system associated with a first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user, determining an intent corresponding to the user request based on the gesture-input by a personalized gesture-classification model associated with the first user, executing one or more tasks based on the determined intent and the speech-input, and sending instructions for presenting execution results of the one or more tasks to the client system responsive the user request.

Abstract: In one embodiment, a method includes receiving, from a client system associated with a user, a user request comprising a reference to a target object, accessing visual data from the client system, wherein the visual data comprises images portraying the target object and one or more additional objects, and wherein attribute information of the target object is recorded in a multimodal dialog state, resolving the reference to the target object based on the attribute information recorded in the multimodal dialog state, determining relational information between the target object and one or more of the additional objects portrayed in the visual data, and sending, to the client system, instructions for presenting a response to the user request, wherein the response comprises the attribute information and the determined relational information.

Abstract: In one embodiment, a method includes receiving a user input at a client system, wherein the user input is associated with one or more intents and one or more slots, generating one or more first dialog acts based on the user input, calculating a task-confidence score based on one or more intent-confidence scores associated with the one or more intents, respectively, and one or more slot-confidence scores associated with the one or more slots, respectively, generating one or more second dialog acts modifying the one or more first dialog acts responsive to the task-confidence score being less than a threshold score, and presenting a response to the user input at the client system, wherein the response is based on one or more of the first dialog acts or the second dialog acts.

Abstract: In one embodiment, a method includes receiving, from a client system associated with a user, a user request comprising a reference to a target object, accessing visual data from the client system, wherein the visual data comprises images portraying the target object and one or more additional objects, and wherein attribute information of the target object is recorded in a multimodal dialog state, resolving the reference to the target object based on the attribute information recorded in the multimodal dialog state, determining relational information between the target object and one or more of the additional objects portrayed in the visual data, and sending, to the client system, instructions for presenting a response to the user request, wherein the response comprises the attribute information and the determined relational information.

Abstract: In one embodiment, a system includes an automatic speech recognition (ASR) module, a natural-language understanding (NLU) module, a dialog manager, one or more agents, an arbitrator, a delivery system, one or more processors, and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to receive a user input, process the user input using the ASR module, the NLU module, the dialog manager, one or more of the agents, the arbitrator, and the delivery system, and provide a response to the user input.

Abstract: In one embodiment, a method includes receiving, at a client system, an audio input, where the audio input comprises a coreference to a target object, accessing visual data from one or more camera associated with the client system, where the visual data comprises images portraying one or more objects, resolving the coreference to the target object from among the one or more objects, resoling the target object to a specific entity, and providing, at the client system, a response to the audio input, where the response comprises information about the specific entity.

Abstract: In one embodiment, a method includes accessing visual data from a client system associated with a user, wherein the visual data comprises images portraying one or more objects, receiving, from the client system, a user request, wherein the user request comprises a coreference to a target object, resolving the coreference to the target object from among the one or more objects, resolving the target object to a specific entity, and sending, to the client system, instructions for providing a response to the user request, wherein the response comprises attribute information about the specific entity.

Abstract: Described herein is a conversation engine that can be used in a system such as a personal digital assistant or search engine that combines a dynamic knowledge graph built during execution of a request and one or more static knowledge graphs holding long term knowledge. The conversation engine comprises a state tracker that holds the dynamic knowledge graph representing the current state of the conversation, a policy engine that selects entities in the dynamic knowledge graph and executes actions provided by those entities to move the state of the conversation toward completion, and a knowledge graph search engine to search the static knowledge graph(s). The conversation is completed by building the dynamic knowledge graph over multiple rounds and chaining together operations that build toward completion of the conversation. Completion of the conversation results in completion of a request by a user.

Abstract: In one embodiment, a method includes, by one or more computing systems, receiving, from a client system, an input in a multi-turn message thread, parsing the input to identify a plurality of hypothesis dialog states associated with the input, generating a plurality of functions corresponding to the plurality of hypothesis dialog states, calculating a plurality of probability scores for the plurality of functions, respectively, based on a prior dialog state associated with the message thread and a current user context, wherein each probability score represents a likelihood that the respective function is associated with an intended dialog state, and storing the plurality of functions if each of the probability scores is below a threshold probability score, wherein the plurality of functions is stored as a hierarchical set of functions that is used as the prior dialog state in response to a next input in the message thread.

Abstract: In one embodiment, a method includes receiving a user request from a first user from a client system associated with a first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user, determining an intent corresponding to the user request based on the gesture-input by a personalized gesture-classification model associated with the first user, executing one or more tasks based on the determined intent and the speech-input, and sending instructions for presenting execution results of the one or more tasks to the client system responsive the user request.

Abstract: In one embodiment, a method includes receiving a user input from a client system associated with a user, determining a task based on the user input and a confidence score associated with the task, generating one or more first dialog acts based on a task policy which specifies dialog acts associated with the task, generating one or more second dialog acts based on an override policy responsive to the confidence score being less than a threshold score, wherein the override policy specifies dialog acts that modify dialog acts specified by the task policy; and sending instructions for presenting a response to the user input to the client system, wherein the response is based on one or more of the first dialog acts or the second dialog acts.

Abstract: In one embodiment, a method includes receiving a user input at a client system, wherein the user input is associated with one or more intents and one or more slots, generating one or more first dialog acts based on the user input, calculating a task-confidence score based on one or more intent-confidence scores associated with the one or more intents, respectively, and one or more slot-confidence scores associated with the one or more slots, respectively, generating one or more second dialog acts modifying the one or more first dialog acts responsive to the task-confidence score being less than a threshold score, and presenting a response to the user input at the client system, wherein the response is based on one or more of the first dialog acts or the second dialog acts.

Abstract: In one embodiment, a method includes, by one or more computing systems, receiving, from a client system, an input in a multi-turn message thread, parsing the input to identify a plurality of hypothesis dialog states associated with the input, generating a plurality of functions corresponding to the plurality of hypothesis dialog states, calculating a plurality of probability scores for the plurality of functions, respectively, based on a prior dialog state associated with the message thread and a current user context, wherein each probability score represents a likelihood that the respective function is associated with an intended dialog state, and storing the plurality of functions if each of the probability scores is below a threshold probability score, wherein the plurality of functions is stored as a hierarchical set of functions that is used as the prior dialog state in response to a next input in the message thread.

Abstract: In one embodiment, a method includes receiving a user request from a first user from a client system associated with a first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user, determining an intent corresponding to the user request based on the gesture-input by a personalized gesture-classification model associated with the first user, executing one or more tasks based on the determined intent and the speech-input, and sending instructions for presenting execution results of the one or more tasks to the client system responsive the user request.

Abstract: Aspects herein provide third party application authors with a user interface authoring platform that automates and simplifies a task definition process while also providing the ability to leverage pre-existing language understanding models and canonicalization and resolution modules that are provided by the operating system on which the CU system resides or as provided by other third parties. In particular, the present disclosure provides a method and system for authoring a task using a user interface authoring platform.

Abstract: In one embodiment, a method includes receiving one or more inputs associated with proactive triggers associated with a first user, determining whether the first user is eligible to receive proactive suggestions based on one or more proactive policies, generating one or more proactive suggestions based on the one or more inputs and user context data associated with the first user, selecting one or more of the proactive suggestions based on task history data associated with the first user, and sending instructions for presenting proactive content to the first user to a client system associated the first user, wherein the proactive content comprises the selected proactive suggestions.