Abstract: In one embodiment, a method includes receiving, at a client system, an audio input, where the audio input comprises a coreference to a target object, accessing visual data from one or more camera associated with the client system, where the visual data comprises images portraying one or more objects, resolving the coreference to the target object from among the one or more objects, resoling the target object to a specific entity, and providing, at the client system, a response to the audio input, where the response comprises information about the specific entity.

Abstract: In one embodiment, a method includes receiving a user request from a client system associated with a user, determining that the user request corresponds to a first suspended task, retrieving a first dialog state of the first suspended task from a dialog history associated with the user, generating a summary of the first suspended task based on the first dialog state using a natural-language generating (NLG) module, and sending instructions to the client system for providing the summary of the first suspended task to the user.

Abstract: In one embodiment, a method includes receiving, from a client system associated with a user, a user request comprising a reference to a target object, accessing visual data from the client system, wherein the visual data comprises images portraying the target object and one or more additional objects, and wherein attribute information of the target object is recorded in a multimodal dialog state, resolving the reference to the target object based on the attribute information recorded in the multimodal dialog state, determining relational information between the target object and one or more of the additional objects portrayed in the visual data, and sending, to the client system, instructions for presenting a response to the user request, wherein the response comprises the attribute information and the determined relational information.

Abstract: In one embodiment, a method includes receiving, from a client system associated with a user, a user request comprising a reference to a target object, accessing visual data from the client system, wherein the visual data comprises images portraying the target object and one or more additional objects, and wherein attribute information of the target object is recorded in a multimodal dialog state, resolving the reference to the target object based on the attribute information recorded in the multimodal dialog state, determining relational information between the target object and one or more of the additional objects portrayed in the visual data, and sending, to the client system, instructions for presenting a response to the user request, wherein the response comprises the attribute information and the determined relational information.

Abstract: In one embodiment, a method includes receiving, at a client system, an audio input, where the audio input comprises a coreference to a target object, accessing visual data from one or more camera associated with the client system, where the visual data comprises images portraying one or more objects, resolving the coreference to the target object from among the one or more objects, resoling the target object to a specific entity, and providing, at the client system, a response to the audio input, where the response comprises information about the specific entity.

Abstract: In one embodiment, a method includes accessing visual data from a client system associated with a user, wherein the visual data comprises images portraying one or more objects, receiving, from the client system, a user request, wherein the user request comprises a coreference to a target object, resolving the coreference to the target object from among the one or more objects, resolving the target object to a specific entity, and sending, to the client system, instructions for providing a response to the user request, wherein the response comprises attribute information about the specific entity.

Abstract: In one embodiment, a method includes accessing visual data from a client system associated with a user, wherein the visual data comprises images portraying one or more objects, receiving, from the client system, a user request, wherein the user request comprises a coreference to a target object, resolving the coreference to the target object from among the one or more objects, resolving the target object to a specific entity, and sending, to the client system, instructions for providing a response to the user request, wherein the response comprises attribute information about the specific entity.

Abstract: In one embodiment, a method includes receiving, from a client system associated with a user, a user request comprising a reference to a target object, accessing visual data from the client system, wherein the visual data comprises images portraying the target object and one or more additional objects, and wherein attribute information of the target object is recorded in a multimodal dialog state, resolving the reference to the target object based on the attribute information recorded in the multimodal dialog state, determining relational information between the target object and one or more of the additional objects portrayed in the visual data, and sending, to the client system, instructions for presenting a response to the user request, wherein the response comprises the attribute information and the determined relational information.