Abstract: In one embodiment, a system includes an automatic speech recognition (ASR) module, a natural-language understanding (NLU) module, a dialog manager, one or more agents, an arbitrator, a delivery system, one or more processors, and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to receive a user input, process the user input using the ASR module, the NLU module, the dialog manager, one or more of the agents, the arbitrator, and the delivery system, and provide a response to the user input.

Abstract: In one embodiment, a system includes an automatic speech recognition (ASR) module, a natural-language understanding (NLU) module, a dialog manager, one or more agents, an arbitrator, a delivery system, one or more processors, and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to receive a user input, process the user input using the ASR module, the NLU module, the dialog manager, one or more of the agents, the arbitrator, and the delivery system, and provide a response to the user input.

Abstract: A caption of a multimodal message (e.g., social media post) can be identified as a named entity using an entity recognition system. The entity recognition system can use an attention-based mechanism that emphasis or de-emphasizes each data type (e.g., image, word, character) in the multimodal message based on each datatypes relevance. The output of the attention mechanism can be used to update a recurrent network to identify one or more words in the caption as being a named entity.

Abstract: A caption of a multimodal message (e.g., social media post) can be identified as a named entity using an entity recognition system. The entity recognition system can use an attention-based mechanism that emphasis or de-emphasizes each data type (e.g., image, word, character) in the multimodal message based on each datatypes relevance. The output of the attention mechanism can be used to update a recurrent network to identify one or more words in the caption as being a named entity.

Abstract: In one embodiment, a method includes accessing visual data from a client system associated with a user, wherein the visual data comprises images portraying one or more objects, receiving, from the client system, a user request, wherein the user request comprises a coreference to a target object, resolving the coreference to the target object from among the one or more objects, resolving the target object to a specific entity, and sending, to the client system, instructions for providing a response to the user request, wherein the response comprises attribute information about the specific entity.

Abstract: In one embodiment, a method includes receiving, from a client system associated with a user, a user request comprising a reference to a target object, accessing visual data from the client system, wherein the visual data comprises images portraying the target object and one or more additional objects, and wherein attribute information of the target object is recorded in a multimodal dialog state, resolving the reference to the target object based on the attribute information recorded in the multimodal dialog state, determining relational information between the target object and one or more of the additional objects portrayed in the visual data, and sending, to the client system, instructions for presenting a response to the user request, wherein the response comprises the attribute information and the determined relational information.

Abstract: In one embodiment, a method includes receiving, by a first client system, from one or more remote servers, a current version of a global neural network model including multiple federated model parameters, accessing, from a local data store, multiple examples and a local personalization model including multiple local model parameters, wherein each of the examples includes one or more features and one or more labels, training the global neural network model and the local personalization model together on the examples to generate multiple updated federated model parameters and multiple updated local model parameters, storing, in the local data store, the trained local personalization model including the updated local model parameters, and sending, to one or more of the remote servers, the trained global neural network model including the updated federated model parameters.

Abstract: In one embodiment, a method includes receiving a query from a user from a client system associated with the user, determining one or more initial memory slots based on the query, accessing a memory graph associated with the user which comprises a plurality of nodes and a plurality of edges connecting the nodes, and wherein one or more of the nodes correspond to one or more episodic memories of the user, respectively, and wherein each edge corresponds to a relationship between the connected nodes, selecting one or more candidate nodes from the memory graph by one or more machine-learning models based on the initial memory slots, generating a response based on the initial memory slots and episodic memories corresponding to the selected candidate nodes, and sending instructions for presenting the response to the client system in response to the query.

Abstract: Systems, methods, and computer program products to perform an operation comprising assigning each of a plurality of images in a blog post and each of a plurality of images in a collection of images to a respective node in a graph, computing an adjacency matrix for the graph, wherein the adjacency matrix defines relationships between images in the blog post and images in the collections of images, and determining a first subset of the images in the collection of images that summarize the images in the image collection, wherein the subset is determined based on the adjacency matrix, wherein the adjacency matrix is computed based on the subset of the images in the collection of images.

Abstract: Systems, methods, and computer program products to perform an operation comprising assigning each of a plurality of images in a blog post and each of a plurality of images in a collection of images to a respective node in a graph, computing an adjacency matrix for the graph, wherein the adjacency matrix defines relationships between images in the blog post and images in the collections of images, and determining a first subset of the images in the collection of images that summarize the images in the image collection, wherein the subset is determined based on the adjacency matrix, wherein the adjacency matrix is computed based on the subset of the images in the collection of images.

Abstract: Physical social actions are conveniently detected, characterized, and provided to a social networking system. This improves the user experience of the social networking system by more thoroughly recording the social actions of users occurring outside a social graph. This has the effect of extending an open social graph, which typically refers to social actions taken outside the social networking system but within a computer network, to physical social actions. By providing characterized physical social actions to the social networking system, the open social graph is extended to include physical social actions.

Abstract: Physical social actions are conveniently detected, characterized, and provided to a social networking system. This improves the user experience of the social networking system by more thoroughly recording the social actions of users occurring outside a social graph. This has the effect of extending an open social graph, which typically refers to social actions taken outside the social networking system but within a computer network, to physical social actions. By providing characterized physical social actions to the social networking system, the open social graph is extended to include physical social actions.