Abstract: Systems and processes for operating a digital assistant are provided. An example process for determining a response includes, at an electronic device having one or more processors and memory, receiving a spoken input including a request, performing a semantic analysis on the spoken input, determining, based on the semantic analysis, a likelihood that the electronic device requires additional contextual data to satisfy the request, and in accordance with the determined likelihood exceeding a threshold, enabling a camera of the electronic device and determining a response to the request based on data captured by the camera of the electronic device.

Abstract: Systems and processes for operating a digital assistant are provided. An example process for determining a response includes, at an electronic device having one or more processors and memory, receiving a spoken input including a request, performing a semantic analysis on the spoken input, determining, based on the semantic analysis, a likelihood that the electronic device requires additional contextual data to satisfy the request, and in accordance with the determined likelihood exceeding a threshold, enabling a camera of the electronic device and determining a response to the request based on data captured by the camera of the electronic device.

Abstract: Systems and processes for operating an intelligent automated assistant are provided. An example process includes receiving an utterance including a user request and determining whether at least a portion of the user request is ambiguous. If at least the portion of the user request is ambiguous then a set of context data based on the ambiguous portion of the user request is determined, metadata is extracted from the context data and a response to the user request is determined based on the extracted metadata.

Abstract: Systems and processes for operating a digital assistant are provided. An example method for processing an image include receiving an image, generating, based on the image, a question corresponding to a first object in the image, generating, based on the image, a caption corresponding to a second object of the image, receiving an utterance from a user, and determining a plurality of speech recognition results from the utterance based on the question and the caption.

Abstract: Systems and processes for operating a digital assistant are provided. An example method for processing an image include receiving an image, generating, based on the image, a question corresponding to a first object in the image, generating, based on the image, a caption corresponding to a second object of the image, receiving an utterance from a user, and determining a plurality of speech recognition results from the utterance based on the question and the caption.

Abstract: Systems and processes for operating a digital assistant are provided. An example process for performing a task includes, at an electronic device having one or more processors and memory, receiving a spoken input including a request, receiving an image input including a plurality of objects, selecting a reference resolution module of a plurality of reference resolution modules based on the request and the image input, determining, with the selected reference resolution module, whether the request references a first object of the plurality of objects based on at least the spoken input, and in accordance with a determination that the request references the first object of the plurality of objects, determining a response to the request including information about the first object.

Abstract: In an exemplary process, a speech input including a referenced virtual object is received. Based on the speech input, a first reference set is obtained. The first reference set is then compared to a plurality of second reference sets. Based on the comparison, a second reference set from the plurality of second reference sets is obtained. The second reference set may be identified based on a matching score between the first reference set and the second reference set. An object is then identified based on the second reference set. Based on the identified object, the reference virtual object is displayed.

Abstract: Systems and processes for operating a digital assistant are provided. An example process for determining a response includes, at an electronic device having one or more processors and memory, receiving a spoken input including a request, performing a semantic analysis on the spoken input, determining, based on the semantic analysis, a likelihood that the electronic device requires additional contextual data to satisfy the request, and in accordance with the determined likelihood exceeding a threshold, enabling a camera of the electronic device and determining a response to the request based on data captured by the camera of the electronic device.

Abstract: A device implementing a system for user-specific candidate responses includes a processor configured to determine candidate responses for a received message based at least in part on a determined category of the received message. The processor is further configured to rank the candidate responses based at least in part on input modalities utilized to input messages previously transmitted via the device, at least one of the previously transmitted messages comprising a respective at least one of the candidate responses. The processor is further configured to provide one or more of the ranked candidate responses for selection by a user.