Abstract: The present disclosure is directed to automating user interactions in an application. In particular, the computing device can determine, based on the input data, an intent associated with performing an action on the computing device. Additionally, the computing device can invoke, using an application-level assistant component configured to provide an application-level interface on the computing device, execution of an application. Moreover, the computing device can transmit, to a system-level assistant component configured to provide a system-level interface with the computing device, instructions to interface with an actionable component of the application. The instructions can be transmitted using a trusted intermediary component configured to facilitate an interface between the system-level assistant component and the application-level assistant component.

Abstract: The present disclosure is directed to automating user interactions in an application. In particular, the computing device can determine, based on the input data, an intent associated with performing an action on the computing device. Additionally, the computing device can invoke, using an application-level assistant component configured to provide an application-level interface on the computing device, execution of an application. Moreover, the computing device can transmit, to a system-level assistant component configured to provide a system-level interface with the computing device, instructions to interface with an actionable component of the application. The instructions can be transmitted using a trusted intermediary component configured to facilitate an interface between the system-level assistant component and the application-level assistant component.

Abstract: Techniques and computing devices are described that automatically caption content directly from audio data being output from content sources, unlike other captioning systems which often rely on information contained in audio signals being sent to speakers. The disclosed techniques and computing devices may analyze metadata to determine whether the audio data is suitable for captioning or whether the audio data is some other type of audio data. Responsive to identifying audio data for captioning, the disclosed techniques and computing devices can generate a description of audible sounds interpreted from the audio data, providing for the automatic captioning of content and making audible content accessible to many users who have difficulty hearing or are otherise unable to listen to content.