Abstract: The present disclosure is directed to input suggestion. In particular, the methods and systems of the present disclosure can: receive, from a first application executed by one or more computing devices, data indicating information that has been presented by and/or input into the first application; generate, based at least in part on the received data, one or more suggested candidate inputs for a second application executed by the computing device(s); provide, in association with the second application, an interface comprising one or more options to select at least one suggested candidate input of the suggested candidate input(s); and responsive to receiving data indicating a selection of a particular suggested candidate input of the suggested candidate input(s) via the interface, communicate, to the second application, data indicating the particular suggested candidate input.

Abstract: Systems and methods for determining identifying semantic entities in audio signals are provided. A method can include obtaining, by a computing device comprising one or more processors and one or more memory devices, an audio signal concurrently heard by a user. The method can further include analyzing, by a machine-learned model stored on the computing device, at least a portion of the audio signal in a background of the computing device to determine one or more semantic entities. The method can further include displaying the one or more semantic entities on a display screen of the computing device.

Abstract: Systems are provided to facilitate continuous detection of words, names, phrases, or other sounds of interest and, responsive to such detection, provide a related user experience. The user experience can include providing links to media, web searches, translation services, journaling applications, or other resources based on detected ambient speech or other sounds. To preserve the privacy of those using and/or proximate to such systems, the system refrains from transmitting any information related to the detected sound unless the system receives permission from a user. Such permission can include the user interacting with a provided web search link, media link, or other user interface element.

Abstract: The present disclosure is directed to input suggestion. In particular, the methods and systems of the present disclosure can: receive, from a first application executed by one or more computing devices, data indicating information that has been presented by and/or input into the first application; generate, based at least in part on the received data, one or more suggested candidate inputs for a second application executed by the computing device(s); provide, in association with the second application, an interface comprising one or more options to select at least one suggested candidate input of the suggested candidate input(s); and responsive to receiving data indicating a selection of a particular suggested candidate input of the suggested candidate input(s) via the interface, communicate, to the second application, data indicating the particular suggested candidate input.

Abstract: The present disclosure provides systems and methods to improve discoverability of selectable user interface elements. In particular, the present disclosure provides computing devices that, in some implementations, intelligently identify user interface elements that are selectable (e.g., based on intelligent understanding of the user interface content) and visually modify one or more of such selectable user interface elements to indicate to the user that they are selectable. The visual modification can highlight or otherwise draw attention to the user interface element(s), thereby improving their discoverability.

Abstract: Provided are methods, systems, and devices for generating semantic objects and an output based on the detection or recognition of the state of an environment that includes objects. State data, based in part on sensor output, can be received from one or more sensors that detect a state of an environment including objects. Based in part on the state data, semantic objects are generated. The semantic objects can correspond to the objects and include a set of attributes. Based in part on the set of attributes of the semantic objects, one or more operating modes, associated with the semantic objects can be determined. Based in part on the one or more operating modes, object outputs associated with the semantic objects can be generated. The object outputs can include one or more visual indications or one or more audio indications.

Abstract: Provided are methods, systems, and devices for generating semantic objects and an output based on the detection or recognition of the state of an environment that includes objects. State data, based in part on sensor output, can be received from one or more sensors that detect a state of an environment including objects. Based in part on the state data, semantic objects are generated. The semantic objects can correspond to the objects and include a set of attributes. Based in part on the set of attributes of the semantic objects, one or more operating modes, associated with the semantic objects can be determined. Based in part on the one or more operating modes, object outputs associated with the semantic objects can be generated. The object outputs can include one or more visual indications or one or more audio indications.

Abstract: Provided are methods, systems, and devices for generating semantic objects and an output based on the detection or recognition of the state of an environment that includes objects. State data, based in part on sensor output, can be received from one or more sensors that detect a state of an environment including objects. Based in part on the state data, semantic objects are generated. The semantic objects can correspond to the objects and include a set of attributes. Based in part on the set of attributes of the semantic objects, one or more operating modes, associated with the semantic objects can be determined. Based in part on the one or more operating modes, object outputs associated with the semantic objects can be generated. The object outputs can include one or more visual indications or one or more audio indications.

Abstract: A method includes outputting a first graphical user interface including application information associated with a particular application of a plurality of applications executable by the computing device. The method also includes receiving an indication of a user input corresponding to a command associated with an operating system. The method further includes generating, based at least in part on the application information displayed as part of the first graphical user interface, at least one task shortcut to an action performable by one or more respective applications of the plurality of applications executable by the computing device. The method includes outputting a second graphical user interface including a graphical element corresponding to the at least one task shortcut.

Abstract: The present disclosure is directed to input suggestion. In particular, the methods and systems of the present disclosure can: receive, from a first application executed by one or more computing devices, data indicating information that has been presented by and/or input into the first application; generate, based at least in part on the received data, one or more suggested candidate inputs for a second application executed by the computing device(s); provide, in association with the second application, an interface comprising one or more options to select at least one suggested candidate input of the suggested candidate input(s); and responsive to receiving data indicating a selection of a particular suggested candidate input of the suggested candidate input(s) via the interface, communicate, to the second application, data indicating the particular suggested candidate input.

Abstract: Methods and apparatus directed to generating one or more interactive content items based on content displayed on a computing device are described herein. In various implementations, content displayed to a user by a display of a computing device may be segmented into semantic region(s) based on respective content of the semantic region(s). User input may be received, e.g., via a biometric sensor of the computing device. The user input may indicate a desire of the user to act upon the semantic region(s). In response to the user input, interactive content item(s) corresponding to the semantic region(s) may be generated and provided for presentation to the user via output device(s) of the computing device. User interaction with a given interactive content item may cause the computing device to perform action(s) that are tailored to the semantic region that corresponds to the given interactive content item.

Abstract: Provided are methods, systems, and devices for activating components of a computing device based on a mode associated with the state of objects including a computing device. Sensor data based sensor outputs from one or more sensors that detect one or more objects can be received. Based in part on the sensor data, the state of the one or more objects can be determined. The state of the one or more objects can include one or more spatial relations of the one or more objects with respect to a computing system. When the state of the one or more objects satisfies one or more operational criteria, a mode associated with activating a component of the computing system can be activated. The component can include a microphone component, a camera component, or a text input component. The component can be associated with performing including input detection.

Abstract: Systems are provided to facilitate continuous detection of words, names, phrases, or other sounds of interest and, responsive to such detection, provide a related user experience. The user experience can include providing links to media, web searches, translation services, journaling applications, or other resources based on detected ambient speech or other sounds. To preserve the privacy of those using and/or proximate to such systems, the system refrains from transmitting any information related to the detected sound unless the system receives permission from a user. Such permission can include the user interacting with a provided web search link, media link, or other user interface element.

Abstract: The present disclosure provides systems and methods to improve discoverability of selectable user interface elements. In particular, the present disclosure provides computing devices that, in some implementations, intelligently identify user interface elements that are selectable (e.g., based on intelligent understanding of the user interface content) and visually modify one or more of such selectable user interface elements to indicate to the user that they are selectable. The visual modification can highlight or otherwise draw attention to the user interface element(s), thereby improving their discoverability.

Abstract: Provided are methods, systems, and devices for activating components of a computing device based on a mode associated with the state of objects including a computing device. Sensor data based sensor outputs from one or more sensors that detect one or more objects can be received. Based in part on the sensor data, the state of the one or more objects can be determined. The state of the one or more objects can include one or more spatial relations of the one or more objects with respect to a computing system. When the state of the one or more objects satisfies one or more operational criteria, a mode associated with activating a component of the computing system can be activated. The component can include a microphone component, a camera component, or a text input component. The component can be associated with performing including input detection.

Abstract: Provided are methods, systems, and devices for generating semantic objects and an output based on the detection or recognition of the state of an environment that includes objects. State data, based in part on sensor output, can be received from one or more sensors that detect a state of an environment including objects. Based in part on the state data, semantic objects are generated. The semantic objects can correspond to the objects and include a set of attributes. Based in part on the set of attributes of the semantic objects, one or more operating modes, associated with the semantic objects can be determined. Based in part on the one or more operating modes, object outputs associated with the semantic objects can be generated. The object outputs can include one or more visual indications or one or more audio indications.

Abstract: Systems and methods of providing content selection are provided. For instance, one or more signals indicative of a user selection of an object displayed within a user interface can be received. Responsive to receiving the one or more signals, a content attribute associated with one or more objects displayed within the user interface can be identified. A content entity can be determined based at least in part on the content attribute and the user selection. One or more relevant actions can then be determined based at least in part on the determined content entity. Data indicative of the relevant actions can then be provided for display.

Abstract: Methods and apparatus directed to generating one or more interactive content items based on content displayed on a computing device are described herein. In various implementations, content displayed to a user by a display of a computing device may be segmented into semantic region(s) based on respective content of the semantic region(s). User input may be received, e.g., via a biometric sensor of the computing device. The user input may indicate a desire of the user to act upon the semantic region(s). In response to the user input, interactive content item(s) corresponding to the semantic region(s) may be generated and provided for presentation to the user via output device(s) of the computing device. User interaction with a given interactive content item may cause the computing device to perform action(s) that are tailored to the semantic region that corresponds to the given interactive content item.