Abstract: Systems and methods for providing audio data, from an initially invoked automated assistant to a subsequently invoked automated assistant. An initially invoked automated assistant may be invoked by a user utterance, followed by audio data that includes a query. The query is provided to a secondary automated assistant for processing. Subsequently, the user can submit a query that is related to the first query. In response, the initially invoked automated assistant provides the query to the secondary automated assistant in lieu of providing the query to other secondary automated assistants based on similarity between the first query and the subsequent query.

Abstract: Some implementations process, using warm word model(s), a stream of audio data to determine a portion of the audio data that corresponds to particular word(s) and/or phrase(s) (e.g., a warm word) associated with an assistant command, process, using an automatic speech recognition (ASR) model, a preamble portion of the audio data (e.g., that precedes the warm word) and/or a postamble portion of the audio data (e.g., that follows the warm word) to generate ASR output, and determine, based on processing the ASR output, whether a user intended the assistant command to be performed. Additional or alternative implementations can process the stream of audio data using a speaker identification (SID) model to determine whether the audio data is sufficient to identify the user that provided a spoken utterance captured in the stream of audio data, and determine if that user is authorized to cause performance of the assistant command.

Abstract: Example aspects of the present disclosure are directed to systems and methods that enable improved adversarial training of machine-learned models. An adversarial training system can generate improved adversarial training examples by optimizing or otherwise tuning one or hyperparameters that guide the process of generating of the adversarial examples. The adversarial training system can determine, solicit, or otherwise obtain a realism score for an adversarial example generated by the system. The realism score can indicate whether the adversarial example appears realistic. The adversarial training system can adjust or otherwise tune the hyperparameters to produce improved adversarial examples (e.g., adversarial examples that are still high-quality and effective while also appearing more realistic). Through creation and use of such improved adversarial examples, a machine-learned model can be trained to be more robust against (e.g.

Abstract: Methods, systems, devices, and tangible non-transitory computer readable media for route familiarity determination and route generation are provided. The disclosed technology can include accessing route data. The route data can include information associated with a starting location and a destination. Routes from the starting location to the destination can be determined based at least in part on the route data. Amounts of familiarity with one or more historical routes that were previously traveled by a user can be determined based on historical route data. Similarities between the one or more routes and the one or more historical routes can be determined based on the historical route data. Suggested routes can be determined based on the routes that satisfy one or more criteria associated with the amounts of familiarity and the similarities. Output including indications associated with the user traveling the one or more suggested routes can be generated.

Abstract: A method includes instructing an always-on first processor to operate in a follow-on query detection mode, and while the always-on first processor operates in the follow-on query detection mode: receiving follow-on audio data captured by the assistant-enabled device; determining, using a voice activity detection (VAD) model executing on the always-on first processor, whether or not the VAD model detects voice activity in the follow-on audio data; performing, using a speaker identification (SID) model executing on the always-on first processor, speaker verification on the follow-on audio data to determine whether the follow-on audio data includes an utterance spoken by the same user. The method also includes initiating a wake-up process on a second processor to determine whether the utterance includes a follow-on query.

Abstract: A method includes receiving user queries directed toward a cloud-based assistant service. For each received user query directed toward the cloud-based assistant service, the method also includes extracting one or more attributes from the user query and logging the user query into one or more of a plurality of category buckets based on the one or more attributes extracted from the user query. The method also includes determining when at least one of the plurality of category buckets includes a threshold number of the user queries logged into the at least one category bucket, and when the at least one of the plurality of category buckets includes the threshold number of the user queries, generating a distilled model of the cloud-based assistant service. The distilled model of the cloud-based assistant service is configured to execute on one or more target client devices.

Abstract: Techniques are disclosed that enable automating user interface input by generating a sequence of actions to perform a task utilizing a multi-agent reinforcement learning framework. Various implementations process an intent associated with received user interface input using a holistic reinforcement policy network to select a software reinforcement learning policy network. The sequence of actions can be generated by processing the intent, as well as a sequence of software client state data, using the selected software reinforcement learning policy network. The sequence of actions are utilized to control the software client corresponding to the selected software reinforcement learning policy network.

Abstract: The present disclosure provides an on-device machine learning platform that enables sharing of machine-learned models between applications on a computing device. For example, a first application which has a machine-learned model for a specific task can expose the model to other applications through a system level application programming interface (API) for the other applications to use. Communications using the API can be handled by the on-device machine learning platform. In some implementations, some exchange of resources (e.g., computing resources) can be provided so that the first application is compensated for sharing the machine-learned model (e.g., on a per model invocation basis).

Abstract: Implementations set forth herein relate to intervening notifications provided by an application for mitigating computationally wasteful application launching behavior that is exhibited by some users. A state of a module of a target application can be identified by emulating user inputs previously provided by the user to the target application. In this way, the state of the module can be determined without visibly launching the target application. When the state of the module is determined to satisfy criteria for providing a notification to the user, the application can render a notification for the user. The application can provide intervening notifications for a variety of different target applications in order to reduce a frequency at which the user launches and closes applications to check for variations in target application content.

Abstract: Methods, systems, and apparatuses for implementing advanced content retrieval are described. Machine learning methods may be implemented so that a system may predict when a user device may experience network disconnections. The system may also predict the type of content one or more applications on the user device may seek to download during the network disconnection period. Neural networks may be trained based on user activity log data and may implement machine-learning techniques to determine user preferences and settings for advanced content retrieval. The system may predict when a user may want to download content in advance, the type of content the user may be interested in, anticipated network connectivity, and anticipated battery consumption. The system may then generate recommendations for the user device based on the predictions. If a user agrees with the recommendations, the system may obtain and cache the content.

Abstract: Implementations described herein relate to configuring a dynamic warm word button, that is associated with a client device, with particular assistant commands based on detected occurrences of warm word activation events at the client device. In response to detecting an occurrence of a given warm word activation event at the client device, implementations can determine whether user verification is required for a user that actuated the warm word button. Further, in response to determining that the user verification is required for the user that actuated the warm word button, the user verification can be performed. Moreover, in response to determining that the user that actuated the warm word button has been verified, implementations can cause an automated assistant to perform the particular assistant command associated with the warm word activation event. Audio-based and/or non-audio-based techniques can be utilized to perform the user verification.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for extending application access across devices. In some implementations, an electronic device receives a request to provide access to the electronic device to a particular user that is not registered as a user of the electronic device. The electronic device receives authentication credentials for the particular user. The electronic device provides the authentication credentials to a server system and receives data from the server system that (i) indicates that the providing access to the electronic device in a guest mode is authorized, and (ii) indicates a state of an instance of an application installed on a second device. The electronic device provides access to the electronic device in the guest mode that provides an interface that at least partially recreates the state of the instance of the application installed on the second device.

Abstract: Some implementations relate to performing speech biasing, NLU biasing, and/or other biasing based on historical assistant interaction(s). It can be determined, for one or more given historical interactions of a given user, whether to affect future biasing for (1) the given user account, (2) additional user account(s), and/or (3) the shared assistant device as a whole. Some implementations disclosed herein additionally and/or alternatively relate to: determining, based on utterance(s) of a given user to a shared assistant device, an association of first data and second data; storing the association as accessible to a given user account of the given user; and determining whether to store the association as also accessible by additional user account(s) and/or the shared assistant device.

Abstract: Implementations can receive audio data corresponding to a spoken utterance of a user, process the audio data to generate a plurality of speech hypotheses, determine an action to be performed by an automated assistant based on the speech hypotheses, and cause the computing device to render an indication of the action. In response to the computing device rendering the indication, implementations can receive additional audio data corresponding to an additional spoken utterance of the user, process the additional audio data to determine that a portion of the spoken utterance is similar to an additional portion of the additional spoken utterance, supplant the action with an alternate action, and cause the automated assistant to initiate performance of the alternate action. Some implementations can determine whether to render the indication of the action based on a confidence level associated with the action.

Abstract: A method for streaming action fulfillment receives audio data corresponding to an utterance where the utterance includes a query to perform an action that requires performance of a sequence of sub-actions in order to fulfill the action. While receiving the audio data, but before receiving an end of speech condition, the method processes the audio data to generate intermediate automated speech recognition (ASR) results, performs partial query interpretation on the intermediate ASR results to determine whether the intermediate ASR results identify an application type needed to perform the action and, when the intermediate ASR results identify a particular application type, performs a first sub-action in the sequence of sub-actions by launching a first application to execute on the user device where the first application is associated with the particular application type. The method, in response to receiving an end of speech condition, fulfills performance of the action.

Abstract: Implementations set forth herein relate to an automated assistant that allows a user to create, edit, and/or share documents without directly interfacing with a document editing application. The user can provide an input to the automated assistant in order to cause the automated assistant to interface with the document editing application and create a document. In order to identify a particular action to perform with respect to a document, and/or identify a particular subsection within the document to direct the action, the automated assistant can rely on semantic annotations. As a user continues to interact with the automated assistant to edit a document, the semantic annotations can be updated according to how the document is changing and/or how the user refers to the document. This can allow the automated assistant to more readily fulfill document-related requests that may lack express details.

Abstract: A computing system can include one or more machine-learned models configured to receive context data that describes one or more entities to be named. In response to receipt of the context data, the machine-learned model(s) can generate output data that describes one or more names for the entity or entities described by the context data. The computing system can be configured to perform operations including inputting the context data into the machine-learned model(s). The operations can include receiving, as an output of the machine-learned model(s), the output data that describes the name(s) for the entity or entities described by the context data. The operations can include storing at least one name described by the output data.

Abstract: Implementations can identify a given assistant device from among a plurality of assistant devices in an ecosystem, obtain device-specific signal(s) that are generated by the given assistant device, process the device-specific signal(s) to generate candidate semantic label(s) for the given assistant device, select a given semantic label for the given semantic device from among the candidate semantic label(s), and assigning, in a device topology representation of the ecosystem, the given semantic label to the given assistant device. Implementations can optionally receive a spoken utterance that includes a query or command at the assistant device(s), determine a semantic property of the query or command matches the given semantic label to the given assistant device, and cause the given assistant device to satisfy the query or command.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for limiting provision and display of redundant digital components on a client. Methods can include storing, by a client device, a list of digital components specifying digital components available for provision to the client device. For a first digital component received within a first application, the client device detects a set of signals specifying a first user interaction with the first digital component and a second user interaction with content provided in response to the first user interaction. Based on whether an affirmative user action was performed, the client device can modify the list of digital components. When a request to access a content page within a second application is received, the client device can receive the second digital component, which can be selected from among digital components included on the modified list of digital components.

Abstract: Implementations set forth herein relate to an automated assistant that can be customized by a user to provide custom assistant responses to certain assistant queries, which may originate from other users. The user can establish certain custom assistant responses by providing an assistant response request to the automated assistant and/or responding to a request from the automated assistant to establish a particular custom assistant response. In some instances, a user can elect to establish a custom assistant response when the user determines or acknowledges that certain common queries are being submitted to the automated assistant—but the automated assistant is unable to resolve the common query. Establishing such custom assistant responses can therefore condense interactions between other users and the automated assistant.