Abstract: Implementations relate to generating a proficiency measure, and utilizing the proficiency measure to adapt one or more automated assistant functionalities. The generated proficiency measure is for a particular class of automated assistant actions, and is specific to an assistant device and/or is specific to a particular user. A generated proficiency measure for a class can reflect a degree of proficiency, of a user and/or of an assistant device, for that class. Various automated assistant functionalities can be adapted, for a particular class, responsive to determining the proficiency measure satisfies a threshold, or fails to satisfy the threshold (or an alternate threshold). The adaptation(s) can make automated assistant processing more efficient and/or improve (e.g., shorten the duration of) user-assistant interaction(s).

Abstract: Implementations relate to an automated assistant that can respond to communications received via a third party application and/or other third party communication modality. The automated assistant can determine that the user is participating in multiple different conversations via multiple different third party communication services. In some implementations, conversations can be processed to identify particular features of the conversations. When the automated assistant is invoked to provide input to a conversation, the automated assistant can compare the input to the identified conversation features in order to select the particular conversation that is most relevant to the input. In this way, the automated assistant can assist with any of multiple disparate conversations that are each occurring via a different third party application.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a prediction of an audio signal. One of the methods includes receiving a request to generate an audio signal conditioned on an input; processing the input using an embedding neural network to map the input to one or more embedding tokens; generating a semantic representation of the audio signal; generating, using one or more generative neural networks and conditioned on at least the semantic representation and the embedding tokens, an acoustic representation of the audio signal; and processing at least the acoustic representation using a decoder neural network to generate the prediction of the audio signal.

Abstract: Implementations set forth herein relate to an automated assistant that can be customized by a user to provide custom assistant responses to certain assistant queries, which may originate from other users. The user can establish certain custom assistant responses by providing an assistant response request to the automated assistant and/or responding to a request from the automated assistant to establish a particular custom assistant response. In some instances, a user can elect to establish a custom assistant response when the user determines or acknowledges that certain common queries are being submitted to the automated assistant—but the automated assistant is unable to resolve the common query. Establishing such custom assistant responses can therefore condense interactions between other users and the automated assistant.

Abstract: Implementations are directed to enabling a representative associated with an entity to quickly and efficiently modify a voice bot associated with the entity. The voice bot can be previously trained to communicate with user(s) on behalf of the entity through various communication channels (e.g., a telephone communication channel, a software application communication channel, a messaging communication channel, etc.). Processor(s) of a computing device can receive, from the representative, representative input to modify behavior(s) and/or parameter(s) that the voice bot utilizes in communicating with the plurality of users via the communication channels, determine whether the representative is authorized to cause the behavior(s) and/or parameter(s) to be modified, and cause the behavior(s) and/or parameter(s) to be modified in response to determining that the representative is authorized.

Abstract: Implementations set forth herein relate to an automated assistant that facilitates the creation of complex messages from user input(s) to the automated assistant. Each message can be created according to a respective template that is selected based on user input that directs the automated assistant to communicate a message to a recipient. Furthermore, sections of a template can be designated for certain content based on prior messages communicated by one or more users to one or more recipients. In this way, in response to a user requesting that the automated assistant send a message, the automated assistant can select a related template and fill out the template accordingly. In some instances, content that is assigned to certain sections of the selected template can come from a variety of different sources and/or may not be explicitly specified in the request from the user to the automated assistant.

Abstract: To determine whether a user is authorized to make a particular audio request during navigation, a client device receives a request for navigation directions from a starting location to a destination location. The client device provides a set of navigation directions for traversing to the destination location along a route. During a navigation session, an audio request related to the route is received from a user. The client device determines an authorization level of the user based on the audio request, and provides a response to the request based on the authorization level of the user.

Abstract: Techniques are described herein for collaborative search sessions through an automated assistant. A method includes: receiving, from a first user of a first client device, a first query in a query session; providing, to the first user, a first set of search results; determining, based on at least one term in the first query, that the first query is relevant to a second user of the first client device; providing, to the second user, a selectable option to join the query session; in response to receiving, from the second user, an indication of acceptance of the selectable option, adding the second user to the query session; receiving, from the second user, additional input; generating, based on the additional input received from the second user, a modified set of search results; and providing, to the first user and the second user, the modified set of search results.

Abstract: The technology relates to integrating web content into a map application. A query is sent from the map application. At least one snippet of web content identified as relevant to the query is received in response to the query, the at least one snippet of content including a portion of media or textual content from a source on the web. The portion of media or textual content is formatted for display in the map application and output for display in the map application.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a prediction of an audio signal. One of the methods includes receiving a request to generate an audio signal conditioned on an input; processing the input using an embedding neural network to map the input to one or more embedding tokens; generating a semantic representation of the audio signal; generating, using one or more generative neural networks and conditioned on at least the semantic representation and the embedding tokens, an acoustic representation of the audio signal; and processing at least the acoustic representation using a decoder neural network to generate the prediction of the audio signal.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for hotword detection on multiple devices are disclosed. In one aspect, a method includes the actions of receiving, by a first computing device, audio data that corresponds to an utterance. The actions further include determining a first value corresponding to a likelihood that the utterance includes a hotword. The actions further include receiving a second value corresponding to a likelihood that the utterance includes the hotword, the second value being determined by a second computing device. The actions further include comparing the first value and the second value. The actions further include based on comparing the first value to the second value, initiating speech recognition processing on the audio data.

Abstract: Voice-based interaction with video content being presented by a media player application is enhanced through the use of an automated assistant capable of identifying when a spoken utterance by a user is a request to playback a specific scene in the video content. A query identified in a spoken utterance may be used to access stored scene metadata associated with video content being presented in the vicinity of the user to identify one or more locations in the video content that correspond to the query, such that a media control command may be issued to the media player application to cause the media player application to seek to a particular location in the video content that satisfies the query.

Abstract: Systems and methods are provided for sharing a screen from a mobile device. For example, a method includes receiving, at a second mobile device, an image of a screen captured from a first mobile device and determining whether to trigger an automated action. The method may also include displaying, responsive to not triggering the automated action, annotation data generated for the image with the image on a display of the second mobile device, the annotation data including at least one visual cue corresponding to content in the image relevant to a user of the second mobile device. The method may further include, responsive to triggering the automated action, determining that a mobile application associated with the image is installed on the second mobile device and replaying user input actions received with the image on the second mobile device starting from a reference screen associated with the mobile application.

Abstract: Techniques are described herein for identifying a failed hotword attempt. A method includes: receiving first audio data; processing the first audio data to generate a first predicted output; determining that the first predicted output satisfies a secondary threshold but does not satisfy a primary threshold; receiving second audio data; processing the second audio data to generate a second predicted output; determining that the second predicted output satisfies the secondary threshold but does not satisfy the primary threshold; in response to the first predicted output and the second predicted output satisfying the secondary threshold but not satisfying the primary threshold, and in response to the first spoken utterance and the second spoken utterance satisfying one or more temporal criteria relative to one another, identifying a failed hotword attempt; and in response to identifying the failed hotword attempt, providing a hint that is responsive to the failed hotword attempt.

Abstract: Implementations set forth herein relate to an automated assistant that can employ various utilities according to whether certain conditions are satisfied for a given utility and/or whether a particular input is determined to have a threshold degree of relevance to the given utility. The automated assistant can be customized by a third party entity, which can make the automated assistant available via a device and/or application. The automated assistant can operate according to certain utilities that have been permitted by the third party entity. However, the third party entity can allow the automated assistant to employ other scalable utilities when a user input is determined to have a threshold degree of relevance and/or when usage of certain scalable utilities has not exceeded a threshold. The utility can refer to a machine learning model and/or other data that can be employed to resolve an input.

Abstract: Some implementations relate to performing speech biasing, NLU biasing, and/or other biasing based on historical assistant interaction(s). It can be determined, for one or more given historical interactions of a given user, whether to affect future biasing for (1) the given user account, (2) additional user account(s), and/or (3) the shared assistant device as a whole. Some implementations disclosed herein additionally and/or alternatively relate to: determining, based on utterance(s) of a given user to a shared assistant device, an association of first data and second data; storing the association as accessible to a given user account of the given user; and determining whether to store the association as also accessible by additional user account(s) and/or the shared assistant device.

Abstract: Implementations are directed to biasing speaker authentication on a per-user basis and on a device-by-device basis and/or contextual feature(s) basis. In some of those implementations, in performing speaker authentication based on a spoken utterance, different biasing parameters are determined for each of multiple different registered users of an assistant device at which the spoken utterance was detected. In those implementations, each of the biasing parameters can be used to make it more likely or less likely (in dependence of the biasing parameter) that a corresponding registered user will be verified using the speaker authentication. Through utilization of biasing parameter(s) in performing speaker authentication, accuracy and/or robustness of speaker authentication can be increased.

Abstract: Systems and methods for determining whether to combine responses from multiple automated assistants. An automated assistant may be invoked by a user utterance, followed by a query, which is provided to a plurality of automated assistants. A first response is received from a first automated assistant and a second response is received from a second automated assistant. Based on similarity between the responses, a primary automated assistant determines whether to combine the responses into a combined response. Once the combined response has been generated, one or more actions are performed in response to the combined response.

Abstract: To identify driving event sounds during navigation, a client device in a vehicle provides a set of navigation directions for traversing from a starting location to a destination location along a route. During navigation to the destination location, the client device identifies audio that includes a driving event sound from within the vehicle or an area surrounding the vehicle. In response to determining that the audio includes the driving event sound, the client device determines whether the driving event sound is artificial. In response to determining that the driving event sound is artificial, the client device presents a notification to the driver indicating that the driving event sound is artificial or masks the driving event sound to prevent the driver from hearing the driving event sound.

Abstract: Methods, systems, and apparatus for receiving a request that includes a user identifier of a user that submitted a search query and an entity identifier of an entity that is referenced by the search query, identifying a plurality of knowledge elements that are related to the entity, identifying, in a consumption database, one or more items that have been indicated as consumed by the user and that are associated with the entity that is referenced by the search query, assigning rank scores to the plurality of knowledge elements, based at least on identifying the one or more items, selecting one or more of the knowledge elements from among the knowledge elements based at least on the rank scores assigned to the knowledge elements, and providing, in response to the request, information associated with the entity and the one or more selected knowledge elements.