Abstract: Implementations set forth relate to suggesting an alternate interface modality when an automated assistant and/or a user is expected to not understand a particular interaction between the user and the automated assistant. In some instances, the automated assistant can pre-emptively determine that a forthcoming and/or ongoing interaction between a user and an automated assistant may experience interference. Based on this determination, the automated assistant can provide an indication that the interaction may not be successful and/or that the user should interact with the automated assistant through a different modality. For example, the automated assistant can render a keyboard interface at a portable computing device when the automated assistant determines that an audio interface of the portable computing device is experiencing interference.

Abstract: A method for detecting freeze words includes receiving audio data that corresponds to an utterance spoken by a user and captured by a user device associated with the user. The method also includes processing, using a speech recognizer, the audio data to determine that the utterance includes a query for a digital assistant to perform an operation. The speech recognizer is configured to trigger endpointing of the utterance after a predetermined duration of non-speech in the audio data. Before the predetermined duration of non-speech, the method includes detecting a freeze word in the audio data. In response to detecting the freeze word in the audio data, the method also includes triggering a hard microphone closing event at the user device. The hard microphone closing event prevents the user device from capturing any audio subsequent to the freeze word.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for obtaining, for each of multiple words or sub-words, audio data corresponding to multiple users speaking the word or sub-word; training, for each of the multiple words or sub-words, a pre-computed hotword model for the word or sub-word based on the audio data for the word or sub-word; receiving a candidate hotword from a computing device; identifying one or more pre-computed hotword models that correspond to the candidate hotword; and providing the identified, pre-computed hotword models to the computing device.

Abstract: In some implementations, a language proficiency of a user of a client device is determined by one or more computers. The one or more computers then determines a text segment for output by a text-to-speech module based on the determined language proficiency of the user. After determining the text segment for output, the one or more computers generates audio data including a synthesized utterance of the text segment. The audio data including the synthesized utterance of the text segment is then provided to the client device for output.

Abstract: Various implementations include determining whether further spoken input is intended to correct at least one word in a candidate text representation of spoken input. Various implementations include receiving audio data capturing spoken input of a user. Various implementations include rendering output based on the candidate text representation to the user. Various implementations include receiving, while the output is being rendered, further audio data capturing the further spoken input. In response to determining the further spoken input is intended to correct the at least one word in the candidate text representation, various implementations include generating a revised text representation of the spoken input by altering at least one word in the candidate text representation based on one or more terms in the further candidate text representation.

Abstract: Automated content switching rules may be generated and/or utilized for automatically switching away from certain interactive content during presentation of that interactive content when one or more switch conditions are met. In some instances, automated content switching rules may define one or more non-temporal switch conditions, e.g., based upon reaching certain points or milestones in certain interactive content, that may be used to initiate actions that switch away from the interactive content. In addition, in some instances, automated content switching rules may be used to not only switch away from particular interactive content, but additionally switch to other interactive content, thereby enabling a user to effectively schedule a workflow across different interactive content, applications and/or other computer-related tasks.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for assigning operations of a computational graph to a plurality of computing devices are disclosed. Data characterizing a computational graph is obtained. Context information for a computational environment in which to perform the operations of the computational graph is received. A model input is generated, which includes at least the context information and the data characterizing the computational graph. The model input is processed using the machine learning model to generate an output defining placement assignments of the operations of the computational graph to the plurality of computing devices. The operations of the computational graph are assigned to the plurality of computing device according to the defined placement assignments.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for implementing hotword recognition and passive assistance are disclosed. In one aspect, a method includes the actions of receiving, by a computing device that is operating in a low-power mode and that includes a display that displays a graphical interface while the computing device is in the low-power mode and that is configured to exit the low-power mode in response to detecting a first hotword, audio data corresponding to an utterance. The method further includes determining that the audio data includes a second, different hotword. The method further includes obtaining a transcription of the utterance by performing speech recognition on the audio data. The method further includes generating an additional user interface. The method further includes providing, for output on the display, the additional graphical interface.

Abstract: A personalized endpointing measure can be used to determine whether a user has finished speaking a spoken utterance. Various implementations include using the personalized endpointing measure to determine whether a candidate endpoint indicates a user has finished speaking the spoken utterance or whether the user has paused and has not finished speaking the spoken utterance. Various implementations include determining the personalized endpointing measure based on a portion of a text representation of the spoken utterance immediately preceding the candidate endpoint and a user-specific measure. Additionally or alternatively, the user-specific measure can be based on the text representation immediately preceding the candidate endpoint and one or more historical interactions with the user. In various implementations, each of the historical interactions are specific to the text representation and the user, and indicate whether a previous instance of the text representation was a previous endpoint for the user.

Abstract: A method for optimizing speech recognition includes receiving a first acoustic segment characterizing a hotword detected by a hotword detector in streaming audio captured by a user device, extracting one or more hotword attributes from the first acoustic segment, and adjusting, based on the one or more hotword attributes extracted from the first acoustic segment, one or more speech recognition parameters of an automated speech recognition (ASR) model. After adjusting the speech recognition parameters of the ASR model, the method also includes processing, using the ASR model, a second acoustic segment to generate a speech recognition result. The second acoustic segment characterizes a spoken query/command that follows the first acoustic segment in the streaming audio captured by the user device.

Abstract: Implementations set forth herein relate to an automated assistant that can solicit other devices for data that can assist with user authentication. User authentication can be streamlined for certain requests by removing a requirement that all authentication be performed at a single device and/or by a single application. For instance, the automated assistant can rely on data from other devices, which can indicate a degree to which a user is predicted to be present at a location of an assistant-enabled device. The automated assistant can process this data to make a determination regarding whether the user should be authenticated in response to an assistant input and/or pre-emptively before the user provides an assistant input. In some implementations, the automated assistant can perform one or more factors of authentication and utilize the data to verify the user in lieu of performing one or more other factors of authentication.

Abstract: Implementations described herein include detecting a stream of audio data that captures a spoken utterance of the user and that captures ambient noise occurring within a threshold time period of the spoken utterance being spoken by the user. Implementations further include processing a portion of the audio data that includes the ambient noise to determine ambient noise classification(s), processing a portion of the audio data that includes the spoken utterance to generate a transcription, processing both the transcription and the ambient noise classification(s) with a machine learning model to generate a user intent and parameter(s) for the user intent, and performing one or more automated assistant actions based on the user intent and using the parameter(s).

Abstract: A method for detecting freeze words includes receiving audio data that corresponds to an utterance spoken by a user and captured by a user device associated with the user. The method also includes processing, using a speech recognizer, the audio data to determine that the utterance includes a query for a digital assistant to perform an operation. The speech recognizer is configured to trigger endpointing of the utterance after a predetermined duration of non-speech in the audio data. Before the predetermined duration of non-speech, the method includes detecting a freeze word in the audio data. In response to detecting the freeze word in the audio data, the method also includes triggering a hard microphone closing event at the user device. The hard microphone closing event prevents the user device from capturing any audio subsequent to the freeze word.

Abstract: A first computing device can implement a method for providing navigation instructions. The method includes initiating a first navigation session for providing a first set of navigation instructions to a user from a starting location to a destination location along a first route. The method also includes detecting a second computing device in proximity to the first computing device, and determining that the second computing device is implementing a second navigation session for providing a second set of navigation instructions to the destination location along a second route. Further, the method includes adjusting the first navigation session in accordance with the second navigation session.

Abstract: Techniques are described herein for warm word arbitration between automated assistant devices. A method includes: determining that warm word arbitration is to be initiated between a first assistant device and one or more additional assistant devices, including a second assistant device; broadcasting, by the first assistant device, to the one or more additional assistant devices, an active set of warm words for the first assistant device; for each of the one or more additional assistant devices, receiving, from the additional assistant device, an active set of warm words for the additional assistant device; identifying a matching warm word included in the active set of warm words for the first assistant device and included in the active set of warm words for the second assistant device; and enabling or disabling detection of the matching warm word by the first assistant device, in response to identifying the matching warm word.

Abstract: An overall endpointing measure can be generated based on an audio-based endpointing measure and (1) an accelerometer-based endpointing measure and/or (2) a gaze-based endpointing measure. The overall endpointing measure can be used in determining whether a candidate endpoint is an actual endpoint. Various implementations include generating the audio-based endpointing measure by processing an audio data stream, capturing a spoken utterance of a user, using an audio model. Various implementations additionally or alternatively include generating the accelerometer-based endpointing measure by processing a stream of accelerometer data using an accelerometer model. Various implementations additionally or alternatively include processing an image data stream using a gaze model to generate the gaze-based endpointing measure.

Abstract: Techniques are described herein for collaborative search sessions through an automated assistant. A method includes: receiving, from a first user of a first client device, a first query in a query session; providing, to the first user, a first set of search results; determining, based on at least one term in the first query, that the first query is relevant to a second user of the first client device; providing, to the second user, a selectable option to join the query session; in response to receiving, from the second user, an indication of acceptance of the selectable option, adding the second user to the query session; receiving, from the second user, additional input; generating, based on the additional input received from the second user, a modified set of search results; and providing, to the first user and the second user, the modified set of search results.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for obtaining, for each of multiple words or sub-words, audio data corresponding to multiple users speaking the word or sub-word; training, for each of the multiple words or sub-words, a pre-computed hotword model for the word or sub-word based on the audio data for the word or sub-word; receiving a candidate hotword from a computing device; identifying one or more pre-computed hotword models that correspond to the candidate hotword; and providing the identified, pre-computed hotword models to the computing device.

Abstract: Systems and methods for determining, based on invocation input that is common to multiple automated assistants, which automated assistant to invoke in lieu of invoking other automated assistants. The invocation input is processed to determine one or more invocation features that may be utilized to determine which, of a plurality of candidate automated assistants, to invoke. Further, additional features are processed that can indicate which, of the plurality of invocable automated assistants, to invoke. Once an automated assistant has been invoked, additional audio data and/or features of additional audio data are provided to the invoked automated assistant for further processing.

Abstract: Implementations relate to interactions between a user and an automated assistant during a dialog between the user and the automated assistant. Some implementations relate to processing received user request input to determine that it is of a particular type that is associated with a source parameter rule and, in response, causing one or more sources indicated as preferred by the source parameter rule and one or more additional sources not indicated by the source parameter rule to be searched based on the user request input. Further, those implementations relate to identifying search results of the search(es), and generating, in dependence on the search results, a response to the user request that includes content from search result(s) of the preferred source(s) and/or content from search result(s) of the additional source(s). Generating the response further includes including, in the response, some indication that indicates whether the source parameter rule was followed or violated in generating the response.