Abstract: Provided are systems, methods, and machine learning models for filling in gaps (e.g., of up to one second) in speech samples by leveraging an auxiliary textual input. Example machine learning models described herein can perform speech inpainting with the appropriate content, while maintaining speaker identity, prosody and recording environment conditions, and generalizing to unseen speakers. This approach significantly outperforms baselines constructed using adaptive TTS, as judged by human raters in side-by-side preference and MOS tests.

Abstract: Implementations relate to an automated assistant that can automate repeatedly performed procedures. The automation can involve communicating with different users, organizations, and/or other automated assistants. The automated assistant, with prior permission from respective user(s), can detect repeated performance of a particular series of manually initiated computational actions. Based on this determination, the automated assistant can determine automated assistant computational action(s) that can be performed by the automated assistant in order to reduce latency in performing a procedure, reduce quantity and/or size of transmissions in performing the procedure, and/or reduce an amount of client device resources required for performing the procedure. Such actions can include communicating with an additional automated assistant that may be associated with another user and/or organization.

Abstract: Provided are systems and methods for continuous training of machine learning (ML) models on changing data. In particular, the present disclosure provides example approaches to model training that take advantage of constantly evolving data that may be available in various ancillary systems that contain large amounts of data, but which are not specific to or dedicated for model training.

Abstract: Implementations utilize a hybrid use of a smaller LLM and a larger LLM to generate and refine content responsive to a user query/request for content generation. In various implementations, the smaller LLM is utilized to process the user query for content generation, to generate initial content responsive to the user query for content generation. The user query for content generation and the initial content can be utilized to generate a text prompt, where the text prompt can be configured to further include a request for focused edit(s). Such a text prompt can be processed using the larger LLM, to generate focused edit(s) to the initial content that refine the initiated content, so that revised content (with improved accuracy) responsive to the user query for content generation is acquired.

Abstract: A method includes, while a user is wearing stereo headphones in an environment, obtaining, from a target digital assistant, a response to a query issued by the user, and obtaining spatial audio preferences of the user. Based on the spatial audio preferences of the user, the method also includes determining a spatially disposed location within a playback sound-field for the user to perceive as a sound-source of the response to the query. The method further includes rendering output audio signals characterizing the response to the query through the stereo headphones to produce the playback sound-field. Here, the user perceives the response to the query as emanating from the sound-source at the spatially disposed location within the playback sound-field.

Abstract: A method for recognizing a voice input includes receiving a first voice input including a plurality of terms, processing the first voice input based on the plurality of terms to obtain a first speech recognition result including one or more candidate terms corresponding to one or more terms from the plurality of terms, receiving a second voice input providing at least one of contextual information relating to the first voice input or confirmation information relating to the one or more candidate terms, and processing the second voice input based on the at least one of the contextual information or the confirmation information to obtain a second speech recognition result including at least one of the one or more candidate terms or one or more new candidate terms, as corresponding to the one or more terms from the plurality of terms.

Abstract: A method including generating base values and delta values based on an image, generating weighted delta values based on the delta values, generating an enhanced image based on the base values and the weighted delta values, and compressing the enhanced image.

Abstract: Implementations can detect respective audio data that captures an acoustic event at multiple assistant devices in an ecosystem that includes a plurality of assistant devices, process the respective audio data locally at each of the multiple assistant devices to generate respective measures that are associated with the acoustic event using respective event detection models, process the respective measures to determine whether the detected acoustic event is an actual acoustic event, and cause an action associated with the actional acoustic event to be performed in response to determining that the detected acoustic event is the actual acoustic event. In some implementations, the multiple assistant devices that detected the respective audio data are anticipated to detect the respective audio data that captures the actual acoustic event based on a plurality of historical acoustic events being detected at each of the multiple assistant devices.

Abstract: In some implementations, a language proficiency of a user of a client device is determined by one or more computers. The one or more computers then determines a text segment for output by a text-to-speech module based on the determined language proficiency of the user. After determining the text segment for output, the one or more computers generates audio data including a synthesized utterance of the text segment. The audio data including the synthesized utterance of the text segment is then provided to the client device for output.

Abstract: A method includes detecting multiple users, receiving a first query issued by a first user, the first query including a command for a digital assistant to perform a first action, and enabling a round robin mode to control performance of actions commanded by queries. The method also includes, while performing the first action, receiving audio data corresponding to a second query including a command to perform a second action, performing speaker identification on the audio data, determining that the second query was spoken by the first user, preventing performing the second action, and prompting at least another user to issue a query. The method further includes receiving a third query issued by a second user, the third query including a command for the digital assistant to perform a third action, and when the digital assistant completes performing the first action, executing performance of the third action.

Abstract: A method using media arbitration includes, while a first assistant-enabled device is performing a first long-standing operation, determining the first assistant-enabled device satisfies a co-presence condition with a second assistant-enabled device, and determining that the second assistant-enabled device is performing a second long-standing operation that conflicts with the first long-standing operation performed by the first assistant-enabled device. Based on determining that the first long-standing operation and the second long-standing operation conflict, the method also includes executing an operation arbitration routine to identify one or more compromise operations for at least one of the first assistant-enabled device or the second assistant-enabled device to perform, and instructing the first assistant-enabled device or the second assistant-enabled device to perform a selected compromise operation among the identified compromise operations.

Abstract: Implementations are directed to dynamically adapting which assistant on-device model(s) are locally stored at assistant devices of an assistant device group and/or dynamically adapting the assistant processing role(s) of the assistant device(s) of the assistant device group. In some of those implementations, the corresponding on-device model(s) and/or corresponding processing role(s), for each of the assistant devices of the group, is determined based on collectively considering individual processing capabilities of the assistant devices of the group. Implementations are additionally or alternatively directed to cooperatively utilizing assistant devices of a group, and their associated post-adaptation on-device model(s) and/or post-adaptation processing role(s), in cooperatively processing assistant requests that are directed to any one of the assistant devices of the group.

Abstract: The technology relates to a navigation system that provides one or more routes from a starting location to a destination location, including a candidate parking location at or near the destination location. The starting location may be the current location of a user or the parking location from a previous navigation session. In some examples, the current location of the user may be the same location as the parking location from the previous navigation session. Each route suggestion may include one or more segments. For example, a route may include a first segment from the location of the user to the candidate parking spot and a second segment may be from the candidate parking location to the destination location. In some examples, there may be a third segment from the user's current location to the parking location from the previous navigation session.

Abstract: Systems and methods for providing dialog data, from an initially invoked automated assistant to a subsequently invoked automated assistant. A first automated assistant may be invoked by a user utterance, followed by a dialog with the user that is processed by the first automated assistant. During the dialog, a request to transfer dialog data to a second automated assistant is received. The request may originate with the user, by the first automated assistant, and/or by the second automated assistant. Once authorized, the first automated assistant provides the previous dialog data to the second automated assistant. The second automated assistant performs one or more actions based on the dialog data.

Abstract: Implementations set forth herein relate to an automated assistant that can solicit other devices for data that can assist with user authentication. User authentication can be streamlined for certain requests by removing a requirement that all authentication be performed at a single device and/or by a single application. For instance, the automated assistant can rely on data from other devices, which can indicate a degree to which a user is predicted to be present at a location of an assistant-enabled device. The automated assistant can process this data to make a determination regarding whether the user should be authenticated in response to an assistant input and/or pre-emptively before the user provides an assistant input. In some implementations, the automated assistant can perform one or more factors of authentication and utilize the data to verify the user in lieu of performing one or more other factors of authentication.

Abstract: To provide navigation directions to one or more points of interest (POIs) for accomplishing a user's goals, a computing device receives an indication of N goals for a user to accomplish, and identifies M POIs for accomplishing the N goals. N is greater than M. The computing device then generates a set of navigation directions for navigating to each of the M POIs, and provides the set of navigation directions for display to the user.

Abstract: Training data is obtained. The training data includes (a) route information indicative of a route from a starting location to a destination location, wherein the route comprises a plurality of route segments comprising a first subset of route segments and a second subset of route segments, and (b) route characteristic information descriptive of one or more route characteristics. At least the first subset of route segments and a portion of the route characteristic information associated with the first subset of route segments is processed with a machine-learned semantic routing model to obtain one or more predicted route segments for the second subset of route segments. One or more parameters of the machine-learned semantic routing model are adjusted based on an optimization function that evaluates a difference between the one or more predicted route segments and the second subset of route segments.

Abstract: Disclosed implementations relate to structures that support an on-demand navigational corpus. An example method involves receiving a navigation request from a client device pertaining to an intent, determining seed content associated with the navigation request, utilizing a large foundational model to create a web page incorporating the seed content, based on a navigation model, and the intent, and delivering the generated web page for presentation on the client device. The method enables efficient and personalized web page generation based on user intent, enhancing user experience and facilitating dynamic navigation using raw seed content.

Abstract: Systems and methods for partitioning a large model that has been configured to use a model-synthesis approach in which multiple basis models are combined to generate a final output. The present technology provides systems and methods for identifying a device-specific or subject-specific subset of those basis models to be used on a given device, such that it need not store the weight matrices for the entire set of basis models, and may perform inference using only the weight matrices of the identified subset of basis models. In some examples, the subset of basis models used by a given device may be updated based on actual usage and feedback. Likewise, in some examples, the model may be trained in a federated setting in which multiple devices each utilize different subsets of the basis models, and share training signals with a full copy of the model.

Abstract: Implementations described herein relate to receiving user input directed to an automated assistant, processing the user input to determine whether data from a server and/or third-party application is needed to perform certain fulfillment of an assistant command included in the user input, and generating a prompt that requests a user consent to transmitting of a request to the server and/or the third-party application to obtain the data needed to perform the certain fulfillment. In implementations where the user consents, the data can be obtained and utilized to perform the certain fulfillment. In implementations where the user does not consent, client data can be generated locally at a client device and utilized to perform alternate fulfillment of the assistant command. In various implementations, the request transmitted to the server and/or third-party application can be modified based on ambient noise captured when the user input is received.