Abstract: An overall endpointing measure can be generated based on an audio-based endpointing measure and (1) an accelerometer-based endpointing measure and/or (2) a gaze-based endpointing measure. The overall endpointing measure can be used in determining whether a candidate endpoint is an actual endpoint. Various implementations include generating the audio-based endpointing measure by processing an audio data stream, capturing a spoken utterance of a user, using an audio model. Various implementations additionally or alternatively include generating the accelerometer-based endpointing measure by processing a stream of accelerometer data using an accelerometer model. Various implementations additionally or alternatively include processing an image data stream using a gaze model to generate the gaze-based endpointing measure.

Abstract: The present disclosure provides a computing device and method for providing personal specific information based on semantic queries. The semantic queries may be input in a natural language form, and may include user specific context, such as by referring to prior or future events related to a place the user is searching for. With the user's authorization, data associated with prior or planned activities of the user may be accessed and information from the accessed data may be identified, wherein the information is correlated with the user specific context. One or more query results are determined based on the identified information and provided for output to the user.

Abstract: Methods, systems, and computer readable media related to generating a combined search query based on search parameters of a current search query of a user and search parameters of one or more previously submitted search quer(ies) of the user that are determined to be of the same line of inquiry as the current search query. Two or more search queries may be determined to share a line of inquiry when it is determined that they are within a threshold level of semantic similarity to one another. Once a shared line of inquiry has been identified and a combined search query generated, users may interact with the search parameters and/or the search results to update the search parameters of the combined search query.

Abstract: A method includes receiving a natural language query specifying an action for an assistant interface to perform and selecting one or more business large language models (LLMs) for the assistant interface to interact with to fulfill performance of the action. For each business LLM, method also includes accessing an adapter module to structure the natural language query into a respective prompt specifically formulated for the corresponding business LLM, issuing, for input to the corresponding business LLM, the respective prompt, and receiving corresponding response content from the corresponding business LLM that conveys details regarding performance of a corresponding portion of the action. The method also includes presenting, for output from the user device, presentation content based on the corresponding response content received from each corresponding business LLM.

Abstract: A computing system can include one or more machine-learned models configured to receive context data that describes one or more entities to be named. In response to receipt of the context data, the machine-learned model(s) can generate output data that describes one or more names for the entity or entities described by the context data. The computing system can be configured to perform operations including inputting the context data into the machine-learned model(s). The operations can include receiving, as an output of the machine-learned model(s), the output data that describes the name(s) for the entity or entities described by the context data. The operations can include storing at least one name described by the output data.

Abstract: Generating audio tracks is provided. The system selects a digital component object having a visual output format. The system determines to convert the digital component object into an audio output format. The system generates text for the digital component object. The system selects, based on context of the digital component object, a digital voice to render the text. The system constructs a baseline audio track of the digital component object with the text rendered by the digital voice. The system generates, based on the digital component object, non-spoken audio cues. The system combines the non-spoken audio cues with the baseline audio form of the digital component object to generate an audio track of the digital component object. The system provides the audio track of the digital component object to the computing device for output via a speaker of the computing device.

Abstract: Generating audio tracks is provided. The system selects a digital component object having a visual output format. The system determines to convert the digital component object into an audio output format. The system generates text for the digital component object. The system selects, based on context of the digital component object, a digital voice to render the text. The system constructs a baseline audio track of the digital component object with the text rendered by the digital voice. The system generates, based on the digital component object, non-spoken audio cues. The system combines the non-spoken audio cues with the baseline audio form of the digital component object to generate an audio track of the digital component object. The system provides the audio track of the digital component object to the computing device for output via a speaker of the computing device.

Abstract: Implementations set forth herein relate to an automated assistant that can selectively communicate audio data to a recipient when a user solicits the automated assistant to send a text message to the recipient. The audio data can include a snippet of audio that characterizes content of the text message, and the automated assistant can communicate the audio data to the recipient when score data for a speech recognition hypothesis does not satisfy a confidence threshold. The score data can correspond to an entirety of content of a text message and/or speech recognition hypothesis, and/or less than an entirety of the content. A recipient device can optionally re-process the audio data using a model that is associated with the recipient device. This can provide more accurate transcripts in some instances, thereby improving accuracy of communications and decreasing a number of corrective messages sent between users.

Abstract: Implementations can identify a given assistant device from among a plurality of assistant devices in an ecosystem, obtain device-specific signal(s) that are generated by the given assistant device, process the device-specific signal(s) to generate candidate semantic label(s) for the given assistant device, select a given semantic label for the given semantic device from among the candidate semantic label(s), and assigning, in a device topology representation of the ecosystem, the given semantic label to the given assistant device. Implementations can optionally receive a spoken utterance that includes a query or command at the assistant device(s), determine a semantic property of the query or command matches the given semantic label to the given assistant device, and cause the given assistant device to satisfy the query or command.

Abstract: Implementations relate to generating a proficiency measure, and utilizing the proficiency measure to adapt one or more automated assistant functionalities. The generated proficiency measure is for a particular class of automated assistant actions, and is specific to an assistant device and/or is specific to a particular user. A generated proficiency measure for a class can reflect a degree of proficiency, of a user and/or of an assistant device, for that class. Various automated assistant functionalities can be adapted, for a particular class, responsive to determining the proficiency measure satisfies a threshold, or fails to satisfy the threshold (or an alternate threshold). The adaptation(s) can make automated assistant processing more efficient and/or improve (e.g., shorten the duration of) user-assistant interaction(s).

Abstract: To select a lane in a multi-lane road segment for a vehicle travelling on the road segment, a system identifies, in multiple lanes and in a region ahead of the vehicle, another vehicle defining a target; the system applies an optical flow technique to track the target during a period of time, to generate an estimate of how fast traffic moves; and the system applies the estimate to machine learning (ML) model to generate a recommendation which one of the plurality of lanes the vehicle is to choose.

Abstract: A method for handling contradictory queries on a shared device includes receiving a first query issued by a first user, the first query specifying a first long-standing operation for a digital assistant to perform, and while the digital assistant is performing the first long-standing operation, receiving a second query, the second query specifying a second long-standing operation for the digital assistant to perform. The method also includes determining that the second query was issued by another user different than the first user and determining, using a query resolver, that performing the second long-standing operation would conflict with the first long-standing operation. The method further includes identifying one or more compromise operations for the digital assistant to perform, and instructing the digital assistant to perform a selected compromise operation among the identified one or more compromise operations.

Abstract: Voice-based interaction with video content being presented by a media player application is enhanced through the use of an automated assistant capable of identifying when a spoken utterance by a user is a request to playback a specific scene in the video content. A query identified in a spoken utterance may be used to access stored scene metadata associated with video content being presented in the vicinity of the user to identify one or more locations in the video content that correspond to the query, such that a media control command may be issued to the media player application to cause the media player application to seek to a particular location in the video content that satisfies the query.

Abstract: Implementations can detect respective audio data that captures an acoustic event at multiple assistant devices in an ecosystem that includes a plurality of assistant devices, process the respective audio data locally at each of the multiple assistant devices to generate respective measures that are associated with the acoustic event using respective event detection models, process the respective measures to determine whether the detected acoustic event is an actual acoustic event, and cause an action associated with the actional acoustic event to be performed in response to determining that the detected acoustic event is the actual acoustic event. In some implementations, the multiple assistant devices that detected the respective audio data are anticipated to detect the respective audio data that captures the actual acoustic event based on a plurality of historical acoustic events being detected at each of the multiple assistant devices.

Abstract: A method includes instructing an always-on first processor to operate in a follow-on query detection mode, and while the always-on first processor operates in the follow-on query detection mode: receiving follow-on audio data captured by the assistant-enabled device; determining, using a voice activity detection (VAD) model executing on the always-on first processor, whether or not the VAD model detects voice activity in the follow-on audio data; performing, using a speaker identification (SID) model executing on the always-on first processor, speaker verification on the follow-on audio data to determine whether the follow-on audio data includes an utterance spoken by the same user. The method also includes initiating a wake-up process on a second processor to determine whether the utterance includes a follow-on query.

Abstract: Implementations described herein relate to pairing a location-based automated assistant with a user device. The user device can include, for example, a headphones apparatus and/or a device that is paired with the headphones apparatus. The user device provides an indication that it is present at a location that is associated with a location-based automated assistant. A trust measure is determined that is indicative of trust between the user device and the location-based automated assistant. User information is provided by the user device to the location-based automated assistant. The location-based automated assistant determines response data to provide, via one or more speakers associated with the user device, that is specific to the location and further based on the user information.

Abstract: Implementations relate to an automated assistant that can respond to communications received via a third party application and/or other third party communication modality. The automated assistant can determine that the user is participating in multiple different conversations via multiple different third party communication services. In some implementations, conversations can be processed to identify particular features of the conversations. When the automated assistant is invoked to provide input to a conversation, the automated assistant can compare the input to the identified conversation features in order to select the particular conversation that is most relevant to the input. In this way, the automated assistant can assist with any of multiple disparate conversations that are each occurring via a different third party application.

Abstract: Implementations set forth herein relate to a robotic computing device that can perform certain operations, such as communicating between users in a common space, according to certain preferences of the users. When interacting with a particular user, the robotic computing device can perform an operation at a preferred location relative to the particular user based on an express or implied preference of that particular user. For instance, certain types of operations can be performed at a first location within a room, and other types of operations can be performed at a second location within the room. When an operation involves following or guiding a user, parameters for driving the robotic computing device can be selected based on preferences of the user and/or a context in which the robotic computing device is interacting with the user (e.g., whether or not the context indicates some amount of urgency).

Abstract: A method includes detecting multiple users, receiving a first query issued by a first user, the first query including a command for a digital assistant to perform a first action, and enabling a round robin mode to control performance of actions commanded by queries. The method also includes, while performing the first action, receiving audio data corresponding to a second query including a command to perform a second action, performing speaker identification on the audio data, determining that the second query was spoken by the first user, preventing performing the second action, and prompting at least another user to issue a query. The method further includes receiving a third query issued by a second user, the third query including a command for the digital assistant to perform a third action, and when the digital assistant completes performing the first action, executing performance of the third action.

Abstract: Techniques are described herein for warm word arbitration between automated assistant devices. A method includes: determining that warm word arbitration is to be initiated between a first assistant device and one or more additional assistant devices, including a second assistant device; broadcasting, by the first assistant device, to the one or more additional assistant devices, an active set of warm words for the first assistant device; for each of the one or more additional assistant devices, receiving, from the additional assistant device, an active set of warm words for the additional assistant device; identifying a matching warm word included in the active set of warm words for the first assistant device and included in the active set of warm words for the second assistant device; and enabling or disabling detection of the matching warm word by the first assistant device, in response to identifying the matching warm word.