Abstract: Implementations described herein relate to causing a client device to initiate streaming of multimedia content and causing the client device to render dialog content before and/or during the streaming of the multimedia content. Processor(s) can generate a structured large language model (LLM) query that can be processed to generate LLM output. The LLM output can include, for example, an indication of the multimedia content and the dialog content. In some implementations, the processors(s) can determine when to generate an additional structured LLM query to continue the streaming of the multimedia content, and proactively cause additional LLM output to be generated based on the additional structured LLM query. In additional or alternative implementations, the processor(s) can determine when to cause the dialog content to be rendered with respect to the streaming of the multimedia content.

Abstract: Implementations are described herein for updating user-specific generative model conditioning data at users' requests. In various implementations, a first input prompt is assembled to include user-specific conditioning data (USCD) built from past user interactions and more recent dialog turns between the user and a generative model application. The first input prompt is processed by one or more generative models to generate output identifying out-of-date portions of the USCD. Based on these identified portions, the USCD is updated and stored to reflect the new dialog turns. This updated USCD is then used for subsequent generative model queries.

Abstract: Implementations are described herein for selectively using retrieval augmented generation (RAG) for generative model prompting. In various implementations, a generative model query of a user may be analyzed to determine whether retrieval augmented generation (RAG) should be used to generate a response. If RAG should be used, a generative model input prompt may be formed with data indicative of the generative model query and data indicative of: (i) user-specific conditioning data (USCD) associated with the user, or (ii) personal RAG data of the user. The user-specific conditioning data may have been built over time based at least in part on the personal RAG data of the user. The prompt may be processed using generative model(s) to generate generative model output, conditioned on one or both of the USCD or the personal RAG data of the user, that includes a response to the generative model query.

Abstract: Implementations described herein personalize a user's interactions with generative model(s), including through an automated assistant, by conditioning its queries and responses on user interactions with various computing devices and appliances over time. A first input prompt is assembled, including user-specific conditioning data built over time and new user interactions. This prompt is processed using generative models to identify outdated portions of the conditioning data. A second prompt is then assembled, including the identified outdated portions and the new interactions. This prompt is processed to generate new versions of the outdated portions. Finally, an updated user-specific conditioning data is stored for subsequent use, incorporating the new versions and unaltered portions of the user-specific conditioning data.

Abstract: Implementations described herein relate to updating user-specific conditioning data (USCD) using mappings to user interactions. In various implementations, Data indicative of new user interactions between a user and computing device(s) may be stored. Based on the new user interactions, portions of USCD representing user attributes may be updated. The USCD may be operable to condition a generative model to generate model output tailored to the user. Mappings between the updated portions of the USCD and the data indicative of the new user interactions may also be stored. When it is determined that user interaction(s) have been altered, the mappings may be used to update corresponding portions of the USCD to reflect the alteration of the user interactions.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using conversation histories to augment a software agent. In particular, the software agent can selectively and adaptively be provided with data from conversation history data that comprises respective segment data for each of one or more past segments of each of a plurality of past conversations between a user and a software agent, the respective segment data comprising a respective segment embedding for the past segment.

Abstract: Implementations described herein relate to causing a client device to initiate streaming of multimedia content and causing the client device to render dialog content before and/or during the streaming of the multimedia content. Processor(s) can generate a structured large language model (LLM) query that can be processed to generate LLM output. The LLM output can include, for example, an indication of the multimedia content and the dialog content. In some implementations, the processors(s) can determine when to generate an additional structured LLM query to continue the streaming of the multimedia content, and proactively cause additional LLM output to be generated based on the additional structured LLM query. In additional or alternative implementations, the processor(s) can determine when to cause the dialog content to be rendered with respect to the streaming of the multimedia content.

Abstract: Implementations provide a method that includes: receiving a user input from a particular user; generating, based on attribute information provided by the particular user, an attribute embedding that numerically represents, but does not reveal, the attribute information of the particular user; processing, using a language model, both the attribute embedding and the user input to generate a language model output; generating, based on the language model output, a response to the user input; and causing the generated response to be rendered at the client device in response to the user input from the particular user.

Abstract: Processor(s) can receive a first natural language (NL) based input as part of a dialog between a user of a client device and an automated assistant that is accessible at the client device; process, using a first machine learning (ML) model, the first NL based input to generate a first inference associated with the user and a confidence indicator for the first inference, the confidence indicator for the first inference indicative of a degree of confidence that the first inference is true; store the first inference and the confidence indicator for the first inference in a database; process, using a second ML model, the first inference and additional data to generate a modified confidence indicator for the first inference and store the modified confidence indicator in the database; and determine, based at least on the modified confidence indicator for the first inference, whether to process, using a first large language model (LLM), the first inference with a second NL based input to generate a first NL based re

Abstract: Processor(s) can receive a first natural language (NL) based input as part of a dialog between a user of a client device and an automated assistant that is accessible at the client device; process, using a first machine learning (ML) model, the first NL based input to generate a first inference associated with the user and a confidence indicator for the first inference, the confidence indicator for the first inference indicative of a degree of confidence that the first inference is true; store the first inference and the confidence indicator for the first inference in a database; process, using a second ML model, the first inference and additional data to generate a modified confidence indicator for the first inference and store the modified confidence indicator in the database; and determine, based at least on the modified confidence indicator for the first inference, whether to process, using a first large language model (LLM), the first inference with a second NL based input to generate a first NL based re

Abstract: Implementations described herein relate to causing a client device to initiate streaming of multimedia content and causing the client device to render dialog content before and/or during the streaming of the multimedia content. Processor(s) can generate a structured large language model (LLM) query that can be processed to generate LLM output. The LLM output can include, for example, an indication of the multimedia content and the dialog content. In some implementations, the processors(s) can determine when to generate an additional structured LLM query to continue the streaming of the multimedia content, and proactively cause additional LLM output to be generated based on the additional structured LLM query. In additional or alternative implementations, the processor(s) can determine when to cause the dialog content to be rendered with respect to the streaming of the multimedia content.

Abstract: A method where one or more objects of a three-dimensional scene are determined in accordance with provided raw data of the three-dimensional scene representing a predefined environment inside and/or outside the vehicle. A two-dimensional image is determined in accordance with the provided raw data of the three-dimensional scene such that the two-dimensional image depicts the determined objects of the three-dimensional scene on a curved plane. The two-dimensional image has a quantity of pixels, each representing at least one part or several of the determined objects of the three-dimensional scene. Data is provided which represents at least one determined field of view of the driver. For at least one of the determined objects, the probability with which the at least one object will be located in the field of view of the driver is determined in accordance with the provided data and the two-dimensional image.

Abstract: A method determines a blockage of a sensor of a plurality of sensors of an ego vehicle. The method determines a prior blockage probability of the sensor of the plurality of sensors; receives sensor data of the sensor of the plurality of sensors; determines a performance of the sensor based on the received sensor data; calculates a posterior blockage probability based on the prior blockage probability of the sensor and the performance of the sensor; and determines the blockage of the sensors using the calculated posterior blockage probability.

Abstract: A main computing device integrated into a vehicle may perform various aspects of the techniques described in this disclosure. The main computing device comprises a memory and a processor. The memory may store a first and second instance of a runtime environment. The processor may execute a first container that enables execution of a first instance of the runtime environment, and execute a second container that enables execution of a second instance of the runtime environment. The first instance of the runtime environment may detect a supporting computing device, transfer, responsive to detecting the supporting computing device, the second container to the supporting computing device, and interface with the second instance of the runtime environment to jointly present the user interface by which the operator of the vehicle controls the functionality of the vehicle.

Abstract: In general, techniques are described by which to provide streaming via a hardware abstraction layer. A cartridge comprising a memory, and a processor may be configured to perform the techniques. The processor may execute a second version of an operating system, and receive a stream, via an interface and to a first version of the operating system executing on a processor of a head unit of a vehicle, information relating to an application executing in the application layer exposed by the first version of the operating system. The information relating to the application may include a first function call that conforms with the first version of the operating system, where the cartridge is communicatively coupled to the head unit. The interface may transform the first function call to a second function call that conforms with the second version of the operating system.

Abstract: In general techniques are described by which to provide switchable communication transport for communication between primary devices and vehicle head units. A primary device comprising a memory and a processor may be configured to perform the techniques. The memory may store an operating system and an application. The processor may execute the operating system to present a single communication interface by which the application establishes a first transport between the primary device and a vehicle head unit that facilitates execution of a mode in which the application provides data for presentation by the vehicle head unit. The processor may also execute the application to transmit, during execution of the mode, the data via the first transport, where the operating system switches, during execution of the mode, from the first transport to a second transport. The application transmits, during execution of the mode, the data via the second transport.

Abstract: A method evaluates a degree of fatigue of a vehicle occupant in a vehicle. A number of first fatigue indicators is provided which are determined according to computation rules from a plurality of first sensor values and each represent a degree of fatigue of the vehicle occupant. The first sensor values represent measured values of the vehicle and/or measured values relating to a current journey. A first metadata record is associated with each of the number of first fatigue indicators, wherein the first metadata records represent information about the characteristics of the sensors. The first sensor values are processed in the respective first fatigue indicators. A number of second fatigue indicators is provided which are determined according to computation rules from one or more second sensor values and each represent a degree of fatigue of the vehicle occupant. The second sensor values represent physiological and/or physical parameters of the vehicle occupants.

Abstract: A main computing device integrated into a vehicle may perform various aspects of the techniques described in this disclosure. The main computing device comprises a memory and a processor. The memory may store a first and second instance of a runtime environment. The processor may execute a first container that enables execution of a first instance of the runtime environment, and execute a second container that enables execution of a second instance of the runtime environment. The first instance of the runtime environment may detect a supporting computing device, transfer, responsive to detecting the supporting computing device, the second container to the supporting computing device, and interface with the second instance of the runtime environment to jointly present the user interface by which the operator of the vehicle controls the functionality of the vehicle.

Abstract: A main computing device integrated into a vehicle may perform various aspects of the techniques described in this disclosure. The main computing device comprises a memory and a processor. The memory may store a first and second instance of a runtime environment. The processor may execute a first container that enables execution of a first instance of the runtime environment, and execute a second container that enables execution of a second instance of the runtime environment. The first instance of the runtime environment may detect a supporting computing device, transfer, responsive to detecting the supporting computing device, the second container to the supporting computing device, and interface with the second instance of the runtime environment to jointly present the user interface by which the operator of the vehicle controls the functionality of the vehicle.

Abstract: In general techniques are described by which to provide switchable communication transport for communication between primary devices and vehicle head units. A primary device comprising a memory and a processor may be configured to perform the techniques. The memory may store an operating system and an application. The processor may execute the operating system to present a single communication interface by which the application establishes a first transport between the primary device and a vehicle head unit that facilitates execution of a mode in which the application provides data for presentation by the vehicle head unit. The processor may also execute the application to transmit, during execution of the mode, the data via the first transport, where the operating system switches, during execution of the mode, from the first transport to a second transport. The application transmits, during execution of the mode, the data via the second transport.