Abstract: Implementations relate to generating tailored multi-modal response(s) through utilization of large language model(s) (LLM(s)). In some implementations, processor(s) of a system can: receive natural language (NL) based input indicative of a request for a set of slides to be generated, generate a multi-modal response, using an LLM, that is responsive to the NL based input, the multi-modal response comprising a generated set of slides, and cause the multi-modal response to be rendered at the client device of the user. In additional or alternative implementations, the NL based input can be indicative of a request for assistance with completing a particular task. In these implementations, the processor(s) can generate the multi-modal response comprising assistive content for assisting the user in performing the particular task. In various implementations, the LLM can be fine-tuned prior to receiving the NL based input.

Abstract: Implementations relate to a method implemented by one or more processors, the method including: receiving an input prompt for a large language model (LLM); generating first LLM output that is usable to generate a set of user interface (UI) each associated with a corresponding sub-prompt of the input prompt; causing, based on the first LLM output, the set of UI elements to be rendered at a user device; receiving further user input based on user interactions with one or more of the UI elements of the set of UI elements; and in response to determining that one or more termination conditions are satisfied: generating a final response to the input prompt based on generating second LLM output that is usable to generate the final response; and causing the final response to be rendered at the user device.

Abstract: The present disclosure is directed to an automated neural architecture search approach for designing new neural network architectures such as, for example, resource-constrained mobile CNN models. In particular, the present disclosure provides systems and methods to perform neural architecture search using a novel factorized hierarchical search space that permits layer diversity throughout the network, thereby striking the right balance between flexibility and search space size. The resulting neural architectures are able to be run relatively faster and using relatively fewer computing resources (e.g., less processing power, less memory usage, less power consumption, etc.), all while remaining competitive with or even exceeding the performance (e.g., accuracy) of current state-of-the-art mobile-optimized models.

Abstract: A method is for operating a surround view camera system for a vehicle towing a trailer. The method includes generating first image data at a first time using a first imaging device operably connected to a front side of the vehicle or a rear side of the trailer, and generating second image data at a second time using the first imaging device. The second time is different from the first time. The method also includes generating third image data using a second imaging device operably connected to a rear side of the vehicle, and receiving vehicle data with a processor as generated by at least one vehicle sensor. The vehicle data corresponds to movement of the vehicle from the first time to the second time. The processor receives the generated first image data, the second image data, and the third image data.

Abstract: A method of operating a surround view camera system for a vehicle includes generating first image data at a first time using a first imaging device operably connected to the vehicle, and generating second image data at a second time using a second imaging device operably connected to the vehicle. The second time is different from the first time, and the second imaging device different from the first imaging device. The method also includes identifying distorted object data in the second image data using a processor configured to receive the first image data and the second image data. The distorted object data corresponds to a predetermined object in a surroundings of the vehicle. The method includes identifying non-distorted object data in the first image data using the processor. The non-distorted object data corresponds to the predetermined object.

Abstract: Some implementations relate to generating a training and/or evaluation dataset with LLM prompts (e.g., derived from user queries) based on a prompt complexity. An input prompt, for example derived from a user query, is received. The input prompt is decomposed into a prompt tree comprising a plurality of nodes. The plurality of nodes comprise: a plurality of leaf nodes corresponding to simple sub-prompts of the input query; a plurality of branch nodes of sub-prompts each corresponding to multiple simple sub-prompts; and a root node corresponding to the input prompt. A prompt complexity is determined based on a path length of the prompt tree. The prompt complexity is compared to a threshold complexity. If the prompt complexity is above the threshold complexity, the input prompt is included in a set of training prompts and/or a set of evaluation prompts.

Abstract: Implementations relate to a method implemented by one or more processors, the method including: receiving natural language (NL) based input associated with a client device; generating, using a large language model (LLM) and based on processing the NL based input, LLM output; determining, based on the LLM output, a sequence of LLM responses, the sequence of LLM responses including at least one intermediate LLM response and a final LLM response. In some implementations, the method may further include causing the final LLM response to be rendered at the client device. In additional or alternative implementations, the method may further include storing, as an instance of training data for fine-tuning the LLM or an additional LLM, the NL based input along with the final LLM response.

Abstract: As part of a dialog session between a user and an automated assistant, implementations can receive a stream of audio data that captures a spoken utterance including an assistant query, determine, based on processing the stream of audio data, a set of assistant outputs that are each predicted to be responsive to the assistant query, process, using large language model (LLM) output(s), the assistant outputs and context of the dialog session to generate a set of modified assistant outputs, and cause given modified assistant output, from among the set of modified assistant outputs, to be provided for presentation to the user in response to the spoken utterance. In some implementations, the LLM output(s) can be generated in an offline manner for subsequent use in an online manner. In additional or alternative implementations, the LLM output(s) can be generated in an online manner when the spoken utterance is received.

Abstract: Implementations relate to improving instruction following capabilities of large language models (LLMs) using instruction decomposition, self-evaluation, and optionally progressive refinement. Processor(s) of a system can: obtain natural language (NL) based input, generate a plurality of candidate responses and evaluate the candidate responses based on instructions included in the NL based input, using an LLM, and progressively refine the candidate responses until it is determined that one or more termination criteria are satisfied. In some implementations, the NL based input can be received from a client device. In these implementations, a given candidate response that is progressively refined can be rendered for presentation at the client device and responsive to the NL base input. In additional or alternative implementations, the NL based input can be obtained from database(s). In these implementations, a given candidate response that is progressively refined can be utilized in fine-tuning of the LLM.

Abstract: As part of a dialog session between a user and an automated assistant, implementations can receive a stream of audio data that captures a spoken utterance including an assistant query, determine, based on processing the stream of audio data, a set of assistant outputs that are each predicted to be responsive to the assistant query, process, using large language model (LLM) output(s), the assistant outputs and context of the dialog session to generate a set of modified assistant outputs, and cause given modified assistant output, from among the set of modified assistant outputs, to be provided for presentation to the user in response to the spoken utterance. In some implementations, the LLM output(s) can be generated in an offline manner for subsequent use in an online manner. In additional or alternative implementations, the LLM output(s) can be generated in an online manner when the spoken utterance is received.

Abstract: Implementations described herein relate to using self-evaluation when utilizing a large language model (LLM) to generate a response to a natural language (NL) based input. The LLM can be used to process the NL based input to generate a plurality of responses, and to generate a critique of those responses by comparing the responses to a set of response evaluation criteria. One of the responses can then be selected based on the comparison with the set of response evaluation criteria which can vary from one NL based input to another. If the NL based input was obtained a user of a client device during an inference stage, then the selected response can be rendered for presentation to the user. If the NL based input was obtained during a training stage, then the selected response can be stored as a training instance and optionally in association with additional data.

Abstract: At least utilizing a custom corpus of documents to condition a large language model (LLM) when generating a response to a user query. In some implementations, a user query associated with a client device is received. An API query for an external application is generated by an LLM based on the user query. The external application has access to a custom corpus of documents comprising a plurality of documents. The external application is queried using the API query. Data representative of one or more documents in the custom corpus of documents is received from the external application in response to the API query. The LLM generates a response to the query that is conditioned on the data representing one or more of the documents in the custom corpus of documents received from the external application. The response to the user query is caused to be rendered on the client device.

Abstract: Methods and systems for obstructive sleep apnea diagnosis and prediction are disclosed. The methods and systems include: obtaining a white noise contaminated sensor signal for a patient; extracting a feature based on the white noise contaminated sensor signal; determining a matrix based on the feature; determining an intermittent forcing signal based on the matrix; determining an overcomplete representation of the intermittent forcing signal; and generating an obstructive sleep apnea indication based on the overcomplete representation and a threshold. Other aspects, embodiments, and features are also claimed and described.

Abstract: Techniques are described herein for personalized multi-response dialog generated using one or more large language models. A method includes: receiving first natural language (NL) based input associated with a client device; generating, based on the first NL based input and using at least one large language model (LLM), one or more instances of first LLM output; determining, based on the one or more instances of first LLM output, at least three responses to the first NL based input; determining, based on at least one scoring criterion, respective scores of the at least three responses to the first NL based input; selecting, based on the respective scores of the at least three responses to the first NL based input, from the at least three responses to the first NL based input, a first subset, the first subset comprising at least two responses to the first NL based input; and causing each of the at least two responses in the first subset to be rendered at the client device.

Abstract: The present disclosure is directed to an automated neural architecture search approach for designing new neural network architectures such as, for example, resource-constrained mobile CNN models. In particular, the present disclosure provides systems and methods to perform neural architecture search using a novel factorized hierarchical search space that permits layer diversity throughout the network, thereby striking the right balance between flexibility and search space size. The resulting neural architectures are able to be run relatively faster and using relatively fewer computing resources (e.g., less processing power, less memory usage, less power consumption, etc.), all while remaining competitive with or even exceeding the performance (e.g., accuracy) of current state-of-the-art mobile-optimized models.

Abstract: A system for predicting and summarizing medical events from electronic health records includes a computer memory storing aggregated electronic health records from a multitude of patients of diverse age, health conditions, and demographics including medications, laboratory values, diagnoses, vital signs, and medical notes. The aggregated electronic health records are converted into a single standardized data structure format and ordered arrangement per patient, e.g., into a chronological order. A computer (or computer system) executes one or more deep learning models trained on the aggregated health records to predict one or more future clinical events and summarize pertinent past medical events related to the predicted events on an input electronic health record of a patient having the standardized data structure format and ordered into a chronological order.

Abstract: The present disclosure is directed to an automated neural architecture search approach for designing new neural network architectures such as, for example, resource-constrained mobile CNN models. In particular, the present disclosure provides systems and methods to perform neural architecture search using a novel factorized hierarchical search space that permits layer diversity throughout the network, thereby striking the right balance between flexibility and search space size. The resulting neural architectures are able to be run relatively faster and using relatively fewer computing resources (e.g., less processing power, less memory usage, less power consumption, etc.), all while remaining competitive with or even exceeding the performance (e.g., accuracy) of current state-of-the-art mobile-optimized models.

Abstract: Generally, the present disclosure is directed to systems and methods that generate augmented training data for machine-learned models via application of one or more augmentation techniques to audiographic images that visually represent audio signals. In particular, the present disclosure provides a number of novel augmentation operations which can be performed directly upon the audiographic image (e.g., as opposed to the raw audio data) to generate augmented training data that results in improved model performance. As an example, the audiographic images can be or include one or more spectrograms or filter bank sequences.

Abstract: Generally, the present disclosure is directed to systems and methods that generate augmented training data for machine-learned models via application of one or more augmentation techniques to audiographic images that visually represent audio signals. In particular, the present disclosure provides a number of novel augmentation operations which can be performed directly upon the audiographic image (e.g., as opposed to the raw audio data) to generate augmented training data that results in improved model performance. As an example, the audiographic images can be or include one or more spectrograms or filter bank sequences.

Abstract: Example aspects of the present disclosure are directed to systems and methods for learning data augmentation strategies for improved object detection model performance. In particular, example aspects of the present disclosure are directed to iterative reinforcement learning approaches in which, at each of a plurality of iterations, a controller model selects a series of one or more augmentation operations to be applied to training images to generate augmented images. For example, the controller model can select the augmentation operations from a defined search space of available operations which can, for example, include operations that augment the training image without modification of the locations of a target object and corresponding bounding shape within the image and/or operations that do modify the locations of the target object and bounding shape within the training image.