Abstract: Implementations relate to selecting a particular device, from an ecosystem of devices, to provide responses to a device-agnostic request of the user while a scenario is occurring. The user specifies a scenario and contextual features are identified from one or more devices of the ecosystem to generate scenario features indicative of the scenario occurring. The scenario features are stored with a correlation to a device that is specified by the user to handle responses while the scenario is occurring. When a subsequent device-agnostic request is received, current contextual features are identified and compared to the scenario features. Based on the comparison, the specified assistant device is selected to respond to the device-agnostic request.

Abstract: A method for facilitating a Parkinson's Disease (“PD”) assessment of a patient includes capturing first video of a patient performing first test movements while holding the mobile device; capturing second video of the patient performing second test movements while maintaining the mobile device on their person; capturing third video of the patient performing third test movements including standing and walking; capturing one or more IMU readings using an IMU of the mobile device; processing the first video, the second video, and the third video according to (i) a hand landmark model to generate one or more hand biomarkers, (ii) a face landmark model to generate one or more face biomarkers, and (iii) a body landmark model to generate one or more body biomarkers; and determining an assessment score based on a standardized PD assessment by processing the biomarkers.

Abstract: A display device may detect an ambient sound from audio data captured by a plurality of microphones on a display device. A display device may determine, based on the audio data, a location of a sound source of the ambient sound. A display device may generate contextual information about the ambient sound based on an audio segment of the audio data that includes the ambient sound. A display device may display the contextual information based on the location of the sound source.

Abstract: Methods, devices, and media for sensing a touch on a display panel including an array of pixels. A method includes controlling first pixels to operate in an illumination state; controlling second pixels to repeatedly switch between operating in the illumination state and operating in a sensing state; generating sensing signals indicative of levels of light detected by the second pixels; detecting a touch input to the display panel based on the generated sensing signals; and in response to detecting the touch input to the display panel, changing at least one of (i) a frequency at which the second pixels switch between operating in the illumination state and in the sensing state, (ii) a duty cycle for operating the second pixels in the sensing state, or (iii) which of the pixels in the array of pixels are controlled to switch between operating in the illumination state and in the sensing state.

Abstract: Techniques disclosed herein are directed towards pairing a first audio data stream of a first client device with a second audio data stream of a second client device based on identifying a user has imitated playback of media at the second client device, where the imitation of the playback by the user occurs at the first client device. Many implementations include the user controlling the media playback at the second client device using one or more automated assistant queries. Various implementations include deleting the pairing between the first audio data stream and the second audio data stream in response to determining whether one or more conditions have been satisfied (e.g., whether the user stops the playback, whether the first client device is a threshold distance away from the second client device, whether a period of time has elapsed, whether the entire media has played at the second client device, whether one or more additional or alternative conditions are satisfied, and/or combinations thereof).

Abstract: Techniques are described herein for multi-device arbitration using pairwise range multilateration between earbuds and a target device. A method includes: transmitting, by a first user-worn device, to a candidate device, a first impulse signal; receiving, by the first user-worn device, from the candidate device, a second impulse signal; determining a physical distance based on an elapsed time between transmitting the first impulse signal and receiving the second impulse signal; transmitting, by a second user-worn device, to the candidate device, a third impulse signal; receiving, by the second user-worn device, from the candidate device, a fourth impulse signal; determining a physical distance based on an elapsed time between transmitting the third impulse signal and receiving the fourth impulse signal; and determining, based on the physical distances, that the candidate device is a query target.

Abstract: A method for a contextual assistant to use mouse pointing or touch cues includes receiving audio data corresponding to a query spoken by a user, receiving, in a graphical user interface displayed on a screen, a user input indication indicating a spatial input applied at a first location on the screen, and processing the audio data to determine a transcription of the query. The method also includes performing query interpretation on the transcription to determine that the query is referring to an object displayed on the screen without uniquely identifying the object, and requesting information about the object. The method further includes disambiguating, using the user input indication indicating the spatial input applied at the first location on the screen, the query to uniquely identify the object that the query is referring to, obtaining the information about the object requested by the query, and providing a response to the query.

Abstract: Implementations relate to an automated assistant that can bypass invocation phrase detection when an estimation of device-to-device distance satisfies a distance threshold. The estimation of distance can be performed for a set of devices, such as a computerized watch and a cellular phone, and/or any other combination of devices. The devices can communicate ultrasonic signals between each other, and the estimated distance can be determined based on when the ultrasonic signals are sent and/or received by each respective device. When an estimated distance satisfies the distance threshold, the automated assistant can operate as if the user is holding onto their cellular phone while wearing their computerized watch. This scenario can indicate that the user may be intending to hold their device to interact with the automated assistant and, based on this indication, the automated assistant can temporarily bypass invocation phrase detection (e.g., invoke the automated assistant).

Abstract: Disclosures are provided for execution of large-language models (LLMs), including systems and methods that allow for increased multi-user efficiency within an execution framework. For example, LLM queries provided by a plurality of users of an LLM may be combined together into a batched prompt that can be received by an LLM. In addition, the LLM receiving the batched prompt may be configured to provide an output that is segmented in accordance with each user's query. Configuration of the LLM may include providing the LLM with prompt-engineering inputs in connection with the batched prompt.

Abstract: Various arrangements for monitoring for contactless human interactions and health using a single radar transmission modulation mode are provided. Radar chirps may be output by a radar sensor operating in a burst mode. The burst mode radar data stream may be monitored for a contactless human interaction performed by an active user. The burst mode radar data stream may be converted to a virtual continuous mode radar data stream. Health monitoring of an inactive user may be performed using the virtual continuous mode radar data stream.

Abstract: Implementations utilize a LLM to generate content responsive to a user query directed to a video and cause audio data for the generated content to be rendered as a response to the user query. Implementations extract a subset of frames from all frames of the video as key frame(s) for the video, and utilize a vision-language model in generating a natural language description for the key frame(s) of the video. A prompt can be generated based on a transcription of the user query and based on the natural language description for the key frame(s) of the video. The prompt is processed as input, using the LLM, to generate the content responsive to the user query directed to the video.

Abstract: Disclosed implementations for generating personalized audio. In response to receiving sensor data corresponding with a physical characteristic of a user, a first function is determined based on a similarity between the physical characteristic of the user and a first model and a second function is determined based on a similarity of the physical characteristic between the user and a second model. A modified function, representing an audio response, is generated by combining the first function and the second function. An audio stream is generated based on the modified function.

Abstract: Implementations described herein relate to utilizing a large language model (LLM) to determine whether a natural language (NL) based input is grammatically incorrect and notifying a user based on the determination. A structured LLM query may be generated, based on the NL based input, that includes an LLM prompt to cause the LLM to generate an LLM response including an indication of whether the NL based input is grammatically incorrect. An LLM response may be generated, based on causing the structured LLM query to be processed using the LLM, that includes the indication of whether the NL based input is grammatically incorrect. Responsive to determining that the NL based input is grammatically incorrect based on the LLM response, a feedback output may be caused to be rendered at the client device, or an additional client device, that indicates the NL based input is grammatically incorrect.

Abstract: A computing device is described that includes one or more processors configured to receive a search result set including a plurality of image frames, and determine, by a machine learning model of the computing device, one or more visual characteristics associated with each image frame of the plurality of image frames. The computing device is further configured to sort each image frame of the plurality of image frames based on the one or more visual characteristics of each image frame to generate an updated search result set.

Abstract: Systems and methods for closed-loop, contactless respiration guidance providing quantified alignment feedback data in real-time are provided. In one embodiment, a computer-implemented method can include receiving, by a computing system operatively coupled to one or more processors, input data including respiration data indicative of an entity's respiration. The computer-implemented method can include converting, by the computing system, the respiration data into an entity respiration signal such that the entity respiration signal tracks the entity's respiration in real-time. The computer-implemented method can include comparing, by the computing system, the entity respiration signal to a suggested respiration signal indicative of a suggested respiration. The computer-implemented method can include providing, by the computing system, alignment feedback data to the entity in real-time based at least in part on the entity's respiration.

Abstract: According to an aspect, a method may include receiving, by a display device, ambient light measurement data from ambient light measurement devices. The method may include calculating distance measurements indicative of a measured distance between the ambient light measurement devices. The method may include generating a light measurement graph based on the ambient light measurement data and the distance measurements. The method may include generating a raw tone mapping function based on the light measurement graph, the generating using a spatially aware graph neural network. The method may include applying the raw tone mapping function to media content to generate a color corrected version of the media content. The method may include displaying on a display of the display device the color corrected version of the media content.

Abstract: A user electronic device comprising: one or more microphones configured to capture raw audio data; and one or more processors and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising: receiving the raw audio data captured by the one or more microphones; processing the raw audio data using a speech transcriber to generate a live transcription of the raw audio data that comprises a plurality of text tokens; processing the raw audio data to generate a speaker identification output that identifies, for each of the text tokens, a respective speaker for each of the text tokens in the live transcription; and processing a first input comprising (i) a first input prompt and (ii) an input text generated from the live transcription using a language model neural network to generate a modified transcription.

Abstract: A computing device, such as a wearable device, may include a light source and a photodetector array. The photodetector array may be used to determine a touch event of a user that occurs during a time interval. A subset of the plurality of photodetectors associated with the touch event may provide detection signals at each of a plurality of times within the time interval, which may be aggregated to obtain a time series of aggregated detection signals. Biometric data of the user may be generated, based on the time series of aggregated detection signals.

Abstract: Methods, systems, apparatus, including computer programs encoded on a computer storage medium, for spatialized audio feedback from automated assistants. In one aspect, the method includes actions of determining that a user input has been received at a client device, identifying, based on sensor data, one or more points of interest of an environment in which the client device is located and an orientation of a user of the client device relative to the one or more points of interest, identifying, based on processing the user input and the sensor data, a natural language response providing information relevant corresponds to a particular point of interest, determining, based on the orientation of the user of the client device relative to the particular point of interest, one or more spatial audio parameters to be used to provision the natural language response to the user, and causing the natural language response to be audibly rendered at the client device using one or more of the spatial audio parameters.

Abstract: To determine the location of a user, a computing device determines a current location of the user and a destination location. The computing device then determines a plurality of paths for traveling from the current location to the destination location. As the user travels to the destination location, the computing device estimates a plurality of waypoints traversed by the user. The computing device compares the estimated plurality of waypoints to each of the plurality of paths to identify a path of the plurality of paths in which the user travels to the destination location. Then the computing device adjusts the estimated plurality of waypoints according to the identified path to determine precise waypoints for the user.