Abstract: This application is directed to an integrated multimodal neural network driven by a natural language prompt. A computer system obtains sensor data from a plurality of sensor devices disposed in a physical environment during a time duration. One or more information items are generated to characterize one or more signature events detected within the time duration in the sensor data. The computer system obtains a natural language prompt. In response to the natural language prompt, the computer system applies a large behavior model (e.g., a large language model, a data processing model) to process the one or more information items and the natural language prompt jointly and generate a multimodal output (e.g., textual statements, software code, an image or video, an information dashboard having a predefined format, a user interface, and a heatmap). The multimodal output associated with the sensor data is represented.

Abstract: This application is directed to compressing sensor data. A computer system obtains the sensor data from a plurality of sensor devices disposed in a physical environment during a time duration, and each sensor device corresponds to a temporal sequence of respective sensor samples. For each of the plurality of sensor devices, the temporal sequence of respective sensor samples is processed to generate an ordered sequence of respective sensor data features defining a respective parametric representation of the temporal sequence of respective sensor samples, independently of a sensor type of the respective sensor device. The computer system detects one or more signature events within the time duration based on the respective parametric representations of the plurality of sensor devices, and generates one or more information items characterizing the one or more signature events detected in the sensor data.

Abstract: This application is directed to integrated multimodal neural networks. A computer system obtains sensor data from a plurality of sensor devices during a time duration, and the plurality of sensor devices include at least two distinct senor types and are disposed in a physical environment. One or more signature events are detected in the sensor data, and one or more information items are generated to characterize the one or more signature events detected in the sensor data, independently of the sensor types of the sensor devices. A large behavior model is applied to process the one or more information items and generate a multimodal output associated with the sensor data. The multimodal output describes the signature events associated with the sensor data in one of a plurality of predefined output modalities. The multimodal output is presented according to the one of the plurality of predefined output modalities.

Abstract: A computing system can be configured to input model input that includes context data into a machine-learned model and receive model output that describes one or more semantic entities referenced by the context data. The computing system can be configured to provide data descriptive of the semantic entity or entities to the computer application(s) and receive application output(s) respectively from the computing application(s) in response to providing the data descriptive of semantic entity or entities to the computer application(s). The application output(s) received from each computer application can describe available action(s) of the corresponding computer application with respect to the semantic entity or entities. The computing system can be configured to provide at least one indicator to a user that describes the available action(s) of the corresponding computer applications with respect to the semantic entity or entities.

Abstract: Various arrangements for performing radar-based measurement of vital signs. Waveform data may be received then filtered of data indicative of static objects to obtain motion-indicative waveform data. The motion-indicative waveform data may be analyzed to determine one or more frequencies of movement present within the motion-indicative waveform data. A spectral analysis may be performed on the motion-indicative waveform data to determine a spectral-analysis state of a monitored region. The spectral-analysis state of the monitored region may be determined to match a predefined spectral-analysis state during which vital sign monitoring is permitted. One or more vital signs of a monitored user present within the monitored region may be determined and output based on analyzing the motion-indicative waveform data.

Abstract: This document describes techniques and systems for authentication management through IMU and radar. The techniques and systems use inertial sensor data from an inertial measurement unit (IMU) and/or radar data to manage authentication for a computing device. By so doing, the techniques conserve power, improve accuracy, or reduce latency relative to many common techniques and systems for computing-device authentication.

Abstract: This document describes techniques and systems for radar-based gesture-recognition with context-sensitive gating and other context-sensitive controls. Sensor data from a proximity sensor and/or a movement sensor produces a context of a user equipment. The techniques and systems enable the user equipment to recognize contexts when a radar system can be unreliable and should not be used for gesture-recognition, enabling the user equipment to automatically disable or “gate” the output from the radar system according to context. The user equipment prevents the radar system from transitioning to a high-power state to perform gesture-recognition in contexts where radar data detected by the radar system is likely due to unintentional input. By so doing, the techniques conserve power, improve accuracy, or reduce latency relative to many common techniques and systems for radar-based gesture-recognition.

Abstract: Various arrangements for performing radar-based measurement of vital signs. Waveform data may be received then filtered of data indicative of static objects to obtain motion-indicative waveform data. The motion-indicative waveform data may be analyzed to determine one or more frequencies of movement present within the motion-indicative waveform data. A spectral analysis may be performed on the motion-indicative waveform data to determine a spectral-analysis state of a monitored region. The spectral-analysis state of the monitored region may be determined to match a predefined spectral-analysis state during which vital sign monitoring is permitted. One or more vital signs of a monitored user present within the monitored region may be determined and output based on analyzing the motion-indicative waveform data.

Abstract: A computing system can be configured to input model input that includes context data into a machine-learned model and receive model output that describes one or more semantic entities referenced by the context data. The computing system can be configured to provide data descriptive of the semantic entity or entities to the computer application(s) and receive application output(s) respectively from the computing application(s) in response to providing the data descriptive of semantic entity or entities to the computer application(s). The application output(s) received from each computer application can describe available action(s) of the corresponding computer application with respect to the semantic entity or entities. The computing system can be configured to provide at least one indicator to a user that describes the available action(s) of the corresponding computer applications with respect to the semantic entity or entities.

Abstract: This document describes techniques and systems for radar-based gesture-recognition with context-sensitive gating and other context-sensitive controls. Sensor data from a proximity sensor and/or a movement sensor produces a context of a user equipment. The techniques and systems enable the user equipment to recognize contexts when a radar system can be unreliable and should not be used for gesture-recognition, enabling the user equipment to automatically disable or “gate” the output from the radar system according to context. The user equipment prevents the radar system from transitioning to a high-power state to perform gesture-recognition in contexts where radar data detected by the radar system is likely due to unintentional input. By so doing, the techniques conserve power, improve accuracy, or reduce latency relative to many common techniques and systems for radar-based gesture-recognition.

Abstract: This document describes techniques and systems for authentication management through IMU and radar. The techniques and systems use inertial sensor data from an inertial measurement unit (IMU) and/or radar data to manage authentication for a computing device. By so doing, the techniques conserve power, improve accuracy, or reduce latency relative to many common techniques and systems for computing-device authentication.

Abstract: Systems and techniques are described for robust radar-based gesture-recognition. A radar system detects radar-based gestures on behalf of application subscribers. A state machine transitions between multiple states based on inertial sensor data. A no-gating state enables the radar system to output radar-based gestures to application subscribers. The state machine also includes a soft-gating state that prevents the radar system from outputting the radar-based gestures to the application subscribers. A hard-gating state prevents the radar system from detecting radar-based gestures altogether. The techniques and systems enable the radar system to determine when not to perform gesture-recognition, enabling user equipment to automatically reconfigure the radar system to meet user demand. By so doing, the techniques conserve power, improve accuracy, or reduce latency relative to many common techniques and systems for radar-based gesture-recognition.

Abstract: A computing system can be configured to input model input that includes context data into a machine-learned model and receive model output that describes one or more semantic entities referenced by the context data. The computing system can be configured to provide data descriptive of the semantic entity or entities to the computer application(s) and receive application output(s) respectively from the computing application(s) in response to providing the data descriptive of semantic entity or entities to the computer application(s). The application output(s) received from each computer application can describe available action(s) of the corresponding computer application with respect to the semantic entity or entities. The computing system can be configured to provide at least one indicator to a user that describes the available action(s) of the corresponding computer applications with respect to the semantic entity or entities.

Abstract: Systems and techniques are described for robust radar-based gesture-recognition. A radar system detects radar-based gestures on behalf of application subscribers. A state machine transitions between multiple states based on inertial sensor data. A no-gating state enables the radar system to output radar-based gestures to application subscribers. The state machine also includes a soft-gating state that prevents the radar system from outputting the radar-based gestures to the application subscribers. A hard-gating state prevents the radar system from detecting radar-based gestures altogether. The techniques and systems enable the radar system to determine when not to perform gesture-recognition, enabling user equipment to automatically reconfigure the radar system to meet user demand. By so doing, the techniques conserve power, improve accuracy, or reduce latency relative to many common techniques and systems for radar-based gesture-recognition.

Abstract: This document describes techniques and systems that enable radar-based authentication status feedback. A radar field is used to enable an electronic device to account for the user's distal physical cues to determine and maintain an awareness of the user's location and movements around the device. This awareness allows the device to anticipate some of the user's intended interactions and provide functionality in a timely and seamless manner, such as preparing an authentication system to authenticate the user before the user touches or speaks to the device. These features also allow the device to provide visual feedback that can help the user understand that the device is aware of the user's location and movements. In some cases, the feedback is provided using visual elements presented on a display.

Abstract: A computing system can be configured to input model input that includes context data into a machine-learned model and receive model output that describes one or more semantic entities referenced by the context data. The computing system can be configured to provide data descriptive of the semantic entity or entities to the computer application(s) and receive application output(s) respectively from the computing application(s) in response to providing the data descriptive of semantic entity or entities to the computer application(s). The application output(s) received from each computer application can describe available action(s) of the corresponding computer application with respect to the semantic entity or entities. The computing system can be configured to provide at least one indicator to a user that describes the available action(s) of the corresponding computer applications with respect to the semantic entity or entities.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving, at a mobile computing device that is associated with a called user, a call from a calling computing device that is associated with a calling user; in response to receiving the call, determining, by the mobile computing device, that data associated with the called user indicates that the called user will not respond to the call; in response to determining that the called user will not respond to the call, inferring, by the mobile computing device, an informational need of the calling user; and automatically providing, from the mobile computing device to the calling computing device, information associated with the called user and that satisfies the inferred informational need of the calling user.

Abstract: This document describes techniques and systems for authentication management through IMU and radar. The techniques and systems use inertial sensor data from an inertial measurement unit (IMU) and/or radar data to manage authentication for a computing device. By so doing, the techniques conserve power, improve accuracy, or reduce latency relative to many common techniques and systems for computing-device authentication.

Abstract: Systems and methods for determining identifying semantic entities in audio signals are provided. A method can include obtaining, by a computing device comprising one or more processors and one or more memory devices, an audio signal concurrently heard by a user. The method can further include analyzing, by a machine-learned model stored on the computing device, at least a portion of the audio signal in a background of the computing device to determine one or more semantic entities. The method can further include displaying the one or more semantic entities on a display screen of the computing device.

Abstract: Systems and methods for determining identifying semantic entities in audio signals are provided. A method can include obtaining, by a computing device comprising one or more processors and one or more memory devices, an audio signal concurrently heard by a user. The method can further include analyzing, by a machine-learned model stored on the computing device, at least a portion of the audio signal in a background of the computing device to determine one or more semantic entities. The method can further include displaying the one or more semantic entities on a display screen of the computing device.