Abstract: A method for a soft acceptance of a hotword receives audio data characterizing a soft hotword event detected by a hotword detector in streaming audio captured by a user device. The method also processes the audio data to determine that the audio data corresponds to a query specifying an action to perform on the user device. Without triggering performance of the action on the user device or the other device, the method provides a notification for output from the user device where the notification prompts a user associated with the user device to provide an affirmative input indication in order to trigger performance of the action on the user device or the other device and, when the user fails to provide the affirmative input indication, instructs the user device or the other device to not perform the action specified by the query.

Abstract: The various embodiments described herein include methods, devices, and systems for power-management on camera devices. In one aspect, a method is performed at a camera device having memory, one or more processors, and an image sensor. The method includes: (1) while a wireless communication component of the camera device is deactivated: (a) capturing a plurality of images containing a motion event; (b) characterizing the motion event; and (c) determining, based on the characterization of the motion event, whether to send video data to a remote computing system; and (2) in accordance with a determination to send video data to the remote computing system: (i) activating the wireless communication component of the camera device; (ii) establishing a wireless connection to the remote computing system via the wireless communication component; and (iii) sending video information to the remote computing system via the established wireless connection.

Abstract: A method includes receiving a sequence of input audio frames and processing each corresponding input audio frame to determine a language ID event that indicates a predicted language. The method also includes obtaining speech recognition events each including a respective speech recognition result determined by a first language pack. Based on determining that the utterance includes a language switch from the first language to a second language, the method also includes loading a second language pack onto the client device and rewinding the input audio data buffered by an audio buffer to a time of the corresponding input audio frame associated with the language ID event that first indicated the second language as the predicted language. The method also includes emitting a first transcription and processing, using the second language pack loaded onto the client device, the rewound buffered audio data to generate a second transcription.

Abstract: A method includes training a client machine learning (ML) model on client training data at a client device. While training the client ML model, the method also includes obtaining, from a server, server model weights of a server ML model trained on server training data, the server training data different that the client training data. While training the client ML model, the method also includes: transmitting, to the server, client model weights of the client ML model; updating the client ML model using the server model weights; obtaining, from the server, updated server model weights of the server ML model, the updated server model weights updated based on the transmitted client model weights; and further updating the client ML model using the updated server model weights.

Abstract: A media application receives, from a server, an identification of a first composition type from a set of compositions to apply to an initial image captured with a user device. Responsive to one or more people being detected in the initial image, the media application generates a modified image, where the one or more people are removed from the initial image to obtain the modified image. The media application scores at least one candidate position within the modified image based on corresponding composition rules for the first composition type. The media application provides a graphical guide on a viewfinder of the user device to guide a user to capture a final image, wherein the graphical guide indicates a recommended position for the one or more people in the final image.

Abstract: A method includes receiving, from a client device, a client machine learning (ML) model and obtaining a set of training data including a plurality of training samples. The client ML model is trained locally on the client device. For each respective training sample in the plurality of training samples, the method also includes determining, using the respective training sample, a first loss of the client ML model; determining, using the respective training sample, a second loss of a server machine learning (ML) model; and determining a respective score based on the first loss and the second loss. The method also includes selecting, based on each respective score of each respective training sample in the plurality of training samples, a subset of training samples from the plurality of training samples and training the server ML model using the subset of training samples.

Abstract: Various configurations of projectors and cameras are disclosed that use shared wafer level optics, in which optical elements, e.g., microlenses, of a projector are fabricated on the same wafer as optical elements, e.g., microlenses, of a camera. Projectors and cameras can be mounted together on a mixed reality headset, e.g., an AR/VR headset, for example, as a feature of smart glasses. Some projectors and/or cameras can be co-located in the arm or temple of the glasses. Some projectors and/or cameras can be co-located near a center point of the frame of the glasses. Use of shared wafer-level optics provides a compact and efficient solution for simultaneously guiding light leaving a projector and light entering a camera.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating network outputs using insertion operations.

Abstract: Systems and methods are provided for a personalized entity repository. For example, a computing device comprises a personalized entity repository having fixed sets of entities from an entity repository stored at a server, a processor, and memory storing instructions that cause the computing device to identify fixed sets of entities that are relevant to a user based on context associated with the computing device, rank the fixed sets by relevancy, and update the personalized entity repository using selected sets determined based on the rank and on set usage parameters applicable to the user. In another example, a method includes generating fixed sets of entities from an entity repository, including location-based sets and topic-based sets, and providing a subset of the fixed sets to a client, the client requesting the subset based on the client's location and on items identified in content generated for display on the client.

Abstract: Role switch between wirelessly paired master/slave devices may be performed without perceived glitches in audio. The devices negotiate an anchor point, such as a point in time or a point in relation to other events, for performing the role switch. To prepare for the role switch prior to the anchor point, the devices communicate a variety of information, such as information for communicating with the host device after the role switch and information for bit processing after the role switch. The slave device may use such information to serve in the master role, without the host knowing that a role switch occurred.

Abstract: Methods and systems for rendering distinct components of a combined user interface (UI) are described herein. A processing device receives a user request for a media item to be presented via a combined UI including one or more video content components and one or more UI control components. The processing device generates a rendering of the video content components and transmits a first stream including the rendering of the video content components to an endpoint device, wherein the first stream is encoded to a first resolution and wherein the rendering of the video content components is to be merged with a rendering of the UI control components into the combined UI at the endpoint device, the combined UI having a second resolution.

Abstract: Methods, systems, and apparatus for receiving a command for controlling a robot, the command referencing an object, receiving sensor data for a portion of an environment of the robot, identifying, from the sensor data, a gesture of a human that indicates a spatial region located outside of the portion of the environment described by the sensor data, searching map data for the object, determining, based at least on searching the map data for the object referenced in the command, that the object referenced in the command is present in the spatial region, and in response to determining that the object referenced in the command is present in the spatial region, controlling the robot to perform an action with respect to the object referenced in the command.

Abstract: This disclosure relates to using probabilistic data structures to enable systems to detect fraud while preserving user privacy. In one aspect, a method includes obtaining a set of frequency filters. Each frequency filter defines a maximum event count for a specified event type over a specified time duration and corresponds to a respective content provider. A subset of the frequency filters are identified as triggered frequency filters for which an actual event count for the specified event type corresponding to the frequency filter exceeds the maximum event count defined by the frequency filter during a time period corresponding to a specified time duration for the frequency filter. A probabilistic data structure that represents at least a portion of the frequency filters in the subset of frequency filters is generated. A request for content is sent to multiple content providers. The request for content includes the probabilistic data structure.