Abstract: A method including identifying a plurality of entities associated with a location within a physical environment, generating a representation for a subset of the plurality of entities based on a feature of the entities included in the subset, and causing a user interface, including the representation, to be rendered on a display of a wearable device.

Abstract: Implementations relate to receiving natural language (NL) input associated with a client device; processing, using a first large language model (LLM), first LLM input to generate corresponding first LLM output, the first LLM input including the NL input; identifying, based on the corresponding first LLM output, a dictation portion of the NL input and an instruction portion of the NL input, where the instruction portion includes one or more instructions for transcription of the dictation portion; processing, using the first LLM or a second LLM, second LLM input to generate corresponding second LLM output, the second LLM input including the dictation portion and the instruction portion; determining, based on the corresponding second LLM output, a transcription of the dictation portion responsive to the one or more instructions for transcription of the dictation portion; and causing the transcription of the dictation portion to be rendered at the client device.

Abstract: According to at least one implementation, a method includes obtaining a query at a wearable device and identifying context associated with the query, the context comprising a set of images captured by the wearable device. The method further includes generating at least one search from the query and the context and obtaining at least one result to the at least one search. The method also generates a response to the query based on at least one result and provides the response to a wearable device user.

Abstract: Systems and methods are disclosed that address the need for adaptive exposure within high dynamic range (HDR) images. Solutions can leverage recent advances in the use of virtual reality (VR) headsets and Augmented Reality (AR) displays equipped with infrared (IR) eye tracking devices. A gaze vector determined by the eye tracking device identifies one or more fixation points on the image that corresponds to an area where there exists a faulty exposure. The exposure around the fixation point can be adaptively corrected using image processing techniques. Using spatial adaptive exposure, the resulting image, a type of foveated image, can be rendered on a low dynamic range (LDR) display with sufficient detail.

Abstract: A method of generating a virtual microphone array according including identifying a plurality of microphones, identifying a relative position in space of each of the plurality of microphones, generating a virtual microphone array based on the plurality of microphones and the relative position in space of each of the plurality of microphones, sensing audio at each of the plurality of microphones, and generating an audio signal of the virtual microphone array based on the sensed audio.

Abstract: According to an aspect, a method for distributed sound/image recognition using a wearable device includes receiving, via at least one sensor device, sensor data, and detecting, by a classifier of the wearable device, whether or not the sensor data includes an object of interest. The classifier configured to execute a first machine learning (ML) model. The method includes transmitting, via a wireless connection, the sensor data to a computing device in response to the object of interest being detected within the sensor data, where the sensor data is configured to be used by a second ML model on the computing device or a server computer for further sound/image classification.

Abstract: Methods and devices are provided to allow for the transfer of a display of a visual representation between a head mounted device and a computing device during the display of a video. A video is displayed on a computing device display of a computing device, a visual representation of a speech for an audio component of the video is received, the visual representation is displayed on the computing device display, and the display of the visual representation is transferred to the head mounted device to display on a head mounted device display upon determining that a head mounted device is in use.

Abstract: Described techniques may be utilized to receive a transcription stream including transcribed text that has been transcribed from speech, and to receive a summary request for a summary to be provided on a display of a device. Extracted text may be identified from the transcribed text and in response to the summary request. The extracted text may be processed using a summarization machine learning (ML) model to obtain a summary of the extracted text, and the summary may be displayed on the display of the device. When an image is captured, an augmented summary may be generated that includes the image together with a visual indication of one or more of an emotion, an entity, or an intent associated with the image, the summary, or the extracted text.

Abstract: Systems and methods are disclosed that address the need for adaptive exposure within high dynamic range (HDR) images. Solutions can leverage recent advances in the use of virtual reality (VR) headsets and Augmented Reality (AR) displays equipped with infrared (IR) eye tracking devices. A gaze vector determined by the eye tracking device identifies one or more fixation points on the image that corresponds to an area where there exists a faulty exposure. The exposure around the fixation point can be adaptively corrected using image processing techniques. Using spatial adaptive exposure, the resulting image, a type of foveated image, can be rendered on a low dynamic range (LDR) display with sufficient detail.

Abstract: Described techniques may be utilized to process transcribed text of a transcription stream (302) in an incremental fashion. For example, a first summary of first transcribed text may be generated, without requiring knowledge or receipt of second or subsequent transcribed text, and without requiring detection of a detected or manual summarization trigger. Similarly, a second summary of second transcribed text may be generated without requiring knowledge or receipt of third transcribed text. Accordingly, a summary stream (310) of summarized text may be provided in a stable fashion, without overwriting previously generated summaries as new summaries are generated.

Abstract: Described techniques including processing input text at a content type classifier machine learning (ML) model to obtain a content type of the input text. The input text and the content type may be processed at a content extractor ML model to obtain extracted content from the input text. The input text, the content type, and the extracted content may be processed at a summarizer ML model to obtain a summary of the input text.

Abstract: Described techniques may be utilized to process transcribed text of a transcription stream using a compression ratio machine learning (ML) model to determine at least two compression ratios. The transcribed text may then be processed by a summarization ML model using the at least two compression ratios to obtain a summary stream that includes first summarized text having a first compression ratio of the at least two compression ratios, relative to a first corresponding portion of the transcribed text, and second summarized text having a second compression ratio of the at least two compression ratios, relative to a second corresponding portion of the transcribed text. The transcribed text may also be summarized by the summarization ML model based on a complexity score determined by a complexity ML model.

Abstract: According to at least one implementation, a method includes obtaining a query at a wearable device and identifying context associated with the query, the context comprising a set of images captured by the wearable device. The method further includes generating at least one search from the query and the context and obtaining at least one result to the at least one search. The method also generates a response to the query based on at least one result and provides the response to a wearable device user.

Abstract: A method can include receiving a message associated with a user account; determining a priority based on content of the message; determining a first status associated with the user account; based on the priority and the first status, withholding notification of the message; determining a second status associated with the user account; and based on the priority and the second status, presenting a summary of the message.

Abstract: Smart devices can be configured to collect and share various forms of context data about where a user is located (e.g., location), what a user will be doing (e.g., schedule), and what a user is currently doing (e.g., activity). This context data may be combined with fingerprint data (e.g., biometrics) to help identify the fingerprint data. For example, a location of a user may help associated speech detected at that location with the user. These associations may be stored in a mapping database that can be updated over time to reduce ambiguities in identification. The mappings in the database may be used to train a machine learning model to recognize fingerprints as identities, which may be useful in applications, such as speaker identification.

Abstract: Described techniques may be utilized to process transcribed text of a transcription stream using a summarization ML model to obtain a summary stream. In this way, a live conversation or other live speech may be provided to a user in real-time, using a display of a head-mounted device (HMD) or other suitable device. A user interface may be provided with a toggle or other stream selector for selecting either or both of the transcription stream or the summary stream at a given point in time. Actionable items may be identified within either or both of the transcription stream or the summary stream, and a scroll bar or other action selector may be provided in the user interface to execute corresponding actions.

Abstract: Aspects of the disclosure provide a hybrid smartwatch that incorporates digital technology with an analog timepiece in a wristwatch form factor. A digital display layer of a non-emissive material is configured to present notices, data, content and other information. An analog display layer includes one or more hands of the timepiece, and overlies the digital display layer. The hands may be controlled by a processor through micro-stepper motors or other actuators. Physical motion of the hands is simultaneously coupled with arrangement of content or other elements on the display layer, which results in numerous types of hybrid visualizations. This includes temporal presentations using hourly, daily, monthly or other time scales. Shape-shifting of the watch hands between 2D and 1D arrangements can linearly focus on certain information. And various content-aware layouts can highlight, bracket, occlude or otherwise emphasize or deemphasize displayed information.

Abstract: Aspects of the technology provide a symbiotic graphical display on a client device such as a smartwatch. The system includes at least one emissive display element and at least one non-emissive display element. The display elements are arrayed in layers or other configurations such that content or other information is concurrently aligned across the respective display surfaces of the different elements. A first set of content is rendered using the non-emissive display element while a second set of content is rendered using the emissive display element. Depending on characteristics or features of a given content item, that item may be rendered by one or both of the display elements. Certain content may be transitioned from the emissive display element to the non-emissive display element according to a time threshold or other criteria.

Abstract: The present disclosure provides computer-implemented methods, systems, and devices for capturing spatial sound for an environment. A computing system captures, using two or more microphones, audio data from an environment around a mobile device. The computing system analyzes the audio data to identify a plurality of sound sources in the environment around the mobile device based on the audio data. The computing system determines, based on characteristics of the audio data and data produced by one or more movement sensors, an estimated location for each respective sound source in the plurality of sound sources. The computing system generates a spatial sound recording of the audio data based, at least in part, on the estimated location of each respective sound source in the plurality of sound sources.

Abstract: A user can interact with sounds and speech in an environment using an augmented reality device. The augmented reality device can be configured to identify objects in the environment and display messages beside the object that are related to sounds produced by the object. For example, the messages may include sound statistics, transcripts of speech, and/or sound detection events. The disclosed approach enables a user to interact with these messages using a gaze and a gesture.