Abstract: Techniques are disclosed relating to display devices. In some embodiments, a display device includes a display system configured to display three-dimensional content to a user. The display device is configured to discover, via a network interface, one or more compute nodes operable to facilitate rendering the three-dimensional content and receive information identifying abilities of the one or more compute nodes to facilitate the rendering. Based on the received information, the display device evaluates a set of tasks to identify one or more of the tasks to offload to the one or more compute nodes for facilitating the rendering and distributes, via the network interface, the identified one or more tasks to the one or more compute nodes for processing by the one or more compute nodes.

Abstract: Techniques are disclosed herein for implementing a novel, low latency, guidance map-free video matting system, e.g., for use in extended reality (XR) platforms. The techniques may be designed to work with low resolution auxiliary inputs (e.g., binary segmentation masks) and to generate alpha mattes (e.g., alpha mattes configured to segment out any object(s) of interest, such as human hands, from a captured image) in near real-time and in a computationally efficient manner. Further, in a domain-specific setting, the system can function on a captured image stream alone, i.e., it would not require any auxiliary inputs, thereby reducing computational costs—without compromising on visual quality and user comfort. Once an alpha matte has been generated, various alpha-aware graphical processing operations may be performed on the captured images according to the generated alpha mattes (e.g.

Abstract: A head-mounted device is provided that includes one or more cameras configured to acquire a raw video feed and one or more displays configured to present a passthrough video feed to a user. Generation of the passthrough video feed can involve processing the raw video feed using an image signal processor and auxiliary compute blocks. One or more of the auxiliary compute blocks can be bypassed in response to detecting one or more failures associated with the auxiliary compute blocks. Configured and operated in this way, the head-mounted device can fall back to a more reliable passthrough video feed without having to power cycle the head-mounted device when a failure occurs.

Abstract: Implementations of the subject technology provides analyzing a recording of content. The subject technology generates metadata information based at least in part on the analyzing. The subject technology identifies, based at least in part on at least one of a user preference or a detected event, a region of interest or an object of interest in the recording of content. Based at least in part on the identified region of interest or object of interest, the subject technology generates a modified version of the recording of content. Further, the subject technology stores the modified version of the recording of content for subsequent playback on an electronic device.

Abstract: Techniques are disclosed relating to encoding recorded content for distribution to other computing devices. In various embodiments, a first computing device records content of a physical environment in which the first computing device is located, the content being deliverable to a second computing device configured to present a corresponding environment based on the recorded content and content recorded by one or more additional computing devices. The first computing device determines a pose of the first computing device within the physical environment and encodes the pose in a manifest usable to stream the content recorded by the first computing device to the second computing device. The encoded pose is usable by the second computing device to determine whether to stream the content recorded by the first computing device.

Abstract: Systems and methods for gaze-driven recording of video are described. Some implementations may include accessing gaze data captured using one or more gaze-tracking sensors; applying a temporal filter to the gaze data to obtain a smoothed gaze estimate; determining a region of interest based on the smoothed gaze estimate, wherein the region of interest identifies a subset of a field of view; accessing a frame of video; recording a portion of the frame associated with the region of interest as an enhanced frame of video, wherein the portion of the frame corresponds to a smaller field of view than the frame; and storing, transmitting, or displaying the enhanced frame of video.

Abstract: Techniques are disclosed relating to display devices. In some embodiments, a display device includes a display system configured to display three-dimensional content to a user. The display device is configured to discover, via a network interface, one or more compute nodes operable to facilitate rendering the three-dimensional content and receive information identifying abilities of the one or more compute nodes to facilitate the rendering. Based on the received information, the display device evaluates a set of tasks to identify one or more of the tasks to offload to the one or more compute nodes for facilitating the rendering and distributes, via the network interface, the identified one or more tasks to the one or more compute nodes for processing by the one or more compute nodes.

Abstract: Techniques are disclosed relating to encoding recorded content for distribution to other computing devices. In some embodiments, a first computing device creates recorded content for transmission to a second computing device configured to present the recorded content. To encode the recorded content, the first computing device detects, via a network interface of the first computing device, one or more computing nodes available to encode the recorded content in one or more formats supported by the second computing device. The first computing device offloads the recorded content via the network interface to the one or more computing nodes for encoding in the one or more formats. In some embodiments, the second computing device receives a request from a user to stream content recorded by a first computing device and requests the content in a first format being encoded by a computing node assisting the first computing device.

Abstract: One implementation forms a composited stream of computer-generated reality (CGR) content using multiple data streams related to a CGR experience to facilitate recording or streaming. A media compositor obtains a first data stream of rendered frames and a second data stream of additional data. The rendered frame content (e.g., 3D models) represents real and virtual content rendered during a CGR experience at a plurality of instants in time. The additional data of the second data stream relates to the CGR experience, for example, relating to audio, audio sources, metadata identifying detected attributes of the CGR experience, image data, data from other devices involved in the CGR experience, etc. The media compositor forms a composited stream that aligns the rendered frame content with the additional data for the plurality of instants in time, for example, by forming time-stamped, n-dimensional datasets (e.g., images) corresponding to individual instants in time.

Abstract: Implementations of the subject technology provides analyzing a recording of content. The subject technology generates metadata information based at least in part on the analyzing. The subject technology identifies, based at least in part on at least one of a user preference or a detected event, a region of interest or an object of interest in the recording of content. Based at least in part on the identified region of interest or object of interest, the subject technology generates a modified version of the recording of content. Further, the subject technology stores the modified version of the recording of content for subsequent playback on an electronic device.

Abstract: A method is provided that includes receiving content data captured by a sensor and receiving a context signal representing a user context. The received content data is scaled using a trained model, wherein the context signal is an input to the trained model, and the scaled content data is provided for presentation to a user.

Abstract: Implementations of the subject technology provide determining an operating mode of an electronic device based at least in part on whether the electronic device is communicatively coupled to an associated base device. Based on the determined operating mode, the subject technology identifies a set of input modalities for initiating a recording of content within a field of view of the electronic device. The subject technology monitors sensor information generated by at least one sensor included in, or communicatively coupled to, the electronic device. Further, the subject technology initiates the recording of content within the field of view of the electronic device when the monitored sensor information indicates that at least one of the identified set of input modalities has been triggered.

Abstract: Aspects of the subject technology relate to encoding of video frames having content with a variable resolution that varies within the video frame. Aspects of the subject technology can provide an efficient encoding by using smaller macroblocks for lower resolution content within the video frame, and larger macroblocks for higher resolution content within the video frame. An encoder may be provided with resolution information for the content of the video frame, which can be used by the encoder to determine macroblock sizes, macroblock divisions, and/or prediction modes for the encoding of the video frame.

Abstract: Implementations of the subject technology provide systems and methods for recording an extended reality experience in a way that allows the experience to be played back at a later time from a different viewpoint or perspective. This allows computer-generated content that was rendered for display to a user during the recording, to be re-rendered during playback at the correct time and location in the recording, but from a different perspective. In order to facilitate this type of viewer-centric playback, the recording includes a computer-generated content track that references resources for re-rendering the computer-generated content at each point in time in the recording.

Abstract: Techniques are disclosed relating to encoding recorded content for distribution to other computing devices. In some embodiments, a first computing device creates recorded content for transmission to a second computing device configured to present the recorded content. To encode the recorded content, the first computing device detects, via a network interface of the first computing device, one or more computing nodes available to encode the recorded content in one or more formats supported by the second computing device. The first computing device offloads the recorded content via the network interface to the one or more computing nodes for encoding in the one or more formats. In some embodiments, the second computing device receives a request from a user to stream content recorded by a first computing device and requests the content in a first format being encoded by a computing node assisting the first computing device.

Abstract: Implementations of the subject technology provide determining an operating mode of an electronic device based at least in part on whether the electronic device is communicatively coupled to an associated base device. Based on the determined operating mode, the subject technology identifies a set of input modalities for initiating a recording of content within a field of view of the electronic device. The subject technology monitors sensor information generated by at least one sensor included in, or communicatively coupled to, the electronic device. Further, the subject technology initiates the recording of content within the field of view of the electronic device when the monitored sensor information indicates that at least one of the identified set of input modalities has been triggered.

Abstract: Techniques are disclosed relating to encoding recorded content for distribution to other computing devices. In various embodiments, a first computing device records content of a physical environment in which the first computing device is located, the content being deliverable to a second computing device configured to present a corresponding environment based on the recorded content and content recorded by one or more additional computing devices. The first computing device determines a pose of the first computing device within the physical environment and encodes the pose in a manifest usable to stream the content recorded by the first computing device to the second computing device. The encoded pose is usable by the second computing device to determine whether to stream the content recorded by the first computing device.

Abstract: Aspects of the subject technology provide electronic devices that operate, in part, based on enrolled user characteristics, and that can be operated by a guest user that has not been enrolled. For example, upon determining that a current user of an electronic device storing a first physical model of a primary user is a guest user different from the primary user, the electronic device may obtain initial physical characteristic data for the guest user and generate a guest physical model of the guest user based on the initial physical characteristic data. In one or more implementations, the electronic device may operate based on guest user inputs and the guest physical model of the guest user, while updating the guest physical model based on the guest user inputs.

Abstract: Techniques are disclosed relating to encoding recorded content for distribution to other computing devices. In various embodiments, a first computing device records content of a physical environment in which the first computing device is located, the content being deliverable to a second computing device configured to present a corresponding environment based on the recorded content and content recorded by one or more additional computing devices. The first computing device determines a location of the first computing device within the physical environment and encodes the location in a manifest usable to stream the content recorded by the first computing device to the second computing device. The encoded location is usable by the second computing device to determine whether to stream the content recorded by the first computing device.

Abstract: Implementations of the subject technology provide systems and methods for recording an extended reality experience in a way that allows the experience to be played back at a later time from a different viewpoint or perspective. This allows computer-generated content that was rendered for display to a user during the recording, to be re-rendered during playback at the correct time and location in the recording, but from a different perspective. In order to facilitate this type of viewer-centric playback, the recording includes a computer-generated content track that references resources for re-rendering the computer-generated content at each point in time in the recording.