Abstract: A device for positional synchronization of virtual and physical cameras may include a processor configured to determine a first position of a physical camera relative to another electronic device in a physical environment. The processor may be configured to initiate positioning of a virtual camera in a second position within a computer-generated environment, wherein the second position relative to a representation of the person in the computer-generated environment coincides with the first position. The processor may be configured to receive an image frame captured by the physical camera and a virtual image frame generated by the virtual camera. The processor may be configured to generate a computer-generated reality image frame that includes at least a portion of the image frame composited with at least a portion of the virtual image frame.

Abstract: Techniques are disclosed relating to encoding recorded content for distribution to other computing devices. In some embodiments, a first computing device creates recorded content for transmission to a second computing device configured to present the recorded content. To encode the recorded content, the first computing device detects, via a network interface of the first computing device, one or more computing nodes available to encode the recorded content in one or more formats supported by the second computing device. The first computing device offloads the recorded content via the network interface to the one or more computing nodes for encoding in the one or more formats. In some embodiments, the second computing device receives a request from a user to stream content recorded by a first computing device and requests the content in a first format being encoded by a computing node assisting the first computing device.

Abstract: Implementations of the subject technology provide systems and methods for recording an extended reality experience in a way that allows the experience to be played back at a later time from a different viewpoint or perspective. This allows computer-generated content that was rendered for display to a user during the recording, to be re-rendered during playback at the correct time and location in the recording, but from a different perspective. In order to facilitate this type of viewer-centric playback, the recording includes a computer-generated content track that references resources for re-rendering the computer-generated content at each point in time in the recording.

Abstract: Various implementations disclosed herein include devices, systems, and methods that generates a stereo view of a three-dimensional (3D) environment by rendering a first eye view and reusing pixels from the first eye view for a second eye view based on a disparity indicative characteristic (e.g., identifying regions of pixel difference/similarity in the views). For example, an example process may include rendering a first view for a first viewpoint (e.g., a left eye view) of a three-dimensional (3D) environment. The process may further include estimating differences between the first view and a second view for a second viewpoint (e.g., a right eye view) of the 3D environment. The process may further include rendering the second view for the second viewpoint based on the estimated differences.

Abstract: A set of planes is determined based on the vertices of a three-dimensional scene. The set of planes is determined independent of any particular orientation with respect to a reference viewpoint of the three-dimensional scene. The vertices are associated with the set of planes and respective textures are associated with the set of planes based on the associated vertices.

Abstract: Implementations of the subject technology provides analyzing a recording of content. The subject technology generates metadata information based at least in part on the analyzing. The subject technology identifies, based at least in part on at least one of a user preference or a detected event, a region of interest or an object of interest in the recording of content. Based at least in part on the identified region of interest or object of interest, the subject technology generates a modified version of the recording of content. Further, the subject technology stores the modified version of the recording of content for subsequent playback on an electronic device.

Abstract: Aspects of the subject technology relate to encoding of video frames having content with a variable resolution that varies within the video frame. Aspects of the subject technology can provide an efficient encoding by using smaller macroblocks for lower resolution content within the video frame, and larger macroblocks for higher resolution content within the video frame. An encoder may be provided with resolution information for the content of the video frame, which can be used by the encoder to determine macroblock sizes, macroblock divisions, and/or prediction modes for the encoding of the video frame.

Abstract: A decoding computing device determines a focus area of a view of volumetric visual content, such as a three-dimensional (3D) object or scene, and indicates the focus area in a request for compressed volumetric visual content. A server provides a depth map and an attribute atlas that includes attribute and/or texture information for the focus area that is signaled at a higher resolution than other portions of the volumetric visual content. The decoding device also applies one or more mesh simplification techniques, wherein a higher resolution mesh is used for the focus area. The textures/attributes are projected onto the mesh representations, wherein the mesh representations and projected textures are used to reconstruct the volumetric visual content. Hole filling techniques are then applied, wherein a more sophisticated hole filling technique is used in the focus area.

Abstract: A decoding computing device receives a bit stream for compressed 3D volumetric content. The bit stream includes video encoded image frames comprising packed attribute patch images and depth maps for the 3D volumetric content. Instead of generating a mesh having a vertex for each depth value signaled in the depth map, the decoder performs a real-time mesh simplification process to reduce a resolution of the mesh, such that the mesh resolution is reduced without exceeding an error threshold, which may be dynamically determined. Additionally, the decoder may perform a re-meshing of particular regions of the mesh for the 3D volumetric content to avoid cracks or gaps.

Abstract: Implementations of the subject technology provide determining an operating mode of an electronic device based at least in part on whether the electronic device is communicatively coupled to an associated base device. Based on the determined operating mode, the subject technology identifies a set of input modalities for initiating a recording of content within a field of view of the electronic device. The subject technology monitors sensor information generated by at least one sensor included in, or communicatively coupled to, the electronic device. Further, the subject technology initiates the recording of content within the field of view of the electronic device when the monitored sensor information indicates that at least one of the identified set of input modalities has been triggered.

Abstract: In one implementation, a camera rig comprises: a first array of image sensors arranged in a planar configuration, wherein the first array of image sensors is provided to capture a first image stream from a first perspective of a physical environment; a second array of image sensors arranged in a non-planar configuration, wherein the second array of image sensors is provided to capture a second image stream from a second perspective of the physical environment different from the first perspective; a buffer provided to store the first and second image streams; and an image processing engine provided to generate a 3D reconstruction of the physical environment based on the first and second image streams.

Abstract: Implementations of the subject technology provides analyzing a recording of content within a field of view of a device, the analyzing including recognition of a set of objects included in the content. The subject technology identifies a subset of the set of objects that are indicated as corresponding to protected content. The subject technology generates a modified version of the recording that obfuscates or filters the subset of the set of objects. Additionally, the subject technology provides the modified version of the recording to a host application for playback.

Abstract: A head-mounted device is provided that includes a variety of subsystems for generating extended reality content, displaying the extended reality content, and recording the extended reality content. The device may include a graphics rendering pipeline configured to render virtual content, tracking sensors configured to obtain user tracking information, a virtual content compositor configured to composite virtual frames based on the virtual content and the user tracking information, cameras configured to capture a video feed, a media merging compositor configured to overlay the composited virtual frames and the video feed, and a recording pipeline configured to record parameters, metadata, raw content, and/or adjusted content in an extended reality recording file. The extended reality recording file may have multiple discrete portions that may each be individually edited. The extended reality recording file may be used to present a replay on the head-mounted device and/or may be exported to an external device.

Abstract: A device may include a processor configured to determine a first position of a physical camera in a physical environment, transmit the first position to a second device, obtain metadata of the physical camera, and cause a modification of a virtual camera of the second device based on the metadata. The processor is also configured to obtain, from the second device, at least one virtual image frame captured by the virtual camera placed at a second position in a computer-generated environment, and to obtain, from the physical camera, at least one image frame captured from the physical environment. The processor is also configured to generate a composite image frame including at least a portion of at least one virtual image frame composited with at least a portion of the at least one image frame, and display the composite image frame.

Abstract: The subject technology provides a framework for privacy-protecting mixed reality and/or virtual reality recordings and/or for providing mixed and/or virtual reality video streams for display on external devices. A first process on an electronic device can receive, from a second process on the electronic device, a frame of a video stream. In some aspects, the frame includes an indicator of privacy-sensitive content The first process can obfuscate at least a portion of the frame based at least in part on the indicator of the privacy-sensitive content. The first process can provide the obfuscated at least the portion of the frame for at least one of: inclusion in a recorded video or for on stream display another electronic device.

Abstract: Aspects of the subject technology may provide localized environmental input sensing for electronic devices. Localized environmental input sensing may include obtaining local lighting condition estimates for one or more local portions of a physical environment. The local lighting conditions may include ambient light levels and/or a light direction of a directional light source. The one or more local portions of the physical environment may be determined based on an identification of a salient region of the physical environment to a user of an electronic device.

Abstract: Cross-platform sharing of displayed content is provided. In one or more implementations, cross-platform sharing of display content may include displaying a first rendering of display content with a first display of a first device, and, responsive to a request to capture and/or share the displayed content, generating a second rendering of the display content, the second rendering different from the first rendering. The second rendering may be different from the first rendering in such a way that, when the second rendering is displayed by a second display of a second device having a different form factor and/or display technology, the displayed second rendering accurately represents the perceived view of the user of the first device.

Abstract: An encoder encodes three-dimensional (3D) volumetric content, such as immersive media, using video encoded attribute patch images packed into a 2D atlas to communicate the attribute values for the 3D volumetric content. The encoder also uses mesh-encoded sub-meshes to communicate geometry information for portions of the 3D object or scene corresponding to the attribute patch images packed into the 2D atlas. The encoder applies decimation operations to the sub-meshes to simplify the sub-meshes before mesh encoding the sub-meshes. A distortion analysis is performed to bound the level to which the sub-meshes are simplified at the encoder. Mesh simplification at the encoder reduces the number of vertices and edges included in the sub-meshes which simplifies rendering at a decoder receiving the encoded 3D volumetric content.

Abstract: A set of planes is determined based on the vertices of a three-dimensional scene. The set of planes is determined independent of any particular orientation with respect to a reference viewpoint of the three-dimensional scene. The vertices are associated with the set of planes and respective textures are associated with the set of planes based on the associated vertices.

Abstract: Implementations of the subject technology provides analyzing a recording of content. The subject technology generates metadata information based at least in part on the analyzing. The subject technology identifies, based at least in part on at least one of a user preference or a detected event, a region of interest or an object of interest in the recording of content. Based at least in part on the identified region of interest or object of interest, the subject technology generates a modified version of the recording of content. Further, the subject technology stores the modified version of the recording of content for subsequent playback on an electronic device.