Abstract: In one implementation, a camera rig comprises: a first array of image sensors arranged in a planar configuration, wherein the first array of image sensors is provided to capture a first image stream from a first perspective of a physical environment; a second array of image sensors arranged in a non-planar configuration, wherein the second array of image sensors is provided to capture a second image stream from a second perspective of the physical environment different from the first perspective; a buffer provided to store the first and second image streams; and an image processing engine provided to generate a 3D reconstruction of the physical environment based on the first and second image streams.

Abstract: A head-mounted device is provided that includes a variety of subsystems for generating extended reality content, displaying the extended reality content, and recording the extended reality content. The device may include a graphics rendering pipeline configured to render virtual content, tracking sensors configured to obtain user tracking information, a virtual content compositor configured to composite virtual frames based on the virtual content and the user tracking information, cameras configured to capture a video feed, a media merging compositor configured to overlay the composited virtual frames and the video feed, and a recording pipeline configured to record parameters, metadata, raw content, and/or adjusted content in an extended reality recording file. The extended reality recording file may have multiple discrete portions that may each be individually edited. The extended reality recording file may be used to present a replay on the head-mounted device and/or may be exported to an external device.

Abstract: The subject technology provides a framework for privacy-protecting mixed reality and/or virtual reality recordings and/or for providing mixed and/or virtual reality video streams for display on external devices. A first process on an electronic device can receive, from a second process on the electronic device, a frame of a video stream. In some aspects, the frame includes an indicator of privacy-sensitive content The first process can obfuscate at least a portion of the frame based at least in part on the indicator of the privacy-sensitive content. The first process can provide the obfuscated at least the portion of the frame for at least one of: inclusion in a recorded video or for on stream display another electronic device.

Abstract: A device may include a processor configured to determine a first position of a physical camera in a physical environment, transmit the first position to a second device, obtain metadata of the physical camera, and cause a modification of a virtual camera of the second device based on the metadata. The processor is also configured to obtain, from the second device, at least one virtual image frame captured by the virtual camera placed at a second position in a computer-generated environment, and to obtain, from the physical camera, at least one image frame captured from the physical environment. The processor is also configured to generate a composite image frame including at least a portion of at least one virtual image frame composited with at least a portion of the at least one image frame, and display the composite image frame.

Abstract: Cross-platform sharing of displayed content is provided. In one or more implementations, cross-platform sharing of display content may include displaying a first rendering of display content with a first display of a first device, and, responsive to a request to capture and/or share the displayed content, generating a second rendering of the display content, the second rendering different from the first rendering. The second rendering may be different from the first rendering in such a way that, when the second rendering is displayed by a second display of a second device having a different form factor and/or display technology, the displayed second rendering accurately represents the perceived view of the user of the first device.

Abstract: An encoder encodes three-dimensional (3D) volumetric content, such as immersive media, using video encoded attribute patch images packed into a 2D atlas to communicate the attribute values for the 3D volumetric content. The encoder also uses mesh-encoded sub-meshes to communicate geometry information for portions of the 3D object or scene corresponding to the attribute patch images packed into the 2D atlas. The encoder applies decimation operations to the sub-meshes to simplify the sub-meshes before mesh encoding the sub-meshes. A distortion analysis is performed to bound the level to which the sub-meshes are simplified at the encoder. Mesh simplification at the encoder reduces the number of vertices and edges included in the sub-meshes which simplifies rendering at a decoder receiving the encoded 3D volumetric content.

Abstract: A set of planes is determined based on the vertices of a three-dimensional scene. The set of planes is determined independent of any particular orientation with respect to a reference viewpoint of the three-dimensional scene. The vertices are associated with the set of planes and respective textures are associated with the set of planes based on the associated vertices.

Abstract: One or more computing devices implement a mesh analysis for evaluating meshes to be rendered when rendering immersive content. The mesh analysis identifies objects in a three-dimensional scene and determines geometrical complexity values for the objects. Objects with similar geometrical complexities are grouped into areas and a mesh vertices budget is determined for the respective areas. Metadata indicating the area definitions and corresponding mesh vertices budgets are generated. The metadata may be uploaded to a server to simplify meshes in the scene prior to streaming to a client, or the metadata may be provided to a client for use in simplifying the meshes as part of rendering the scene.

Abstract: Aspects of the subject technology relate to encoding of video frames having content with a variable resolution that varies within the video frame. Aspects of the subject technology can provide an efficient encoding by using smaller macroblocks for lower resolution content within the video frame, and larger macroblocks for higher resolution content within the video frame. An encoder may be provided with resolution information for the content of the video frame, which can be used by the encoder to determine macroblock sizes, macroblock divisions, and/or prediction modes for the encoding of the video frame.

Abstract: A system comprises an encoder configured to compress and encode data for three-dimensional volumetric content. The encoder also is configured to segment the three-dimensional volumetric content based on viewing areas, wherein different ones of the viewing areas correspond to visible portions of the volumetric content. The system may provide metadata to a client device to support viewport adaptive rendering of the three-dimensional volumetric content or may adaptively stream portions of the three-dimensional volumetric content to a rending device based on viewing areas of the three-dimensional volumetric content that are to be rendered at the rendering device.