Abstract: Systems and techniques are described herein for processing frames. The systems and techniques can be implemented by various types of systems, such as by an extended reality (XR) system or device. In some cases, a process can include obtaining feature information associated with a feature in a current frame, wherein the feature information is based on one or more previous frames; determining an estimated pose of the apparatus associated with the current frame; obtaining a distance associated with the feature in the current frame; and determining an estimated scale of the feature in the current frame based on the feature information associated with the feature, the estimated pose, and the distance associated with the feature.

Abstract: Several methods and systems for chroma residual data prediction for encoding blocks corresponding to video data are disclosed. In an embodiment, at least one coefficient correlating reconstructed luma residual samples and corresponding reconstructed chroma residual samples is computed for one or more encoded blocks of video data. Predicted chroma residual samples are generated for encoding a block of video data based on corresponding reconstructed luma residual samples and the at least one coefficient.

Abstract: Techniques and systems are provided for capturing images by a first device. For instance, a process can include obtaining a first image from a first camera, the first image being associated with a first capture time based on a first clock; mapping the first capture time to a second clock to obtain a second capture time, the second capture time based on a second clock, and wherein the second clock is based on a network time; associating the second capture time with the first image; obtaining a second image from a second camera of another device, the second image including a third capture time based on the second clock; determining phase delta information based on a time difference between the second capture time and the third capture time; and outputting the phase delta information to adjust a next capture time of at least one of the first or second camera.

Abstract: Systems and techniques are described herein for generating models of faces. For instance, a method for generating models of faces is provided. The method may include obtaining one or more images of one or both eyes of a face of a user; obtaining audio data based on utterances of the user; and generating, using a machine-learning model, a three-dimensional model of the face of the user based on the one or more images and the audio data.

Abstract: Systems and techniques are described herein for using head mounted display (HMD) hinge motion to calibrate the head mounted device. For example, an device (or component of the device) can obtain, via one or more sensors, sensor measurements during rotation of a display of the HMD between a first end of a range of motion of a hinge of the display about an axis and a second end of the range of motion of the hinge about the axis; determine parameters of the HMD based on the sensor measurements and the rotation of the display between the first end of the range of motion of the hinge and the second end of the range of motion of the hinge about the axis; and determine pose information for the HMD based on the determined parameters.

Abstract: Systems and techniques are provided for performing user authentication. For example, a process can include obtaining a plurality of images associated with a face and a facial expression of the user, wherein each respective image of the plurality of images includes a different portion of the face. An encoder neural network can be used to generate one or more predicted three-dimensional (3D) facial modeling parameters, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images. A reference 3D facial model associated with the face and the facial expression can be obtained. An error can be determined between the one or more predicted 3D facial modeling parameters and the reference 3D facial model, and the user can be authenticated user based on the error being less than a pre-determined authentication threshold.

Abstract: Systems and techniques are described herein for processing image data. For instance, a method for processing image data is provided. The method may include capturing an image; determining at least one region of interest of the image based on at least one of object recognition, object tracking, hand tracking, semantic segmentation, saliency detection, or computer vision; encoding a first portion of the image according to a first parameter to generate first encoded data, the first portion of the image corresponding to the at least one region of interest; encoding a second portion of the image according to a second parameter to generate second encoded data; and transmitting, to a computing device, the first encoded data and the second encoded data.

Abstract: Methods, devices, and apparatuses are provided to facilitate a positioning of an item of virtual content in an extended reality environment. For example, a placement position for an item of virtual content can be transmitted to one or more of a first device and a second device. The placement position can be based on correlated map data generated based on first map data obtained from the first device and second map data obtained from the second device. In some examples, the first device can transmit the placement position to the second device.

Abstract: Imaging systems and techniques are described. A system receives an image of an environment captured using an image sensor according to an image capture setting, and receives motion data captured using a motion sensor. The system determines a weight associated with at least one of a plurality of features of the environment in the image based on an estimated motion blur level for the at least one of the features of the environment in the image. The estimated motion blur level is based on the motion data and the image capture setting. The system tracks the features of the environment across a plurality of images (that includes the received image) according to respective weights (that include the determined weight) for the features of the environment across the plurality of images. The system can use the tracked features for mapping the environment and/or determining the pose of the system.

Abstract: Methods, devices, and apparatuses are provided to facilitate a positioning of an item of virtual content in an extended reality environment. For example, a placement position for an item of virtual content can be transmitted to one or more of a first device and a second device. The placement position can be based on correlated map data generated based on first map data obtained from the first device and second map data obtained from the second device. In some examples, the first device can transmit the placement position to the second device.

Abstract: Systems and techniques are provided for performing user authentication. For example, a process can include obtaining a plurality of images associated with a face and a facial expression of the user, wherein each respective image of the plurality of images includes a different portion of the face. An encoder neural network can be used to generate one or more predicted three-dimensional (3D) facial modeling parameters, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images. A reference 3D facial model associated with the face and the facial expression can be obtained. An error can be determined between the one or more predicted 3D facial modeling parameters and the reference 3D facial model, and the user can be authenticated user based on the error being less than a pre-determined authentication threshold.

Abstract: Disclosed are systems, apparatuses, processes, and computer-readable media to capture images with subjects at different depths. A method of processing image data includes obtaining, at an imaging device, a first image of an environment from an image sensor of the imaging device; determining a region of interest of the first image based on features depicted in the first image, wherein the features are associated with the environment; determining a representative luma value associated with the first image based on image data in the region of interest of the first image; determining one or more exposure control parameters based on the representative luma value; and obtaining, at the imaging device, a second image captured based on the one or more exposure control parameters.

Abstract: Systems and techniques are described for establishing one or more virtual sessions between users. For instance, a first device can transmit, to a second device, a call establishment request for a virtual representation call for a virtual session and can receive, from the second device, a call acceptance indicating acceptance of the call establishment request. The first device can transmit, to the second device, first mesh information for a first virtual representation of a first user of the first device and first mesh animation parameters for the first virtual representation. The first device can receive, from the second device, second mesh information for a second virtual representation of a second user of the second device and second mesh animation parameters for the second virtual representation. The first device can generate, based on the second mesh information and the second mesh animation parameters, the second virtual representation of the second user.

Abstract: The architecture shown can perform global search, local search and local sub pixel search in a parallel or in a pipelined mode. All operations are in a streaming mode without the requirement of external intermediate data storage.

Abstract: Systems and techniques are provided for generating a texture for a three-dimensional (3D) facial model. For example, a process can include obtaining a first frame, the first frame including a first portion of a face. In some aspects, the process can include generating a 3D facial model based on the first frame and generating a first facial feature corresponding to the first portion of the face. In some examples, the process includes obtaining a second frame, the second frame including a second portion of the face. In some cases, the second portion of the face at least partially overlaps the first portion of the face. In some examples, the process includes combining the first facial feature with the second facial feature to generate an enhanced facial feature, wherein the combining is performed to enhance an appearance of select areas of the enhanced facial feature.

Abstract: Systems and techniques are provided for generating a three-dimensional (3D) facial model. For example, a process can include obtaining at least one input image associated with a face. In some aspects, the process can include obtaining a pose for a 3D facial model associated with the face. In some examples, the process can include generating, by a machine learning model, the 3D facial model associated with the face. In some cases, one or more parameters associated with a shape component of the 3D facial model are conditioned on the pose. In some implementations, the 3D facial model is configured to vary in shape based on the pose for the 3D facial model associated with the face.

Abstract: Imaging systems and techniques are described. An imaging system receives, from an image sensor, image(s) of a user (e.g., in a pose and/or with a facial expression). The image sensor captures the first set of image(s) in a first electromagnetic (EM) frequency domain, such as the infrared and/or near-infrared domain. The imaging system generates a representation of the user in the first pose in a second EM frequency domain (e.g., visible light domain) at least in part by inputting the image(s) into one or more trained machine learning models. The representation of the user is based on an image property associated with image data of at least the part of the user in the second EM frequency domain. The imaging system outputs the representation of the user in the pose in the second EM frequency domain.

Abstract: The architecture shown can perform global search, local search and local sub pixel search in a parallel or in a pipelined mode. All operations are in a streaming mode without the requirement of external intermediate data storage.

Abstract: Systems and techniques are described herein for processing images. The systems and techniques can be implemented by various types of systems, such as by an extended reality (XR) system or device. In some cases, a first processor receives an image of an environment captured by an image sensor, identifies features depicted in the image, and generates descriptors for the features. The first processor sends the descriptors to a second processor, which may be more powerful than the first processor. The second processor receives the descriptors. The second processor associates the plurality of features with a map of the environment based on at least a subset of the plurality of descriptors. For example, the second processor can track at least a subset of the features based on at least a subset of the descriptors and based on feature information from one or more additional images of the environment.

Abstract: Systems and techniques are provided for determining and applying corrected poses in digital content experiences. An example method can include receiving, from one or more sensors associated with an apparatus, inertial measurements and one or more frames of a scene; based on the one or more frames and the inertial measurements, determining, via a first filter, an angular and linear motion of the apparatus and a gravity vector indicating a direction of gravitational force interacting with the apparatus; when a motion of the apparatus is below a threshold, determining, via a second filter, an updated gravity vector indicating a direction of gravitational force interacting with the apparatus; determining, based on the updated gravity vector, parameters for aligning an axis of the scene with a gravity direction in a real-world spatial frame; and aligning, using the parameters, the axis of the scene with the gravity direction in the real-world spatial frame.