Abstract: System and techniques are described herein for processing images to detect expressions of a subject. In one illustrative example, a method of recognizing facial expressions in one or more images includes obtaining, by a computing device, a first image of a person; obtaining expression information based on the first image and an anchor image associated with the person; and determining an expression classification associated with the first image based on the expression information.

Abstract: Systems and techniques are described for establishing one or more virtual sessions between users. For instance, a first device can transmit, to a second device, a call establishment request for a virtual representation call for a virtual session and can receive, from the second device, a call acceptance indicating acceptance of the call establishment request. The first device can transmit, to the second device, first mesh information for a first virtual representation of a first user of the first device and first mesh animation parameters for the first virtual representation. The first device can receive, from the second device, second mesh information for a second virtual representation of a second user of the second device and second mesh animation parameters for the second virtual representation. The first device can generate, based on the second mesh information and the second mesh animation parameters, the second virtual representation of the second user.

Abstract: Techniques and systems are provided for three-dimensional model segmentation. For instances, a process can include: rendering a three-dimensional (3D) model of an object based on a 3D mesh model and a texture of the object; generating a view of the 3D model based on the rendered 3D model; generating a texture space mask based on the texture of the object using a first machine learning (ML) model; generating a labeled mask based on a second ML model; assigning labels to the texture space mask based on the labeled mask to obtain a labeled texture space mask; and generating a 3D mask based on the labeled texture space mask.

Abstract: Systems and techniques are described herein for generating image data. For instance, a method for generating image data is provided. The method may include selecting enrollment images from among a plurality of enrollment images of a subject based on a target viewing angle, wherein each enrollment image of the plurality of enrollment images represents the subject as viewed from a different respective viewing angle; determining, from the selected enrollment images, appearance features representing at least one characteristic of the subject in the selected enrollment images; combining the appearance features based on a three-dimensional geometry of the subject to generate combined appearance features; and generating an image of the subject from the target viewing angle based on a texture image of the subject and the combined appearance features.

Abstract: Systems and techniques are described herein for generating three-dimensional (3D) representations. For instance, a method for generating three-dimensional (3D) representations is provided. The method may include generating a Gaussian-splat representation of a subject based on a point-cloud representation of the subject; and iteratively adjusting the Gaussian-splat representation and a mask indicative of Gaussian splats of the Gaussian-splat representation to render images by rendering images based on the Gaussian-splat representation and the mask, comparing the rendered images to input images of the subject, and adjusting parameters of the Gaussian-splat representation and values of the mask based on the comparison.

Abstract: Systems and techniques are described herein for generating an image. For instance, a method for generating an image is provided. The method may include obtaining a source image of a face having source attributes and exhibiting a source pose and source gaze; obtaining at least one of a target pose and a target gaze; and generating a modified image of the face having the source attributes and exhibiting at least one of the target pose and the target gaze.

Abstract: Systems and techniques are disclosed for generating a three-dimensional (3D) model. For example, a process can include estimating a plurality of features associated with at least a portion of images; inverse warping the plurality of features into reference pose features having a reference pose; generating filtered reference pose features by selecting features from the reference pose features based on a distance of the selected features from corresponding features from the reference pose; generating modified reference pose features by modifying the filtered reference pose features based on a feature grid associated with a reference model associated with the reference pose; projecting the filtered reference pose features into one or more two dimensional (2D) planes; identifying first features associated with the person from the one or more 2D planes; and generating a 3D model of the person having a pose using the first features, the modified reference pose features, and pose information.

Abstract: Certain aspects of the present disclosure provide techniques for gaze zone detection. A method generally includes estimating a first gaze direction of a user based on at least one or more first two-dimensional (2D) images of a face of the user, wherein the first gaze direction is represented as a first yaw angle and a first pitch angle in a camera coordinate system; and associating the first gaze direction of the user with a first gaze zone among a plurality of gaze zones based on: the first yaw angle; the first pitch angle; and a first Gaussian distribution of the first gaze zone, wherein the first Gaussian distribution is based on a plurality of gaze directions, of a plurality of users, associated with the first gaze zone.

Abstract: System and techniques are described herein for processing images to detect objects in the provided images. In one illustrative example, a method of processing image data includes obtaining an image including at least a first object. The method can include generating a feature map based on providing the image to a neural network. The method can further include identifying a plurality of objects based on the feature map, the plurality of objects including a first part of the first object. The method can include identifying a first set of object parts within the plurality of objects corresponding to the first object.

Abstract: Systems and techniques are described herein for segmenting images. For instance, a method for segmenting images is provided. The method may include encoding, using a machine-learning-model image encoder, an image to generate a plurality of image features; selecting a first image feature from among the plurality of image features; determining a first image point related to the first image feature; encoding, using a machine-learning-model prompt encoder, the first image point as a first encoded prompt; generating, using a machine-learning-model image decoder, a first mask based on the plurality of image features and the first encoded prompt, wherein the first mask is indicative of pixels of the image that are semantically similar to the first image point; and storing the first mask.

Abstract: Systems and techniques are described herein for human-body-model shape modification. For instance, a method for human-body-model shape modification is provided. The method may include obtaining a three-dimensional (3D) model of a body of a person; obtaining body pixels based on an image of the body of the person; generating projected body points by projecting points of the 3D model into an image plane; determining a body-point loss based on a comparison of the body pixels and the projected body points; and modifying the 3D model based on the body-point loss to generate a first modified 3D model.

Abstract: Systems and techniques are provided for generating a three-dimensional (3D) facial model. For example, a process can include obtaining at least one input image associated with a face. In some aspects, the process can include obtaining a pose for a 3D facial model associated with the face. In some examples, the process can include generating, by a machine learning model, the 3D facial model associated with the face. In some cases, one or more parameters associated with a shape component of the 3D facial model are conditioned on the pose. In some implementations, the 3D facial model is configured to vary in shape based on the pose for the 3D facial model associated with the face.

Abstract: Techniques and systems are provided for three-dimensional model segmentation. For instances, a process can include: rendering a three-dimensional (3D) model of an object based on a 3D mesh model and a texture of the object; generating a view of the 3D model based on the rendered 3D model; generating a texture space mask based on the texture of the object using a first machine learning (ML) model; generating a labeled mask based on a second ML model; assigning labels to the texture space mask based on the labeled mask to obtain a labeled texture space mask; and generating a 3D mask based on the labeled texture space mask.

Abstract: Systems and techniques are described herein for generating image data. For instance, a method for generating image data is provided. The method may include selecting enrollment images from among a plurality of enrollment images of a subject based on a target viewing angle, wherein each enrollment image of the plurality of enrollment images represents the subject as viewed from a different respective viewing angle; determining, from the selected enrollment images, appearance features representing at least one characteristic of the subject in the selected enrollment images; combining the appearance features based on a three-dimensional geometry of the subject to generate combined appearance features; and generating an image of the subject from the target viewing angle based on a texture image of the subject and the combined appearance features.

Abstract: Techniques and systems are provided for image processing. For instance, a process can include obtaining, from one or more image sensors, a first image of an environment; determining semantic labels for a plurality of pixels of the first image based on whether each pixel of the plurality of pixels is associated with an object in the first image to generate a semantically segmented image; applying a two-dimensional line representation of the first image to the semantically segmented image to generate a labeled 2D line representation; back-projecting the labeled 2D line representation to three dimensions to generate a line representation model; fusing the line representation model and the semantically segmented image to generate a labeled line representation model; and outputting the labeled line representation model.

Abstract: This disclosure provides systems, devices, apparatus, and methods, including computer programs encoded on storage media, for delivering stored objects for XR applications. A processor transmits, to an object library, a query for supported forms of enroll data of an avatar of a second user, where the first device is associated with a first user and a second device is associated with the second user. The processor receives, from the object library and based on the query, a list of the supported forms of the enroll data. The processor transmits, to the second device, an identifier of a supported form in the list of the supported forms. The processor performs an avatar call with the second device based on the transmitted identifier of the supported form.

Abstract: Systems and techniques are provided for generating a texture for a three-dimensional (3D) facial model. For example, a process can include obtaining a first frame, the first frame including a first portion of a face. In some aspects, the process can include generating a 3D facial model based on the first frame and generating a first facial feature corresponding to the first portion of the face. In some examples, the process includes obtaining a second frame, the second frame including a second portion of the face. In some cases, the second portion of the face at least partially overlaps the first portion of the face. In some examples, the process includes combining the first facial feature with the second facial feature to generate an enhanced facial feature, wherein the combining is performed to enhance an appearance of select areas of the enhanced facial feature.

Abstract: Systems and techniques are disclosed for generating a three-dimensional (3D) model. For example, a process can include estimating a plurality of features associated with at least a portion of images; inverse warping the plurality of features into reference pose features having a reference pose; generating filtered reference pose features by selecting features from the reference pose features based on a distance of the selected features from corresponding features from the reference pose; generating modified reference pose features by modifying the filtered reference pose features based on a feature grid associated with a reference model associated with the reference pose; projecting the filtered reference pose features into one or more two dimensional (2D) planes; identifying first features associated with the person from the one or more 2D planes; and generating a 3D model of the person having a pose using the first features, the modified reference pose features, and pose information.

Abstract: Systems and techniques are provided for performing scene segmentation and object tracking. For example, a method for processing one or more frames. The method may include determining first one or more features from a first frame. The first frame includes a target object. The method may include obtaining a first mask associated with the first frame. The first mask includes an indication of the target object. The method may further include generating, based on the first mask and the first one or more features, a representation of a foreground and a background of the first frame. The method may include determining second one or more features from a second frame and determining, based on the representation of the foreground and the background of the first frame and the second one or more features, a location of the target object in the second frame.

Abstract: Systems and techniques are described for image processing. An imaging system receives an identity image and an attribute image. The identity image depicts a first person having an identity. The attribute image depicts a second person having an attribute, such as a facial feature, an accessory worn by the second person, and/or an expression. The imaging system uses trained machine learning model(s) to generate a combined image based on the identity image and the attribute image. The combined image depicts a virtual person having both the identity of the first person and the attribute of the second person. The imaging system outputs the combined image, for instance by displaying the combined image or sending the combined image to a receiving device. In some examples, the imaging system updates the trained machine learning model(s) based on the combined image.