Abstract: Systems and techniques are provided for generating one or more models. For example, a process can include obtaining a plurality of input images corresponding to faces of one or more people during a training interval. The process can include determining a value of the coefficient representing at least the portion of the facial expression for each of the plurality of input images during the training interval. The process can include determining, from the determined values of the coefficient representing at least the portion of the facial expression for each of the plurality of input images during the training interval, an extremum value of the coefficient representing at least the portion of the facial expression during the training interval. The process can include generating an updated bounding value for the coefficient representing at least the portion of the facial expression based on the initial bounding value and the extremum value.

Abstract: Systems and techniques are described herein for generating virtual representation (e.g., avatar). For example, a process can include obtaining data describing a virtual representation, the data including a hierarchical set of nodes, wherein a first node of the set of nodes includes type information, source information, and a mapping, and wherein a child node of the hierarchical set of nodes includes data associated with a segment of the virtual representation; identifying, based on the type information, a format associated with the virtual representation of a user; identifying, based on the mapping, the child node in the hierarchical set of nodes; identifying, based on the source information, a portion of the data associated with the child node; and processing the data associated with the segment of the virtual representation of the child node based on a corresponding format for the virtual representation to generate a segment of the virtual representation.

Abstract: Systems and techniques are provided for deep learning model optimizations. An example process can include generating, based on processing data using a neural network model, a plurality of prediction outputs associated with a plurality of output channels corresponding to a multi-channel prediction target; determining that a prediction output from the plurality of prediction outputs has a value that is outside of a quantization range and one or more remaining prediction outputs from the plurality of prediction outputs have a respective value that is within the quantization range; clamping the prediction output based on the quantization range; and generating a single channel output based on the clamped prediction output and the one or more remaining prediction outputs.

Abstract: A device includes a memory and one or more processors configured to process image data corresponding to a user's face to generate face data. The one or more processors are configured to process sensor data to generate feature data and to generate a representation of an avatar based on the face data and the feature data. The one or more processors are also configured to generate an audio output for the avatar based on the sensor data.

Abstract: A device includes a memory and one or more processors configured to process sensor data to determine a semantical context associated with the sensor data. The one or more processors are also configured to generate adjusted face data based on the determined semantical context and face data. The adjusted face data includes an avatar facial expression that corresponds to the semantical context.

Abstract: Systems and techniques are described for establishing one or more virtual sessions between users. For instance, a first device can transmit, to a second device, a call establishment request for a virtual representation call for a virtual session and can receive, from the second device, a call acceptance indicating acceptance of the call establishment request. The first device can transmit, to the second device, first mesh information for a first virtual representation of a first user of the first device and first mesh animation parameters for the first virtual representation. The first device can receive, from the second device, second mesh information for a second virtual representation of a second user of the second device and second mesh animation parameters for the second virtual representation. The first device can generate, based on the second mesh information and the second mesh animation parameters, the second virtual representation of the second user.

Abstract: Systems and techniques are provided for generating a texture for a three-dimensional (3D) facial model. For example, a process can include obtaining a first frame, the first frame including a first portion of a face. In some aspects, the process can include generating a 3D facial model based on the first frame and generating a first facial feature corresponding to the first portion of the face. In some examples, the process includes obtaining a second frame, the second frame including a second portion of the face. In some cases, the second portion of the face at least partially overlaps the first portion of the face. In some examples, the process includes combining the first facial feature with the second facial feature to generate an enhanced facial feature, wherein the combining is performed to enhance an appearance of select areas of the enhanced facial feature.

Abstract: Techniques are provided for generating three-dimensional models of objects from one or more images or frames. For example, at least one frame of an object in a scene can be obtained. A portion of the object is positioned on a plane in the at least one frame. The plane can be detected in the at least one frame and, based on the detected plane, the object can be segmented from the plane in the at least one frame. A three-dimensional (3D) model of the object can be generated based on segmenting the object from the plane. A refined mesh can be generated for a portion of the 3D model corresponding to the portion of the object positioned on the plane.

Abstract: Systems and techniques are provided for generating a three-dimensional (3D) facial model. For example, a process can include obtaining at least one input image associated with a face. In some aspects, the process can include obtaining a pose for a 3D facial model associated with the face. In some examples, the process can include generating, by a machine learning model, the 3D facial model associated with the face. In some cases, one or more parameters associated with a shape component of the 3D facial model are conditioned on the pose. In some implementations, the 3D facial model is configured to vary in shape based on the pose for the 3D facial model associated with the face.

Abstract: Methods, systems, and apparatuses are provided to automatically reconstruct an image, such as a 3D image. For example, a computing device may obtain an image, and may apply a first trained machine learning process to the image to generate coefficient values characterizing the image in a plurality of dimensions. Further, the computing device may generate a mesh based on the coefficient values. The computing device may apply a second trained machine learning process to the coefficient values and the image to generate a displacement map. Based on the mesh and the displacement map, the computing device may generate output data characterizing an aligned mesh. The computing device may store the output data within a data repository. In some examples, the computing device provides the output data for display.

Abstract: A method performed by an electronic device is described. The method includes obtaining sensor data corresponding to multiple occupants from an interior of a vehicle. The method also includes obtaining, by a processor, at least one occupant status for at least one of the occupants based on a first portion of the sensor data. The method further includes identifying, by the processor, at least one vehicle operation in response to the at least one occupant status. The method additionally includes determining, by the processor, based at least on a second portion of the sensor data, whether to perform the at least one vehicle operation. The method also includes performing the at least one vehicle operation in a case that it is determined to perform the at least one vehicle operation.

Abstract: Systems and techniques are provided for facial expression recognition. In some examples, a system receives an image frame corresponding to a face of a person. The system also determines, based on a three-dimensional model of the face, landmark feature information associated with landmark features of the face. The system then inputs, to at least one layer of a neural network trained for facial expression recognition, the image frame and the landmark feature information. The system further determines, using the neural network, a facial expression associated with the face.

Abstract: Techniques are provided for generating three-dimensional models of objects from one or more images or frames. For example, at least one frame of an object in a scene can be obtained. A portion of the object is positioned on a plane in the at least one frame. The plane can be detected in the at least one frame and, based on the detected plane, the object can be segmented from the plane in the at least one frame. A three-dimensional (3D) model of the object can be generated based on segmenting the object from the plane. A refined mesh can be generated for a portion of the 3D model corresponding to the portion of the object positioned on the plane.

Abstract: The present disclosure generally relates to systems and techniques for constructing three-dimensional (3D) models. Certain aspects of the present disclosure provide an apparatus for model generation. The apparatus generally includes a memory, and one or more processors coupled to the memory. The one or more processors and the memory may be configured to receive one or more images depicting an object to be modeled, determine a category associated with the object to be modeled, select a shape of a space based on the category, and generate a 3D model of the object at least in part by carving one or more points associated with the space based on the one or more images depicting the object.

Abstract: A method performed by an electronic device is described. The method includes obtaining sensor data corresponding to multiple occupants from an interior of a vehicle. The method also includes obtaining, by a processor, at least one occupant status for at least one of the occupants based on a first portion of the sensor data. The method further includes identifying, by the processor, at least one vehicle operation in response to the at least one occupant status. The method additionally includes determining, by the processor, based at least on a second portion of the sensor data, whether to perform the at least one vehicle operation. The method also includes performing the at least one vehicle operation in a case that it is determined to perform the at least one vehicle operation.

Abstract: Disclosed are techniques for processing image data. In some aspects, a three-dimensional model can be determined corresponding to an object in an input image. Based on the three-dimensional model, an estimated focal length can be determined that corresponds to the input image. An estimated depth associated with the object in the input image can be calculated based on the estimated focal length and an input image focal length.

Abstract: Techniques are provided for generating one or more three-dimensional (3D) models. In one example, an image of an object (e.g., a face or other object) is obtained, and a 3D model of the object in the image is generated. The 3D model includes geometry information. Color information for the 3D model is determined, and a fitted 3D model of the object is generated based on a modification of the geometry information and the color information for the 3D model. In some cases, the color information (e.g., determination and/or modification of the color information) and the fitted 3D model can be based on one or more vertex-level fitting processes. A refined 3D model of the object is generated based on the fitted 3D model and depth information associated with the fitted 3D model. In some cases, the refined 3D model can be based on a pixel-level refinement or fitting process.

Abstract: Systems and techniques are provided for generating one or more models. For example, a process can include obtaining a plurality of input images corresponding to faces of one or more people during a training interval. The process can include determining a value of the coefficient representing at least the portion of the facial expression for each of the plurality of input images during the training interval. The process can include determining, from the determined values of the coefficient representing at least the portion of the facial expression for each of the plurality of input images during the training interval, an extremum value of the coefficient representing at least the portion of the facial expression during the training interval. The process can include generating an updated bounding value for the coefficient representing at least the portion of the facial expression based on the initial bounding value and the extremum value.

Abstract: Techniques are provided for generating three-dimensional models of objects from one or more images or frames. For example, at least one frame of an object in a scene can be obtained. A portion of the object is positioned on a plane in the at least one frame. The plane can be detected in the at least one frame and, based on the detected plane, the object can be segmented from the plane in the at least one frame. A three-dimensional (3D) model of the object can be generated based on segmenting the object from the plane. A refined mesh can be generated for a portion of the 3D model corresponding to the portion of the object positioned on the plane.

Abstract: Systems and techniques are provided for facial expression recognition. In some examples, a system receives an image frame corresponding to a face of a person. The system also determines, based on a three-dimensional model of the face, landmark feature information associated with landmark features of the face. The system then inputs, to at least one layer of a neural network trained for facial expression recognition, the image frame and the landmark feature information. The system further determines, using the neural network, a facial expression associated with the face.