Abstract: Examples are disclosed that relate to generating expressive avatars using multi-modal three-dimensional face modeling and tracking. One example includes a computer system comprising a processor coupled to a storage system that stores instructions. Upon execution by the processor, the instructions cause the processor to receive initialization data describing an initial state of a facial model. The instructions further cause the processor to receive a plurality of multi-modal data signals. The instructions further cause the processor to perform a fitting process using the initialization data and the plurality of multi-modal data signals. The instructions further cause the processor to determine a set of parameters based on the fitting process, wherein the determined set of parameters describes an updated state of the facial model.

Abstract: Computing an image depicting a face having an expression with wrinkles is described. A 3D polygon mesh model of a face has a non-neutral expression. A tension map is computed from the 3D polygon mesh model. A neutral texture, a compressed wrinkle texture and an expanded wrinkle texture are computed or obtained from a library. The neutral texture comprises a map of the first face with a neutral expression. The compressed wrinkle texture is a map of the first face formed by aggregating maps of the first face with different expressions using the tension map, and the expanded wrinkle texture comprises a map of the first face formed by aggregating maps of the first face with different expressions using the tension map. A graphics engine may be used to apply the wrinkle textures to the 3D model according to the tension map; and render the image from the 3D model.

Abstract: Keypoints are predicted in an image. Predictions are generated for each of the keypoints of an image as a 2D random variable, normally distributed with location (x, y) and standard deviation sigma. A neural network is trained to maximize a log-likelihood that samples from each of the predicted keypoints equal a ground truth. The trained neural network is used to predict keypoints of an image without generating a heatmap.

Abstract: This document relates to head mounted display devices. One example can include a layer of individually controllable pixels that can be energized to emit light and a layer of lenses that are physically aligned over the pixels. The example can also include a layer of shutters interposed between the pixels and the lenses and configured to be independently transitioned between a transmissive state and an opaque state to limit paths of the emitted light that reach the layer of lenses.

Abstract: This document relates to head mounted display devices. In one example the head mounted display device includes a light engine including an array of individually controllable pixels that can be energized to emit light. The example also includes an optical assembly physically aligned with the light engine and including a set of focusing elements facing toward the light engine and a different set of focusing elements facing away from the light engine.

Abstract: This document relates to head mounted display devices. One example can include a layer of individually controllable pixels that can be energized to emit light and a layer of lenses that are physically aligned over the pixels. The example can also include a layer of shutters interposed between the pixels and the lenses and configured to be independently transitioned between a transmissive state and an opaque state to limit paths of the emitted light that reach the layer of lenses.

Abstract: A head-mounted, near-eye display device includes a central display and a peripheral display. The central display creates a central image of a first resolution in a central eyebox. The peripheral display creates a peripheral image of a second resolution, lower than the first resolution, in a peripheral eyebox, different than the central eyebox.

Abstract: A method of fitting a three dimensional (3D) model to input data is described. Input data comprises a 3D scan and associated appearance information. The 3D scan depicts a composite object having elements from at least two classes. A texture model is available which, given an input vector, computes, for each of the classes, a texture and a mask. A joint optimization is computed to find values of the input vector and values of parameters of the 3D model, where the optimization enforces that the 3D model, instantiated by the values of the parameters, gives a simulated texture which agrees with the input data in a region specified by the mask associated with the 3D model; such that the 3D model is fitted to the input data.

Abstract: This document relates to head mounted display devices. In one example the head mounted display device includes a light engine including an array of individually controllable pixels that can be energized to emit light. The example also includes an optical assembly physically aligned with the light engine and including a set of focusing elements facing toward the light engine and a different set of focusing elements facing away from the light engine.

Abstract: A holographic display system includes an eye tracker configured to determine a position of a feature of an eye, a light source configured to output image light, and a digital dynamic hologram. The digital dynamic hologram is configured to receive the image light from the light source. The digital dynamic hologram is further configured to spatially modulate the image light based on a target image to form a reconstructed image in the eye. The reconstructed image includes noise that is non-uniformly distributed across the reconstructed image based on the position of the feature of the eye.

Abstract: A head-mounted, near-eye display device includes a central display and a peripheral display. The central display creates a central image of a first resolution in a central eyebox. The peripheral display creates a peripheral image of a second resolution, lower than the first resolution, in a peripheral eyebox, different than the central eyebox.

Abstract: A holographic display system includes an eye tracker configured to determine a position of a feature of an eye, a light source configured to output image light, and a digital dynamic hologram. The digital dynamic hologram is configured to receive the image light from the light source. The digital dynamic hologram is further configured to spatially modulate the image light based on a target image to form a reconstructed image in the eye. The reconstructed image includes noise that is non-uniformly distributed across the reconstructed image based on the position of the feature of the eye.