Abstract: The disclosed matting technique comprises receiving a video feed comprising a plurality of temporally ordered video frames as the video frames are captured by a video capture device, generating, using one or more machine learning models, an image mask corresponding to each video frame included in the video feed and a depth estimate corresponding to each video frame included in the video feed, and, for each video frame in the video feed, transmitting, in real-time, the video frame, the corresponding image mask, and the corresponding depth estimate to a compositing system. The compositing system composites a computer-generated element with the video frame based on the corresponding image mask and the corresponding depth estimate.

Abstract: One embodiment of the present invention sets forth a technique for training a neural motion controller. The technique includes determining a first set of features associated with a first control signal for a virtual character. The technique also includes matching the first set of features to a first sequence of motions included in a plurality of sequences of motions. The technique further includes training the neural motion controller based on one or more motions included in the first sequence of motions and the first control signal.

Abstract: A method and controller for autonomous control and station-keeping of an unmanned vehicle uses sensor data corresponding to at least geographical location and vertical position of the unmanned vehicle within a fluid environment and computes a gradient of the unmanned vehicle movement, uses the gradient to estimate a vertical profile of a feature of the fluid environment that affects the gradient of the unmanned vehicle, identifies a favourable position in the vertical profile where the feature of the fluid environment would transport the unmanned vehicle in a direction that minimizes a performance metric, and outputs a control signal based on the favourable position in the vertical profile. The control signal controls at least one actuator that causes the unmanned vehicle to ascend or descend to the favourable position in the vertical profile. The unmanned vehicle may be a high altitude platform (HAP).

Abstract: Various embodiments disclosed herein provide techniques for generating image data of a three-dimensional (3D) animatable asset. A rendering module executing on a computer system accesses a machine learning model that has been trained via first image data of the 3D animatable asset generated from first rig vector data. The rendering module receives second rig vector data. The rendering module generates, via the machine learning model, a second image data of the 3D animatable asset based on the second rig vector data.

Abstract: Techniques are disclosed for learning a machine learning model that maps control data, such as renderings of skeletons, and associated three-dimensional (3D) information to two-dimensional (2D) renderings of a character. The machine learning model may be an adaptation of the U-Net architecture that accounts for 3D information and is trained using a perceptual loss between images generated by the machine learning model and ground truth images. Once trained, the machine learning model may be used to animate a character, such as in the context of previsualization or a video game, based on control of associated control points.

Abstract: An image processing system includes a computing platform having a hardware processor and a system memory storing an image augmentation software code, a three-dimensional (3D) shapes library, and/or a 3D poses library. The image processing system also includes a two-dimensional (2D) pose estimation module communicatively coupled to the image augmentation software code. The hardware processor executes the image augmentation software code to provide an image to the 2D pose estimation module and to receive a 2D pose data generated by the 2D pose estimation module based on the image. The image augmentation software code identifies a 3D shape and/or a 3D pose corresponding to the image using an optimization algorithm applied to the 2D pose data and one or both of the 3D poses library and the 3D shapes library, and may output the 3D shape and/or 3D pose to render an augmented image on a display.

Abstract: In various embodiments, a training application generates training items for three-dimensional (3D) pose estimation. The training application generates multiple posed 3D models based on multiple 3D poses and a 3D model of a person wearing a costume that is associated with multiple visual attributes. For each posed 3D model, the training application performs rendering operation(s) to generate synthetic image(s). For each synthetic image, the training application generates a training item based on the synthetic image and the 3D pose associated with the posed 3D model from which the synthetic image was rendered. The synthetic images are included in a synthetic training dataset that is tailored for training a machine-learning model to compute estimated 3D poses of persons from two-dimensional (2D) input images. Advantageously, the synthetic training dataset can be used to train the machine-learning model to accurately infer the orientations of persons across a wide range of environments.

Abstract: An image processing system includes a computing platform having a hardware processor and a system memory storing an image augmentation software code, a three-dimensional (3D) shapes library, and/or a 3D poses library. The image processing system also includes a two-dimensional (2D) pose estimation module communicatively coupled to the image augmentation software code. The hardware processor executes the image augmentation software code to provide an image to the 2D pose estimation module and to receive a 2D pose data generated by the 2D pose estimation module based on the image. The image augmentation software code identifies a 3D shape and/or a 3D pose corresponding to the image using an optimization algorithm applied to the 2D pose data and one or both of the 3D poses library and the 3D shapes library, and may output the 3D shape and/or 3D pose to render an augmented image on a display.

Abstract: One embodiment of the present invention sets forth a technique for generating a pose estimation model. The technique includes generating one or more trained components included in the pose estimation model based on a first set of training images and a first set of labeled poses associated with the first set of training images, wherein each labeled pose includes a first set of positions on a left side of an object and a second set of positions on a right side of the object. The technique also includes training the pose estimation model based on a set of reconstructions of a second set of training images, wherein the set of reconstructions is generated by the pose estimation model from a set of predicted poses outputted by the one or more trained components.

Abstract: In various embodiments, a training application generates training items for three-dimensional (3D) pose estimation. The training application generates multiple posed 3D models based on multiple 3D poses and a 3D model of a person wearing a costume that is associated with multiple visual attributes. For each posed 3D model, the training application performs rendering operation(s) to generate synthetic image(s). For each synthetic image, the training application generates a training item based on the synthetic image and the 3D pose associated with the posed 3D model from which the synthetic image was rendered. The synthetic images are included in a synthetic training dataset that is tailored for training a machine-learning model to compute estimated 3D poses of persons from two-dimensional (2D) input images. Advantageously, the synthetic training dataset can be used to train the machine-learning model to accurately infer the orientations of persons across a wide range of environments.

Abstract: Various embodiments disclosed herein provide techniques for generating image data of a three-dimensional (3D) animatable asset. A rendering module executing on a computer system accesses a machine learning model that has been trained via first image data of the 3D animatable asset generated from first rig vector data. The rendering module receives second rig vector data. The rendering module generates, via the machine learning model, a second image data of the 3D animatable asset based on the second rig vector data.

Abstract: Techniques are disclosed for estimating poses from images. In one embodiment, a machine learning model, referred to herein as the “detector,” is trained to estimate animal poses from images in a bottom-up fashion. In particular, the detector may be trained using rendered images depicting animal body parts scattered over realistic backgrounds, as opposed to renderings of full animal bodies. In order to make appearances of the rendered body parts more realistic so that the detector can be trained to estimate poses from images of real animals, the body parts may be rendered using textures that are determined from a translation of rendered images of the animal into corresponding images with more realistic textures via adversarial learning. Three-dimensional poses may also be inferred from estimated joint locations using, e.g., inverse kinematics.

Abstract: Techniques are disclosed for learning a machine learning model that maps control data, such as renderings of skeletons, and associated three-dimensional (3D) information to two-dimensional (2D) renderings of a character. The machine learning model may be an adaptation of the U-Net architecture that accounts for 3D information and is trained using a perceptual loss between images generated by the machine learning model and ground truth images. Once trained, the machine learning model may be used to animate a character, such as in the context of previsualization or a video game, based on control of associated control points.

Abstract: An automated costume augmentation system includes a computing platform having a hardware processor and a system memory storing a software code. The hardware processor executes the software code to provide an image including a posed figure to an artificial neural network (ANN), receive from the ANN a 2D skeleton data including joint positions corresponding to the posed figure, and determine a 3D pose corresponding to the posed figure using an optimization algorithm applied to the skeleton data. The software code further identifies one or more proportion(s) of the posed figure based on the skeleton data, determines bone directions corresponding to the posed figure using another optimization algorithm applied to the 3D pose, parameterizes a costume for the posed figure based on the 3D pose, the proportion(s), and the bone directions, and outputs an enhanced image including the posed figure augmented with the fitted costume for rendering on a display.

Abstract: Techniques are disclosed for estimating poses from images. In one embodiment, a machine learning model, referred to herein as the “detector,” is trained to estimate animal poses from images in a bottom-up fashion. In particular, the detector may be trained using rendered images depicting animal body parts scattered over realistic backgrounds, as opposed to renderings of full animal bodies. In order to make appearances of the rendered body parts more realistic so that the detector can be trained to estimate poses from images of real animals, the body parts may be rendered using textures that are determined from a translation of rendered images of the animal into corresponding images with more realistic textures via adversarial learning. Three-dimensional poses may also be inferred from estimated joint locations using, e.g., inverse kinematics.

Abstract: An automated costume augmentation system includes a computing platform having a hardware processor and a system memory storing a software code. The hardware processor executes the software code to provide an image including a posed figure to an artificial neural network (ANN), receive from the ANN a 2D skeleton data including joint positions corresponding to the posed figure, and determine a 3D pose corresponding to the posed figure using an optimization algorithm applied to the skeleton data. The software code further identifies one or more proportion(s) of the posed figure based on the skeleton data, determines bone directions corresponding to the posed figure using another optimization algorithm applied to the 3D pose, parameterizes a costume for the posed figure based on the 3D pose, the proportion(s), and the bone directions, and outputs an enhanced image including the posed figure augmented with the fitted costume for rendering on a display.

Abstract: Techniques for generating locomotion data for animating a virtual quadruped model, starting at an origin point and travelling to a destination point along a defined path. A virtual skeletal structure for the virtual quadruped model is analyzed to identify a torso region, limbs each ending in a respective end effector, and limb attributes. A predefined locomotion template for virtual quadruped characters is retrieved and mapped to the virtual quadruped model by aligning the torso region and the plurality of limbs of the virtual skeletal structure for the virtual quadruped with a second torso region and a second plurality of limbs of the predefined locomotion template. Locomotion data is generated for the virtual quadruped model based on the defined path and by upscaling the mapped predefined locomotion template, based at least on the set of limb attributes determined by analyzing the virtual skeletal structure for the virtual quadruped model.

Abstract: An image processing system includes a computing platform having a hardware processor and a system memory storing an image augmentation software code, a three-dimensional (3D) shapes library, and/or a 3D poses library. The image processing system also includes a two-dimensional (2D) pose estimation module communicatively coupled to the image augmentation software code. The hardware processor executes the image augmentation software code to provide an image to the 2D pose estimation module and to receive a 2D pose data generated by the 2D pose estimation module based on the image. The image augmentation software code identifies a 3D shape and/or a 3D pose corresponding to the image using an optimization algorithm applied to the 2D pose data and one or both of the 3D poses library and the 3D shapes library, and may output the 3D shape and/or 3D pose to render an augmented image on a display.

Abstract: A simulation engine models interactions between a simulated character and real-world objects to produce a physically realistic augmented reality (AR) simulation. The simulation engine recognizes a given real-world object and then identifies, within a library of object models, an object model corresponding to that object. The simulation engine projects the object model onto the real-world object such that the object model is geometrically aligned with the real-world object. When the simulated character encounters the real-world object, the simulation engine models interactions between the simulated character and the real-world object by adjusting the kinematics of the simulated character relative to the object model associated with the real-world object.

Abstract: A simulation engine models interactions between a simulated character and real-world objects to produce a physically realistic augmented reality (AR) simulation. The simulation engine recognizes a given real-world object and then identifies, within a library of object models, an object model corresponding to that object. The simulation engine projects the object model onto the real-world object such that the object model is geometrically aligned with the real-world object. When the simulated character encounters the real-world object, the simulation engine models interactions between the simulated character and the real-world object by adjusting the kinematics of the simulated character relative to the object model associated with the real-world object.