Abstract: In various examples, artificial intelligence (AI) agents can be generated to synthesize more natural motion by simulated actors in various visualizations (such as video games or simulations). AI agents may employ one or more machine learning models and techniques, such as reinforcement learning, to enable synthesis of motion with enhanced realism. The AI agent can be trained based on widely-available broadcast video data, without the need for more costly and limited motion capture data. To account for the lower quality of such video data, various techniques can be employed, such as taking into account the motion of joints, and applying physics-based constraints on the actors, resulting in higher quality, more lifelike motion.

Abstract: Systems and techniques are described related to training one or more machine learning models for use in control of a robot. In at least one embodiment, one or more machine learning models are trained based at least on simulations of the robot and renderings of such simulations—which may be performed using one or more ray tracing algorithms, operations, or techniques.

Abstract: This disclosure presents a process to generate one or more video frames through guiding the movements of a target object in an environment controlled by physics-based constraints. The target object is guided by the movements of a reference object from a motion capture (MOCAP) video clip. As disturbances, environmental factors, or other physics-based constraints interfere with the target object mimicking the reference object. A tracking agent, along with a corresponding neural network, can be used to compensate and modify the movements of the target object. Should the target object diverge significantly from the reference object, such as falling down, a recovery agent, along with a corresponding neural network, can be used to move the target object back into an approximate alignment with the reference object before resuming the tracking process.

Abstract: Apparatuses, systems, and techniques are described that estimate the pose of an object while the object is being manipulated by a robotic appendage. In at least one embodiment, a sample-based optimization algorithm tracks in-hand object poses during manipulation via contact feedback and a GPU-accelerated robotic simulation is developed. In at least one embodiment, parallel simulations concurrently model object pose changes that may be caused by complex contact dynamics. In at least one embodiment, the optimization algorithm tunes simulation parameters during object pose tracking to further improve tracking performance. In various embodiments, real-world contact sensing may be improved by utilizing vision in-the-loop.

Abstract: This disclosure presents a process to generate one or more video frames through guiding the movements of a target object in an environment controlled by physics-based constraints. The target object is guided by the movements of a reference object from a motion capture (MOCAP) video clip. As disturbances, environmental factors, or other physics-based constraints interfere with the target object mimicking the reference object. A tracking agent, along with a corresponding neural network, can be used to compensate and modify the movements of the target object. Should the target object diverge significantly from the reference object, such as falling down, a recovery agent, along with a corresponding neural network, can be used to move the target object back into an approximate alignment with the reference object before resuming the tracking process.

Abstract: A machine-learning control system is trained to perform a task using a simulation. The simulation is governed by parameters that, in various embodiments, are not precisely known. In an embodiment, the parameters are specified with an initial value and expected range. After training on the simulation, the machine-learning control system attempts to perform the task in the real world. In an embodiment, the results of the attempt are compared to the expected results of the simulation, and the parameters that govern the simulation are adjusted so that the simulated result matches the real-world attempt. In an embodiment, the machine-learning control system is retrained on the updated simulation. In an embodiment, as additional real-world attempts are made, the simulation parameters are refined and the control system is retrained until the simulation is accurate and the control system is able to successfully perform the task in the real world.