Abstract: A handheld device includes: an input control configured to control and modify a virtual scene including a virtual camera; and a display that shows a representation of the controlled and modified virtual scene generated by the virtual camera. A system includes: a computer system configured to execute program instructions for generating a virtual scene including a virtual camera; and handheld device configured to communicate with the computer system for controlling and modifying the virtual scene, the handheld device comprising: an input control configured to control and modify the virtual scene; and a display that shows a representation of the controlled and modified virtual scene generated by the virtual camera.

Abstract: System and methods are provided for a non-coherent point tracking process that allows unknown camera motion to be estimated. One or more edges can be identified in images captured by a camera when shooting a scene. For each of the identified edge in the images, at least one tracking object can be placed arbitrarily on the edge. The positions of tracking objects in the images can then be used to estimate a camera motion. In some embodiments, two tracking objects can be placed arbitrarily on the edge to represent the edge and move along the edge arbitrarily from image to image where the edge appears. Multiple of such edges can be identified in the images and camera motions in multiple directions can be estimated based on the identified edges and combined to obtain a combined camera motion indicating the camera's movement in a 3D space when shooting the scene.

Abstract: A method may include rendering a first view of a three-dimensional (3-D) virtual scene comprising a view of first content being displayed on a virtual display device from a location in the 3-D virtual scene. The method may also include rendering a second view comprising one or more content objects. The second view may be rendered from the location in the 3-D virtual scene, and the second scene may include a view of the display device as would be seen through a pair of augmented-reality glasses that display the one or more content objects. The method may additionally include generating a composite view by combing the first view and the second view. The method may further include causing the composite view to be displayed on a virtual-reality headset.

Abstract: In some embodiments a method of transferring facial expressions from a subject to a computer-generated character is provided where the method includes receiving positional information from a motion capture session of the subject representing a performance having facial expressions to be transferred to the computer-generated character, receiving a first animation model that represents the subject, and receiving a second animation model that represents the computer-generated character. Each of the first and second animation models can include a plurality of adjustable controls that define geometries of the model and that can be adjusted to present different facial expressions on the model, and where the first and second animation models are designed so that setting the same values for the same set of adjustable controls in each model generates a similar facial poses on the models.

Abstract: Systems and techniques are provided for transferring changes in animation data between geometric models of a character having different resolutions. For example, systems and techniques are provided for transferring changes in geometric properties between the geometric models. A method may include obtaining a first geometric model of the character and a second geometric model of the character, the geometric models having different resolutions with different numbers of data points. The method may further include determining one or more correspondences between data points of the first geometric model and data points of the second geometric model. The correspondences include one or more data points of the first geometric model that overlap with one or more data points of the second geometric model.

Abstract: Systems and techniques for dynamically capturing and reconstructing lighting are provided. The systems and techniques may be based on a stream of images capturing the lighting within an environment as a scene is shot. Reconstructed lighting data may be used to illuminate a character in a computer-generated environment as the scene is shot. For example, a method may include receiving a stream of images representing lighting of a physical environment. The method may further include compressing the stream of images to reduce an amount of data used in reconstructing the lighting of the physical environment and may further include outputting the compressed stream of images for reconstructing the lighting of the physical environment using the compressed stream, the reconstructed lighting being used to render a computer-generated environment.

Abstract: Methods and systems efficiently apply known distortion, such as of a camera and lens, to source image data to produce data of an output image with the distortion. In an embodiment, an output image field is segmented into regions so that on each segment the distortion function is approximately linear, and segmentation data is stored in a quadtree. The distortion function is applied to the segmented image field to produce a segmented rendered distortion image (SRDI) and a corresponding look-up table. To distort a source image, a location in the output image field is selected, and the uniquely colored segment at the same location in the SRDI is found. The look-up table provides the local linear inverse of the distortion function, which is applied to determine from where in the source image to take image texture data for the distorted output image.

Abstract: Methods, systems and computer program products pertaining to simulating liquid bodies are presented. The subject matter of this document can be embodied in a method that includes obtaining one or more data arrays representing low frequency spatial features of a simulated volume of liquid, and up-sampling the one or more data arrays to produce corresponding high resolution data arrays. The method also includes obtaining procedural data representing high frequency spatial features of a simulated liquid surface, and modifying the one or more high resolution data arrays using the procedural data to produce corresponding modified data arrays that reflect both the high frequency and the low frequency spatial features of the simulated volume of liquid.

Abstract: A method of motion capture may include accessing a 3D model of a subject, and associating the 3D model of the subject with a 2D representation of the subject in a plurality of frames. The method may also include identifying a change to the 2D representation of the subject between two or more of the plurality of frames, and deforming the 3D model in a virtual 3D space. In some embodiments, the deforming may be based on the identified change to the 2D representation and at least one constraint restricting how the 3D model can be deformed.

Abstract: Among other aspects, on computer-implemented method includes: receiving at least one command in a computer system from a handheld device; positioning a virtual camera and controlling a virtual scene according to the command; and in response to the command, generating an output to the handheld device for displaying a view of the virtual scene as controlled on a display of the handheld device, the view captured by the virtual camera as positioned.

Abstract: A method may include presenting a scene from linear content on one or more display devices in an immersive environment, and receiving, from a user within the immersive environment, input to change an aspect of the scene. The method may also include accessing 3-D virtual scene information previously used to render the scene, and changing the 3-D virtual scene information according to the changed aspect of the scene. The method may additionally include rending the 3-D virtual scene to incorporate the changed aspect, and presenting the rendered scene in real time in the immersive user environment.

Abstract: A system includes a computing device that includes a memory configured to store instructions. The system also includes a processor configured to execute the instructions to perform a method that includes receiving multiple representations of an object. Each of the representations includes position information of the object and corresponds to an instance in time. For at least one of the representations, the method includes defining a contour that represents a movable silhouette of a surface feature of the object. The method also includes producing a deformable model of the surface of the object from the defined contour and from the at least one representation of the object.

Abstract: A method of compressing a deep image representation may include receiving a deep image, where the deep image may include multiple pixels, and where each pixel in the deep image may include multiple samples. The method may also include compressing the deep image by combining samples in each pixel that are associated with the same primitives. This process may be repeated on a pixel-by-pixel basis. Some embodiments may use primitive IDs to match pixels to primitives through the rendering and compositing process.

Abstract: An animation analyzer is configured to receive an animation sequence and to identify a subsample of the frames that are to be rendered. A rendering engine is configured to render the subsample of the frames. The rendering engine is further configured to identify the frames that have not been rendered and to generate in-betweens for the frames that have not been rendered. The rendering engine is further configured to assemble the subsample of frames and the in-betweens into a video sequence depicting the animation sequence.

Abstract: A method may include displaying, on one or more display devices in a virtual-reality environment, a visual representation of a 3-D virtual scene from the perspective of a subject location in the virtual-reality environment. The method may also include displaying, on the one or more display devices, a chroma-key background with the visual representation. The method may further include recording, using a camera, an image of the subject in the virtual-reality environment against the chroma-key background.

Abstract: Methods are disclosed for the computer generation of data for images that include hair, fur, or other strand-like material. A volume for the hair is specified, having a plurality of surfaces. A fluid flow simulation is performed within the volume, with a first surface of the volume being a source area through which fluid is simulated to enter the volume, and a second surface being an exit surface through which fluid is simulated as exiting the volume. The fluid flow simulation may be used to produce fluid flow lines, such as from a velocity vector field for the fluid. Fluid flow lines are selected, and image data of hairs that follow the fluid flow lines are generated. Other embodiments include generating animation sequences by generating images wherein the volume and surfaces vary between frames.

Abstract: A method may include receiving a plurality of objects from a 3-D virtual scene. The plurality of objects may be arranged in a hierarchy. The method may also include generating a plurality of identifiers for the plurality of objects. The plurality of identifiers may include a first identifier for a first object in the plurality of objects, and the identifier may be generated based on a position of the first object in the hierarchy. The method may additionally include performing a rendering operation on the plurality of objects to generate a deep image. The deep image may include a plurality of samples that correspond to the first object. The method may further include propagating the plurality of identifiers through the rendering operation such that each of the plurality of samples in the deep image that correspond to the first object are associated with the identifier.

Abstract: A method of rendering stereo images includes receiving a first image of an object comprising rendered color information. The first image of the object is rendered using a first camera position. The method also includes receiving a second image of the object. The second image of the object is rendered using a second camera position, and pixels in the second image comprise locations of corresponding pixels in the first image. The method additionally includes coloring the pixels in the second image using colors of the corresponding pixels in the first image. The second image and the third image are stereo images for a 3-D presentation.

Abstract: A handheld device includes: an input control configured to control and modify a virtual scene including a virtual camera; and a display that shows a representation of the controlled and modified virtual scene generated by the virtual camera. A system includes: a computer system configured to execute program instructions for generating a virtual scene including a virtual camera; and handheld device configured to communicate with the computer system for controlling and modifying the virtual scene, the handheld device comprising: an input control configured to control and modify the virtual scene; and a display that shows a representation of the controlled and modified virtual scene generated by the virtual camera.

Abstract: A multi-channel tracking pattern is provided along with techniques and systems for performing motion capture using the multi-channel tracking pattern. The multi-channel tracking pattern includes a plurality of shapes having different colors on different portions of the pattern. The portions with the unique shapes and colors allow a motion capture system to track motion of an object bearing the pattern across a plurality of video frames.