Abstract: The techniques described herein disclose a system that is configured to detect and track the three-dimensional pose of an object (e.g., a head-mounted display device) in a color image using an accessible three-dimensional model of the object. The system uses the three-dimensional pose of the object to repair pixel depth values associated with a region (e.g., a surface) of the object that is composed of material that absorbs light emitted by a time-of-flight depth sensor to determine depth. Consequently, a color-depth image (e.g., a Red-Green-Blue-Depth image or RGB-D image) can be produced that does not include dark holes on and around the region of the object that is composed of material that absorbs light emitted by the time-of-flight depth sensor.

Abstract: Keypoints are predicted in an image. Predictions are generated for each of the keypoints of an image as a 2D random variable, normally distributed with location (x, y) and standard deviation sigma. A neural network is trained to maximize a log-likelihood that samples from each of the predicted keypoints equal a ground truth. The trained neural network is used to predict keypoints of an image without generating a heatmap.

Abstract: Memory facilitation using directed acyclic graphs is described, for example, where a plurality of directed acyclic graphs are trained for gesture recognition from human skeletal data, or to estimate human body joint positions from depth images for gesture detection. In various examples directed acyclic graphs are grown during training using a training objective which takes into account both connection patterns between nodes and split function parameter values. For example, a layer of child nodes is grown and connected to a parent layer of nodes using an initialization strategy. In examples, various local search processes are used to find good combinations of connection patterns and split function parameters.

Abstract: Region of interest detection in raw time of flight images is described. For example, a computing device receives at least one raw image captured for a single frame by a time of flight camera. The raw image depicts one or more objects in an environment of the time of flight camera (such as human hands, bodies or any other objects). The raw image is input to a trained region detector and in response one or more regions of interest in the raw image are received. A received region of interest comprises image elements of the raw image which are predicted to depict at least part of one of the objects. A depth computation logic computes depth from the one or more regions of interest of the raw image.

Abstract: A tracker is described which comprises an input configured to receive captured sensor data depicting an object. The tracker has a processor configured to access a rigged, smooth-surface model of the object and to compute values of pose parameters of the model by calculating an optimization to fit the model to data related to the captured sensor data. Variables representing correspondences between the data and the model are included in the optimization jointly with the pose parameters.

Abstract: Model fitting from raw time of flight image data is described, for example, to track position and orientation of a human hand or other entity. In various examples, raw image data depicting the entity is received from a time of flight camera. A 3D model of the entity is accessed and used to render, from the 3D model, simulations of raw time of flight image data depicting the entity in a specified pose/shape. The simulated raw image data and at least part of the received raw image data are compared and on the basis of the comparison, parameters of the entity are computed.

Abstract: Examples describe an apparatus for calibrating a three dimensional (3D) mesh model of an articulated object. The articulated object is an instance of a specified object class. The apparatus comprises an input configured to receive captured sensor data depicting the object. The apparatus has a calibration engine configured to compute values of shape parameters of the 3D mesh model which indicate which member of the object class is depicted in the captured sensor data, in order to calibrate the 3D mesh model. The calibration engine is configured to compute the values of the shape parameters with an optimization process to find at least one potential local or global minimum of an energy function, the energy function expressing a degree of similarity between data rendered from the model and the received sensor data.

Abstract: Tracking hand or body pose from image data is described, for example, to control a game system, natural user interface or for augmented reality. In various examples a prediction engine takes a single frame of image data and predicts a distribution over a pose of a hand or body depicted in the image data. In examples, a stochastic optimizer has a pool of candidate poses of the hand or body which it iteratively refines, and samples from the predicted distribution are used to replace some candidate poses in the pool. In some examples a best candidate pose from the pool is selected as the current tracked pose and the selection processes uses a 3D model of the hand or body.

Abstract: Region of interest detection in raw time of flight images is described. For example, a computing device receives at least one raw image captured for a single frame by a time of flight camera. The raw image depicts one or more objects in an environment of the time of flight camera (such as human hands, bodies or any other objects). The raw image is input to a trained region detector and in response one or more regions of interest in the raw image are received. A received region of interest comprises image elements of the raw image which are predicted to depict at least part of one of the objects. A depth computation logic computes depth from the one or more regions of interest of the raw image.

Abstract: Region of interest detection in raw time of flight images is described. For example, a computing device receives at least one raw image captured for a single frame by a time of flight camera. The raw image depicts one or more objects in an environment of the time of flight camera (such as human hands, bodies or any other objects). The raw image is input to a trained region detector and in response one or more regions of interest in the raw image are received. A received region of interest comprises image elements of the raw image which are predicted to depict at least part of one of the objects. A depth computation logic computes depth from the one or more regions of interest of the raw image.

Abstract: Examples describe an apparatus for calibrating a three dimensional (3D) mesh model of an articulated object. The articulated object is an instance of a specified object class. The apparatus comprises an input configured to receive captured sensor data depicting the object. The apparatus has a calibration engine configured to compute values of shape parameters of the 3D mesh model which indicate which member of the object class is depicted in the captured sensor data, in order to calibrate the 3D mesh model. The calibration engine is configured to compute the values of the shape parameters with an optimization process to find at least one potential local or global minimum of an energy function, the energy function expressing a degree of similarity between data rendered from the model and the received sensor data.

Abstract: A tracker is described which comprises an input configured to receive captured sensor data depicting an object. The tracker has a processor configured to access a rigged, smooth-surface model of the object and to compute values of pose parameters of the model by calculating an optimization to fit the model to data related to the captured sensor data. Variables representing correspondences between the data and the model are included in the optimization jointly with the pose parameters.

Abstract: Memory facilitation using directed acyclic graphs is described, for example, where a plurality of directed acyclic graphs are trained for gesture recognition from human skeletal data, or to estimate human body joint positions from depth images for gesture detection. In various examples directed acyclic graphs are grown during training using a training objective which takes into account both connection patterns between nodes and split function parameter values. For example, a layer of child nodes is grown and connected to a parent layer of nodes using an initialization strategy. In examples, various local search processes are used to find good combinations of connection patterns and split function parameters.

Abstract: Tracking hand or body pose from image data is described, for example, to control a game system, natural user interface or for augmented reality. In various examples a prediction engine takes a single frame of image data and predicts a distribution over a pose of a hand or body depicted in the image data. In examples, a stochastic optimizer has a pool of candidate poses of the hand or body which it iteratively refines, and samples from the predicted distribution are used to replace some candidate poses in the pool. In some examples a best candidate pose from the pool is selected as the current tracked pose and the selection processes uses a 3D model of the hand or body.

Abstract: Tracking using sensor data is described, for example, where a plurality of machine learning predictors are used to predict a plurality of complementary, or diverse, parameter values of a process describing how the sensor data arises. In various examples a selector selects which of the predicted values are to be used, for example, to control a computing device. In some examples the tracked parameter values are pose of a moving camera or pose of an object moving in the field of view of a static camera; in some examples the tracked parameter values are of a 3D model of a hand or other articulated or deformable entity. The machine learning predictors have been trained in series, with training examples being reweighted after training an individual predictor, to favor training examples on which the set of predictors already trained performs poorly.

Abstract: Depth image compression is described for example, to enable body-part centers of players of a game to be detected in real time from depth images or for other applications such as augmented reality, and human-computer interaction. In an embodiment, depth images which have associated body-part probabilities, are compressed using probability mass which is related to the depth of an image element and a probability of a body part for the image element. In various examples, compression of the depth images using probability mass enables body part center detection, by clustering output elements, to be speeded up. In some examples, the scale of the compression is selected according to a depth of a foreground region and in some cases different scales are used for different image regions. In some examples, certainties of the body-part centers are calculated using probability masses of clustered image elements.

Abstract: Tracking hand or body pose from image data is described, for example, to control a game system, natural user interface or for augmented reality. In various examples a prediction engine takes a single frame of image data and predicts a distribution over a pose of a hand or body depicted in the image data. In examples, a stochastic optimizer has a pool of candidate poses of the hand or body which it iteratively refines, and samples from the predicted distribution are used to replace some candidate poses in the pool. In some examples a best candidate pose from the pool is selected as the current tracked pose and the selection processes uses a 3D model of the hand or body.

Abstract: Memory facilitation using directed acyclic graphs is described, for example, where a plurality of directed acyclic graphs are trained for gesture recognition from human skeletal data, or to estimate human body joint positions from depth images for gesture detection. In various examples directed acyclic graphs are grown during training using a training objective which takes into account both connection patterns between nodes and split function parameter values. For example, a layer of child nodes is grown and connected to a parent layer of nodes using an initialization strategy. In examples, various local search processes are used to find good combinations of connection patterns and split function parameters.

Abstract: Model fitting from raw time of flight image data is described, for example, to track position and orientation of a human hand or other entity. In various examples, raw image data depicting the entity is received from a time of flight camera. A 3D model of the entity is accessed and used to render, from the 3D model, simulations of raw time of flight image data depicting the entity in a specified pose/shape. The simulated raw image data and at least part of the received raw image data are compared and on the basis of the comparison, parameters of the entity are computed.

Abstract: Region of interest detection in raw time of flight images is described. For example, a computing device receives at least one raw image captured for a single frame by a time of flight camera. The raw image depicts one or more objects in an environment of the time of flight camera (such as human hands, bodies or any other objects). The raw image is input to a trained region detector and in response one or more regions of interest in the raw image are received. A received region of interest comprises image elements of the raw image which are predicted to depict at least part of one of the objects. A depth computation logic computes depth from the one or more regions of interest of the raw image.