Abstract: A control system comprises a memory storing a sequence of sensor data received from one or more sensors. The control system has a processor which processes the sensor data to compute a sequence of derived sensor data values. An autoencoder receives the sequence of derived sensor data values and computes a forward prediction of the sequence of derived sensor data values, the autoencoder having been trained imposing a relationship on positions of the derived sensor data values encoded in a latent space of the autoencoder. A processor initiates control of an apparatus using the forward prediction.

Abstract: A method for virtually representing human body poses includes receiving positioning data detailing parameters of one or more body parts of a human user based at least in part on input from one or more sensors. One or more mapping constraints are maintained that relate a model articulated representation to a target articulated representation. A model pose of the model articulated representation and a target pose of the target articulated representation are concurrently estimated based at least in part on the positioning data and the one or more mapping constraints. The previously-trained pose optimization machine is trained with training positioning data having ground truth labels for the model articulated representation. The target articulated representation is output for display with the target pose as a virtual representation of the human user.

Abstract: A method for virtually representing human body poses includes receiving positioning data detailing parameters of one or more body parts of a human user based at least in part on input from one or more sensors. One or more mapping constraints are maintained that relate a model articulated representation to a target articulated representation. A model pose of the model articulated representation and a target pose of the target articulated representation are concurrently estimated based at least in part on the positioning data and the one or more mapping constraints. The previously-trained pose optimization machine is trained with training positioning data having ground truth labels for the model articulated representation. The target articulated representation is output for display with the target pose as a virtual representation of the human user.

Abstract: Keypoints are predicted in an image. Predictions are generated for each of the keypoints of an image as a 2D random variable, normally distributed with location (x, y) and standard deviation sigma. A neural network is trained to maximize a log-likelihood that samples from each of the predicted keypoints equal a ground truth. The trained neural network is used to predict keypoints of an image without generating a heatmap.

Abstract: Keypoints are predicted in an image. A neural network is executed that is configured to predict each of the keypoints as a 2D random variable, normally distributed with a 2D position and 2×2 covariance matrix. The neural network is trained to maximize a log-likelihood that samples from each of the predicted keypoints equal a ground truth. The trained neural network is used to predict keypoints of an image without generating a heatmap.

Abstract: A method for predicting the pose of an articulated object includes receiving spatial information for n joints of the articulated object. The spatial information for the n joints is passed to a machine learning model previously trained to receive spatial information for n+m joints as input, wherein m>=1. From the machine learning model, a pose prediction for the articulated object is received as output based at least on the spatial information for the n joints, and without spatial information for the m joints.

Abstract: In various examples there is an apparatus for aligning three-dimensional, 3D, representations of people. The apparatus comprises at least one processor and a memory storing instructions that, when executed by the at least one processor, perform a method comprising accessing a first 3D representation which is an instance of a parametric model of a person; accessing a second 3D representation which is a photoreal representation of the person; computing an alignment of the first and second 3D representations; and computing and storing a hologram from the aligned first and second 3D representations such that the hologram depicts parts of the person which are observed in only one of the first and second 3D representations; or controlling an avatar representing the person where the avatar depicts parts of the person which are observed in only one of the first and second 3D representations.

Abstract: A control system comprises a memory storing a sequence of sensor data received from one or more sensors. The control system has a processor which processes the sensor data to compute a sequence of derived sensor data values. An autoencoder receives the sequence of derived sensor data values and computes a forward prediction of the sequence of derived sensor data values, the autoencoder having been trained imposing a relationship on positions of the derived sensor data values encoded in a latent space of the autoencoder. A processor initiates control of an apparatus using the forward prediction.

Abstract: In various examples there is an apparatus for aligning three-dimensional, 3D, representations of people. The apparatus comprises at least one processor and a memory storing instructions that, when executed by the at least one processor, perform a method comprising accessing a first 3D representation which is an instance of a parametric model of a person; accessing a second 3D representation which is a photoreal representation of the person; computing an alignment of the first and second 3D representations; and computing and storing a hologram from the aligned first and second 3D representations such that the hologram depicts parts of the person which are observed in only one of the first and second 3D representations; or controlling an avatar representing the person where the avatar depicts parts of the person which are observed in only one of the first and second 3D representations.

Abstract: A control system comprises a memory storing a sequence of sensor data received from one or more sensors. The control system has a processor which processes the sensor data to compute a sequence of derived sensor data values. An autoencoder receives the sequence of derived sensor data values and computes a forward prediction of the sequence of derived sensor data values, the autoencoder having been trained imposing a relationship on positions of the derived sensor data values encoded in a latent space of the autoencoder. A processor initiates control of an apparatus using the forward prediction.

Abstract: A tracker is described which comprises a processor configured to receive captured sensor data depicting an object. The processor is configured to access a rigged polygon mesh model of the object and to compute a plurality of approximate surface normals of a limit surface of the rigged polygon mesh. The processor is configured to compute values of pose parameters of the model by calculating an optimization to fit the model to the captured sensor data where the optimization uses an evaluation function based on the plurality of approximate surface normals.

Abstract: There is an apparatus for detecting pose of an object. The apparatus comprises a processor configured to receive captured sensor data depicting the object. It also has a memory storing a parameterized model of a class of 3D shape of which the object is a member, where an instance of the model is given as a mapping from a point in a 2D rectangular grid to a 3D position. The processor is configured to compute values of the parameters of the model by calculating an optimization to fit the model to the captured sensor data, using the parametrized mapping. The processor is configured to output the computed values of the parameters comprising at least global position and global orientation of the object.

Abstract: A method of fitting a three dimensional (3D) model to input data is described. Input data comprises a 3D scan and associated appearance information. The 3D scan depicts a composite object having elements from at least two classes. A texture model is available which, given an input vector, computes, for each of the classes, a texture and a mask. A joint optimization is computed to find values of the input vector and values of parameters of the 3D model, where the optimization enforces that the 3D model, instantiated by the values of the parameters, gives a simulated texture which agrees with the input data in a region specified by the mask associated with the 3D model; such that the 3D model is fitted to the input data.

Abstract: In various examples there is an apparatus for detecting position and orientation of an object. The apparatus comprises a memory storing at least one frame of captured sensor data depicting the object. The apparatus also comprises a trained machine learning system configured to receive the frame of the sensor data and to compute a plurality of two dimensional positions in the frame. Each predicted two dimensional position is a position of sensor data in the frame depicting a keypoint, where a keypoint is a pre-specified 3D position relative to the object. At least one of the keypoints is a floating keypoint depicting a pre-specified position relative to the object, lying inside or outside the object's surface. The apparatus comprises a pose detector which computes the three dimensional position and orientation of the object using the predicted two dimensional positions and outputs the computed three dimensional position and orientation.

Abstract: An apparatus for detecting pose of an object is described. The apparatus has a processor configured to receive captured sensor data depicting the object. The apparatus has a memory storing a model of a class of object of which the depicted object is a member, the model comprising a plurality of parameters specifying the pose, comprising global position and global orientation, of the model. The processor is configured to compute values of the parameters of the model by calculating an optimization to fit the model to the captured sensor data, wherein the optimization comprises iterated computation of updates to the values of the parameters and updates to values of variables representing correspondences between the captured sensor data and the model, the updates being interdependent in computation. The processor is configured to discard updates to values of the variables representing correspondences without applying the updates.

Abstract: There is an apparatus for detecting pose of an object. The apparatus comprises a processor configured to receive captured sensor data depicting the object. It also has a memory storing a parameterized model of a class of 3D shape of which the object is a member, where an instance of the model is given as a mapping from a point in a 2D rectangular grid to a 3D position. The processor is configured to compute values of the parameters of the model by calculating an optimization to fit the model to the captured sensor data, using the parametrized mapping. The processor is configured to output the computed values of the parameters comprising at least global position and global orientation of the object.

Abstract: A control system comprises a memory storing a sequence of sensor data received from one or more sensors. The control system has a processor which processes the sensor data to compute a sequence of derived sensor data values. An autoencoder receives the sequence of derived sensor data values and computes a forward prediction of the sequence of derived sensor data values, the autoencoder having been trained imposing a relationship on positions of the derived sensor data values encoded in a latent space of the autoencoder. A processor initiates control of an apparatus using the forward prediction.

Abstract: A tracker is described which comprises a processor configured to receive captured sensor data depicting an object. The processor is configured to access a rigged polygon mesh model of the object and to compute a plurality of approximate surface normals of a limit surface of the rigged polygon mesh. The processor is configured to compute values of pose parameters of the model by calculating an optimization to fit the model to the captured sensor data where the optimization uses an evaluation function based on the plurality of approximate surface normals.

Abstract: An apparatus for detecting pose of an object is described. The apparatus has a processor configured to receive captured sensor data depicting the object. The apparatus has a memory storing a model of a class of object of which the depicted object is a member, the model comprising a plurality of parameters specifying the pose, comprising global position and global orientation, of the model. The processor is configured to compute values of the parameters of the model by calculating an optimization to fit the model to the captured sensor data, wherein the optimization comprises iterated computation of updates to the values of the parameters and updates to values of variables representing correspondences between the captured sensor data and the model, the updates being interdependent in computation. The processor is configured to discard updates to values of the variables representing correspondences without applying the updates.

Abstract: In various examples there is an apparatus for detecting position and orientation of an object. The apparatus comprises a memory storing at least one frame of captured sensor data depicting the object. The apparatus also comprises a trained machine learning system configured to receive the frame of the sensor data and to compute a plurality of two dimensional positions in the frame. Each predicted two dimensional position is a position of sensor data in the frame depicting a keypoint, where a keypoint is a pre-specified 3D position relative to the object. At least one of the keypoints is a floating keypoint depicting a pre-specified position relative to the object, lying inside or outside the object's surface. The apparatus comprises a pose detector which computes the three dimensional position and orientation of the object using the predicted two dimensional positions and outputs the computed three dimensional position and orientation.