Patents by Inventor Pushmeet Kohli

Pushmeet Kohli has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20160127715
    Abstract: Model fitting from raw time of flight image data is described, for example, to track position and orientation of a human hand or other entity. In various examples, raw image data depicting the entity is received from a time of flight camera. A 3D model of the entity is accessed and used to render, from the 3D model, simulations of raw time of flight image data depicting the entity in a specified pose/shape. The simulated raw image data and at least part of the received raw image data are compared and on the basis of the comparison, parameters of the entity are computed.
    Type: Application
    Filed: October 30, 2014
    Publication date: May 5, 2016
    Inventors: Jamie Daniel Joseph Shotton, Toby Sharp, Jonathan James Taylor, Pushmeet Kohli, Shahram Izadi, Andrew William Fitzgibbon, Reinhard Sebastian Bernhard Nowozin
  • Publication number: 20160104031
    Abstract: Region of interest detection in raw time of flight images is described. For example, a computing device receives at least one raw image captured for a single frame by a time of flight camera. The raw image depicts one or more objects in an environment of the time of flight camera (such as human hands, bodies or any other objects). The raw image is input to a trained region detector and in response one or more regions of interest in the raw image are received. A received region of interest comprises image elements of the raw image which are predicted to depict at least part of one of the objects. A depth computation logic computes depth from the one or more regions of interest of the raw image.
    Type: Application
    Filed: October 14, 2014
    Publication date: April 14, 2016
    Inventors: Jamie Daniel Joseph Shotton, Cem Keskin, Christoph Rhemann, Toby Sharp, Duncan Paul Robertson, Pushmeet Kohli, Andrew William Fitzgibbon, Shahram Izadi
  • Publication number: 20160104070
    Abstract: An inference engine is described for efficient machine learning. For example, an inference engine executes a plurality of ordered steps to carry out inference on the basis of observed data. For each step, a plurality of inputs to the step are received. A predictor predicts an output of the step and computes uncertainty of the prediction. Either the predicted output or a known output is selected on the basis of the uncertainty. If the known output is selected, the known output is computed, (for example, using a resource intensive, accurate process). The predictor is retrained using the known output and the plurality of inputs of the step as training data. For example, computing the prediction is fast and efficient as compared with computing the known output.
    Type: Application
    Filed: October 14, 2014
    Publication date: April 14, 2016
    Inventors: Seyed Mohammadali Eslami, Daniel Stefan Tarlow, Pushmeet Kohli, John Winn
  • Publication number: 20160085310
    Abstract: Tracking hand or body pose from image data is described, for example, to control a game system, natural user interface or for augmented reality. In various examples a prediction engine takes a single frame of image data and predicts a distribution over a pose of a hand or body depicted in the image data. In examples, a stochastic optimizer has a pool of candidate poses of the hand or body which it iteratively refines, and samples from the predicted distribution are used to replace some candidate poses in the pool. In some examples a best candidate pose from the pool is selected as the current tracked pose and the selection processes uses a 3D model of the hand or body.
    Type: Application
    Filed: September 23, 2014
    Publication date: March 24, 2016
    Inventors: Jamie Daniel Joseph Shotton, Cem Keskin, Jonathan James Taylor, Toby Sharp, Shahram Izadi, Andrew William Fitzgibbon, Pushmeet Kohli, Duncan Paul Robertson
  • Publication number: 20160034840
    Abstract: Crowdsourcing using active learning is described, for example, to select pairs of tasks and groups of workers so that information gained about answers to the tasks in the pool is optimized. In various examples a machine learning system learns variables describing characteristics of communities of workers, characteristics of workers, task variables and uncertainty of these variables. In various examples, the machine learning system predicts task variables and uncertainty of the predicted task variables for possible combinations of communities of workers and tasks. In examples the predicted variables and uncertainty are used to calculate expected information gain of the possible combinations and to rank the possible combinations. In examples, the crowdsourcing system uses the expected information gain to allocate tasks to worker communities and observe the results; the results may then be used to update the machine learning system.
    Type: Application
    Filed: July 31, 2014
    Publication date: February 4, 2016
    Inventors: Matteo Venanzi, John Philip Guiver, Pushmeet Kohli
  • Patent number: 9247238
    Abstract: Systems and methods for reducing interference between multiple infra-red depth cameras are described. In an embodiment, the system comprises multiple infra-red sources, each of which projects a structured light pattern into the environment. A controller is used to control the sources in order to reduce the interference caused by overlapping light patterns. Various methods are described including: cycling between the different sources, where the cycle used may be fixed or may change dynamically based on the scene detected using the cameras; setting the wavelength of each source so that overlapping patterns are at different wavelengths; moving source-camera pairs in independent motion patterns; and adjusting the shape of the projected light patterns to minimize overlap. These methods may also be combined in any way. In another embodiment, the system comprises a single source and a mirror system is used to cast the projected structured light pattern around the environment.
    Type: Grant
    Filed: January 31, 2011
    Date of Patent: January 26, 2016
    Assignee: Microsoft Technology Licensing, LLC
    Inventors: Shahram Izadi, David Molyneaux, Otmar Hilliges, David Kim, Jamie Daniel Joseph Shotton, Stephen Edward Hodges, David Alexander Butler, Andrew Fitzgibbon, Pushmeet Kohli
  • Patent number: 9242171
    Abstract: Real-time camera tracking using depth maps is described. In an embodiment depth map frames are captured by a mobile depth camera at over 20 frames per second and used to dynamically update in real-time a set of registration parameters which specify how the mobile depth camera has moved. In examples the real-time camera tracking output is used for computer game applications and robotics. In an example, an iterative closest point process is used with projective data association and a point-to-plane error metric in order to compute the updated registration parameters. In an example, a graphics processing unit (GPU) implementation is used to optimize the error metric in real-time. In some embodiments, a dense 3D model of the mobile camera environment is used.
    Type: Grant
    Filed: February 23, 2013
    Date of Patent: January 26, 2016
    Assignee: Microsoft Technology Licensing, LLC
    Inventors: Richard Newcombe, Shahram Izadi, David Molyneaux, Otmar Hilliges, David Kim, Jamie Daniel Joseph Shotton, Pushmeet Kohli, Andrew Fitzgibbon, Stephen Edward Hodges, David Alexander Butler
  • Publication number: 20160019711
    Abstract: Surface reconstruction contour completion embodiments are described which provide dense reconstruction of a scene from images captured from one or more viewpoints. Both a room layout and the full extent of partially occluded objects in a room can be inferred using a Contour Completion Random Field model to augment a reconstruction volume. The augmented reconstruction volume can then be used by any surface reconstruction pipeline to show previously occluded objects and surfaces.
    Type: Application
    Filed: September 28, 2015
    Publication date: January 21, 2016
    Inventors: Lior Shapira, Ran Gal, Eyal Ofek, Pushmeet Kohli, Nathan Silberman
  • Publication number: 20150356774
    Abstract: A “Layout Optimizer” provides various real-time iterative constraint-satisfaction methodologies that use constraint-based frameworks to generate optimized layouts that map or embed virtual objects into environments. The term environment refers to combinations of environmental characteristics, including, but not limited to, 2D or 3D scene geometry or layout, scene colors, patterns, and/or textures, scene illumination, scene heat sources, fixed or moving people, objects or fluids, etc., any of which may evolve or change over time. A set of parameters are specified or selected for each object. Further, the environmental characteristics are determined automatically or specified by users. Relationships between objects and/or the environment derived from constraints associated with objects and the environment are then used to iteratively determine optimized self-consistent and scene-consistent object layouts.
    Type: Application
    Filed: June 9, 2014
    Publication date: December 10, 2015
    Inventors: Ran Gal, Pushmeet Kohli, Eyal Ofek, Lior Shapira
  • Publication number: 20150347846
    Abstract: Tracking using sensor data is described, for example, where a plurality of machine learning predictors are used to predict a plurality of complementary, or diverse, parameter values of a process describing how the sensor data arises. In various examples a selector selects which of the predicted values are to be used, for example, to control a computing device. In some examples the tracked parameter values are pose of a moving camera or pose of an object moving in the field of view of a static camera; in some examples the tracked parameter values are of a 3D model of a hand or other articulated or deformable entity. The machine learning predictors have been trained in series, with training examples being reweighted after training an individual predictor, to favour training examples on which the set of predictors already trained performs poorly.
    Type: Application
    Filed: June 2, 2014
    Publication date: December 3, 2015
    Applicant: Microsoft Corporation
    Inventors: Abner GUZMÁN-RIVERA, Pushmeet KOHLI, Benjamin Michael GLOCKER, Jamie Daniel Joseph SHOTTON, Shahram IZADI, Toby SHARP, Andrew William FITZGIBBON
  • Patent number: 9171403
    Abstract: Surface reconstruction contour completion embodiments are described which provide dense reconstruction of a scene from images captured from one or more viewpoints. Both a room layout and the full extent of partially occluded objects in a room can be inferred using a Contour Completion Random Field model to augment a reconstruction volume. The augmented reconstruction volume can then be used by any surface reconstruction pipeline to show previously occluded objects and surfaces.
    Type: Grant
    Filed: February 13, 2014
    Date of Patent: October 27, 2015
    Assignee: Microsoft Technology Licensing, LLC
    Inventors: Lior Shapira, Ran Gal, Eyal Ofek, Pushmeet Kohli, Nathan Silberman
  • Publication number: 20150302317
    Abstract: Non-greedy machine learning for high accuracy is described, for example, where one or more random decision trees are trained for gesture recognition in order to control a computing-based device. In various examples, a random decision tree or directed acyclic graph (DAG) is grown using a greedy process and is then post-processed to recalculate, in a non-greedy process, leaf node parameters and split function parameters of internal nodes of the graph. In various examples the very large number of options to be assessed by the non-greedy process is reduced by using a constrained objective function. In examples the constrained objective function takes into account a binary code denoting decisions at split nodes of the tree or DAG. In examples, resulting trained decision trees are more compact and have improved generalization and accuracy.
    Type: Application
    Filed: April 22, 2014
    Publication date: October 22, 2015
    Applicant: Microsoft Corporation
    Inventors: Mohammad Norouzi, Pushmeet Kohli
  • Publication number: 20150296152
    Abstract: Filtering sensor data is described, for example, where filters conditioned on a local appearance of the signal are predicted by a machine learning system, and used to filter the sensor data. In various examples the sensor data is a stream of noisy video image data and the filtering process denoises the video stream. In various examples the sensor data is a depth image and the filtering process refines the depth image which may then be used for gesture recognition or other purposes. In various examples the sensor data is one dimensional measurement data from an electric motor and the filtering process denoises the measurements. In examples the machine learning system comprises a random decision forest where trees of the forest store filters at their leaves. In examples, the random decision forest is trained using a training objective with a data dependent regularization term.
    Type: Application
    Filed: April 14, 2014
    Publication date: October 15, 2015
    Inventors: Sean Ryan Francesco Fanello, Cem Keskin, Pushmeet Kohli, Shahram Izadi, Jamie Daniel Joseph Shotton, Antonio Criminisi
  • Publication number: 20150248765
    Abstract: A method of sensing depth using an RGB camera. In an example method, a color image of a scene is received from an RGB camera. The color image is applied to a trained machine learning component which uses features of the image elements to assign all or some of the image elements a depth value which represents the distance between the surface depicted by the image element and the RGB camera. In various examples, the machine learning component comprises one or more entangled geodesic random decision forests.
    Type: Application
    Filed: February 28, 2014
    Publication date: September 3, 2015
    Applicant: Microsoft Corporation
    Inventors: Antonio Criminisi, Duncan Paul Robertson, Peter Kontschieder, Pushmeet Kohli, Henrik Turbell, Adriana Dumitras, Indeera Munasinghe, Jamie Daniel Joseph Shotton
  • Publication number: 20150248764
    Abstract: A method of sensing depth using an infrared camera. In an example method, an infrared image of a scene is received from an infrared camera. The infrared image is applied to a trained machine learning component which uses the intensity of image elements to assign all or some of the image elements a depth value which represents the distance between the surface depicted by the image element and the infrared camera. In various examples, the machine line component comprises one or more random decision forests.
    Type: Application
    Filed: February 28, 2014
    Publication date: September 3, 2015
    Inventors: Cem Keskin, Sean Ryan Francesco Fanello, Shahram Izadi, Pushmeet Kohli, David Kim, David Sweeney, Jamie Daniel Joesph Shotton, Duncan Paul Robertson, Sing Bing Kang
  • Patent number: 9117281
    Abstract: Surface segmentation from RGB and depth images is described. In one example, a computer receives an image of a scene. The image has pixels which each have an associated color value and an associated depth value representing a distance between from an image sensor to a surface in the scene. The computer uses the depth values to derive a set of three-dimensional planes present within the scene. A cost function is used to determine whether each pixel belongs to one of the planes, and the image elements are labeled accordingly. The cost function has terms dependent on the depth value of a pixel, and the color values of the pixels and at least one neighboring pixel. In various examples, the planes can be extended until they intersect to determine the extent of the scene, and pixels not belonging to a plane can be labeled as objects on the surfaces.
    Type: Grant
    Filed: November 2, 2011
    Date of Patent: August 25, 2015
    Assignee: Microsoft Corporation
    Inventors: Derek Hoiem, Pushmeet Kohli
  • Publication number: 20150228114
    Abstract: Surface reconstruction contour completion embodiments are described which provide dense reconstruction of a scene from images captured from one or more viewpoints. Both a room layout and the full extent of partially occluded objects in a room can be inferred using a Contour Completion Random Field model to augment a reconstruction volume. The augmented reconstruction volume can then be used by any surface reconstruction pipeline to show previously occluded objects and surfaces.
    Type: Application
    Filed: February 13, 2014
    Publication date: August 13, 2015
    Applicant: Microsoft Corporation
    Inventors: Lior Shapira, Ran Gal, Eyal Ofek, Pushmeet Kohli, Nathan Silberman
  • Publication number: 20150213360
    Abstract: Crowdsourcing systems with machine learning are described, for example, to aggregate answers to a crowdsourced task in a manner achieving good accuracy even where observed data about past behavior of crowd members is sparse. In various examples a machine learning system jointly learns variables describing characteristics of both individual crowd workers and communities of the workers. In various examples, the machine learning system learns aggregated labels. In examples learnt variables describing characteristics of an individual crowd worker are related, by addition of noise, to learnt variables describing characteristics of a community of which the individual is a member. In examples the crowdsourcing system uses the learnt variables describing characteristics of individual workers and of communities of workers for any one or more of: active learning, targeted training of workers, targeted issuance of tasks, calculating and issuing rewards.
    Type: Application
    Filed: January 24, 2014
    Publication date: July 30, 2015
    Applicant: Microsoft Corporation
    Inventors: Matteo Venanzi, John Philip Guiver, Gabriella Kazai, Pushmeet Kohli, Milad Shokouhi
  • Patent number: 9070047
    Abstract: A tractable model solves certain labeling problems by providing potential functions having arbitrary dependencies upon an observed dataset (e.g., image data). The model uses decision trees corresponding to various factors to map dataset content to a set of parameters used to define the potential functions in the model. Some factors define relationships among multiple variable nodes. When making label predictions on a new dataset, the leaf nodes of the decision tree determine the effective weightings for such potential functions. In this manner, decision trees define non-parametric dependencies and can represent rich, arbitrary functional relationships if sufficient training data is available. Decision trees training is scalable, both in the training set size and by parallelization. Maximum pseudolikelihood learning can provide for joint training of aspects of the model, including feature test selection and ordering, factor weights, and the scope of the interacting variable nodes used in the graph.
    Type: Grant
    Filed: December 27, 2011
    Date of Patent: June 30, 2015
    Assignee: Microsoft Technology Licensing, LLC
    Inventors: Reinhard Sebastian Bernhard Nowozin, Carsten Curt Eckard Rother, Bangpeng Yao, Toby Leonard Sharp, Pushmeet Kohli
  • Patent number: 9053571
    Abstract: Generating computer models of 3D objects is described. In one example, depth images of an object captured by a substantially static depth camera are used to generate the model, which is stored in a memory device in a three-dimensional volume. Portions of the depth image determined to relate to the background are removed to leave a foreground depth image. The position and orientation of the object in the foreground depth image is tracked by comparison to a preceding depth image, and the foreground depth image is integrated into the volume by using the position and orientation to determine where to add data derived from the foreground depth image into the volume. In examples, the object is hand-rotated by a user before the depth camera. Hands that occlude the object are integrated out of the model as they do not move in sync with the object due to re-gripping.
    Type: Grant
    Filed: June 6, 2011
    Date of Patent: June 9, 2015
    Assignee: Microsoft Corporation
    Inventors: Jamie Daniel Joseph Shotton, Shahram Izadi, Otmar Hilliges, David Kim, David Molyneaux, Pushmeet Kohli, Andrew Fitzgibbon, Stephen Edward Hodges