Abstract: The described implementations relate to images and depth information and generating useful information from the images and depth information. One example can identify planes in a semantically-labeled 3D voxel representation of a scene. The example can infer missing information by extending planes associated with structural elements of the scene. The example can also generate a watertight manifold representation of the scene at least in part from the inferred missing information.

Abstract: Improved techniques for performing object segmentation are disclosed. Surface reconstruction (SR) data corresponding to an environment is accessed. This SR data is used to generate a detailed three-dimensional (3D) representation of the environment. The SR data is also used to infer a high-level 3D structural representation of the environment. The high-level 3D structural representation is inferred using machine learning that is performed on the surface reconstruction data to identify a structure of the environment. The high-level 3D structural representation is then cut from the detailed 3D representation. This cutting process generates a clutter mesh comprising objects that remain after the cut and that are distinct from the structure. Object segmentation is then performed on the remaining objects to identify those objects.

Abstract: Improved techniques for performing object segmentation are disclosed. Surface reconstruction (SR) data corresponding to an environment is accessed. This SR data is used to generate a detailed three-dimensional (3D) representation of the environment. The SR data is also used to infer a high-level 3D structural representation of the environment. The high-level 3D structural representation is inferred using machine learning that is performed on the surface reconstruction data to identify a structure of the environment. The high-level 3D structural representation is then cut from the detailed 3D representation. This cutting process generates a clutter mesh comprising objects that remain after the cut and that are distinct from the structure. Object segmentation is then performed on the remaining objects to identify those objects.

Abstract: The described implementations relate to images and depth information and generating useful information from the images and depth information. One example can identify planes in a semantically-labeled 3D voxel representation of a scene. The example can infer missing information by extending planes associated with structural elements of the scene. The example can also generate a watertight manifold representation of the scene at least in part from the inferred missing information.

Abstract: Examples are disclosed herein that relate to calibrating a user's eye for a stereoscopic display. One example provides, on a head-mounted display device including a see-through display, a method of calibrating a stereoscopic display for a user's eyes, the method including for a first eye, receiving an indication of alignment of a user-controlled object with a first eye reference object viewable via the head-mounted display device from a perspective of the first eye, determining a first ray intersecting the user-controlled object and the first eye reference object from the perspective of the first eye, and determining a position of the first eye based on the first ray. The method further includes repeating such steps for a second eye, determining a position of the second eye based on a second ray, and calibrating the stereoscopic display based on the position of the first eye and the position of the second eye.

Abstract: Examples are disclosed herein that relate to calibrating a user's eye for a stereoscopic display. One example provides, on a head-mounted display device including a see-through display, a method of calibrating a stereoscopic display for a user's eyes, the method including for a first eye, receiving an indication of alignment of a user-controlled object with a first eye reference object viewable via the head-mounted display device from a perspective of the first eye, determining a first ray intersecting the user-controlled object and the first eye reference object from the perspective of the first eye, and determining a position of the first eye based on the first ray. The method further includes repeating such steps for a second eye, determining a position of the second eye based on a second ray, and calibrating the stereoscopic display based on the position of the first eye and the position of the second eye.

Abstract: Pose and gesture detection and classification of a human poses and gestures using a discriminative ferns ensemble classifier is provided. Sample image data in one or more channels includes a human image. A processing device operates on the sample image data using the discriminative ferns ensemble classifier. The classifier has set of classification tables and matching bit features (ferns) which are developed using a first set of training data and optimized by a weighting of the tables using an SVM linear classifier configured based on the first or a second set of pose training data. The tables allow computation of a score per pose class for the image in the sample data and the processor outputs a determination of the pose in the sample depth image data. The determination enables the manipulation of a natural user interface.

Abstract: A system and method are disclosed for calibrating a depth camera in a natural user interface. The system in general obtains an objective measurement of true distance between a capture device and one or more objects in a scene. The system then compares the true depth measurement to the depth measurement provided by the depth camera at one or more points and determines an error function describing an error in the depth camera measurement. The depth camera may then be recalibrated to correct for the error. The objective measurement of distance to one or more objects in a scene may be accomplished by a variety of systems and methods.

Abstract: Technology for testing a target recognition, analysis, and tracking system is provided. A searchable repository of recorded and synthesized depth clips and associated ground truth tracking data is provided. Data in the repository is used by one or more processing devices each including at least one instance of a target recognition, analysis, and tracking pipeline to analyze performance of the tracking pipeline. An analysis engine provides at least a subset of the searchable set responsive to a request to test the pipeline and receives tracking data output from the pipeline on the at least subset of the searchable set. A report generator outputs an analysis of the tracking data relative to the ground truth in the at least subset to provide an output of the error relative to the ground truth.

Abstract: Technology for testing a target recognition, analysis, and tracking system is provided. A searchable repository of recorded and synthesized depth clips and associated ground truth tracking data is provided. Data in the repository is used by one or more processing devices each including at least one instance of a target recognition, analysis, and tracking pipeline to analyze performance of the tracking pipeline. An analysis engine provides at least a subset of the searchable set responsive to a request to test the pipeline and receives tracking data output from the pipeline on the at least subset of the searchable set. A report generator outputs an analysis of the tracking data relative to the ground truth in the at least subset to provide an output of the error relative to the ground truth.

Abstract: Technology for testing a target recognition, analysis, and tracking system is provided. A searchable repository of recorded and synthesized depth clips and associated ground truth tracking data is provided. Data in the repository is used by one or more processing devices each including at least one instance of a target recognition, analysis, and tracking pipeline to analyze performance of the tracking pipeline. An analysis engine provides at least a subset of the searchable set responsive to a request to test the pipeline and receives tracking data output from the pipeline on the at least subset of the searchable set. A report generator outputs an analysis of the tracking data relative to the ground truth in the at least subset to provide an output of the error relative to the ground truth.

Abstract: Technology for testing a target recognition, analysis, and tracking system is provided. A searchable repository of recorded and synthesized depth clips and associated ground truth tracking data is provided. Data in the repository is used by one or more processing devices each including at least one instance of a target recognition, analysis, and tracking pipeline to analyze performance of the tracking pipeline. An analysis engine provides at least a subset of the searchable set responsive to a request to test the pipeline and receives tracking data output from the pipeline on the at least subset of the searchable set. A report generator outputs an analysis of the tracking data relative to the ground truth in the at least subset to provide an output of the error relative to the ground truth.

Abstract: A system and method are disclosed for calibrating a depth camera in a natural user interface. The system in general obtains an objective measurement of true distance between a capture device and one or more objects in a scene. The system then compares the true depth measurement to the depth measurement provided by the depth camera at one or more points and determines an error function describing an error in the depth camera measurement. The depth camera may then be recalibrated to correct for the error. The objective measurement of distance to one or more objects in a scene may be accomplished by a variety of systems and methods.