Abstract: A system configured to improve the operations associated with generating virtual representations on limited resources of a mobile device. In some cases, the system may utilize viewpoint bundles that include collection of image data with an associated pose in relative physical proximity to each other to render a virtual scene. In other cases, the system may utilize 2.5D manifolds including 2D image data and a weighted depth value to render the 3D environment.

Abstract: A system configured to assist a user in scanning a physical environment in order to generate a three-dimensional scan or model. In some cases, the system may include an interface to assist the user in capturing data usable to determine a scale or depth of the physical environment and to perform a scan in a manner that minimizes gaps.

Abstract: A system configured to generate a three-dimensional scene of a physical environment. In some cases, the system may present the three-dimensional scene such that the user is able to view the model in various different types. The system may also allow the user to capture various measurements associated with the scanned physical environment by selecting different portions of the scene.

Abstract: A method for automatic registration of 3D image data, captured by a 3D image capture system having an RGB camera and a depth camera, includes capturing 2D image data with the RGB camera at a first pose; capturing depth data with the depth camera at the first pose; performing an initial registration of the RGB camera to the depth camera; capturing 2D image data with the RGB camera at a second pose; capturing depth data at the second pose; and calculating an updated registration of the RGB camera to the depth camera.

Abstract: A three-dimensional (3D) sensing apparatus together with a projector subassembly is provided. The 3D sensing apparatus includes two cameras, which may be configured to capture ultraviolet and/or near-infrared light. The 3D sensing apparatus may also contain an optical filter and one or more computing processors that signal a simultaneous capture using the two cameras and processing the captured images into depth. The projector subassembly of the 3D sensing apparatus includes a laser diode, one or optical elements, and a photodiode that are useable to enable 3D capture.

Abstract: A method for reducing power consumption of a 3D image capture system includes capturing 3D image data with the 3D image capture system while the 3D image capture system is in a first power state, detecting a power state change trigger, and switching from the first power state to a second power state based on the power state change trigger, wherein the 3D image capture system consumes less power in the second power state than in the first power state.

Abstract: A system configured to improve the operations associated with generating virtual representations on limited resources of a mobile device. In some cases, the system may utilize viewpoint bundles that include collection of image data with an associated pose in relative physical proximity to each other to render a virtual scene. In other cases, the system may utilize 2.5D manifolds including 2D image data and a weighted depth value to render the 3D environment.

Abstract: A system configured to determine one or more boundaries of the space, for example, in 3D applications. In some cases, the system may generate point data including a set of safe points in a space and a set of unsafe points in the space, the space surrounding a user device of the system, generate a triangulation over a union of the set of safe points and the set of unsafe points, determine triangles of the triangulation that include at least one safe point and determine edges of determined triangles which are part of a single triangle that include at least one safe point. The system may then determine one or more boundaries of the space using the determined edges.

Abstract: A system configured to provide a three-dimensional representation of a physical environment. The three-dimensional representation including annotation data associated with particular objects and/or viewpoints of the three-dimensional representation. In some cases, the viewpoints may be rendered using image data associated with a photograph captured from a corresponding viewpoint within the physical environment.

Abstract: A system configured to provide a three-dimensional representation of a physical environment. The three-dimensional representation including annotation data associated with particular objects and/or viewpoints of the three-dimensional representation. In some cases, the viewpoints may be rendered using image data associated with a photograph captured from a corresponding viewpoint within the physical environment.

Abstract: A virtual reality or mixed reality system configured to preform object detection using a monocular camera. The system configured to make the user aware of the detected objects by showing edges or lines of the object within a virtual scene. Thus, the user the user is able to avoid injury or collision while immersed in the virtual scene. In some cases, the system may also detect and correct for drift in the six degree of freedom pose of the user using corrections based on the current motion of the users.

Abstract: A system configured to improve the operations associated with generating virtual representations of physical environments to recognize the physical environments and/or relocalize within the virtual representations in a substantially real time system. In some cases, the system may use a first pre-training phase of descriptors and/or split nodes of regression forests using features common across various scenes to learn general image appearance, and a second training phase of descriptors and/or leaf nodes of regression forests to learn scene specific features. The system may align the features using an orientation vector, correct for camera perspective and lens distortion of the features as well as learn robust illumination invariant features from real and synthetic data.

Abstract: A method for automatic registration of 3D image data, captured by a 3D image capture system having an RGB camera and a depth camera, includes capturing 2D image data with the RGB camera at a first pose; capturing depth data with the depth camera at the first pose; performing an initial registration of the RGB camera to the depth camera; capturing 2D image data with the RGB camera at a second pose; capturing depth data at the second pose; and calculating an updated registration of the RGB camera to the depth camera.

Abstract: A three-dimensional (3D) sensing apparatus together with a projector subassembly is provided. The 3D sensing apparatus includes two cameras, which may be configured to capture ultraviolet and/or near-infrared light. The 3D sensing apparatus may also contain an optical filter and one or more computing processors that signal a simultaneous capture using the two cameras and processing the captured images into depth. The projector subassembly of the 3D sensing apparatus includes a laser diode, one or optical elements, and a photodiode that are useable to enable 3D capture.

Abstract: A system configured to determine a six-degree of freedom pose of a physical object in a physical environment and to utilize the six-degree of freedom pose as an within a virtual environment or mixed reality environment. In some cases, the system may utilize one or more cameras on a headset device to track the pose of a controller or other objects and one or more cameras on the controller itself to track the pose of the headset device or the user. In one example, the system may capture image data of a physical object having a constellation or pattern on the external source. The system may analyze the image data to identify image points associated with the constellation or pattern and to determine the pose of the object based on a location of the points in the image.

Abstract: A system configured to determine a pose of a physical object in a physical environment and to utilize the pose as an input to control or manipulate a virtual environment or mixed reality environment. In some cases, the system may be capture image data of a physical object having a constellation or pattern on the external source. The system may analyze the image data to identify points associated with the constellation or pattern and to determine the pose based on the identified points and a stored object model associated with the physical object.

Abstract: A virtual environment or mixed reality environment system including a physical tracking enhancement tag that may be identified by the system to assist with localization of the system with respect to the physical environment. The tracking enhancement tags may be encoded or coded in a high frequency and low frequency that is readable by the system to provide different types of information related the environment. For example, the encoded information may include a unique identifier of the tag and a relative direction with regards to the physical environment.

Abstract: A virtual environment or mixed reality environment system including a headset device and controller. Both the headset device and the controller are configured to access and update a shared scene map. In some cases, the headset device may implement pose tracking with respect to the headset device and the controller and the controller may also implement pose tracking with respect to the headset device and the controller.

Abstract: A system configured to improve the operations associated with generating virtual representations of physical environments to recognize the physical environments and/or relocalize within the virtual representations in a substantially real time system. In some cases, the system may use a first pre-training phase of descriptors and/or split nodes of regression forests using features common across various scenes to learn general image appearance, and a second training phase of descriptors and/or leaf nodes of regression forests to learn scene specific features. The system may align the features using an orientation vector, correct for camera perspective and lens distortion of the features as well as learn robust illumination invariant features from real and synthetic data.

Abstract: A head mounted display is provided that includes a main body, a sensor, and a door. The door of the head mounted display includes a computing device or, in some cases, an embedded screen. The computing device includes a camera and a display interface. The sensor and the camera of the mobile computing device are rigidly positioned with fixed displacements.