AUGMENTED REALITY SYSTEM AND METHOD FOR POSITIONING AND MAPPING
An augmented reality and virtual reality head mounted display is described. The head mounted display comprises a camera array in communication with a processor to map the physical environment for rendering an augmented reality of the physical environment.
The following relates generally to systems and methods for augmented and virtual reality environments, and more specifically to systems and methods for mapping a virtual or augmented environment based on a physical environment, and displaying the virtual or augmented environment on a head mounted device.
BACKGROUNDThe range of applications for augmented reality (AR) and virtual reality (VR) visualization has increased with the advent of wearable technologies and 3-dimensional (3D) rendering techniques. AR and VR exist on a continuum of mixed reality visualization.
SUMMARYIn embodiments, a method is described for mapping a physical environment surrounding a user wearing a wearable display for augmented reality. The method comprises: (i) capturing, by at least one depth camera disposed upon the user, depth information for the physical environment; (ii) by a processor, obtaining the depth information, determining the orientation of the at least one depth camera relative to the wearable display, and assigning coordinates for the depth information in a map of the physical environment based on the orientation of the at least one depth camera.
In further embodiments, a system is described for mapping a physical environment surrounding a user wearing a wearable display for augmented reality. The system comprises: (i) at least one depth camera disposed upon the user, to capture depth information for the physical environment; and (ii) at least one processor in communication with the at least one depth camera, to obtain the depth information from the at least one depth camera, determine the orientation of the at least one depth camera relative to the wearable display, and assign coordinates for the depth information in a map of the physical information based on the orientation of the at least one depth camera.
In still further embodiments, a system is described for displaying a rendered image stream in combination with a physical image stream of a region of a physical environment captured in the field of view of at least one image camera disposed upon a user wearing a wearable display for augmented reality. The system comprises a processor configured to: (i) obtain a map of the physical environment; (ii) determine the orientation and location of the wearable display within the physical environment; (iii) determine, from the orientation and location of the wearable display, the region of the physical environment captured in the field of view of the at least one image camera; (iv) determine a region of the map corresponding to the captured region of the physical environment; and (iv) generate rendered stream comprising augmented reality for the corresponding region of the map.
In yet further embodiments, a method is described for displaying a rendered image stream in combination with a physical image stream of a region of a physical environment captured in the field of view of at least one image camera disposed upon a user wearing a wearable display for augmented reality. The method comprises, by a processor: (i) obtaining a map of the physical environment; (ii) determining the orientation and location of the wearable display within the physical environment; (iii) determining, from the orientation and location of the wearable display, the region of the physical environment captured in the field of view of the at least one image camera; (iv) determining a region of the map corresponding to the captured region of the physical environment; and (v) generating rendered stream comprising augmented reality for the corresponding region of the map.
A greater understanding of the embodiments will be had with reference to the Figures, in which:
It will be appreciated that for simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Also, the description is not to be considered as limiting the scope of the embodiments described herein.
It will also be appreciated that any module, unit, component, server, computer, terminal or device exemplified herein that executes instructions may include or otherwise have access to computer readable media such as storage media, computer storage media, or data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the device or accessible or connectable thereto. Any application or module herein described may be implemented using computer readable/executable instructions that may be stored or otherwise held by such computer readable media and executed by the one or more processors.
The present disclosure is directed to systems and methods for augmented reality (AR). However, the term “AR” as used herein may encompass several meanings. In the present disclosure, AR includes: the interaction by a user with real physical objects and structures along with virtual objects and structures overlaid thereon; and the interaction by a user with a fully virtual set of objects and structures that are generated to include renderings of physical objects and structures and that may comply with scaled versions of physical environments to which virtual objects and structures are applied, which may alternatively be referred to as an “enhanced virtual reality”. Further, the virtual objects and structures could be dispensed with altogether, and the AR system may display to the user a version of the physical environment which solely comprises an image stream of the physical environment. Finally, a skilled reader will also appreciate that by discarding aspects of the physical environment, the systems and methods presented herein are also applicable to virtual reality (VR) applications, which may be understood as “pure” VR. For the reader's convenience, the following refers to “AR” but is understood to include all of the foregoing and other variations recognized by the skilled reader.
A head mounted display (HMD) or other wearable display worn by a user situated in a physical environment may comprise a display system and communicate with: at least one depth camera disposed upon or within the HMD, or worn by (i.e., disposed upon) the user, to generate depth information for the physical environment; and at least one processor disposed upon, or within, the HMD, or located remotely from the HMD (such as, for example, a processor of a central console, or a server) to generate a map of the physical environment from the depth information. The processor may generate the map as, for example, a point cloud, in which the points correspond to the obtained depth information for the physical environment.
Mapping a physical environment from a scanning system tied to the user may be referred to as inside-out mapping or first-person-view mapping. In contrast, outside-in mapping involves mapping a physical environment from one or more scanning systems situated in the physical environment and directed to scan towards one or more users. It has been found that user engagement with an AR may be enhanced by allowing a user to move throughout a physical environment in an unconstrained manner. Inside-out mapping may provide greater portability because mapping of a physical environment is performed by equipment tied to a user rather than the physical environment.
The processor further generates an AR (also referred to as “rendered”) image stream comprising computer-generated imagery (CGI) for the map, and provides the AR image stream to the display system for display to the user. The processor may continuously adapt the rendered image stream to correspond to the user's actual position and orientation within the physical environment. The processor may therefore obtain real-time depth information from the depth camera to determine the user's real-time orientation and location with the physical environment, as described herein in greater detail. The processor provides the rendered image stream to the display system for display to the user.
The display system of the HMD may display an image stream of the physical environment, referred to herein as a “physical image stream”, to the user. The display system obtains the image stream from at least one image camera disposed upon the HMD or the user, either directly, or by way of the processor. The at least one image camera may be any suitable image capture device operable to capture visual images of the physical environment in digital format, such as, for example, a colour camera or video camera. In operation, the at least one image camera dynamically captures the physical image stream for transmission to the display system.
The display system may further simultaneously display the physical image stream provided by the at least one image camera, and the rendered image stream obtained from the processor. Further systems and methods are described herein.
Referring now to
In certain applications, the user views an AR comprising a completely rendered version of the physical environment (i.e., “enhanced VR”). In such applications, the user may determine the locations for obstacles or boundaries in the physical environment based solely on the rendering displayed to the user in the display system 121 of the user's HMD 12.
As shown in
If the at least one depth camera 227 of the HMD 212 has a combined field of view that is less than 360 degrees about the HMD 212, as shown in
As shown in
At block 303, a processor receives the depth information obtained at blocks 300 and 301 during the user's rotation and “stitches” the depth information received during the user's rotation. Stitching comprises aligning subsequent frames in the continuously captured depth information to create a substantially seamless map, as outlined herein with reference to blocks 303 and 305.
The region of the physical space captured within the depth camera's field of view at time t=0 is illustrated by the image 320; similarly, the region of the physical space captured within the depth camera's field of view at time t=1 is illustrated by the image 321. It will be appreciated that the user capturing the sequence shown in
It will be appreciated that accuracy may be enhanced if, instead of identifying a single topographical feature common to subsequent depth information captures, the processor identifies more than one common topographical feature between frames. Further, capture frequency may be increased to enhance accuracy.
Alternatively, or in addition, to identifying common features between subsequent frames captured by the at least one depth camera, the processor may obtain real-time orientation information from an orientation detecting system for the HMD, as shown in
As shown in
The HMD may further comprise at least one imaging camera to capture a physical image stream of the physical environment. The processor may enhance the map of the physical environment generated using depth information from the at least one depth camera by adding information from the physical image stream of the physical environment. During mapping according to the previously described mapping methods, the processor may further obtain a physical image stream of the physical environment from the at least one imaging camera, as shown in
Once the processor has mapped the physical environment, the processor may track changes in the user's orientation and position due to the user's movements. As shown in
Alternatively, the HMD may comprise a local positioning system and/or an orientation detecting system, such as, for example, a 3D magnetic positioning system, laser positioning system and/or inertial measurement unit to determine the real-time position and orientation of the user.
Augmented reality involves combining CGI (also understood as renderings generated by a processor) with a physical image stream of the physical environment. An HMD for AR and VR applications is shown in
The display system of an HMD may display the rendered image stream alone (enhanced VR) or overlaid over the physical image stream to combine the visual and typographic aspects of the physical environment (AR).
In an enhanced VR application, the processor may enhance a user's interaction with the physical environment by accounting for the user's real-time location and orientation within the physical environment when generating the rendered image stream. As the user moves about the physical environment, the VR of that physical environment displayed to the user will reflect changes in the user's position and/or orientation. As shown in
In an AR application, the display system of an HMD may display the rendered image stream overlaid over, or combined with, the physical image stream. When the at least one image camera captures an image stream of the physical environment, the captured physical image stream at any given moment will comprise elements of the physical environment lying within the field of view of the camera at that time.
The physical image stream obtained by the camera is either transmitted to the processor for processing and/or transmission to the display system, or directly to the display system for display to the user.
Referring now to
If the fields of view of the notional and physical cameras are substantially aligned and identical, simultaneous and combined display of both image streams provides a combined stream that is substantially matched.
In embodiments, the processor may increase or decrease the signal strength of one or the other of the physical and rendered image streams to vary the effective transparency.
In embodiments, the processor only causes the display system to display the physical image stream upon the user selecting display of the physical image stream. In further embodiments, the processor causes the display system to display the physical image stream in response to detecting proximity to an obstacle in the physical environment. In still further embodiments, the processor increases the transparency of the rendered image stream in response to detecting proximity to an obstacle in the physical environment. Conversely, the processor may reduce the transparency of the rendered image stream as the HMD moves away from obstacles in the physical environment.
In still further embodiments, the display system displays the physical and rendered image streams according to at least two of the techniques described herein.
Although the following has been described with reference to certain specific embodiments, various modifications thereto will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the appended claims. The entire disclosures of all references recited above are incorporated herein by reference.
Claims
1. A method for mapping a physical environment in which a user wearing a wearable display for augmented reality is situated, the method comprising:
- (a) capturing, by at least one depth camera disposed upon the user, depth information for the physical environment;
- (b) by a processor, obtaining the depth information, determining the orientation of the at least one depth camera relative to the wearable display, and assigning coordinates for the depth information in a map of the physical environment based on the orientation of the at least one depth camera;
- wherein: i) the capturing comprises continuously capturing a sequence of frames of depth information for the physical environment during rotation and translation of the at least one depth camera in the physical environment; ii) the obtaining further comprises continuously determining the translation and the rotation of the at least one depth camera between each of the frames, and; iii) the assigning comprises assigning first coordinates to the depth information from a first frame and assigning subsequent coordinates to the depth information from each of the subsequent frames according to the rotation and translation of the at least one depth camera between each of the frames.
2. (canceled)
3. The method of claim 1, further comprising:
- (a) identifying topography shared between first and second ones of subsequent frames;
- (b) assigning shared coordinates to the shared topography for each of the first and second ones of the subsequent frames; and
- (c) assigning coordinates for the second one of the subsequent frames with reference to the coordinates for the shared topography.
4. The method of claim 1, further comprising:
- (a) capturing, by at least one image camera disposed upon the user, a physical image stream of the physical environment;
- (b) obtaining the physical image stream, determining the orientation of the at least one image camera relative to the wearable display, and assigning coordinates to a plurality of pixels in the physical image stream in the map of the physical environment based on the orientation of the at least one image camera.
5. A system for mapping a physical environment surrounding a user wearing a wearable display for augmented reality, the system comprising:
- (a) at least one depth camera disposed upon the user, to capture depth information for the physical environment;
- (b) at least one processor in communication with the at least one depth camera, to obtain the depth information from the at least one depth camera, determine the orientation of the at least one depth camera relative to the wearable display, and assign coordinates for the depth information in a map of the physical environment based on the orientation of the at least one depth camera;
- wherein the at least one depth camera is configured to continuously capture a sequence of frames of depth information for the physical environment during rotation and translation of the at least one depth camera in the physical environment, and the processor is configured to: i) continuously determine the rotation and the translation of the at least one depth camera between each of the frames; ii) assign coordinates for the depth information by assigning first coordinates to the depth information from a first frame and assigning subsequent coordinates to the depth information from each of the subsequent frames according to the rotation and translation of the at least one depth camera between each of the frames.
6. (canceled)
7. The system of claim 5, wherein the processor is further configured to:
- (a) identify topography shared between first and second ones of subsequent frames;
- (b) assign shared coordinates to the shared topography for each of the first and second ones of the subsequent frames; and
- (c) assign coordinates for the second one of the subsequent frames with reference to the coordinates for the shared topography.
8. The system of claim 5, further comprising at least one image camera disposed upon the user, operable to capture a physical image stream of the physical environment, and wherein the processor is configured to:
- (a) obtain the physical image stream, determine the orientation of the at least one image camera relative to the wearable display; and
- (b) assign coordinates to a plurality of pixels in the physical image stream in the map of the physical environment based on the orientation of the at least one image camera.
9. (canceled)
10. (canceled)
Type: Application
Filed: Oct 3, 2014
Publication Date: Jul 21, 2016
Inventors: Dhanushan BALACHANDRESWARAN (Richmond Hill), Kibaya Mungai NJENGA (Markham), Jian ZHANG (Oakville)
Application Number: 14/778,855