Abstract: A server receives, from a first display device of a first user, first content data, first sensor data, and a request for assistance identifying a context of the first display device. The server identifies a second display device of a second user based on the context of the first display device. The server receives second content data and second sensor data from the second display device. The first content data is synchronized with the second content data based on the first and second sensor data. Playback parameters are formed based on the context of the first display device. An enhanced playback session is generated using the synchronized first and second content data in response to determining that the first sensor data meet the playback parameters. The enhanced playback session is communicated to the first display device.

Abstract: A device has a biometric sensor, a display, a light source, and a hardware processor. The biometric sensor measures biometric information of a user of the device. The display displays augmented reality (AR) content. The light source outputs a visual signal. The hardware processor performs operations comprising: identifying a task being performed by the user of the device, determining a disruption level based on a combination of the AR content, the task, and the biometric information, and generating, using the light source, the visual signal corresponding to the disruption level. The visual indicator visually alerts other users of other devices to avoid engaging the user of the device.

Abstract: A server for content creation is described. A content creation tool of the server receives, from a first device, a content identifier of a physical object, a virtual object content, and a selection of a template corresponding to an interactive feature for the virtual object content. The content creation tool generates a content dataset based on the content identifier of the physical object, the virtual object content, and the selected template. The content creation tool provides the content dataset to a second device, the second device configured to display the interactive feature corresponding to the selected template.

Abstract: A head mounted device (HMD) has a transparent display. The HMD detects an event external to the HMD and determines a location of the event relative to the location of the HMD. The HMD generates augmented reality (AR) content identifying the event and a direction of the location of the event relative to the location of the HMD and displays the AR content in the transparent display.

Abstract: A device can determine a distance to an object. The device can use the determined distance to vary a focal length of a first adjustable element so that the first adjustable element directs light from the object into a first waveguide and onto a detector, and forms an image of the object at the detector. The device can produce an image, such as augmented content, on a panel. The device can direct light from the panel into a second waveguide. The device can use the determined distance to vary a focal length of a second adjustable element so that the second adjustable element directs light out of the second waveguide and forms a virtual image of the panel in a plane coincident with the object. The device can operate as an augmented reality headset. The adjustable elements can be phase modulators, or acoustically responsive material with surface acoustic wave transducers.

Abstract: A predictive augmented reality assistance system is described. A device generates and renders augmented reality content in a display of the device. The device tracks user interactions with the augmented reality content of a user of the device. A context of the user interactions with the augmented reality content is determined based on the user interactions. A behavioral analysis of a user of the device is generated based on the context of the user interaction. A predictive model of the user of the device is generated based on the behavioral analysis. The augmented reality content is modified based on the predictive model of the user of the device.

Abstract: A head-mounted device (HMD) has a transparent display and an array of microphones. The HMD determines a task-based context of the HMD based on a task being performed by a user of the HMD, and determines an ambient-based context of the HMD based on audio data produced by the array of microphones. The HMD detects a first notification configured to be displayed in the transparent display of the HMD. The HMD intercepts the first notification in response to detecting the first notification and generates a second notification based on a priority level of the first notification, the task-based context, and the ambient-based context. The second notification includes a combination of an audio notification and a visual notification (AR content). The HMD replaces the first notification with the second notification and provides the second notification at the HMD.

Abstract: A visualization system that uses deep learning can store data representing a scene in multiple layers of representation. Each layer can include a different category of data from the other layers. The categories of data can increase in complexity and specificity from a lowermost layer to an uppermost layer. For instance, first, second, and third layers can store information corresponding to edges, corners, and surface finishes present in the scene, respectively. The visualization system can retrieve data representing the scene from multiple layers, and augment a point cloud representation of the scene in response to the retrieved data. The point cloud can be augmented to increase a point density in one or more point cloud regions that lack data or include only sparse data. Downstream, the visualization system can use the augmented point cloud to create improved images of the scene from desired points of view.

Abstract: Interface-based modeling and design of three dimensional spaces using two dimensional representations are provided herein. An example method includes converting a three dimensional space into a two dimensional space using a map projection schema, where the two dimensional space is bounded by ergonomic limits of a human, and the two dimensional space is provided as an ergonomic user interface, receiving an anchor position within the ergonomic user interface that defines a placement of an asset relative to the three dimensional space when the two dimensional space is re-converted back to a three dimensional space, and re-converting the two dimensional space back into the three dimensional space for display along with the asset, within an optical display system.

Abstract: Fixed position optical devices for displaying augmented reality images are provided herein. In one embodiment an optical device includes a AIIE having a waveguide that reflects a computer generated image along a central viewing axis, the computer generated image being received from an image generator optically coupled to the waveguide, and a fixed lens assembly for coupling a background image with the computer generated image to create the augmented reality display, the fixed lens assembly including a proximal lens disposed on one side of the waveguide, the proximal lens being fixedly spaced apart from the waveguide at a first distance, and a distal lens disposed on an opposing side of the AIIE from the one side, the distal lens being fixedly spaced apart from the waveguide at a second distance.

Abstract: A system and method for manipulating a virtual object based on thought is described. A display of a device displays a first virtual object based on an identification of a physical object detected by the device. The device monitors brain activity data of a user of the device. The brain activity data is generated in response to the first virtual object being displayed in the display. The device identifies a second virtual object based on the brain activity data of the user of the device and displays the second virtual object in the display.

Abstract: A system and method for manipulating a virtual object based on intent is described. A reference identifier from a physical object is captured. The reference identifier is communicated via a network to a remote server. The remote server includes virtual object data associated with the reference identifier. The virtual object data is received at the computing device. The virtual image is displayed in a virtual landscape using the virtual object data. In response to relative movement between the computing device and the physical object caused by a user, the virtual image is modified. Brain activity data of the user is received. A state of the virtual object in the virtual landscape is changed based on the brain activity data.

Abstract: A system and method for offloading augmented reality processing is described. A first sensor of a server generates a first set of sensor data corresponding to a location and an orientation of a display device. The server receives a request from the display device to offload a combination of at least one of a tracking process and a rendering process from the display device. The server generates offloaded processed data based on a combination of at least one of the first set of sensor data and a second set of sensor data. The second set of sensor data is generated by a second sensor at the display device. The server streams the offloaded processed data to the display device.

Abstract: An application generates instructions to a wearable device to remotely activate a sensor in the wearable device and to receive sensor data from the sensor. A query related to a physical object is received. Instructions to wearable devices are generated to remotely activate at least one sensor of the wearable devices in response to the query. Sensor data is received from at least one of the wearable devices in response to that wearable device being within a range of the physical object.

Abstract: An image localization system is described. A receiver receives visual inertial navigation (VIN) data and corresponding image data from devices, the VIN data indicating VIN states and corresponding poses of the devices. A training module generates a localization model based on the VIN data and corresponding image data from the plurality of devices. The image data includes images, the localization model correlating the VIN states and poses with each image among the plurality of images. An image localizer receives a query for a geographic location and a pose of a device. The query includes a picture. The image localizer compares the picture with images from the localization model, identifies an image based on the localization model, the image matching the picture in the query, and determines the geographic location and the pose of the device based on the VIN state and pose corresponding to the identified image.

Abstract: A system and method for generating a dynamic sensor array for an augmented reality system is described. A head mounted device includes one or more sensors, an augmented reality (AR) application, and a sensor array module. The sensor array module identifies available sensors from other head mounted devices that are geographically located within a predefined area. A dynamic sensor array is formed based on the available sensors and the one or more sensors. The dynamic sensor array is updated based on an operational status of the available sensors and the one or more sensors. The AR application generates AR content based on data from the dynamic sensor array. A display of the head mounted device displays the AR content.

Abstract: A system and method for multi-spectrum segmentation for computer vision is described. A first sensor captures an image within a first spectrum range and generates first sensor data. A second sensor captures an image within a second spectrum range different than the first spectrum range and generates second sensor data. A multi-spectrum segmentation module identifies a segmented portion of the image within the second spectrum range based on: the second sensor data, a subset of the first sensor data corresponding to the segmented portion of the image within the second spectrum range, and a segmented portion of the image at the second spectrum range corresponding to the subset of the first sensor data. The multi-spectrum segmentation module identifies a physical object in the segmented portion of the image within the second spectrum range, and a device generates augmented reality content based on the identified physical object.

Abstract: A system and method for augmented reality content adapted to changes in real world space geometry are described. A device captures an image of a local environment and maps a real world space geometry of the local environment using the image of the local environment. The device generates a visualization of a virtual object in the display relative to the mapped real world space geometry of the local environment. A content of the virtual object is adjusted to changes in the real world space geometry of the local environment.