Abstract: Augmented reality glasses include near eye displays the include sources of imagewise amplitude modulated light optical coupled to spatial phase modulators or active zone plate modulators and optically coupled to eye coupling optics. The sources of imagewise amplitude modulated light can include emissive 2D display panels or light sources coupled to imagewise amplitude modulators. The eye coupling optics can include volume holographic diffraction gratings.

Abstract: The invention relates to a viewing system for use in a surgical environment. Various real object detection devices detect locations of real objects in a real environment, such as a patient and body part of patient, medical staff, robots, a cutting tool on a robot, implant transferred by robot into body part, surgical tools, and disposable items. A map generator generates a map that forms a digital representation or a digital twin of the real environment. Various guiding modules including a room setup module, an anatomy registration module, a surgical planning module, and a surgical execution module make use of the digital representation to guide virtual or real objects based on the digital representation.

Abstract: Techniques are disclosed for using and training a descriptor network. An image may be received and provided to the descriptor network. The descriptor network may generate an image descriptor based on the image. The image descriptor may include a set of elements distributed between a major vector comprising a first subset of the set of elements and a minor vector comprising a second subset of the set of elements. The second subset of the set of elements may include more elements than the first subset of the set of elements. A hierarchical normalization may be imposed onto the image descriptor by normalizing the major vector to a major normalization amount and normalizing the minor vector to a minor normalization amount. The minor normalization amount may be less than the major normalization amount.

Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

Abstract: An optical system comprises an optically transmissive substrate comprising a metasurface which comprises a grating comprising a plurality of unit cells. Each unit cell comprises a laterally-elongated first nanobeam having a first width; and a laterally-elongated second nanobeam spaced apart from the first nanobeam by a gap, the second nanobeam having a second width larger than the first width. A pitch of the unit cells is 10 nm to 1 ?m. The heights of the first and the second nanobeams are: 10 nm to 450 nm where a refractive index of the substrate is more than 3.3; and 10 nm to 1 ?m where the refractive index is 3.3 or less.

Abstract: Described are scanning micromirror devices, methods of making scanning micromirror devices, two-dimensional optical scanning systems that incorporate scanning micromirror devices, and methods of projecting light and images using two-dimensional optical scanning systems. The disclosed two-dimensional optical scanning systems can incorporate a first scanning micromirror device oscillating at a relatively higher frequency, which directs reflected light onto a second scanning micromirror device oscillating at a relatively lower frequency, which directs reflected light for projection.

Abstract: An augmented reality (AR) device can be configured to generate a virtual representation of a user's physical environment. The AR device can capture images of the user's physical environment to generate a mesh map. The AR device can project graphics at designated locations on a virtual bounding box to guide the user to capture images of the user's physical environment. The AR device can provide visual, audible, or haptic guidance to direct the user of the AR device to look toward waypoints to generate the mesh map of the user's environment.

Abstract: A virtual or augmented reality display system that controls power inputs to the display system as a function of image data. Image data itself is made of a plurality of image data frames, each with constituent color components of, and depth planes for displaying on, rendered content. Light sources or spatial light modulators to relay illumination from the light sources may receive signals from a display controller to adjust a power setting to the light source or spatial light modulator based on control information embedded in an image data frame.

Abstract: Systems and methods for cuboid detection and keypoint localization in images are disclosed. In one aspect, a deep cuboid detector can be used for simultaneous cuboid detection and keypoint localization in monocular images. The deep cuboid detector can include a plurality of convolutional layers and non-convolutional layers of a trained convolution neural network for determining a convolutional feature map from an input image. A region proposal network of the deep cuboid detector can determine a bounding box surrounding a cuboid in the image using the convolutional feature map. The pooling layer and regressor layers of the deep cuboid detector can implement iterative feature pooling for determining a refined bounding box and a parameterized representation of the cuboid.

Abstract: Disclosed herein are systems and methods for presenting audio content in mixed reality environments. A method may include receiving a first input from an application program; in response to receiving the first input, receiving, via a first service, an encoded audio stream; generating, via the first service, a decoded audio stream based on the encoded audio stream; receiving, via a second service, the decoded audio stream; receiving a second input from one or more sensors of a wearable head device; receiving, via the second service, a third input from the application program, wherein the third input corresponds to a position of one or more virtual speakers; generating, via the second service, a spatialized audio stream based on the decoded audio stream, the second input, and the third input; presenting, via one or more speakers of the wearable head device, the spatialized audio stream.

Abstract: Disclosed herein are systems and methods for sharing and synchronizing virtual content. A method may include receiving, from a host application via a wearable device comprising a transmissive display, a first data package comprising first data; identifying virtual content based on the first data; presenting a view of the virtual content via the transmissive display; receiving, via the wearable device, first user input directed at the virtual content; generating second data based on the first data and the first user input; sending, to the host application via the wearable device, a second data package comprising the second data, wherein the host application is configured to execute via one or more processors of a computer system remote to the wearable device and in communication with the wearable device.

Abstract: Examples of systems and methods for improved hand tracking of a user in a mixed reality environment are disclosed. The systems and methods may be configured to estimate the hand pose and shape of a user's hands for applications such as animating a hand on a user's avatar. Data from multiple sources, such as a totem internal measurement unit (“IMU”), external totem location tracking, vision cameras, and depth sensors, may be manipulated using a set of rules that are based on historical data, ergonomics data, and motion data.

Abstract: A fan assembly is disclosed. The fan assembly can include a first support frame. The fan assembly can comprise a shaft assembly having a first end coupled with the first support frame and a second end disposed away from the first end. A second support frame can be coupled with the first support frame and disposed at or over the second end of the shaft assembly. An impeller can have fan blades coupled with a hub, the hub being disposed over the shaft assembly for rotation between the first and second support frames about a longitudinal axis. Transverse loading on the shaft assembly can be controlled by the first and second support frames.

Abstract: Architectures are provided for selectively outputting light for forming images, the light having different wavelengths and being outputted with low levels of crosstalk. In some embodiments, light is incoupled into a waveguide and deflected to propagate in different directions, depending on wavelength. The incoupled light then outcoupled by outcoupling optical elements that outcouple light based on the direction of propagation of the light. In some other embodiments, color filters are between a waveguide and outcoupling elements. The color filters limit the wavelengths of light that interact with and are outcoupled by the outcoupling elements. In yet other embodiments, a different waveguide is provided for each range of wavelengths to be outputted. Incoupling optical elements selectively incouple light of the appropriate range of wavelengths into a corresponding waveguide, from which the light is outcoupled.

Abstract: A smartphone may be freely moved in three dimensions as it captures a stream of images of an object. Multiple image frames may be captured in different orientations and distances from the object and combined into a composite image representing an image of the object. The image frames may be formed into the composite image based on representing features of each image frame as a set of points in a three dimensional point cloud. Inconsistencies between the image frames may be adjusted when projecting respective points in the point cloud into the composite image. Quality of the image frames may be improved by processing the image frames to correct errors. Reflections and shadows may be detected and removed. Further, optical character recognition may be applied. As the scan progresses, a direction for capturing subsequent image frames is provided to a user as a real-time feedback.

Abstract: Examples of the disclosed systems and methods may provide for improved and more realistic rendering of virtual characters and a more realistic interaction between a user and virtual characters. For example, the systems and methods describe techniques for mathematically generating a map used for animating facial expressions in a multidimensional animation blendspace. As another example, the systems and methods describe a transition system for dynamically transitioning facial expressions across a face of the virtual character. As another example, realistic physical movements can be added to a virtual character's facial expressions to provide interactivity with other virtual characters.

Abstract: A method of presenting an audio signal to a user of a mixed reality environment is disclosed, the method comprising the steps of detecting a first audio signal in the mixed reality environment, where the first audio signal is a real audio signal; identifying a virtual object intersected by the first audio signal in the mixed reality environment; identifying a listener coordinate associated with the user; determining, using the virtual object and the listener coordinate, a transfer function; applying the transfer function to the first audio signal to produce a second audio signal; and presenting, to the user, the second audio signal.

Abstract: A multiple degree of freedom hinge system is provided, which is particularly well adapted for eyewear, such as spatial computing headsets. In the context of such spatial computing headsets having an optics assembly supported by opposing temple arms, the hinge system provides protection against over-extension of the temple arms or extreme deflections that may otherwise arise from undesirable torsional loading of the temple arms. The hinge systems also allow the temple arms to splay outwardly to enable proper fit and enhanced user comfort.

Abstract: Disclosed are methods and systems for managing mixed-reality systems. These methods or systems identify, at a management software application, a set of managed mixed-reality devices and validate a device management license against a managed mixed-reality device in the set of managed mixed-reality devices. Whether a software application installed or to be installed on the managed mixed-reality device is amenable to a software management function is verified based at least in part upon an operation mode of the managed mixed-reality device and a profile that is applied to the managed mixed-reality device or is associated with the authorized user. The software management function is programmatically performed by the management software application for the software application based at least in part upon a result of verifying whether the software application is amenable to the software management operation.

Abstract: An electromagnetic tracking system includes a handheld controller including a first phased array element characterized by a first phase and a second phased array element characterized by a second phase different than the first phase. The first phased array element and the second phased array element are configured to generate a steerable electromagnetic beam characterized by an electromagnetic field pattern. The electromagnetic tracking system also includes a head mounted augmented reality display including an electromagnetic sensor configured to sense the electromagnetic field pattern.