Abstract: Aspects of the present disclosure are for an interaction mode system that provides multiple interaction modes in an artificial reality environment with automatic, context-specific transitions between interaction modes. An interaction mode can specify how the interaction mode system determines direction indications and movement within an artificial reality environment and interactions for making selections or performing other actions. In some implementations, the interaction mode system can control at least four interaction modes including a no-hands, 3DoF mode, a no-hands, 6DoF mode, a gaze and gesture mode, and a ray casting mode. The interaction mode system can employ a mapping of interaction mode context factors (e.g., which components are enabled, mode settings, lighting or other environment conditions, current body positions, etc.) to interaction modes to control transitioning between particular interaction modes.

Abstract: A system can include a device and a graphics processing unit (GPU). The device can be configured to receive a first image from one or more cameras corresponding to a first view and a second image from the one or more cameras corresponding to a second view. The GPU can include a motion estimator and configured to receive the first image and the second image and be configured to receive the first image and the second image. The motion estimator can be configured to determine first disparity offsets for the first image and second disparity offsets for the second image. The device can be configured to generate, for rendering 3D image using the first image and the second image, a first depth buffer for the first image derived from the first disparity offsets and a second depth buffer for the second image derived from the second disparity offsets.

Abstract: In one embodiment, a method includes determining a first time interval for rendering pixel values for a first portion of a display, where the first portion of the display is one of a plurality of portions of the display, accessing a representation of virtual objects in a virtual scene, and determining that the first time interval is insufficient for rendering pixel values for the first portion of the display based on the representation of virtual objects. The method further includes, for each of the portions of the display, determining a current viewpoint of a user, rendering, based on the current viewpoint of the user, pixel values for that portion of the display, and sending the rendered pixel values to be output by the display, where the rendered pixel values for the first portion of the display are rendered based on a simplified representation of virtual objects in the virtual scene.

Abstract: In one embodiment, a method includes receiving a user input comprising one or more words at a client system, wherein each word comprises one or more characters, inputting the words to a convolutional neural network (CNN) model stored on the client system, accessing a plurality of character-embeddings for a plurality of characters, respectively, from a data store of the client system, generating one or more word-embeddings for the one or more words, respectively, based on the accessed character-embeddings by processing the accessed character-embeddings with one or more convolutional layers and one or more gated linear units of the CNN model, determining one or more tasks corresponding to the user input for execution based on an analysis of the one or more word-embeddings by the CNN model, and providing an output responsive to the user input based on the execution of the one or more tasks at the client system.

Abstract: An audio system for providing content to a user. The system includes a first and a second transducer assembly of a plurality of transducer assemblies, an acoustic sensor, and a controller. The first transducer assembly couples to a portion of an auricle of the user's ear and vibrates over a first range of frequencies based on a first set of audio instructions. The vibration causes the portion of the ear to create a first range of acoustic pressure waves. The second transducer assembly is configured to vibrate over a second range of frequencies to produce a second range of acoustic pressure waves based on a second set of audio instructions. The acoustic sensor detects acoustic pressure waves at an entrance of the ear. The controller generates the audio instructions based on audio content to be provided to the user and the detected acoustic pressure waves from the acoustic sensor.

Abstract: A sensor assembly for determining one or more features of a local area is presented herein. The sensor assembly includes a plurality of stacked sensor layers. A first sensor layer of the plurality of stacked sensor layers located on top of the sensor assembly includes an array of pixels. The top sensor layer can be configured to capture one or more images of light reflected from one or more objects in the local area. The sensor assembly further includes one or more sensor layers located beneath the top sensor layer. The one or more sensor layers can be configured to process data related to the captured one or more images. A plurality of sensor assemblies can be integrated into an artificial reality system, e.g., a head-mounted display.

Abstract: In one embodiment, a method includes creating, by a computing device, an inter process communication (IPC) channel for communication from a producer process to a consumer process. The IPC channel includes a message buffer mapped to a first virtual address space of a kernel of an operating system and to a second virtual address space of the consumer process in a user space of the operating system. A sender handle for the message buffer is sent to the producer process. The kernel receives a request, which includes the sender handle, to send a message from the producer process to the consumer process through the IPC channel. Responsive to a determination that the request is permitted, the kernel writes the message to the message buffer using the sender handle, such that message written in the message buffer is accessible to the consumer process through the second address space.

Abstract: The disclosed computer-implemented method may include applying, via a sound reproduction system, sound cancellation that reduces an amplitude of various sound signals. The method further includes identifying, among the sound signals, an external sound whose amplitude is to be reduced by the sound cancellation. The method then includes analyzing the identified external sound to determine whether the identified external sound is to be made audible to a user and, upon determining that the external sound is to be made audible to the user, the method includes modifying the sound cancellation so that the identified external sound is made audible to the user. Various other methods, systems, and computer-readable media are also disclosed.

Abstract: The technology disclosed relates to a machine learning based speech-to-text transcription intermediary which, from among multiple speech recognition engines, selects a speech recognition engine for accurately transcribing an audio channel based on sound and speech characteristics of the audio channel.

Abstract: In one embodiment, a method includes accessing a plurality of points, wherein each point (1) corresponds to a spatial location associated with an observed feature of a physical environment and (2) is associated with a patch representing the observed feature, determining a density associated with each of the plurality of points based on the spatial locations of the plurality of points, scaling the patch associated with each of the plurality of points based on the density associated with the point, and reconstructing a scene of the physical environment based on at least the scaled patches.

Abstract: In one embodiment, a method includes receiving a user input from a user from a client system associated with the user, wherein the client system comprises one or more cameras, determining one or more points of interest in a field of view of the one or more cameras based on one or more machine-learning models and sensory data captured by the one or more cameras, generating a plurality of media files based on the one or more points of interest, wherein each media file is a recording of at least one of the one or more points of interest, generating one or more highlight files based on the plurality of media files, wherein each highlight file comprises a media file that satisfies a predefined quality standard, and sending instructions for presenting the one or more highlight files to the client system.

Abstract: A capacitive sensing assembly comprises a nonconductive interface material, a conductive plastic, an electrode, and a controller. The capacitive sensing assembly is configured to detect a proximity of skin to the nonconductive interface material. The nonconductive interface material comprises a first surface and a second surface that are opposite one another. The first surface receives contact with the skin. The conductive plastic is coupled to the second surface of the interface material. The electrode is electrically coupled to the conductive plastic and configured to provide an electrical signal. The electrical signal corresponds to a surface capacitance of the nonconductive interface material that is based in part on the conductive plastic, the nonconductive interface material, and the proximity of the skin to the interface material. The controller determines the proximity of the skin based on the electrical signal that corresponds to the surface capacitance.

Abstract: Techniques disclosed herein relate to eye illumination for eye tracking in a near-eye display system. One example of an eye illumination system includes a substrate transparent to visible light and infrared light and configured to be placed in front of an eye of a user, and a shortwave-pass filter on a first surface of the substrate. The shortwave-pass filter includes regions configured to transmit visible light and reflect infrared light in ambient light, and a plurality of windows configured to transmit both visible light and infrared light in the ambient light.

Abstract: Techniques for overcoating slanted structures and devices obtained using the techniques are disclosed. In some embodiments, a method of forming an overcoat layer on a surface-relief structure on a substrate includes receiving the substrate with the surface-relief structure. The surface-relief structure includes a plurality of ridges slanted with respect to the substrate, and a plurality of grooves each between two adjacent ridges. The method further includes depositing, in each cycle of a plurality of cycles, a uniform layer of an overcoat material on surfaces of the plurality of ridges and bottoms of the plurality of grooves. The deposited layers of the overcoat material and the plurality of ridges collectively form a light-coupling structure on the substrate. A surface of the overcoat layer is planar.

Abstract: This disclosure describes an artificial reality system that presents artificial reality content or artificial reality effects based on, or in response to, interactions with one or more physical objects within a physical environment. In one example, this disclosure describes a method that includes determining that a user has performed a trigger action on a trigger object; responsive to determining that the user has performed the trigger action, generating artificial reality content; presenting the artificial reality content on a display associated with a head-mounted display; determining that the user has performed a de-trigger action; and responsive to determining that the user has performed the de-trigger action, discontinuing presentation of the artificial reality content.

Abstract: In one embodiment, a computing system may generate and display a virtual reality environment to a user. The computing system may determine a head pose of the user based on headset tracking data associated with a headset worn by the user. The computing system may determine a hand pose of the user based on hand tracking data associated with a device held or worn by a hand of the user. The computing system may access scene information associated with the displayed virtual reality environment. The computing system may determine a predicted focal point of the user within the virtual reality environment by processing the head pose, the hand pose, and the scene information using a machine-learning model.

Abstract: In one embodiment, a method includes, generating rays for casting into an artificial reality scene that includes one or more surfaces to determine whether the one or more surfaces are visible from a viewpoint. An origin and a trajectory of each ray are based on the viewpoint. The method includes applying a geometric transformation to the rays to modify their respective trajectory into the artificial reality scene. The geometric transformation is based on one or more distortion characteristics of a display system. The method includes determining, based on the modified trajectories of the rays, points of intersection of rays with the one or more surfaces in the artificial reality scene. The method includes providing, for display by the display system, color values generated based on the determined points of intersection.

Abstract: In one embodiment, a computing system may access an image to be displayed by a display. The system may determine one or more characteristics associated with a content of the image. The one or more characteristics may include a spatial frequency of the content in a spatial frequency domain. The system may determine a display persistence time period for the display to display the image based on the one or more characteristics associated with the content of the image. The system may configure the display to display the image using the display persistence time period.

Abstract: Techniques disclosed herein relate to micro light emitting diodes (micro-LEDs) for a display system. A display system includes an array of micro light emitting diodes (micro-LEDs), an array of output couplers optically coupled to the array of micro-LEDs and configured to extract light emitted by respective micro-LEDs in the array of micro-LEDs, a waveguide display, and display optics configured to couple the light emitted by the array of micro-LEDs and extracted by the array of output couplers into the waveguide display. Each output coupler in the array of output couplers is configured to direct a chief ray of the light emitted by a respective micro-LED in the array of micro-LEDs to a different respective direction.

Abstract: An optical system for a head mounted display (HMD) includes a display layer, an illumination layer, an optical combiner, and an active optics block. The display layer is configured to emit display light and the illumination layer includes an array of point light sources configured to emit calibration light. The optical combiner is configured to pass the display light to a front side of the optical system and to direct the calibration light to a camera. The active optics block includes an adjustable lens disposed between the illumination layer and the optical combiner, where the adjustable lens is configured to pass the calibration light to the optical combiner and to focus the display light for a user of the HMD. The active optics block is further configured to adjust an optical power of the adjustable lens based on a calibration image captured by the camera responsive to the calibration light.