Abstract: An augmented reality system generates computer-mediated reality on a client device. The client device has sensors including a camera configured to capture image data of an environment. The augmented reality system generates a first 3D map of the environment around the client device based on captured image data. The server receives image data captured from a second client device in the environment and generates a second 3D map of the environment. The server links the first and second 3D together in a singular 3D map. The singular 3D map may be a graphical representation of the real world using nodes that represent 3D maps generated by image data captured at client devices and edges that represent transformations between the nodes.

Abstract: An AR client device generates and uses a background model to identify portions of images that depict the sky. A background model is a model that represents where the sky is visible for the client device. To identify a sky background portion of an image, a client device can map an image onto the background model and thereby determine which portion of the image represents the sky. The client device can use the identified sky background portion to augment the image to include AR content in the sky. To generate the background model, the client device applies a background detection model to a set of images to generate background probability images. The background probability images are mapped onto a background model using orientation data captured by the client device to update the background model based on the background probability image.

Abstract: A parallel-reality game uses a virtual game board having tiles placed over an identified traversable space corresponding to flat regions of a scene. A game board generation module receives one or more images of the scene captured by a camera of a mobile device. The game board generation module obtains a topographical mesh of the scene based on the received one or more images. The game board generation module then identifies a traversable space within the scene based on the obtained topographical mesh. The game board generation module determines a location for each of a set of polygon tiles in the identified traversable space. The game board generation module also allows for queries to identify parts of the game board that meet one or more provided criterion.

Abstract: A client device and a controller allow a user to control AR content by selecting real-world locations or objects. The client receives position data indicating a position and orientation of the controller, the position data defining an axis of the controller. The client device performs ray casting to determine a location in a 3D map of a real world that intersects the axis. The client device receives a selection indication (e.g., a user pressing a button on the controller). The client device selects, subsequent to the selection indication, the location in the 3D map that intersects the axis as a waypoint. The client device defines a route for a virtual object based on the waypoint.

Abstract: The present disclosure describes approaches to camera re-localization that improve the accuracy of re-localization determinations by performing simulated consistency checks for three-dimensional maps. Client devices associated with users of a location-based application transmit image scans to a game server, which divides the received scan data into mapping sets used to generate 3D maps of environments and validation sets used to test the accuracy of the maps. To perform the testing, the game server identifies query scans in the validation set having GPS coordinates within a threshold distance of the mapped location and uses the 3D map of the environment to generate a pose estimate for each frame. The results of the localization queries are analyzed by comparing differences between the localization pose estimates and differences between the poses of independent pairs of frames in the query scan to evaluate the accuracy of the 3D map.

Abstract: A head-mounted device (HMD) may include a front portion, a rear portion and one or more bands. The front portion may include an optical display configured to output image light to a user's eyes. The rear portion may be arranged with the front portion to balance the HMD in weight. The one or more bands connect the front portion and the rear portion so that the two portions rest on either side of the user's head.

Abstract: A model predicts the geometry of both visible and occluded traversable surfaces from input images. The model may be trained from stereo video sequences, using camera poses, per-frame depth, and semantic segmentation to form training data, which is used to supervise an image to image network. In various embodiments, the model is applied to a single RGB image depicting a scene to produce information describing traversable space of the scene that includes occluded traversable. The information describing traversable space can include a segmentation mask of traversable space (both visible and occluded) and non-traversable space and a depth map indicating an estimated depth to traversable surfaces corresponding to each pixel determined to correspond to traversable space.

Abstract: A system determines an accuracy of a set containing a plurality of sensor location measurements of a client device and generating a set of sensor location measurements labeled with an associated accuracy estimate, is presented. The system receives a plurality of sensor location measurements of the client device generated by a location sensor. The system may determine the accuracy of a sensor location measurement by comparing the sensor location measurement to a reference location measurement, such as VIO location measurements computed using VIO data. The system computes a first set of location translations for the set of sensor location measurements and a second set of location translations for the set of VIO location measurements. The system may calculate a measurement difference between each corresponding pair of location translations from the first and second set, identify measurement differences that exceed a threshold, and label the corresponding sensor location measurement as inaccurate.

Abstract: A reference image and recorded sound of an environment of a client device are obtained. The recorded sound may be captured by a microphone of the client device in a period of time after generation of a localization sound by the client device. The location of the client device in the environment may be determined using the reference image and the recorded sound.

Abstract: Methods and systems for creating accurate three-dimensional representations of environments using neural radiance fields regularized by denoising diffusion models are disclosed. Receiving a plurality of images representing an environment, a scene representation model is trained to create the three-dimensional model of the environment. Using these images, the virtual rays are sampled from training viewpoints within the environment. The scene representation model is then applied to these rays to generate simulated images of the environment from the training viewpoints. These simulated images undergo a regularization process that uses a denoising diffusion model to determine color gradients and depth gradients in each simulated image. The scene representation model is trained with this data to create the final three-dimensional model of the environment. This model is provided to the requesting client device to generate the three-dimensional representation and create a virtual object within the environment.

Abstract: A depth prediction model for predicting a depth map from an input image is disclosed. The depth prediction model leverages wavelet decomposition to minimize computations. The depth prediction model comprises a plurality of encoding layers, a coarse prediction layer, a plurality of decoding layers, and a plurality of inverse discrete wavelet transforms (IDWTs). The encoding layers are configured to input the image and to downsample the image into feature maps including a coarse feature map. The coarse depth prediction layer is configured to input the coarse feature map and to output a coarse depth map. The decoding layers are configured to input the feature maps and to predict wavelet coefficients based on the feature maps. The IDWTs are configured to upsample the coarse depth map based on the predicted wavelet coefficients to the final depth map at the same resolution as the input image.

Abstract: The present disclosure describes an online system that applies template-based facial augmentations to images as part of an augmented reality experience. A facial augmentation template is a template for an augmented reality (AR) modification to apply to a user's face. These templates include a structure for how the augmentation should be applied to the face and display parameters that a user can set for how the templates should be rendered (e.g., color, specularity, or opacity). The online system identifies feature points on a 3D model of the user's face that correspond to features on the user's actual face, and localizes the template relative to the 3D model by mapping anchor points of the template to corresponding feature points on the 3D model. The online system renders the facial augmentation template at the localization to generate an augmented image of the user's face.

Abstract: A machine learning model classifies points of interest in a parallel reality game hosted by a server. The server generates training data sets that include verified properties for points of interest. The machine learning model may predict unverified properties for points of interest. Players in the parallel reality game may input properties for the points of interest. The machine learning model use the received properties from players as inputs to the machine learning model to verify unverified properties or generate new properties for the points of interest. The server may classify the points of interest as suitable for particular activities, and the server may use the classifications for future activities within the parallel reality game.

Abstract: An online system uses a pose prior model and a pose objective function to estimate the pose of a client device. A pose prior model is a model for prior information known about client devices and their poses without reference to a particular client device and its pose data. The online system receives pose data from a client device and computes an estimated pose for the client device based on the received pose data, the pose prior model, and a generated initial candidate pose for the client device. The online system uses these as inputs to a pose objective function and optimizes the pose objective function to estimate a pose for the client device. The online system transmits this estimated pose to the client device, and may use the estimated pose as the pose for the client device for the purposes of delivering content to the user.

Abstract: A system includes a first sensor system and a second sensor system. The first sensor system includes a first internal clock, a first sensor configured to generate data describing an environment, and a Universal Serial Bus (USB) module. The second sensor system includes a second internal clock and a controller configured to perform a precision time protocol (PTP) to synchronize the second internal clock with the first internal clock. The precision time protocol includes transmitting timestamped messages encoded in USB bulk messages between the first sensor system and the second sensor system. The timestamped messages may be encoded in USB musical instrument digital interface (MIDI) bulk messages.

Abstract: The present disclosure describes a method for estimating a pose of a client device using a magnetic field vector map. The method includes receiving a plurality of magnetic field measurements from a plurality of client devices, each magnetic field measurement describing a magnetic field vector at a geographic location. The method further includes grouping the magnetic field measurements into one or more region groups, aggregating the magnetic field measurements in each region group to generate a probability distribution of magnetic field vectors associated with the geographic region, determining a magnetic field vector within each geographic region, and generating a magnetic field vector map. Based on the magnetic field vector map, the method may include estimating a pose of a client device based on a user location of the client device and received magnetic field vector from the client device.

Abstract: The present disclosure describes approaches to camera re-localization that improve the speed and accuracy with which pose estimates are generated by fusing output of a computer vision algorithm with data from a prior model of a geographic area in which a user is located. For each candidate pose estimate output by the algorithm, a game server maps the estimate to a position on the prior model (e.g., a specific cell on a heatmap-style histogram) and retrieves a probability corresponding to the mapped position. A data fusion module fuses, for each candidate pose estimate, a confidence score generated by the computer vision algorithm with the location probability from the prior model to generate an updated confidence score. If an updated confidence score meets or exceeds a score threshold, a re-localization module initiates a location-based application (e.g., a parallel reality game) based on the associated candidate pose estimate.