Abstract: An augmented reality system generates computer-mediated reality on a client device. The client device has sensors including a camera configured to capture image data of an environment and a location sensor to capture location data describing a geolocation of the client device. The client device creates a three-dimensional (3-D) map with the image data and the location data for use in generating virtual objects to augment reality. The client device transmits the created 3-D map to an external server that may utilize the 3-D map to update a world map stored on the external server. The external server sends a local portion of the world map to the client device. The client device determines a distance between the client device and a mapping point to generate a computer-mediated reality image at the mapping point to be displayed on the client device.

Abstract: The present disclosure describes approaches to camera re-localization using a graph neural network (GNN). A re-localization model includes encoding an input image into a feature map. The model retrieves reference images from an image database of a previously scanned environment based on the feature map of the image. The model builds a graph based on the image and the reference images, wherein nodes represent the image and the reference images, and edges are defined between the nodes. The model may iteratively refine the graph through auto-aggressive edge-updating and message passing between nodes. With the graph built, the model predicts a pose of the image based on the edges of the graph. The pose may be a relative pose in relation to the reference images, or an absolute pose.

Abstract: A method of determining a position for a virtual object is described. A location of a client device is determined, and, based on the determined location a set of map segments is retrieved. A virtual object is determined to be displayed on the client device. Relation vectors between the virtual object and each map segment of the retrieved set of map segments are obtained. Each relation vector is weighted based on object parameters of the virtual object. A position to display the virtual object is determined based on the weighted relation vectors. The virtual object is provided for display on the client device as the determined position.

Abstract: The present disclosure describes approaches for evaluating interest points for localization uses based on a repeatability of the detection of the interest point in images capturing a scene that includes the interest point. The repeatability of interest points is determined by using a trained repeatability model. The repeatability model is trained by analyzing a time series of images of a scene and determining repeatability functions for each interest point in the scene. The repeatability function is determined by identifying which images in the time series of images allowed for the detection of the interest point by an interest point detection model.

Abstract: A client device associated with a user of a location-based application detects client devices associated with other users which are within proximity of the client device. This detection of other client devices may result in various application actions occurring, such as triggering a multi-user activity between users. Detection of client devices may be performed using personal area network devices of the client devices, such as Bluetooth. Proximity detection can occur when client devices are disconnected from an online system hosting the location-based application with the detection later being reported to the online system by one or both devices.

Abstract: An augmented reality (“AR”) device applies smooth correction methods to correct the location of the virtual objects presented to a user. The AR device may apply an angular threshold to determine whether a virtual object can be moved from an original location to a target location. An angular threshold is a maximum angle by which a line from the AR device to the virtual object can change within a timestep. Similarly, the AR device may apply a motion threshold, which is a maximum on the distance that a virtual object's location can be corrected based on the motion of the virtual object. Furthermore, the AR device may apply a pixel threshold to the correction of the virtual object's location. A pixel threshold is a maximum on the distance that a pixel projection of the virtual object can change based on the virtual object's change in location.

Abstract: Systems and methods for generating and storing metrics are described herein. In particular, a game server may receive game activity data from one or more client devices connected to the game server via a network. The game server sends the activity data to an anticheat server that uses one or more nodes to each calculate a portion of the activity data to generate one or more metrics. The metrics may indicate whether the data is indicative of cheating behavior within a parallel reality game. The nodes add their respective generated metrics to a relational database using prepared insert statements. The anticheat server may instruct the game server to take action with respect to one or more client devices if the metrics indicate that cheating behavior was exhibited within the data.

Abstract: A scene reconstruction model is disclosed that outputs a heightfield for a series of input images. The model, for each input image, predicts a depth map and extracts a feature map. The model builds a 3D model utilizing the predicted depth maps and camera poses for the images. The model raycasts the 3D model to determine a raw heightfield for the scene. The model utilizes the raw heightfield to sample features from the feature maps corresponding to positions on the heightfield. The model aggregates the sampled features into an aggregate feature map. The model regresses a refined heightfield based on the aggregate feature map. The model determines the final heightfield based on a combination of the raw heightfield and the refined heightfield. With the final heightfield, a client device may generate virtual content augmented on real-world images captured by the client device.

Abstract: A client device associated with a player of a location-based game detects client devices associated with other players which are within proximity of the player device. This detection of other player client devices may result in various game actions occurring, such as an exchange of game elements between players, game progress for a player, access to a game feature, or establishing a connection between players. Detection of player devices may be performed using personal area network devices of the client devices, such as Bluetooth. Proximity detection can occur when player client devices are disconnected from an online system hosting the location-based game with the detection later being reported to the online system by one or both devices.

Abstract: A system enables an arbitrary number of items to be indexed in a geographic region that provides a predictable query response time across a sharded database. Items indexed to the geographic region are stored on a single shard and additional items are added to that shard as long as an overflow condition indicative of undesirable query response times is not met. If the overflow condition is met the system expands the storage of items indexed to the geographic region to one or more additional shards in order to maintain predictable query response times. The system may maintain a shard count representing the total number of shards being used to store items corresponding to a geographic region, which can be used to query one or more relevant shards. The system may apply deterministic hashing in order to evenly distribute shards across database nodes of the sharded database.

Abstract: Events in an interactive application (e.g., a location-based parallel reality game) are triggered by a user's real-world activity meeting one or more criteria. A background process executing on the user's client device periodically extracts activity data from an activity monitoring application (e.g., a fitness application) and provides it to a server hosting the interactive application. The server determines whether received activity data triggers an event and, if so, sends a notification of the event to the client device.

Abstract: A multi-frame depth estimation model is disclosed. The model is trained and configured to receive an input image and an additional image. The model outputs a depth map for the input image based on the input image and the additional image. The model may extract a feature map for the input image and an additional feature map for the additional image. For each of a plurality of depth planes, the model warps the feature map to the depth plane based on relative pose between the input image and the additional image, the depth plane, and camera intrinsics. The model builds a cost volume from the warped feature maps for the plurality of depth planes. A decoder of the model inputs the cost volume and the input image to output the depth map.

Abstract: An image matching system for determining visual overlaps between images by using box embeddings is described herein. The system receives two images depicting a 3D surface with different camera poses. The system inputs the images (or a crop of each image) into a machine learning model that outputs a box encoding for the first image and a box encoding for the second image. A box encoding includes parameters defining a box in an embedding space. Then the system determines an asymmetric overlap factor that measures asymmetric surface overlaps between the first image and the second image based on the box encodings. The asymmetric overlap factor includes an enclosure factor indicating how much surface from the first image is visible in the second image and a concentration factor indicating how much surface from the second image is visible in the first image.

Abstract: A player may send virtual characters to travel to remote locations in a parallel-reality game. The remote locations are virtual locations corresponding to real-world locations other than where the player is located. The player can therefore send a virtual character to a virtual location without traveling to the corresponding real-world location themselves. A travel time and a path the character will travel along to reach the remote location is determined. Once the virtual character has reached the remote location, it stays there for a duration of time. Client devices of players near the remote location may display a visual representation of the virtual character. The virtual character may collect virtual items and interact with other players and return to the player with any virtual items it has collected or been given.

Abstract: The virtual location of a player in a location-based game is determined from the real-world location of the player's client device. The location-based game provides the player access to one or more chat room based on their location. To determine the locations of chat room, a server analyzes player locations in a geographic region, clusters player locations to identify centroids, and adjusts the clusters based on constraints. The server selects chat room locations (e.g., at points of interest) to more evenly balance the number of players in each chat room while complying with one or more restraints on the size of the geographic area served by each chat room.

Abstract: The present disclosure describes approaches to camera re-localization that improve the accuracy of re-localization determinations by performing simulated consistency checks for three-dimensional maps. Client devices associated with users of a location-based application transmit image scans to a game server, which divides the received scan data into mapping sets used to generate 3D maps of environments and validation sets used to test the accuracy of the maps. To perform the testing, the game server identifies query scans in the validation set having GPS coordinates within a threshold distance of the mapped location and uses the 3D map of the environment to generate a pose estimate for each frame. The results of the localization queries are analyzed by comparing differences between the localization pose estimates and differences between the poses of independent pairs of frames in the query scan to evaluate the accuracy of the 3D map.

Abstract: Computer-implemented methods and systems for locating virtual elements that can be used or collected by players of a parallel reality game having a virtual world that parallels at least a portion of the real world are provided. In particular, the location of virtual elements in the virtual world is determined based on data associated with one or more real world conditions. Virtual elements can be located in the virtual world at locations corresponding to locations in the real world that encourage safe and effective game play. Locating virtual elements in the virtual world based on data associated with real world conditions improves the link between the parallel virtual world and the real world, enhancing the illusion that the virtual world is another dimension of the real world that the player can interact with through the parallel reality game.

Abstract: An image localization system receives an image of a scene and generates a depth map for the image by inputting the image to a model trained for generating depth maps for images. The system determines surface normal vectors for the pixels in the depth map. The system clusters the surface normal vectors to identify regions in the image corresponding to planar surfaces. The system partitions the image into patches, each of which is a region of connected pixels in the image and corresponds to a cluster of surface normal vectors. The system rectifies the perspective distortion of patches and extracts perspective corrected features from the rectified patches. The system matches the perspective corrected features of the image with perspective corrected features of other images for three-dimensional re-localization.