Abstract: A push gateway brokers messages between backend services and client devices connected to the gateway. The gateway includes a plurality of messaging frontend servers, a messaging database, and a plurality of topic servers. The messaging frontend servers manage the socket connections connecting the client devices to the push gateway. The messaging frontend servers may further monitor the socket connections, collecting analytical data on the client devices and the connections. The messaging database is a key-value database that subscribes messaging frontend servers to topics. The topic servers receive publish requests from the backend servers to identify subscribed messaging frontend servers in the messaging database to push messages out to. The frontend messaging servers subsequently push out the message to the client devices subscribed to the topic.

Abstract: An augmented reality system generates computer-mediated reality on a client device. The client device has sensors including a camera configured to capture image data of an environment. The augmented reality system generates a first 3D map of the environment around the client device based on captured image data. The server receives image data captured from a second client device in the environment and generates a second 3D map of the environment. The server links the first and second 3D together in a singular 3D map. The singular 3D map may be a graphical representation of the real world using nodes that represent 3D maps generated by image data captured at client devices and edges that represent transformations between the nodes.

Abstract: The present disclosure describes approaches to camera re-localization that improve the speed and accuracy with which pose estimates are generated by fusing output of a computer vision algorithm with data from a prior model of a geographic area in which a user is located. For each candidate pose estimate output by the algorithm, a game server maps the estimate to a position on the prior model (e.g., a specific cell on a heatmap-style histogram) and retrieves a probability corresponding to the mapped position. A data fusion module fuses, for each candidate pose estimate, a confidence score generated by the computer vision algorithm with the location probability from the prior model to generate an updated confidence score. If an updated confidence score meets or exceeds a score threshold, a re-localization module initiates a location-based application (e.g., a parallel reality game) based on the associated candidate pose estimate.

Abstract: A wearable augmented reality device is described. The device comprises a headband, brim, optical displays, and environmental sensors. The headband wraps at least partially around a head of a user and distributes the weight of components of the device across the head of the user, supporting the device. The brim is coupled to the headband such that the brim extends away from the headband, over the eyes of the user. The brim includes a heatsink that releases heat from the electronics of the device into the environment. The optical displays are coupled to the underside of the brim. The optical displays both present image light and allow environmental light to pass through. The environmental sensors are couples to the forward edge of the brim and capture data about the environment.

Abstract: An augmented reality (“AR”) device applies smooth correction methods to correct the location of the virtual objects presented to a user. The AR device may apply an angular threshold to determine whether a virtual object can be moved from an original location to a target location. An angular threshold is a maximum angle by which a line from the AR device to the virtual object can change within a timestep. Similarly, the AR device may apply a motion threshold, which is a maximum on the distance that a virtual object's location can be corrected based on the motion of the virtual object. Furthermore, the AR device may apply a pixel threshold to the correction of the virtual object's location. A pixel threshold is a maximum on the distance that a pixel projection of the virtual object can change based on the virtual object's change in location.

Abstract: A system and method for determining a location of a client device is described herein. In particular, a client device receives images captured by a camera at the client device. The client device identifies features in the images. The features may be line junctions, lines, curves, or any other features found in images. The client device retrieves a 3D map of the environment from the map database and compares the identified features to the 3D map of the environment, which includes map features such as map line junctions, map lines, map curves, and the like. The client device identifies a correspondence between the features identified from the images and the map features and determines a location of the client device in the real world based on the correspondence. The client device may display visual data representing a location in a virtual world corresponding to the location in the real world.

Abstract: Systems and methods for providing a shared augmented reality environment are provided. In particular, the latency of communication is reduced by using a peer-to-peer protocol to determine where to send datagrams. Datagrams describe actions that occur within the shared augmented reality environment, and the processing of datagrams is split between an intermediary node of a communications network (e.g., a cell tower) and a server. As a result, the intermediary node may provide updates to a local state of a client device when a datagram is labelled peer-to-peer, and otherwise provides updates to the master state on the server. This may reduce the latency of communication and allow users of the location-based parallel reality game to see actions occur more quickly in the shared augmented reality environment.

Abstract: An image matching system for determining visual overlaps between images by using box embeddings is described herein. The system receives two images depicting a 3D surface with different camera poses. The system inputs the images (or a crop of each image) into a machine learning model that outputs a box encoding for the first image and a box encoding for the second image. A box encoding includes parameters defining a box in an embedding space. Then the system determines an asymmetric overlap factor that measures asymmetric surface overlaps between the first image and the second image based on the box encodings. The asymmetric overlap factor includes an enclosure factor indicating how much surface from the first image is visible in the second image and a concentration factor indicating how much surface from the second image is visible in the first image.

Abstract: A computer system generates stereo image data from monocular images. The system generates depth maps for single images using a monocular depth estimation method. The system converts the depth maps to disparity maps and uses the disparity maps to generate additional images forming stereo pairs with the monocular images. The stereo pairs can be used to form a stereo image training data set for training various models, including depth estimation models or stereo matching models.

Abstract: Processing of actions within a shared augmented reality experience is split between an edge node of a communications network (e.g., a cell tower) and a server. As a result, computation of the current state may be sharded naturally based on real-world location, with state updates generally provided by the edge node and the server providing conflict resolution based on a master state (e.g., where actions connected to different edge nodes potentially interfere with each other). In this way, latency may be reduced as game actions are communicated between clients connected to the same edge node using a peer-to-peer (P2P) protocol without routing the actions via the game server.

Abstract: A method and system for verifying a client device's location in a parallel reality game hosted by a server. The client transmits its location to the server and receives verification instructions comprising a landmark and a verification pathway. The client prompts a player to capture image data of the landmark and, in response, receives a first set of image data of the landmark from an initial perspective. The client determines whether the first set of image data matches to the landmark before prompting the player to move along the verification pathway while capturing image data. The client receives a second set of image data of the landmark from a moving perspective. The client determines whether the second set of image data matches to an expected change in perspective of the landmark. Upon completion of the verification instructions, the client confirms to the server the client's location.

Abstract: A dataflow hierarchy protocol is implemented by one or more devices to optimize how the one or more devices process datagrams for network communications. The dataflow hierarchy considers various available network pathways for dataflow. A device implementing the dataflow hierarchy selects one or more of the available network pathways to provide low latency in data communication with other devices. The device may sample various available network pathways to determine pathway metrics (e.g., latency) and select one or more network pathways based on the metrics. The available network pathways can include pathways through one or more intermediary nodes, such as pathways through a game server, pathways through a cell tower, and pathways through a network.

Abstract: A client device can display various augmented reality and virtual reality images to a user. For example, the client device captures images of an external environment and identifies objects in the environment. If an object satisfies a virtual environment criterion, the client device displays a window to a virtual environment at the location of the object in the environment. In another example, the client device includes a light assembly that can generate light. If the user turns the light assembly on, virtual objects displayed in the environment will gather to a region highlighted by the light from the light assembly. In another example, the client device estimates depth information of an object in the environment. The client device then displays a virtual object in the environment based on the depth information and may also display the virtual object interacting with an object in the external environment.

Abstract: A model predicts the geometry of both visible and occluded traversable surfaces from input images. The model may be trained from stereo video sequences, using camera poses, per-frame depth, and semantic segmentation to form training data, which is used to supervise an image to image network. In various embodiments, the model is applied to a single RGB image depicting a scene to produce information describing traversable space of the scene that includes occluded traversable. The information describing traversable space can include a segmentation mask of traversable space (both visible and occluded) and non-traversable space and a depth map indicating an estimated depth to traversable surfaces corresponding to each pixel determined to correspond to traversable space.

Abstract: A location-based parallel reality game may use regions to determine how to disperse virtual elements, locate virtual experiences, and/or define game objectives. To determine the regions of the geographic area, a server divides the geographic region into cells. The server assigns real-world data to each of the cells and constructs a graph of nodes and edges based on the cells. Each node corresponds to a cell, and each edge connects nodes that correspond to adjacent cells. The server partitions the graph into contiguous segments based on the real-world data. The server defines regions of the geographic area based on the contiguous segments and stores the defined regions in a data store. The server may hierarchically partition the regions into higher order regions until each region meets a set of criteria.

Abstract: The virtual location of a player in a location-based game is determined from the real-world location of the player's client device. The location-based game provides the player access to one or more chat room based on their location. To determine the locations of chat room, a server analyzes player locations in a geographic region, clusters player locations to identify centroids, and adjusts the clusters based on constraints. The server selects chat room locations (e.g., at points of interest) to more evenly balance the number of players in each chat room while complying with one or more restraints on the size of the geographic area served by each chat room.

Abstract: A method for training a depth estimation model with depth hints is disclosed. For each image pair: for a first image, a depth prediction is determined by the depth estimation model and a depth hint is obtained; the second image is projected onto the first image once to generate a synthetic frame based on the depth prediction and again to generate a hinted synthetic frame based on the depth hint; a primary loss is calculated with the synthetic frame; a hinted loss is calculated with the hinted synthetic frame; and an overall loss is calculated for the image pair based on a per-pixel determination of whether the primary loss or the hinted loss is smaller, wherein if the hinted loss is smaller than the primary loss, then the overall loss includes the primary loss and a supervised depth loss between depth prediction and depth hint. The depth estimation model is trained by minimizing the overall losses for the image pairs.

Abstract: A client device associated with a player of a location-based game detects client devices associated with other players which are within proximity of the player device. This detection of other player client devices may result in various game actions occurring, such as an exchange of game elements between players, game progress for a player, access to a game feature, or establishing a connection between players. Detection of player devices may be performed using personal area network devices of the client devices, such as Bluetooth. Proximity detection can occur when player client devices are disconnected from an online system hosting the location-based game with the detection later being reported to the online system by one or both devices.

Abstract: A system enables an arbitrary number of items to be indexed in a geographic region that provides a predictable query response time across a sharded database. Items indexed to the geographic region are stored on a single shard and additional items are added to that shard as long as an overflow condition indicative of undesirable query response times is not met. If the overflow condition is met the system expands the storage of items indexed to the geographic region to one or more additional shards in order to maintain predictable query response times. The system may maintain a shard count representing the total number of shards being used to store items corresponding to a geographic region, which can be used to query one or more relevant shards. The system may apply deterministic hashing in order to evenly distribute shards across database nodes of the sharded database.

Abstract: A system provides an augmented reality (AR) experience in a parallel-reality application in which a geography of a virtual world parallels a geography of the real world. The system receives a connection request from a client device and receives a route that the client device traversed in the real world. The route comprises a plurality of locations in the real world. The system determines a plurality of virtual locations to place virtual elements at, where each virtual element corresponds to a location in the real world of the route. The system updates a global state of the AR experience to include the plurality of virtual elements at the plurality of virtual locations. The system provides, to a second client device, AR data including some of the virtual locations for display of some of the virtual elements at the corresponding locations in the real world.