Abstract: A scene representation model uses a 3D diffusion prior to push predicted 3D scene points towards plausible geometries during training. The plausible geometries are learned by training a second model (e.g., a diffusion model) on a set of scene agnostic training data from a variety of scenes. In other words, the second model encodes information about what geometries are plausible in the real world, enabling inferences to be made about the geometry for novel scenes where incomplete data is available.

Abstract: A system uses models to relocalize a mobile device. The system accesses an input image of a scene in a real-world environment, where the image was captured by a mobile device. The system applies a two-dimensional (2D) foundation model to the input image. The 2D foundation model is trained to determine an image vector representing characteristics of the input image. The system accesses a map representation of the real-world environment, where the map representation includes visual data that describes the real-world environment. The system applies a three-dimensional (3D) geospatial model to the map representation and the image vector. The 3D geospatial model is configured to output 3D splats representing the real-world environment. The system determines a pose of a camera that captured the input image using the 3D splats.

Abstract: A reference image and recorded sound of an environment of a client device are obtained. The recorded sound may be captured by a microphone of the client device in a period of time after generation of a localization sound by the client device. The location of the client device in the environment may be determined using the reference image and the recorded sound.

Abstract: A system performs image-based localization with an ensemble of localizers. The system receives a target frame from image data captured by a camera assembly of a client device. The system deploys an ensemble of localizers, each disparately trained to output a pose of the target frame and a model-specific confidence for the pose. The system calibrates each model-specific confidence by applying a model-specific calibration transformation to transform the model-specific confidence to a calibrated confidence. The system determines a final pose for the target frame by aggregating the poses output by the ensemble based on the calibrated confidences. The system may provide a visual positioning service (VPS) with the image-based localization. The system may also leverage the image-based localization to generate augmented reality content for presentation to a user.

Abstract: A reference image and recorded sound of an environment of a client device are obtained. The recorded sound may be captured by a microphone of the client device in a period of time after generation of a localization sound by the client device. The location of the client device in the environment may be determined using the reference image and the recorded sound.

Abstract: A method for determining a metric relative pose between a target image and a reference image is disclosed. The method includes receiving the target image depicting a scene captured by a camera assembly of a client device. The method includes applying a machine-learning model to the target image to determine a metric relative pose between the target image and a reference image, wherein the metric relative pose represents a transformation from a pose of the reference image to a pose of the target image that is scaled to physical dimensions of the scene. The machine-learning model may include a keypoint network for determining a keypoint distribution including the spatial coordinates of keypoints extracted from each image. The machine-learning model may establish correspondences between the keypoint distribution of the target image and the keypoint distribution of the reference image. Based on the identified correspondences, the machine-learning model may regress the metric relative pose.

Abstract: This disclosure pertains to a scene-agnostic, map-relative pose regression method. The pose regressor is conditioned on a scene-specific map representation such that its pose predictions are relative to the scene map. This allows training of the pose regressor across multiple scenes to learn the generic relation between a scene-specific map representation and the camera pose. The map-relative pose regressor can then be applied to new map representations.

Abstract: A relocalizer model for an environment is trained using an iterative process. To initialize the relocalizer model, an initial image is registered with its camera pose established as the reference. In each subsequent iteration of training, the relocalizer model is applied to additional images to predict pose estimates for the images. The images and their pose estimates are then leveraged in retraining of the relocalizer model. In general, the training of the relocalizer model entails extracting scene coordinates for pixels of a training image. The scene coordinates are then projected into a projection based on the pose estimate of the training image. A loss is calculated between the projection and the training image. And parameters of the relocalizer model are adjusted to minimize the loss. The iterative training may continue until an end condition is met. The trained relocalizer model is configured to input an image of the environment and to output the camera pose for the image.

Abstract: The present disclosure describes approaches to camera re-localization using a graph neural network (GNN). A re-localization model includes encoding an input image into a feature map. The model retrieves reference images from an image database of a previously scanned environment based on the feature map of the image. The model builds a graph based on the image and the reference images, wherein nodes represent the image and the reference images, and edges are defined between the nodes. The model may iteratively refine the graph through auto-aggressive edge-updating and message passing between nodes. With the graph built, the model predicts a pose of the image based on the edges of the graph. The pose may be a relative pose in relation to the reference images, or an absolute pose.

Abstract: A machine learned model may calculate a relative pose between a pair of overlapping images of a scene. The model may be applied to predict one or more errors (e.g., translation error and/or rotation error) in the relative pose between the pair of overlapping images. The model may leverage epipolar geometry to compare features of the overlapping images in a dense manner. For example, the two-view geometry model may incorporate the epipolar geometry into an attention layer of a neural network for one or more different fundamental matrix hypotheses. The model may output one or more predicted errors for the pair of images along with a proposed fundamental matrix hypothesis. A client device may select a fundamental matrix associated with the lowest predicted one or more errors. The client device may then display content that accounts for the predicted one or more errors.

Abstract: A set of training images of one or more environments and corresponding metadata are received. The metadata includes camera pose and intrinsics. A relocalizer model is trained using the set of training images and the corresponding metadata to generate predict scene coordinates corresponding to pixels in an image of an environment. The relocalizer model includes a scene-agnostic convolutional network and a scene-specific regression network. A set of query images of an environment is received and the trained relocalizer model is applied to the set of query images of the environment to generate predicted scene coordinates corresponding to the pixels in a query image. A pose solver algorithm is applied to the predicted scene coordinates to generate a camera pose.

Abstract: A reference image and recorded sound of an environment of a client device are obtained. The recorded sound may be captured by a microphone of the client device in a period of time after generation of a localization sound by the client device. The location of the client device in the environment may be determined using the reference image and the recorded sound.

Abstract: A method or a system for map-free visual relocalization of a device. The system obtains a reference image of an environment captured by a reference pose. The system also receives a query image taken by a camera of the device. The system determines a relative pose of the camera of the device relative to the reference camera based in part on the reference image and the query image. The system determines a pose of the query camera in the environment based on the reference pose and the relative pose.

Abstract: The present disclosure describes approaches to camera re-localization using a graph neural network (GNN). A re-localization model includes encoding an input image into a feature map. The model retrieves reference images from an image database of a previously scanned environment based on the feature map of the image. The model builds a graph based on the image and the reference images, wherein nodes represent the image and the reference images, and edges are defined between the nodes. The model may iteratively refine the graph through auto-aggressive edge-updating and message passing between nodes. With the graph built, the model predicts a pose of the image based on the edges of the graph. The pose may be a relative pose in relation to the reference images, or an absolute pose.