Abstract: Examples of the disclosure describe systems and methods for presenting virtual content on a wearable head device. In some embodiments, a state of a wearable head device is determined by minimizing a total error based on a reduced weight associated with a reprojection error. A view reflecting the determined state of the wearable head device is presented via a display of the wearable head device. In some embodiments, a wearable head device calculates a preintegration term based on the image data received via a sensor of the wearable head device and the inertial data received via a first IMU and a second IMU of the wearable head device. The wearable head device estimates a position of the device based on the preintegration term, and the wearable head device presents the virtual content based on the position of the device.

Abstract: The present disclosure generally relates to systems and methods for searching media content. In some implementation examples, a search system receives an input query, generates a query embedding of the input query, and generates a bias mitigation transformation associated with a sensitive attribute. Based on the query embedding and the bias mitigation transformation, the search system generates a transformed query embedding that suppresses at least a portion of the query embedding related to the sensitive attribute. Using the transformed query embedding, the search system executes a similarity search in a media embedding model to identify one or more media embeddings that are similar to the transformed query embedding and transmits the one or more media embeddings.

Abstract: A first region within image data corresponding to a first portion of a user is determined, and a classifier associated with the first portion of the user is determined. A second region within the image data that includes the first region is determined. Within the second region, first pixels corresponding to the first portion of the user, second pixels corresponding to a second portion of the user adjoined to the first portion of the user, and third pixels corresponding to an environment of the user are identified. Based on identifying the second pixels, the classifier associated with the first portion of the user is confirmed. Upon the confirmation an item is rendered on the first portion of the user.

Abstract: To determine the head pose of a user, a head-mounted display system having an imaging device can obtain a current image of a real-world environment, with points corresponding to salient points which will be used to determine the head pose. The salient points are patch-based and include: a first salient point being projected onto the current image from a previous image, and with a second salient point included in the current image being extracted from the current image. Each salient point is subsequently matched with real-world points based on descriptor-based map information indicating locations of salient points in the real-world environment. The orientation of the imaging devices is determined based on the matching and based on the relative positions of the salient points in the view captured in the current image. The orientation may be used to extrapolate the head pose of the wearer of the head-mounted display system.

Abstract: Methods and apparatus for providing a representation of an environment, for example, in an XR system, and any suitable computer vision and robotics applications. A representation of an environment may include one or more planar features. The representation of the environment may be provided by jointly optimizing plane parameters of the planar features and sensor poses that the planar features are observed at. The joint optimization may be based on a reduced matrix and a reduced residual vector in lieu of the Jacobian matrix and the original residual vector.

Abstract: The present disclosure describes a system and method for implementing a virtual shoe try-on feature. The method includes predicting, using a machine learning model, a first plurality of pixels including pixels of an image corresponding to a foot in the image and adding a first virtual cylindrical object to a three-dimensional model of a shoe such that a portion of the first virtual cylindrical object is positioned within the three-dimensional model of the shoe. The method also include determining a set of pixels including pixels of the first plurality of pixels that intersect with the first virtual cylindrical object and removing, from the three-dimensional model of the shoe, a first portion of the three-dimensional model of the shoe that intersects with the set of pixels.

Abstract: A method of efficiently and accurately computing a pose of an image with respect to other image information. The image may be acquired with a camera on a portable device and the other information may be a map, such that the computation of pose localizes the device relative to the map. Such a technique may be applied in a cross reality system to enable devices to efficiently and accurately access previously persisted maps. Localizing with respect to a map may enable multiple cross reality devices to render virtual content at locations specified in relation to those maps, providing an enhanced experience for uses of the system. The method may be used in other devices and for other purposes, such as for navigation of autonomous vehicles.

Abstract: Methods and apparatus for providing a representation of an environment, for example, in an XR system, and any suitable computer vision and robotics applications. A representation of an environment may include one or more planar features. The representation of the environment may be provided by jointly optimizing plane parameters of the planar features and sensor poses that the planar features are observed at. The joint optimization may be based on a reduced matrix and a reduced residual vector in lieu of the Jacobian matrix and the original residual vector.

Abstract: The present disclosure describes a system and method for implementing a virtual shoe try-on feature. The method includes receiving, from a database, a three-dimensional model of a shoe with a first annotated point on the three-dimensional model of the shoe, a second annotated point on the three-dimensional model of the shoe, and a third annotated point on the three-dimensional model of the shoe. The method also includes predicting, using a machine learning model and based on an image of a foot, one or more features of the foot, generating, based on the first annotated point, the second annotated point, the third annotated point, and the one or more features, a two-dimensional model of the shoe using the three-dimensional model of the shoe, and superimposing the two-dimensional model of the shoe onto the image of the foot.

Abstract: A cross reality system enables any of multiple devices to efficiently and accurately access previously stored maps and render virtual content specified in relation to those maps. The cross reality system may include a cloud-based localization service that responds to requests from devices to localize with respect to a stored map. The request may include one or more sets of feature descriptors extracted from an image of the physical world around the device. Those features may be posed relative to a coordinate frame used by the local device. The localization service may identify one or more stored maps with a matching set of features. Based on a transformation required to align the features from the device with the matching set of features, the localization service may compute and return to the device a transformation to relate its local coordinate frame to a coordinate frame of the stored map.

Abstract: One or more computing devices, systems, and/or methods for end-to-end encryption for multiple recipient devices are provided. A first registration, comprising a first device public key, is created for a first device. A second registration, comprising a second device public key, is created for a second device. A first notify message of the second registration and second device public key is provided to the first device. A second notify message is provided to the second device of the first registration and first device public key. A secure communication invite is routed to the first device. An encrypted message, comprising a first device private key, is routed from the first device to the second device. End-to-end encrypted communication between a sender device and the first device and the second device using the first device private key is facilitated.

Abstract: A cross reality system that provides an immersive user experience by storing persistent spatial information about the physical world that one or multiple user devices can access to determine position within the physical world and that applications can access to specify the position of virtual objects within the physical world. Persistent spatial information enables users to have a shared virtual, as well as physical, experience when interacting with the cross reality system. Further, persistent spatial information may be used in maps of the physical world, enabling one or multiple devices to access and localize into previously stored maps, reducing the need to map a physical space before using the cross reality system in it. Persistent spatial information may be stored as persistent coordinate frames, which may include a transformation relative to a reference orientation and information derived from images in a location corresponding to the persistent coordinate frame.

Abstract: To determine the head pose of a user, a head-mounted display system having an imaging device can obtain a current image of a real-world environment, with points corresponding to salient points which will be used to determine the head pose. The salient points are patch-based and include: a first salient point being projected onto the current image from a previous image, and with a second salient point included in the current image being extracted from the current image. Each salient point is subsequently matched with real-world points based on descriptor-based map information indicating locations of salient points in the real-world environment. The orientation of the imaging devices is determined based on the matching and based on the relative positions of the salient points in the view captured in the current image. The orientation may be used to extrapolate the head pose of the wearer of the head-mounted display system.

Abstract: Examples of the disclosure describe systems and methods for presenting virtual content on a wearable head device. In some embodiments, a state of a wearable head device is determined by minimizing a total error based on a reduced weight associated with a reprojection error. A view reflecting the determined state of the wearable head device is presented via a display of the wearable head device. In some embodiments, a wearable head device calculates a preintegration term based on the image data received via a sensor of the wearable head device and the inertial data received via a first IMU and a second IMU of the wearable head device. The wearable head device estimates a position of the device based on the preintegration term, and the wearable head device presents the virtual content based on the position of the device.

Abstract: A system form localizing an electronic device with dynamic buffering identifies, from the buffer, a first set of features that is extracted from a first image captured by the electronic device and receives, at the system, a second set of features that is extracted from a second image captured by the electronic device. The system further determines a first characteristic for the first set of features and a second characteristic for the second set of features and determines whether a triggering condition for dynamically changing a size of the buffer is satisfied based at least in part upon the first characteristic for the first set of features and the second characteristic for the second set of features.

Abstract: A cross reality system enables any of multiple devices to efficiently and accurately access previously stored maps and render virtual content specified in relation to those maps. The cross reality system may include a cloud-based localization service that responds to requests from devices to localize with respect to a stored map. The request may include one or more sets of feature descriptors extracted from an image of the physical world around the device. Those features may be posed relative to a coordinate frame used by the local device. The localization service may identify one or more stored maps with a matching set of features. Based on a transformation required to align the features from the device with the matching set of features, the localization service may compute and return to the device a transformation to relate its local coordinate frame to a coordinate frame of the stored map.

Abstract: A cross reality system that provides an immersive user experience by storing persistent spatial information about the physical world that one or multiple user devices can access to determine position within the physical world and that applications can access to specify the position of virtual objects within the physical world. Persistent spatial information enables users to have a shared virtual, as well as physical, experience when interacting with the cross reality system. Further, persistent spatial information may be used in maps of the physical world, enabling one or multiple devices to access and localize into previously stored maps, reducing the need to map a physical space before using the cross reality system in it. Persistent spatial information may be stored as persistent coordinate frames, which may include a transformation relative to a reference orientation and information derived from images in a location corresponding to the persistent coordinate frame.

Abstract: A method includes extracting, using a backbone of a machine learning model, a plurality of features from an image of a foot and predicting, using a first portion of the machine learning model and based on one or more features of the plurality of features, a first aspect of the foot. The method also includes predicting, using a second portion of the machine learning model and based on one or more features of the plurality of features, a second aspect of the foot different from the first aspect, generating, using at least the first aspect and the second aspect, a two-dimensional model of a shoe, and superimposing the two-dimensional model of the shoe onto the image of the foot.

Abstract: To determine the head pose of a user, a head-mounted display system having an imaging device can obtain a current image of a real-world environment, with points corresponding to salient points which will be used to determine the head pose. The salient points are patch-based and include: a first salient point being projected onto the current image from a previous image, and with a second salient point included in the current image being extracted from the current image. Each salient point is subsequently matched with real-world points based on descriptor-based map information indicating locations of salient points in the real-world environment. The orientation of the imaging devices is determined based on the matching and based on the relative positions of the salient points in the view captured in the current image. The orientation may be used to extrapolate the head pose of the wearer of the head-mounted display system.

Abstract: One or more computing devices, systems, and/or methods for end-to-end encryption for multiple recipient devices are provided. A first registration, comprising a first device public key, is created for a first device. A second registration, comprising a second device public key, is created for a second device. A first notify message of the second registration and second device public key is provided to the first device. A second notify message is provided to the second device of the first registration and first device public key. A secure communication invite is routed to the first device. An encrypted message, comprising a first device private key, is routed from the first device to the second device. End-to-end encrypted communication between a sender device and the first device and the second device using the first device private key is facilitated.