Abstract: Techniques of rendering an image of a mobile device in a virtual environment include a combination of depth images, ultra-wideband (UWB) signals, and inertial measurement unit (IMU) signals to track the position and orientation of a mobile device. The depth images may be generated using an RGB image acquired by the head-mounted display (HMD). The UWB and IMU signals may be generated by the HMD, although in some implementations UWB and HMD signals may also be generated by the mobile device. In some implementations, the depth images are sufficient to track the position and orientation of the mobile device. In some implementations, a combination of UWB and IMU signals are sufficient to track the position and orientation of the mobile device. In some implementations, a combination of the depth images and UWB and IMU signals are used to track the position and orientation of the mobile device.

Abstract: Systems and methods for generating and using primary-ray based implicit neural representations of three-dimensional objects. In some examples, an implicit neural representation of an object may be generated by training a neural network using training examples that include a reference point, a ray, a perpendicular foot of the ray relative to the reference point, and a distance indicator indicating the shortest distance along the ray between the perpendicular foot and a surface of the object, and, optionally, an intersection indicator indicating whether the ray will intersect with the object. Once trained, the neural network may then be used to generate predictions of whether a ray will intersect with the object, and/or predictions of the shortest distance along a ray between its perpendicular foot and a surface of an object, based on a given reference point, ray, and perpendicular foot of the ray relative to the reference point.

Abstract: A method performed by a computing system comprises generating text from audio data and determining an end portion of the text to include in a summarization of the text based on a length of a portion of the audio data from which the text was generated and which ends with a proposed end portion and a time value associated with the proposed end portion, the proposed end portion including a word from the text.

Abstract: Systems and methods for training models to predict dense correspondences across images such as human images. A model may be trained using synthetic training data created from one or more 3D computer models of a subject. In addition, one or more geodesic distances derived from the surfaces of one or more of the 3D models may be used to generate one or more loss values, which may in turn be used in modifying the model's parameters during training.

Abstract: Methods and devices are provided to allow for the transfer of a display of a visual representation between a head mounted device and a computing device during the display of a video. A video is displayed on a computing device display of a computing device, a visual representation of a speech for an audio component of the video is received, the visual representation is displayed on the computing device display, and the display of the visual representation is transferred to the head mounted device to display on a head mounted device display upon determining that a head mounted device is in use.

Abstract: According to an aspect, a method includes generating at least one surface for a subject based on image data, the image data comprising a first image from a first perspective and a second image from a second perspective. The method further includes identifying a distribution of light intensity over the at least one surface and identifying a view independent feature based on the distribution of light intensity. The method also provides generating a texture map for a sequence of images based on the view independent feature.

Abstract: Nonlinear peri-codec optimization for image and video coding includes obtaining a source image including pixel values expressed in a first defined image sample space, generating a neuralized image representing the source image, the neuralized image including pixel values that are expressed as neural latent space values, encoding the input image wherein the neural latent space values are used as pixel values in a second defined image sample space and the input image is in an operative image format of the encoder, such that a decoder decodes the encoded image to obtain a reconstructed image in the second defined image sample space, wherein the reconstructed image is a reconstructed neuralized image including reconstructed neural latent space values, such that a deneuralized reconstructed image corresponding to the source image is obtained by a nonlinear post-codec image processor in the first defined image sample space.

Abstract: According to at least one implementation, a method includes determining an intent of a user to interact with an application on a wearable device. The method further includes determining an interface for the application based on the intent of the user, and overlaying the interface on a body portion of the user.

Abstract: A system and method for relighting of an augmented reality scene are provided. Image data and depth data corresponding to a physical environment may be captured by an image sensor of a computing device, for display in an AR scene generated by the computing device. The system may use the image data and depth data to generate a live depth map of the physical environment. A virtual light source may be placed in the AR scene of the physical environment in response to a user input. The system may relight the AR scene in response to the placement of the virtual light source based on the image data and the depth map.

Abstract: A visual programming platform can leverage a machine learning-based coding system to generate an initial set of programming-language code for further graphical editing by a human user. As an example, the visual programming platform can obtain a natural language description of a task to be performed by a computational pipeline. The visual programming platform can process the natural language description of the task with a machine learning coding system that includes one or more machine-learned models to generate, as an output of the machine learning coding system, a set of pseudocode that describes performance of the task. The platform can process the set of pseudocode that describes performance of the task with a compiler to generate a set of programming-language code that defines the computational pipeline for performing the task. The visual programming platform can generate a graphical visualization of the computational pipeline defined by the set of programming-language code.

Abstract: A method can include determining, by a head-mounted device mounted on a head of a user, a number of degrees of freedom of an input modality of an auxiliary device; determining an input modality of the head-mounted device based on the number of degrees of freedom of the input modality of the auxiliary device; and presenting the input modality of the head-mounted device to the user.

Abstract: A system including a computer vision model configured to perform a machine learning task is described. The computer vision model includes multiple wrapped convolutional layers, in which each wrapped convolutional layer includes a respective convolutional layer configured to receive, for each time step of multiple time steps, a layer input and to process the layer input to generate an initial output for the current time step, and a respective note-taking module configured to receive the initial output and to process the initial output to generate a feature vector for the current time step, the feature vector representing local information of the wrapped convolutional layer. The model includes a summarization module configured to receive the feature vectors and to process the feature vectors to generate a revision vector for the current time step, the revision vector representing global information of the plurality of wrapped convolutional layers.

Abstract: In described techniques, a depth of an object may be determined with respect to a head-mounted device (HMD). A gaze duration with respect to the object may be determined, relative to a gaze duration threshold. Then, based on the depth and on the gaze duration exceeding the gaze duration threshold, virtual content may be rendered at a virtual depth with respect to the HMD that is different than the depth.

Abstract: A visual programming platform can leverage a machine learning-based coding system to generate an initial set of programming-language code for further graphical editing by a human user. As an example, the visual programming platform can obtain a natural language description of a task to be performed by a computational pipeline. The visual programming platform can process the natural language description of the task with a machine learning coding system that includes one or more machine-learned models to generate, as an output of the machine learning coding system, a set of pseudocode that describes performance of the task. The platform can process the set of pseudocode that describes performance of the task with a compiler to generate a set of programming-language code that defines the computational pipeline for performing the task. The visual programming platform can generate a graphical visualization of the computational pipeline defined by the set of programming-language code.

Abstract: A method including determining a gaze direction of a user of a wearable device, capturing an image using a forward-looking camera of the wearable device, detecting a surroundings of the user based on the image, determining whether or not the user is distracted based on the gaze direction and the surroundings, and in response to determining the user is distracted, causing an operation to be performed on the wearable device, the operation configured to cause the user to change the user's attention.

Abstract: A method comprises selecting a two-dimensional image based on input to a computing device, generating multiple two-dimensional views of an object based on the two-dimensional image, and generating a three-dimensional virtual object based on the multiple two-dimensional views.

Abstract: A media stream comprising audio data and first lyric data associated with the audio data is received by a processing device. A set of user data associated with a user of a client device is identified. The first lyric data and the set of user data are provided as input to a generative machine learning model. An output of the generative machine learning model is obtained. The output comprises second lyric data. The second lyric data is a version of the first lyric data that is customized for the user. The second lyric data and the media stream are caused to be presented in a graphical user interface on the client device.

Abstract: Systems and methods are disclosed that address the need for adaptive exposure within high dynamic range (HDR) images. Solutions can leverage recent advances in the use of virtual reality (VR) headsets and Augmented Reality (AR) displays equipped with infrared (IR) eye tracking devices. A gaze vector determined by the eye tracking device identifies one or more fixation points on the image that corresponds to an area where there exists a faulty exposure. The exposure around the fixation point can be adaptively corrected using image processing techniques. Using spatial adaptive exposure, the resulting image, a type of foveated image, can be rendered on a low dynamic range (LDR) display with sufficient detail.

Abstract: According to an aspect, a method includes receiving, by a head-mounted display device, an authentication code associated with multi-factor authentication, receiving image data from an image camera on the head-mounted display device, detecting, by the head-mounted display device, that the image data includes an interface for receiving the authentication code, and displaying, by the head-mounted display device, the authentication code at a location that corresponds to the interface.

Abstract: Methods and devices are provided to allow for the transfer of a display of a visual representation between a head mounted device and a computing device during the display of a video. A video is displayed on a computing device display of a computing device, a visual representation of a speech for an audio component of the video is received, the visual representation is displayed on the computing device display, and the display of the visual representation is transferred to the head mounted device to display on a head mounted device display upon determining that a head mounted device is in use.