Abstract: A method includes obtaining an image captured by a see-through camera of a video see-through (VST) augmented reality (AR) device. The method also includes identifying a first distortion created by at least one see-through camera lens and identifying a second distortion created by at least one display lens of the VST AR device. The method further includes determining a combined distortion based on the first distortion and the second distortion. The method also includes pre-warping the image to offset the combined distortion such that the image is not distorted when the pre-warped image is viewed by a user through the at least one display lens of the VST AR device. In addition, the method includes presenting the pre-warped image to the user on at least one display of the VST AR device.

Abstract: A method includes obtaining images of a scene captured using a stereo pair of imaging sensors of an XR device and depth data associated with the images, where the scene includes multiple objects. The method also includes obtaining volume-based 3D models of the objects. The method further includes, for one or more first objects, performing depth-based reprojection of the one or more 3D models of the one or more first objects to left and right virtual views based on one or more depths of the one or more first objects. The method also includes, for one or more second objects, performing constant-depth reprojection of the one or more 3D models of the one or more second objects to the left and right virtual views based on a specified depth. In addition, the method includes rendering the left and right virtual views for presentation by the XR device.

Abstract: A method includes obtaining first image frames having a first dynamic range captured using at least one imaging sensor of a VST XR device. The method also includes, for each of at least one of the first image frames, generating a tone mapping filter configured to provide noise reduction and edge preservation while being guided by image and feature information and applying the tone mapping filter to the first image frame in order to transform the first image frame into a second image frame having a second dynamic range smaller than the first dynamic range. The tone mapping filter is applied to a luminance channel and not chrominance channels associated with the first image frame. The method further includes presenting one or more rendered images or videos based on the second image frame for each of at least one of the first image frames using at least one display.

Abstract: A method includes obtaining, using at least one imaging sensor of an electronic device, a first image frame of a scene. The method also includes determining, using at least one processing device of the electronic device, a low-light image score indicative of a brightness of the first image frame. The method further includes, in response to the low-light image score indicating that the brightness of the first image frame is below a threshold, applying, using the at least one processing device, a low-light visibility enhancement model to the first image frame in order to generate a second image frame having a higher brightness than the first image frame. The low-light visibility enhancement model is trained using at least one dataset that includes image frames obtained using the at least one imaging sensor of the electronic device.

Abstract: A method includes capturing reflections of illumination from an eye of a user wearing an extended reality (XR) headset. The method also includes detecting movement of the XR headset based on changes in positions of the reflections of the illumination from the user's eye, where the movement changes a pose of the XR headset relative to the user's eye. The method further includes determining a direction based on the detected movement and causing content presented on at least one display of the XR headset to change based on the determined direction.

Abstract: A method includes obtaining one or more image frames of a scene and data associated with the one or more image frames where the data includes user eye behavior data. The method also includes applying passthrough transformations on the one or more image frames to generate one or more transformed image frames. The method further includes identifying an attentional region in the one or more transformed image frames based on the user eye behavior data. The method also includes adjusting lightness of the one of more transformed image frames using a weighting distribution to generate one or more modified image frames where the lightness is attenuated from a center point of the attentional region towards edges of the one or more transformed image frames. In addition, the method includes rendering one or more images for display based on the one or more modified image frames.

Abstract: A method includes obtaining a first image frame captured at a first time and first depth data associated with the first image frame, where the first image frame has a higher resolution than the first depth data. The method also includes predicting motion of the electronic device between the first time and a second time and generating second depth data based on the first depth data, the first image frame, and the predicted motion, where the second depth data has a higher resolution than the first depth data. The method further includes reprojecting the first image frame using the second depth data to generate a second image frame and displaying a rendered image based on the second image frame. Generating the second depth data includes performing depth densification and super-resolution in order to increase the resolution of the second depth data relative to the resolution of the first depth data.

Abstract: A method includes obtaining a 2D image captured using an imaging sensor. The 2D image is associated with an imaging sensor pose. The method also includes providing the 2D image, the imaging sensor pose, and one or more additional imaging sensor poses to at least one machine learning model that is trained to generate a texture map and a depth map for the imaging sensor pose and each additional imaging sensor pose. The method further includes generating a stereo image pair based on the texture maps and the depth maps. The stereo image pair represents a 2.5D view of the 2D image. The 2.5D view includes a pair of images each including multiple collections of pixels and, for each collection of pixels, a common depth associated with the pixels in the collection of pixels. In addition, the method includes initiating display of the stereo image pair on an XR device.

Abstract: A method includes obtaining, using multiple imaging sensors of an electronic device, a left image frame and a right image frame forming a stereo pair of image frames. The method also includes identifying, using at least one processing device of the electronic device, extrinsic parameters associated with relative positions and orientations of the imaging sensors. The method further includes performing, using the at least one processing device, an online stereo rectification of the stereo pair of image frames based on the extrinsic parameters such that epipolar lines of the left and right image frames are horizontally aligned to generate a rectified stereo pair of image frames. In addition, the method includes rendering, using the at least one processing device, one or more images for display based on the rectified stereo pair of image frames.

Abstract: A method includes obtaining images of a scene and corresponding position data of a device that captures the images. The method also includes determining position data and direction data associated with camera rays passing through keyframes of the images. The method further includes using a position-dependent multilayer perceptron (MLP) and a direction-dependent MLP to create sparse feature vectors. The method also includes storing the sparse feature vectors in at least one data structure. The method further includes receiving a request to render the scene on an augmented reality (AR) device associated with a viewing direction. In addition, the method includes rendering the scene associated with the viewing direction using the sparse feature vectors in the at least one data structure.

Abstract: A method includes obtaining an image frame using at least one see-through camera of a VST XR device. The method also includes applying a first correction to the image frame based on one or more intrinsic parameters of the at least one see-through camera. The method further includes applying a second correction to the image frame based on one or more intrinsic parameters of at least one display lens of the VST XR device. In addition, the method includes, after applying the first correction and the second correction, displaying the image frame on at least one display visible through the at least one display lens. The one or more intrinsic parameters of the at least one see-through camera are determined using a first machine learning model, and the one or more intrinsic parameters of the at least one display lens are determined by a second machine learning model.

Abstract: A method includes obtaining, using at least one processing device, images of a scene captured using one or more imaging sensors of a video see-through (VST) extended reality (XR) device. The method also includes identifying, using the at least one processing device, a region of the scene on which a user is focused. The method further includes generating, using the at least one processing device, a mask for each image based on the region of the scene on which the user is focused, where different masks are associated with different resolutions and/or different shapes. The method also includes mapping, using the at least one processing device, at least some image data of each image onto a mesh based on the mask associated with that image. In addition, the method includes rendering, using the at least one processing device, final views of the scene using the mapped image data.

Abstract: A method includes identifying a transformation associated with a video see-through (VST) extended reality (XR) device using registration and parallax errors. The registration and parallax errors are based on one or more differences between (i) one or more actual positions of contents of a scene as imaged using a see-through camera and (ii) one or more perceived positions of the contents of the scene at a virtual camera associated with a viewpoint of a user when viewing a display panel. The method also includes obtaining an image that captures an object using the see-through camera, applying the transformation to the image in order to generate a modified image, and rendering the modified image for presentation on the display panel. Applying the transformation modifies the image such that a perceived position of the object substantially matches an actual position of the object when the display panel is viewed by the user.

Abstract: A method includes obtaining a first tile corresponding to a first portion of an image frame and a second tile corresponding to a second portion of the image frame after the first tile. The method also includes mapping the first and second tiles onto first and second distortion tile meshes, respectively. The method further includes predicting a head pose of a user when the image frame will be displayed. The method also includes transforming the first and second distortion tile meshes based on the predicted head pose. The second distortion tile mesh is transformed after the first distortion tile mesh. The method further includes rendering the first and second tiles based on the first and second transformed distortion tile meshes, respectively. The second tile is rendered after the first tile. In addition, the method includes initiating display of the first and second rendered tiles on at least one display panel.

Abstract: A method includes determining at an extended reality (XR) device, a first set of vertex adjustment values of a distortion mesh and receiving image frame data of a scene captured at a first time and at a first head pose using a see-through camera of the XR device. The method further includes applying the first set of vertex adjustment values of the distortion mesh to the image frame data to obtain intermediate image data, and predicting a second head pose at a second time subsequent to the first time. The method also includes generating, based on the predicted second head pose, a second set of vertex adjustment values of the distortion mesh, applying the second set of vertex adjustment values of the distortion mesh to the intermediate image data to generate a rendered virtual frame and displaying the rendered virtual frame by the XR device at the second time.

Abstract: A method includes obtaining first and second image data of a real-world scene, performing feature extraction to obtain first and second feature maps, and performing pose tracking based on at least one of the first image data, second image data, and pose data to obtain a 6DOF pose of an apparatus. The method also includes generating, based on the 6DOF pose, first feature map, and second feature map, a disparity map between the image data and generating an initial depth map based on the disparity map. The method further includes generating a dense depth map based on the initial depth map and a camera model and generating, based on the dense depth map, a three-dimensional reconstruction of at least pail of the scene. In addition, the method includes rendering an AR or XR display that includes one or more virtual objects positioned to contact one or more surfaces of the reconstruction.

Abstract: An electronic device includes at least one processing device configured to obtain an image. The at least one processing device is also configured to determine high-frequency features of the image using the image. The at least one processing device is further configured to determine a weighting map based on blurriness of at least some pixels in the image, where the weighting map represents how much to sharpen the at least some pixels in the image. The at least one processing device is also configured to apply the weighting map to the high-frequency features of the image to generate weighted high-frequency features. In addition, the at least one processing device is configured to combine the weighted high-frequency features with the at least some pixels in the image to generate an enhanced image.

Abstract: A method includes obtaining, using at least one imaging sensor, an image frame. The method also includes mapping, using at least one processing device, the image frame to a mesh including multiple vertices. The method further includes performing, using the at least one processing device, noise reduction to determine color data of pixels located on the vertices of the mesh. Performing the noise reduction includes using a denoising filter to denoise the image frame. The method also includes determining, using the at least one processing device, color data of remaining pixels not located on the vertices of the mesh based on the determined color data of the pixels located on the vertices to generate a denoised image. In addition, the method includes performing, using the at least one processing device, image enhancement of the denoised image to enhance at least part of the denoised image and generate an enhanced image.

Abstract: A method includes obtaining, from an image sensor, image data of a real-world scene; obtaining, from a depth sensor, sparse depth data of the real-world scene; and passing the image data to a first neural network to obtain one or more object regions of interest (ROIs) and one or more feature map ROIs. Each object ROI includes at least one detected object. The method also includes passing the image data and sparse depth data to a second neural network to obtain one or more dense depth map ROIs; aligning the one or more object ROIs, one or more feature map ROIs, and one or more dense depth map ROIs; and passing the aligned ROIs to a fully convolutional network to obtain a segmentation of the real-world scene. The segmentation contains one or more pixelwise predictions of one or more objects in the real-world scene.

Abstract: A method includes obtaining, from a memory of an electronic device connected to a head mounted display (HMD), a first reference frame, wherein the first reference frame comprises a first set of pixels associated with a first time. The method includes, rendering, at the electronic device, a source image as a new frame, wherein the new frame includes a second set of pixels associated with a display to be provided by the HMD at a second time, and generating, by the electronic device, a differential frame, wherein the differential frame is based on a difference operation between pixels of the new frame with pixels of the first reference frame to identify pixels unique to the new frame. Still further, the method includes sending the differential frame to the HMD, and storing the new frame in the memory of the electronic device as a second reference frame.