Abstract: A method includes obtaining split-pixel image data including a first sub-image and a second sub-image. The method also includes determining, for each respective pixel of the split-pixel image data, a corresponding position of a scene feature represented by the respective pixel relative to a depth of field, and identifying, based on the corresponding positions, out-of-focus pixels. The method additionally includes determining, for each respective out-of-focus pixel, a corresponding pixel value based on the corresponding position, a location of the respective out-of-focus pixel within the split-pixel image data, and at least one of: a first value of a corresponding first pixel in the first sub-image or a second value of a corresponding second pixel in the second sub-image. The method further includes generating, based on the corresponding pixel values, an enhanced image having an extended depth of field.

Abstract: Implementations described herein relate to a computer-implemented method that includes capturing image data using one or more cameras, wherein the image data includes a primary image and associated depth values. The method further includes encoding the image data in an image format. The encoded image data includes the primary image encoded in the image format and image metadata that includes a device element that includes a profile element indicative of an image type and a first camera element, wherein the first camera element includes an image element and a depth map based on the depth values. The method further includes, after the encoding, storing the image data in a file container based on the image format. The method further includes causing the primary image to be displayed.

Abstract: A method includes obtaining dual-pixel image data that includes a first sub-image and a second sub-image, and generating an in-focus image, a first kernel corresponding to the first sub-image, and a second kernel corresponding to the second sub-image. A loss value may be determined using a loss function that determines a difference between (i) a convolution of the first sub-image with the second kernel and (ii) a convolution of the second sub-image with the first kernel, and/or a sum of (i) a difference between the first sub-image and a convolution of the in-focus image with the first kernel and (ii) a difference between the second sub-image and a convolution of the in-focus image with the second kernel. Based on the loss value and the loss function, the in-focus image, the first kernel, and/or the second kernel, may be updated and displayed.

Abstract: Implementations described herein relate to a computer-implemented method that includes capturing image data using one or more cameras, wherein the image data includes a primary image and associated depth values. The method further includes encoding the image data in an image format. The encoded image data includes the primary image encoded in the image format and image metadata that includes a device element that includes a profile element indicative of an image type and a first camera element, wherein the first camera element includes an image element and a depth map based on the depth values. The method further includes, after the encoding, storing the image data in a file container based on the image format. The method further includes causing the primary image to be displayed.

Abstract: Systems and methods for augmenting images can utilize one or more image augmentation models and one or more texture transfer blocks. The image augmentation model can process input images and one or more segmentation masks to generate first output data. The first output data and the one or more segmentation masks can be processed with the texture transfer block to generate an augmented image. The input image can depict a scene with one or more occlusions, and the augmented image can depict the scene with the one or more occlusions replaced with predicted pixel data.

Abstract: A method includes receiving an input image. The input image corresponds to one or more masked regions to be inpainted. The method includes providing the input image to a first neural network. The first neural network outputs a first inpainted image at a first resolution, and the one or more masked regions are inpainted in the first inpainted image. The method includes creating a second inpainted image by increasing a resolution of the first inpainted image from the first resolution to a second resolution. The second resolution is greater than the first resolution such that the one or more inpainted masked regions have an increased resolution. The method includes providing the second inpainted image to a second neural network. The second neural network outputs a first refined inpainted image at the second resolution, and the first refined inpainted image is a refined version of the second inpainted image.

Abstract: Generally, the present disclosure is directed to systems and methods for measuring heart rate and respiratory rate using a camera such as, for example, a smartphone camera or other consumer-grade camera. Specifically, the present disclosure presents and validates two algorithms that make use of smartphone cameras (or the like) for measuring heart rate (HR) and respiratory rate (RR) for consumer wellness use. As an example, HR can be measured by placing the finger of a subject over the rear-facing camera. As another example, RR can be measured via a video of the subject sitting still in front of the front-facing camera.

Abstract: A method includes obtaining split-pixel image data including a first sub-image and a second sub-image. The method also includes determining, for each respective pixel of the split-pixel image data, a corresponding position of a scene feature represented by the respective pixel relative to a depth of field, and identifying, based on the corresponding positions, out-of-focus pixels. The method additionally includes determining, for each respective out-of-focus pixel, a corresponding pixel value based on the corresponding position, a location of the respective out-of-focus pixel within the split-pixel image data, and at least one of: a first value of a corresponding first pixel in the first sub-image or a second value of a corresponding second pixel in the second sub-image. The method further includes generating, based on the corresponding pixel values, an enhanced image having an extended depth of field.

Abstract: Apparatus and methods related to using machine learning to determine depth maps for dual pixel images of objects are provided. A computing device can receive a dual pixel image of at least a foreground object. The dual pixel image can include a plurality of dual pixels. A dual pixel of the plurality of dual pixels can include a left-side pixel and a right-side pixel that both represent light incident on a single dual pixel element used to capture the dual pixel image. The computing device can be used to train a machine learning system to determine a depth map associated with the dual pixel image. The computing device can provide the trained machine learning system.

Abstract: A system includes a computing device. The computing device is configured to perform a set of functions. The set of functions includes receiving an image, wherein the image comprises a two-dimensional array of data. The set of functions includes extracting, by a two-dimensional neural network, a plurality of two-dimensional features from the two-dimensional array of data. The set of functions includes generating a linear combination of the plurality of two-dimensional features to form a single three-dimensional input feature. The set of functions includes extracting, by a three-dimensional neural network, a plurality of three-dimensional features from the single three-dimensional input feature. The set of functions includes determining a two-dimensional depth map. The two-dimensional depth map contains depth information corresponding to the plurality of three-dimensional features.

Abstract: A method includes obtaining dual-pixel image data that includes a first sub-image and a second sub-image, and generating an in-focus image, a first kernel corresponding to the first sub-image, and a second kernel corresponding to the second sub-image. A loss value may be determined using a loss function that determines a difference between (i) a convolution of the first sub-image with the second kernel and (ii) a convolution of the second sub-image with the first kernel, and/or a sum of (i) a difference between the first sub-image and a convolution of the in-focus image with the first kernel and (ii) a difference between the second sub-image and a convolution of the in-focus image with the second kernel. Based on the loss value and the loss function, the in-focus image, the first kernel, and/or the second kernel, may be updated and displayed.

Abstract: Example implementations relate to joint depth prediction from dual cameras and dual pixels. An example method may involve obtaining a first set of depth information representing a scene from a first source and a second set of depth information representing the scene from a second source. The method may further involve determining, using a neural network, a joint depth map that conveys respective depths for elements in the scene. The neural network may determine the joint depth map based on a combination of the first set of depth information and the second set of depth information. In addition, the method may involve modifying an image representing the scene based on the joint depth map. For example, background portions of the image may be partially blurred based on the joint depth map.

Abstract: Implementations described herein relate to a computer-implemented method that includes capturing image data using one or more cameras, wherein the image data includes a primary image and associated depth values. The method further includes encoding the image data in an image format. The encoded image data includes the primary image encoded in the image format and image metadata that includes a device element that includes a profile element indicative of an image type and a first camera element, wherein the first camera element includes an image element and a depth map based on the depth values. The method further includes, after the encoding, storing the image data in a file container based on the image format. The method further includes causing the primary image to be displayed.

Abstract: A camera may capture an image of a scene and use the image to generate a first and a second subpixel image of the scene. The pair of subpixel images may be represented by a first set of subpixels and a second set of subpixels from the image respectively. Each pixel of the image may include two green subpixels that are respectively represented in the first and second subpixel images. The camera may determine a disparity between a portion of the scene as represented by the pair of subpixel images and may estimate a depth map of the scene that indicates a depth of the portion relative to other portions of the scene based on the disparity and a baseline distance between the two green subpixels. A new version of the image may be generated with a focus upon the portion and with the other portions of the scene blurred.

Abstract: A handheld user device includes a monocular camera to capture a feed of images of a local scene and a processor to select, from the feed, a keyframe and perform, for a first image from the feed, stereo matching using the first image, the keyframe, and a relative pose based on a pose associated with the first image and a pose associated with the keyframe to generate a sparse disparity map representing disparities between the first image and the keyframe. The processor further is to determine a dense depth map from the disparity map using a bilateral solver algorithm, and process a viewfinder image generated from a second image of the feed with occlusion rendering based on the depth map to incorporate one or more virtual objects into the viewfinder image to generate an AR viewfinder image. Further, the processor is to provide the AR viewfinder image for display.

Abstract: Example embodiments allow for training of artificial neural networks (ANNs) to generate depth maps based on images. The ANNs are trained based on a plurality of sets of images, where each set of images represents a single scene and the images in such a set of images differ with respect to image aperture and/or focal distance. An untrained ANN generates a depth map based on one or more images in a set of images. This depth map is used to generate, using the image(s) in the set, a predicted image that corresponds, with respect to image aperture and/or focal distance, to one of the images in the set. Differences between the predicted image and the corresponding image are used to update the ANN. ANNs tramed in this manner are especially suited for generating depth maps used to perform simulated image blur on small-aperture images.

Abstract: Example embodiments allow for training of artificial neural networks (ANNs) to generate depth maps based on images. The ANNs are trained based on a plurality of sets of images, where each set of images represents a single scene and the images in such a set of images differ with respect to image aperture and/or focal distance. An untrained ANN generates a depth map based on one or more images in a set of images. This depth map is used to generate, using the image(s) in the set, a predicted image that corresponds, with respect to image aperture and/or focal distance, to one of the images in the set. Differences between the predicted image and the corresponding image are used to update the ANN. ANNs tramed in this manner are especially suited for generating depth maps used to perform simulated image blur on small-aperture images.

Abstract: Structural health monitoring (SHM) is essential but can be expensive to perform. In an embodiment, a method includes sensing vibrations at a plurality of locations of a structure by a plurality of time-synchronized sensors. The method further includes determining a first set of dependencies of all sensors of the time-synchronized sensors at a first sample time to any sensors of a second sample time, and determining a second set of dependencies of all sensors of the time-synchronized sensors at the second sample time to any sensors of a third sample time. The second sample time is later than the first sample time, and the third sample time is later than the second sample time. The method then determines whether the structure has changed if the first set of dependencies is different from the second set of dependencies. Therefore, automated SHM can ensure safety at a lower cost to building owners.

Abstract: Apparatus and methods related to using machine learning to determine depth maps for dual pixel images of objects are provided. A computing device can receive a dual pixel image of at least a foreground object. The dual pixel image can include a plurality of dual pixels. A dual pixel of the plurality of dual pixels can include a left-side pixel and a right-side pixel that both represent light incident on a single dual pixel element used to capture the dual pixel image. The computing device can be used to train a machine learning system to determine a depth map associated with the dual pixel image. The computing device can provide the trained machine learning system.

Abstract: A handheld user device includes a monocular camera to capture a feed of images of a local scene and a processor to select, from the feed, a keyframe and perform, for a first image from the feed, stereo matching using the first image, the keyframe, and a relative pose based on a pose associated with the first image and a pose associated with the keyframe to generate a sparse disparity map representing disparities between the first image and the keyframe. The processor further is to determine a dense depth map from the disparity map using a bilateral solver algorithm, and process a viewfinder image generated from a second image of the feed with occlusion rendering based on the depth map to incorporate one or more virtual objects into the viewfinder image to generate an AR viewfinder image. Further, the processor is to provide the AR viewfinder image for display.