Abstract: Particular embodiments perform a light path analysis of an image comprising a scene, wherein the scene comprises at least one refractive or reflective object. The image may be decomposed based on the light path analysis into a plurality of components, each of the components representing a contribution to lighting in the scene by a different type of light interaction. For each of the components, one or more motion vectors are extracted for each of the components in order to capture motion in the scene. Finally, a final contribution of each of the components to the image is computed based on the motion vectors.

Abstract: Particular embodiments decompose an image comprising a scene into a diffuse component and a specular component. Each of the components represent a contribution to lighting in the scene. A set of motion vectors may be extracted in order to capture motion in the scene. Finally, a final contribution of each of the components to the image may be computed based on the motion vectors.

Abstract: According to one implementation, a video processing system includes a computing platform having a hardware processor and a system memory storing a sample-based video denoising software code. The hardware processor executes the sample-based video denoising software code to receive a video sequence, and select a reference frame of the video sequence to denoise. For each pixel of the reference frame, the hardware processor executes the sample-based video denoising software code to map the pixel to a sample pixel in each of other frames of the video sequence, identify a first confidence value corresponding to each of the sample pixels based on the mapping, identify a second confidence value corresponding to each of the sample pixels based on the frame that includes the sample pixel, and denoise the pixel based on a weighted combination of the sample pixels determined using the first confidence values and the second confidence values.

Abstract: According to one implementation, a video processing system includes a computing platform having a hardware processor and a system memory storing a sample-based video sharpening software code. The sample-based video sharpening software code receives a video sequence, and classifies frames of the video sequence as sharp or unsharp. For each pixel of an unsharp frame, the sample-based video sharpening software code determines a mapping of the pixel to another pixel in some or all of the sharp frames, determines a reverse mapping of each of the other pixels to the pixel, identifies a first confidence value corresponding to each of the other pixels based on the mapping, identifies a second confidence value corresponding to each of the other pixels based on the mapping and the reverse mapping, and sharpens the pixel based on a weighted combination of the other pixels determined using the first and second confidence values.

Abstract: Enhanced removing of noise and outliers from one or more point sets generated by image-based 3D reconstruction techniques is provided. In accordance with the disclosure, input images and corresponding depth maps can be used to remove pixels that are geometrically and/or photometrically inconsistent with the colored surface implied by the input images. This allows standard surface reconstruction methods (such as Poisson surface reconstruction) to perform less smoothing and thus achieve higher quality surfaces with more features. In some implementations, the enhanced point-cloud noise removal in accordance with the disclosure can include computing per-view depth maps, and detecting and removing noisy points and outliers from each per-view point cloud by checking if points are consistent with the surface implied by the other input views.

Abstract: There are provided systems and methods for an interactive synchronization of multiple videos. An example system includes a memory storing a first video and a second video, the first video including first video clips and the second video including second video clips. The system further includes a processor configured to calculate a histogram based on a number of features that are similar between the first video clips and the second video clips, generate a cost matrix based on the histogram, generate a first graph that includes first nodes based on the cost matrix, compute a path through the graph using the nodes, and align the first video with the second video using the path, where the path corresponds to playback speeds for the first video and the second video.

Abstract: Enhanced removing of noise and outliers from one or more point sets generated by image-based 3D reconstruction techniques is provided. In accordance with the disclosure, input images and corresponding depth maps can be used to remove pixels that are geometrically and/or photometrically inconsistent with the colored surface implied by the input images. This allows standard surface reconstruction methods (such as Poisson surface reconstruction) to perform less smoothing and thus achieve higher quality surfaces with more features. In some implementations, the enhanced point-cloud noise removal in accordance with the disclosure can include computing per-view depth maps, and detecting and removing noisy points and outliers from each per-view point cloud by checking if points are consistent with the surface implied by the other input views.

Abstract: The disclosure provides an approach for image segmentation from an uncalibrated camera array. In one aspect, a segmentation application computes a pseudo depth map for each frame of a video sequence recorded with a camera array based on dense correspondences between cameras in the array. The segmentation application then fuses such pseudo depth maps computed for satellite cameras of the camera array to obtain a pseudo depth map at a central camera. Further, the segmentation application interpolates virtual green screen positions for an entire frame based on user input which provides control points and pseudo depth thresholds at the control points. The segmentation application then computes an initial segmentation based on a thresholding using the virtual green screen positions, and refines the initial segmentation by solving a binary labeling problem in a Markov random field to better align the segmentation with image edges and provide temporal coherency for the segmentation.

Abstract: According to one implementation, a video processing system includes a computing platform having a hardware processor and a system memory storing a sample-based video sharpening software code. The sample-based video sharpening software code receives a video sequence, and classifies frames of the video sequence as sharp or unsharp. For each pixel of an unsharp frame, the sample-based video sharpening software code determines a mapping of the pixel to another pixel in some or all of the sharp frames, determines a reverse mapping of each of the other pixels to the pixel, identifies a first confidence value corresponding to each of the other pixels based on the mapping, identifies a second confidence value corresponding to each of the other pixels based on the mapping and the reverse mapping, and sharpens the pixel based on a weighted combination of the other pixels determined using the first and second confidence values.

Abstract: According to one implementation, a video processing system includes a computing platform having a hardware processor and a system memory storing a sample-based video denoising software code. The hardware processor executes the sample-based video denoising software code to receive a video sequence, and select a reference frame of the video sequence to denoise. For each pixel of the reference frame, the hardware processor executes the sample-based video denoising software code to map the pixel to a sample pixel in each of other frames of the video sequence, identify a first confidence value corresponding to each of the sample pixels based on the mapping, identify a second confidence value corresponding to each of the sample pixels based on the frame that includes the sample pixel, and denoise the pixel based on a weighted combination of the sample pixels determined using the first confidence values and the second confidence values.

Abstract: Enhanced removing of noise and outliers from one or more point sets generated by image-based 3D reconstruction techniques is provided. In accordance with the disclosure, input images and corresponding depth maps can be used to remove pixels that are geometrically and/or photometrically inconsistent with the colored surface implied by the input images. This allows standard surface reconstruction methods (such as Poisson surface reconstruction) to perform less smoothing and thus achieve higher quality surfaces with more features. In some implementations, the enhanced point-cloud noise removal in accordance with the disclosure can include computing per-view depth maps, and detecting and removing noisy points and outliers from each per-view point cloud by checking if points are consistent with the surface implied by the other input views.

Abstract: The present disclosure includes a method for modifying an input image to map to a three dimensional shape prior to forming the three dimensional shape. The method includes receiving by a processor at least two pre-forming images of a locally unique non-repeating pattern printed on an input material. The at least two pre-forming images are captured prior to the input material being formed into the three dimensional shape. The method further includes receiving by the processor at least two post-forming images of the pattern printed on the input material, wherein the post-forming images are captured after the input material has been formed into the three dimensional shape and analyzing by the processor the at least pre-forming images and the at least two post-forming images to determine a translation table.

Abstract: Enhanced removing of noise and outliers from one or more point sets generated by image-based 3D reconstruction techniques is provided. In accordance with the disclosure, input images and corresponding depth maps can be used to remove pixels that are geometrically and/or photometrically inconsistent with the colored surface implied by the input images. This allows standard surface reconstruction methods (such as Poisson surface reconstruction) to perform less smoothing and thus achieve higher quality surfaces with more features. In some implementations, the enhanced point-cloud noise removal in accordance with the disclosure can include computing per-view depth maps, and detecting and removing noisy points and outliers from each per-view point cloud by checking if points are consistent with the surface implied by the other input views.

Abstract: Systems and methods for generating a panoramic video from unstructured camera arrays. The systems and methods are configured to statically align corresponding image-frames of respective input video streams, warp the aligned image-frames according to a warping-order, and relax the warped image-frames thereby generating a temporally coherent panoramic video. Methods according to embodiments this invention utilize a new parallax-warping-error metric that is devised to capture structural differences created by parallax artifacts. The parallax-warping-error metric is effective in finding an optimal warping-order and in driving the warping process, resulting in a panoramic video with minimal parallax artifacts.

Abstract: The disclosure provides an approach for image segmentation from an uncalibrated camera array. In one aspect, a segmentation application computes a pseudo depth map for each frame of a video sequence recorded with a camera array based on dense correspondences between cameras in the array. The segmentation application then fuses such pseudo depth maps computed for satellite cameras of the camera array to obtain a pseudo depth map at a central camera. Further, the segmentation application interpolates virtual green screen positions for an entire frame based on user input which provides control points and pseudo depth thresholds at the control points. The segmentation application then computes an initial segmentation based on a thresholding using the virtual green screen positions, and refines the initial segmentation by solving a binary labeling problem in a Markov random field to better align the segmentation with image edges and provide temporal coherency for the segmentation.

Abstract: The disclosure provides an approach for estimating depth in a scene. According to one aspect, regions where the depth estimation is expected to perform well may first be identified in full-resolution epipolar-plane images (EPIs) generated from a plurality of images of the scene. Depth estimates for EPI-pixels with high edge confidence are determined by testing a number of discrete depth hypotheses and picking depths that lead to highest color density of sampled EPI-pixels. The depth estimate may also be propagated throughout the EPIs. This process of depth estimation and propagation may be iterated until all EPI-pixels with high edge confidence have been processed, and all EPIs may also be processed in this manner. The EPIs are then iteratively downsampled to coarser resolutions, at which edge confidence for EPI-pixels not yet processed are determined, depth estimates of EPI-pixels with high edge confidence made, and depth estimates propagated throughout the EPIs.

Abstract: Interpolating frames of a video may provide a technique for one or more of frame rate conversion, temporal upsampling for generating slow motion video, image morphing, virtual view synthesis, and/or other video applications. A system may be configured to interpolated frames of a video by leveraging frequency domain representations of individual frames. The frequency domain representations may be decomposed into set of discrete functions that make up the frequency domain representations. Corresponding functions from sets of functions associated with frames with which an interpolated frame is to be determined may be identified. Phase differences between corresponding functions may be determined. Interpolated functions between the corresponding functions may be determined based on the determined phased differences. Information describing spatial domain representations of interpolated frames may be determined based on the interpolated functions.

Abstract: Particular embodiments perform a light path analysis of an image comprising a scene, wherein the scene comprises at least one refractive or reflective object. The image may be decomposed based on the light path analysis into a plurality of components, each of the components representing a contribution to lighting in the scene by a different type of light interaction. For each of the components, one or more motion vectors are extracted for each of the components in order to capture motion in the scene. Finally, a final contribution of each of the components to the image is computed based on the motion vectors.

Abstract: Particular embodiments decompose an image comprising a scene into a diffuse component and a specular component. Each of the components represent a contribution to lighting in the scene. A set of motion vectors may be extracted in order to capture motion in the scene. Finally, a final contribution of each of the components to the image may be computed based on the motion vectors.

Abstract: Systems and methods to generate stereoscopic panoramas obtain images based on captured images. The obtained images may be processed and/or preprocessed, for example to compensate for perspective distortion caused by the non-ideal camera orientation during capturing, to reduce vertical parallax, to align adjacent images, and/or to reduce rotational and/or positional drift between adjacent images. The obtained images may be used for interpolating virtual in-between images on the fly to reduce visible artifacts in the resulting panorama. Obtained and/or interpolated images (or image fragments) may be stitched together to form a stereoscopic panorama.