Abstract: Systems, apparatus, and methods for augmenting dense depth maps using sparse data. Various embodiments combine single-image depth estimation (SIDE) techniques with structure-from-motion techniques for improved depth accuracy. In some examples, a machine learning (ML) model is used to generate a dense depth map based on one or more frames/images of a video. Structure-from-motion (SfM) analysis is performed on the video to determine depth information from camera movement in the video. The structure-from-motion techniques may generate more accurate data than the ML model, however, the data from the ML model may be denser compared with the SFM depth data. The dense ML model depth map may be augmented by the SFM depth data. Augmentation may include fitting the relative depths determined by the ML model depth map to the absolute depth in the SFM data resulting in more accurate dense depth information.

Abstract: Systems, apparatus, and methods for adding post-processing motion blur to video. Conventional post-processing techniques relied on the filmmaker to select and stage their shots. Different motion blur techniques were designed to fix certain types of footage. Vector blur is one technique that “smears” pixel information in the direction of movement. Frame interpolation and stacking attempts to create motion blur by stacking interpolated frames together. Each technique has its own set of limitations. Various embodiments use a combination of motion blur techniques in post-processing for better, more realistic outcomes with faster/more efficient rendering times. In some cases, this may enable adaptive quality post-processing that may be performed in mobile/embedded ecosystems.

Abstract: Systems, apparatus, and methods for post-processing video e.g. frame denoising. Noise reduction techniques may be employed to improve the quality of digital video. Frames may be extracted from a video. Synthetic frames may be created using motion data between the extracted frames. Synthetic frames may be masked to exclude pixels from the composite frame. Thresholds used in masking may vary based on the temporal distance of the extracted frame used to create the synthetic frame and the extracted frame. Masking may be based on frame differences between extracted and synthetic frames (e.g., sub-pixel/luminance differences), areas of lower quality motion data (e.g., occlusions), or edge detection in the extracted frames. Synthetic and extracted frames may be composited generating frames having less noise. The composited frame may be based on averaging pixel values across the synthetic and extracted frames. Composited frames may be compiled and encoded into denoised video.

Abstract: An image/video captured by an image capture device may depict an object with a halo artifact. The halo artifact within the image/video may be blurred using a mask. A high-fidelity version of the image/video may be generated by enhancing the blurred image/video. Super-resolution or sharpening may be applied to the blurred image/video to generate the high-fidelity version of the image/video. The high-fidelity version of the image/video may depict the object with a reduced halo artifact, which has a smaller visual impact than the halo artifact in the original image/video.

Abstract: Methods and apparatus for post-processing in-camera stabilized video. Embodiments of the present disclosure reconstruct and re-stabilize an in-camera stabilized video to provide for improved stabilization (e.g., a wider crop, etc.). In-camera sensor data may be stored and used to re-calculate orientation metadata in post-production. In-camera stabilization provides several benefits (e.g., the ability to share stabilized videos from the camera without additional post-processing as well as reduced file sizes of the shared videos). Camera-aware post-processing can reuse portions of the in-camera stabilized videos while providing additional benefits (e.g., the ability to regenerate the original captured videos in post-production and re-stabilize the videos). Camera-aware post-processing can also improve orientation metadata and remove sensor error. The disclosed techniques also enable assisted optical flow-based stabilization using the refined metadata.

Abstract: Methods and apparatus for stabilizing image data based on a lens polynomial. Non-rectilinear footage can be captured and rectified in-camera; the rectified images may be stabilized to provide rectified stable video. In one exemplary embodiment, the footage is rectified and stabilized based on a lens polynomial and the camera's own movement. In some variants, the rectified stable video may be stored along with its margin track. In-camera rectified stable video provides several benefits over traditional techniques (e.g., the ability to share rectilinear content from the camera without additional post-processing, as well as reduced file sizes of the shared videos). Lens-aware post-processing can reuse portions of the in-camera rectified stable videos while providing additional benefits (e.g., the ability to re-frame the video in post-production).

Abstract: An image capture device may capture a video while experiencing motion. Motion of the image capture device during video capture may be determined based on optical flow and structure from motion solve of the video. Optical flow derived motion and/or the motion solve derived motion of the image capture device may be selected to perform stabilization of the video.

Abstract: Systems, apparatus, and methods for piggyback camera calibration. Existing piggybacked capture techniques use a “beauty camera” and an “action camera” to capture raw footage. The user directly applies the EIS stabilization track of a piggybacked action camera to the cinematic footage to create desired stable footage. Unfortunately, since the action camera may have been slightly offset from the cinematic video camera, the EIS stabilization data will only roughly approximate the necessary corrections. In other words, the user must manually fine tune the corrections. The disclosed embodiments use a calibration sequence to estimate a physical offset between the beauty camera and the action camera. Then, the estimated physical offset can be used to calculate an offset camera orientation for stabilizing the beauty camera. The foregoing process can be performed in-the-field before actual capture. This allows the user to check their set-up and fix any issues before capturing the desired footage.

Abstract: Methods and apparatus for stabilizing image data based on a lens polynomial. Non-rectilinear footage can be captured and rectified in-camera; the rectified images may be stabilized to provide rectified stable video. In one exemplary embodiment, the footage is rectified and stabilized based on a lens polynomial and the camera's own movement. In some variants, the rectified stable video may be stored along with its margin track. In-camera rectified stable video provides several benefits over traditional techniques (e.g., the ability to share rectilinear content from the camera without additional post-processing, as well as reduced file sizes of the shared videos). Lens-aware post-processing can reuse portions of the in-camera rectified stable videos while providing additional benefits (e.g., the ability to re-frame the video in post-production).

Abstract: Systems, apparatus, and methods for augmenting dense depth maps using sparse data. Various embodiments combine single-image depth estimation (SIDE) techniques with structure-from-motion techniques for improved depth accuracy. In some examples, a machine learning (ML) model is used to generate a dense depth map based on one or more frames/images of a video. Structure-from-motion (SfM) analysis is performed on the video to determine depth information from camera movement in the video. The structure-from-motion techniques may generate more accurate data than the ML model, however, the data from the ML model may be denser compared with the SFM depth data. The dense ML model depth map may be augmented by the SFM depth data. Augmentation may include fitting the relative depths determined by the ML model depth map to the absolute depth in the SFM data resulting in more accurate dense depth information.

Abstract: Systems, apparatus, and methods for encoding telemetry data as audio waveforms. Action cameras may be used to capture telemetry data, which in some applications, may be more useful than its audio/visual media capture. For example, a user may desire the “look-and-feel” of one camera but would like to use the telemetry track information from an action camera. In so-called “piggybacking” operation, the user may rigidly affix the two cameras and use the video track with the telemetry track in post-processing. Unfortunately, time-aligning the two tracks can result in relative drift over time. Various embodiments of the present disclosure transfer telemetry data as data-over-audio. The data-over-audio may be sampled according to the recipient device and recorded at capture. Subsequent post-processing may be performed with the time-aligned tracks.

Abstract: Video frames are captured by an image capture device and stabilized to generate stabilized video frames. Multiple stabilized video frames are combined into single motion blurred video frames. Combination of multiple stabilized video frames into single motion blurred video frames produces motion blur within the single motion blurred video frames that is both physical and real.

Abstract: Image captured through a non-rectilinear lens may exhibit distortions. The distortions may be reduced by warping the image. However, warping the image may degrade the fidelity of the image. The warped image may be enhanced to increase the fidelity of the image. The enhancement may be applied to the portions of the image that were degraded from the warping.

Abstract: Systems, apparatus, and methods for motion transfer between different media. Two videos shot simultaneously are combined with aspects of each used to produce a higher-resolution frame interpolated video than would otherwise be possible by the devices alone. The interpolated video may be used in slow motion footage, a higher frame rate (e.g., virtual reality) application, or to add a realistic motion blur effect in post-processing (rather than in capture). A post-processing device may receive high-speed video from a high-speed camera and a low-speed video from a low-speed camera. The post-processing device may use motion data from the high-speed video to interpolate frames of (high-resolution) low-speed video producing low-speed interpolated video.

Abstract: Systems, apparatus, and methods for adding post-processing motion blur to video. Conventional post-processing techniques relied on the filmmaker to select and stage their shots. Different motion blur techniques were designed to fix certain types of footage. Vector blur is one technique that “smears” pixel information in the direction of movement. Frame interpolation and stacking attempts to create motion blur by stacking interpolated frames together. Each technique has its own set of limitations. Various embodiments use a combination of motion blur techniques in post-processing for better, more realistic outcomes with faster/more efficient rendering times. In some cases, this may enable adaptive quality post-processing that may be performed in mobile/embedded ecosystems.

Abstract: Systems, apparatus, and methods adding post-processing motion blur to video and/or frame interpolation with occluded motion. Conventional post-processing techniques relied on the filmmaker to select and stage their shots. Different motion blur techniques were designed to fix certain types of footage. Vector blur is one technique that “smears” pixel information in the direction of movement. Frame interpolation and stacking attempts to create motion blur by stacking interpolated frames together. Each technique has its own set of limitations. Various embodiments use a combination of motion blur techniques in post-processing for better, more realistic outcomes with faster/more efficient rendering times. In some cases, this may enable adaptive quality post-processing that may be performed in mobile/embedded ecosystems. Various embodiments use a combination of video frame interpolation techniques for better interpolated frames with faster/more efficient rendering times.

Abstract: Methods and apparatus for post-processing in-camera stabilized video. Embodiments of the present disclosure reconstruct and re-stabilize an in-camera stabilized video to provide for improved stabilization (e.g., a wider crop, etc.). In-camera sensor data may be stored and used to re-calculate orientation metadata in post-production. In-camera stabilization provides several benefits (e.g., the ability to share stabilized videos from the camera without additional post-processing as well as reduced file sizes of the shared videos). Camera-aware post-processing can reuse portions of the in-camera stabilized videos while providing additional benefits (e.g., the ability to regenerate the original captured videos in post-production and re-stabilize the videos). Camera-aware post-processing can also improve orientation metadata and remove sensor error. The disclosed techniques also enable assisted optical flow-based stabilization using the refined metadata.

Abstract: Methods and apparatus for post-processing in-camera stabilized video. Embodiments of the present disclosure reconstruct and re-stabilize an in-camera stabilized video to provide for improved stabilization (e.g., a wider crop, etc.) In-camera sensor data may be stored and used to re-calculate orientation metadata in post-production. In-camera stabilization provides several benefits (e.g., the ability to share stabilized videos from the camera without additional post-processing as well as reduced file sizes of the shared videos). Camera-aware post-processing can reuse portions of the in-camera stabilized videos while providing additional benefits (e.g., the ability to regenerate the original captured videos in post-production and re-stabilize the videos). Camera-aware post-processing can also improve orientation metadata and remove sensor error. The disclosed techniques also enable assisted optical flow-based stabilization using the refined metadata.

Abstract: Systems, apparatus, and methods for piggyback camera calibration. Existing piggybacked capture techniques use a “beauty camera” and an “action camera” to capture raw footage. The user directly applies the EIS stabilization track of a piggybacked action camera to the cinematic footage to create desired stable footage. Unfortunately, since the action camera may have been slightly offset from the cinematic video camera, the EIS stabilization data will only roughly approximate the necessary corrections. In other words, the user must manually fine tune the corrections. The disclosed embodiments use a calibration sequence to estimate a physical offset between the beauty camera and the action camera. Then, the estimated physical offset can be used to calculate an offset camera orientation for stabilizing the beauty camera. The foregoing process can be performed in-the-field before actual capture. This allows the user to check their set-up and fix any issues before capturing the desired footage.

Abstract: Image captured through a non-rectilinear lens may exhibit distortions. The distortions may be reduced by warping the image. However, warping the image may degrade the fidelity of the image. The warped image may be enhanced to increase the fidelity of the image. The enhancement may be applied to the portions of the image that were degraded from the warping.