Abstract: Systems and methods are provided for harmonizing a foreground region with a background image during background replacement and content editing. A sensor image including a foreground region and a background region is received at an image signal processor (ISP), along with a background image having different visual characteristics. The sensor image and the background image are downscaled and processed by a harmonization neural network that predicts one or more ISP hardware block parameters. The predicted parameters are used to configure ISP hardware blocks such that the foreground region of a processed image is harmonized to match visual characteristics of the background image while preserving full sensor bit depth. The processed image is then blended with the background image to generate a harmonized output image.

Abstract: Implementing video manipulation that can deliver professional-quality results while maintaining real-time performance is not trivial. To address this technical challenge, a dual parallel path processing system can be implemented. In a pipeline parallel to a main image processing pipeline, low-resolution images of a video can be processed, such as by a neural network, to generate gain maps that specify spatially-varying corrections for a variety of applications such as face relighting, content emphasis/de-emphasis, graphics overlay effects, and scene tone-mapping. These gain maps are upscaled through existing image processing unit hardware, and applied on or to the full-resolution, high bit-depth images. This approach achieves superior image quality with minimal computational overhead, little to no degradation to processing latency, and minimal additional power consumption.

Abstract: Systems and methods are provided for change detection in low-resolution video streams, which can be used for applications such as high resolution video restoration and processing. The techniques effectively detect changes by leveraging a large receptive field and lightweight computation, which are achieved by working with low-resolution images. In particular, the techniques include extracting features from a change detection model and a semantic segmentation model, and integrating the extracted feature outputs from the models to produce a robust change detection map. A pre-processing phase can be employed to optimize the input for each model, ensuring minimal complexity and enhanced performance. The change detection model can be implemented as a deep neural network, and methods are provided for generating ground truth (GT) data, which semantically guides the change detection neural network to perform change detection inpainting during training.

Abstract: Systems and methods for a patch management system that combines the benefits of working on a low-resolution image with added cues from the high-resolution image. The patch management system collects high-resolution patches during the downscaling process. The high-resolution patches are analyzed using a Deep Neural Network to detect fine details that are lost in the downscaled image. By fusing the high-resolution patch-level information with semantic segmentation results, the ISP blocks are provided with both global context and local details, improving texture reproduction and temporal noise reduction while adding minimal overhead compared to standard downscaled processing. The patch management system can also be used for tasks such as optical flow calculation from a downscaled image. By strategically selecting image areas for high-resolution patches, the system minimizes computational overhead as compared to processing a full-resolution image.

Abstract: Systems and methods for a skin segmentation framework that leverages facial crop as prior knowledge. In particular, the system employs a cross-attention mechanism to transfer features extracted from the face region to guide the main segmentation network. By utilizing the face as a reference point for skin tone and lighting conditions, the model learns to adapt to diverse environmental scenarios and varying skin appearances. This approach significantly enhances skin segmentation accuracy and robustness compared to traditional color-based and deep learning methods, particularly in challenging lighting conditions. Any changes to pixels representing skin (e.g., white balance, auto exposure) are consistent with the changes in the face crop, and thus skin segmentation colors can be updated to reflect the changes.

Abstract: Methods, systems, articles of manufacture, and apparatus to enhance image depth confidence maps are disclosed. An apparatus includes a gradient analyzer to: generate an image gradient based on a digital image of a scene; and generate a depth gradient based on a depth map associated with the digital image. The depth map is to define depth values corresponding to pixels in the digital image. The apparatus further includes an enhanced confidence map analyzer to determine an enhanced confidence map for the depth map based on the image gradient, the depth gradient, and an initial confidence map for the depth map.

Abstract: Methods, systems, articles of manufacture, and apparatus to enhance image depth confidence maps are disclosed. An apparatus includes a gradient analyzer to: generate an image gradient based on a digital image of a scene; and generate a depth gradient based on a depth map associated with the digital image. The depth map is to define depth values corresponding to pixels in the digital image. The apparatus further includes an enhanced confidence map analyzer to determine an enhanced confidence map for the depth map based on the image gradient, the depth gradient, and an initial confidence map for the depth map.

Abstract: Techniques related to color matching and color conversion are discussed. Such techniques may include determining operators for converting input pixel values to output pixel values in a color space such that the operators may include a set of color conversion matrix operators, a chroma adaptation operator, and/or a luma adaptation operators determined based on one or more perceptual color space values associated with the input pixel values.

Abstract: Techniques related to color matching and color conversion are discussed. Such techniques may include determining operators for converting input pixel values to output pixel values in a color space such that the operators may include a set of color conversion matrix operators, a chroma adaptation operator, and/or a luma adaptation operators determined based on one or more perceptual color space values associated with the input pixel values.