Abstract: Systems and methods are provided for harmonizing a foreground region with a background image during background replacement and content editing. A sensor image including a foreground region and a background region is received at an image signal processor (ISP), along with a background image having different visual characteristics. The sensor image and the background image are downscaled and processed by a harmonization neural network that predicts one or more ISP hardware block parameters. The predicted parameters are used to configure ISP hardware blocks such that the foreground region of a processed image is harmonized to match visual characteristics of the background image while preserving full sensor bit depth. The processed image is then blended with the background image to generate a harmonized output image.

Abstract: Systems and methods for a patch management system that combines the benefits of working on a low-resolution image with added cues from the high-resolution image. The patch management system collects high-resolution patches during the downscaling process. The high-resolution patches are analyzed using a Deep Neural Network to detect fine details that are lost in the downscaled image. By fusing the high-resolution patch-level information with semantic segmentation results, the ISP blocks are provided with both global context and local details, improving texture reproduction and temporal noise reduction while adding minimal overhead compared to standard downscaled processing. The patch management system can also be used for tasks such as optical flow calculation from a downscaled image. By strategically selecting image areas for high-resolution patches, the system minimizes computational overhead as compared to processing a full-resolution image.

Abstract: Systems and methods for adaptive emotion-based image signal processor parameter tuning. A neural network is provided to detect the emotional context of a scene. The neural network can identify facial micro expressions of a person in the scene and tune the ISP parameters based on detected emotional cues. The mood of a scene can also be extracted by analyzing a transcription of the conversation using a natural language processing model. Adaptive emotion-based camera ISP tuning can be added as a software component. The ISP tuning system can process an image, classify the emotion of the scene in the image using a deep neural network, and dynamically adjust the ISP parameters to convey the mood of the scene. ISP parameters for ISP blocks such as a tone mapping block, a color correction matrix block, and a sharpening block can be dynamically adjusted based on the mood of the scene.