Patents by Inventor Federico PERAZZI
Federico PERAZZI has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Patent number: 11854206Abstract: A Video Semantic Segmentation System (VSSS) is disclosed that performs accurate and fast semantic segmentation of videos using a set of temporally distributed neural networks. The VSSS receives as input a video signal comprising a contiguous sequence of temporally-related video frames. The VSSS extracts features from the video frames in the contiguous sequence and based upon the extracted features, selects, from a set of labels, a label to be associated with each pixel of each video frame in the video signal. In certain embodiments, a set of multiple neural networks are used to extract the features to be used for video segmentation and the extraction of features is distributed among the multiple neural networks in the set. A strong feature representation representing the entirety of the features is produced for each video frame in the sequence of video frames by aggregating the output features extracted by the multiple neural networks.Type: GrantFiled: May 3, 2022Date of Patent: December 26, 2023Assignee: Adobe Inc.Inventors: Federico Perazzi, Zhe Lin, Ping Hu, Oliver Wang, Fabian David Caba Heilbron
-
Patent number: 11783184Abstract: Certain embodiments involve techniques for efficiently estimating denoising kernels for generating denoised images. For instance, a neural network receives a noisy reference image to denoise. The neural network uses a kernel dictionary of base kernels and generates a coefficient vector for each pixel in the reference image such that the coefficient vector includes a coefficient value for each base kernel in the kernel dictionary, where the base kernels are combined to generate a denoising kernel and each coefficient value indicates a contribution of a given base kernel to a denoising kernel. The neural network calculates the denoising kernel for a given pixel by applying the coefficient vector for that pixel to the kernel dictionary. The neural network applies each denoising kernel to the respective pixel to generate a denoised output image.Type: GrantFiled: February 2, 2022Date of Patent: October 10, 2023Assignee: Adobe Inc.Inventors: Federico Perazzi, Zhihao Xia, Michael Gharbi, Kalyan Sunkavalli
-
Publication number: 20230259778Abstract: The disclosure describes one or more implementations of a neural network architecture pruning system that automatically and progressively prunes neural networks. For instance, the neural network architecture pruning system can automatically reduce the size of an untrained or previously-trained neural network without reducing the accuracy of the neural network. For example, the neural network architecture pruning system jointly trains portions of a neural network while progressively pruning redundant subsets of the neural network at each training iteration. In many instances, the neural network architecture pruning system increases the accuracy of the neural network by progressively removing excess or redundant portions (e.g., channels or layers) of the neural network. Further, by removing portions of a neural network, the neural network architecture pruning system can increase the efficiency of the neural network.Type: ApplicationFiled: April 28, 2023Publication date: August 17, 2023Inventors: Shikun Liu, Zhe Lin, Yilin Wang, Jianming Zhang, Federico Perazzi
-
Patent number: 11710042Abstract: The present disclosure relates to shaping the architecture of a neural network. For example, the disclosed systems can provide a neural network shaping mechanism for at least one sampling layer of a neural network. The neural network shaping mechanism can include a learnable scaling factor between a sampling rate of the at least one sampling layer and an additional sampling function. The disclosed systems can learn the scaling factor based on a dataset while jointly learning the network weights of the neural network. Based on the learned scaling factor, the disclosed systems can shape the architecture of the neural network by modifying the sampling rate of the at least one sampling layer.Type: GrantFiled: February 5, 2020Date of Patent: July 25, 2023Assignee: Adobe Inc.Inventors: Shikun Liu, Zhe Lin, Yilin Wang, Jianming Zhang, Federico Perazzi
-
Patent number: 11663481Abstract: The disclosure describes one or more implementations of a neural network architecture pruning system that automatically and progressively prunes neural networks. For instance, the neural network architecture pruning system can automatically reduce the size of an untrained or previously-trained neural network without reducing the accuracy of the neural network. For example, the neural network architecture pruning system jointly trains portions of a neural network while progressively pruning redundant subsets of the neural network at each training iteration. In many instances, the neural network architecture pruning system increases the accuracy of the neural network by progressively removing excess or redundant portions (e.g., channels or layers) of the neural network. Further, by removing portions of a neural network, the neural network architecture pruning system can increase the efficiency of the neural network.Type: GrantFiled: February 24, 2020Date of Patent: May 30, 2023Assignee: Adobe Inc.Inventors: Shikun Liu, Zhe Lin, Yilin Wang, Jianming Zhang, Federico Perazzi
-
Publication number: 20230058793Abstract: The present disclosure relates to an image retouching system that automatically retouches digital images by accurately correcting face imperfections such as skin blemishes and redness. For instance, the image retouching system automatically retouches a digital image through separating digital images into multiple frequency layers, utilizing a separate corresponding neural network to apply frequency-specific corrections at various frequency layers, and combining the retouched frequency layers into a retouched digital image. As described herein, the image retouching system efficiently utilizes different neural networks to target and correct skin features specific to each frequency layer.Type: ApplicationFiled: October 11, 2022Publication date: February 23, 2023Inventors: Federico Perazzi, Jingwan Lu
-
Patent number: 11568544Abstract: The present disclosure relates to utilizing a neural network having a two-stream encoder architecture to accurately generate composite digital images that realistically portray a foreground object from one digital image against a scene from another digital image. For example, the disclosed systems can utilize a foreground encoder of the neural network to identify features from a foreground image and further utilize a background encoder to identify features from a background image. The disclosed systems can then utilize a decoder to fuse the features together and generate a composite digital image. The disclosed systems can train the neural network utilizing an easy-to-hard data augmentation scheme implemented via self-teaching. The disclosed systems can further incorporate the neural network within an end-to-end framework for automation of the image composition process.Type: GrantFiled: September 23, 2021Date of Patent: January 31, 2023Assignee: Adobe Inc.Inventors: Zhe Lin, Jianming Zhang, He Zhang, Federico Perazzi
-
Patent number: 11521299Abstract: The present disclosure relates to an image retouching system that automatically retouches digital images by accurately correcting face imperfections such as skin blemishes and redness. For instance, the image retouching system automatically retouches a digital image through separating digital images into multiple frequency layers, utilizing a separate corresponding neural network to apply frequency-specific corrections at various frequency layers, and combining the retouched frequency layers into a retouched digital image. As described herein, the image retouching system efficiently utilizes different neural networks to target and correct skin features specific to each frequency layer.Type: GrantFiled: October 16, 2020Date of Patent: December 6, 2022Assignee: Adobe Inc.Inventors: Federico Perazzi, Jingwan Lu
-
Publication number: 20220270370Abstract: A Video Semantic Segmentation System (VSSS) is disclosed that performs accurate and fast semantic segmentation of videos using a set of temporally distributed neural networks. The VSSS receives as input a video signal comprising a contiguous sequence of temporally-related video frames. The VSSS extracts features from the video frames in the contiguous sequence and based upon the extracted features, selects, from a set of labels, a label to be associated with each pixel of each video frame in the video signal. In certain embodiments, a set of multiple neural networks are used to extract the features to be used for video segmentation and the extraction of features is distributed among the multiple neural networks in the set. A strong feature representation representing the entirety of the features is produced for each video frame in the sequence of video frames by aggregating the output features extracted by the multiple neural networks.Type: ApplicationFiled: May 3, 2022Publication date: August 25, 2022Inventors: Federico Perazzi, Zhe Lin, Ping Hu, Oliver Wang, Fabian David Caba Heilbron
-
Patent number: 11354906Abstract: A Video Semantic Segmentation System (VSSS) is disclosed that performs accurate and fast semantic segmentation of videos using a set of temporally distributed neural networks. The VSSS receives as input a video signal comprising a contiguous sequence of temporally-related video frames. The VSSS extracts features from the video frames in the contiguous sequence and based upon the extracted features, selects, from a set of labels, a label to be associated with each pixel of each video frame in the video signal. In certain embodiments, a set of multiple neural networks are used to extract the features to be used for video segmentation and the extraction of features is distributed among the multiple neural networks in the set. A strong feature representation representing the entirety of the features is produced for each video frame in the sequence of video frames by aggregating the output features extracted by the multiple neural networks.Type: GrantFiled: April 13, 2020Date of Patent: June 7, 2022Assignee: Adobe Inc.Inventors: Federico Perazzi, Zhe Lin, Ping Hu, Oliver Wang, Fabian David Caba Heilbron
-
Publication number: 20220156588Abstract: Certain embodiments involve techniques for efficiently estimating denoising kernels for generating denoised images. For instance, a neural network receives a noisy reference image to denoise. The neural network uses a kernel dictionary of base kernels and generates a coefficient vector for each pixel in the reference image such that the coefficient vector includes a coefficient value for each base kernel in the kernel dictionary, where the base kernels are combined to generate a denoising kernel and each coefficient value indicates a contribution of a given base kernel to a denoising kernel. The neural network calculates the denoising kernel for a given pixel by applying the coefficient vector for that pixel to the kernel dictionary. The neural network applies each denoising kernel to the respective pixel to generate a denoised output image.Type: ApplicationFiled: February 2, 2022Publication date: May 19, 2022Inventors: Federico Perazzi, Zhihao Xia, Michael Gharbi, Kalyan Sunkavalli
-
Publication number: 20220148135Abstract: A plurality of pixel-based sampling points are identified within an image, wherein sampling points of a pixel are distributed within the pixel. For individual sampling points of individual pixels, a corresponding radiance vector is estimated. A radiance vector includes one or more radiance values characterizing light received at a sampling point. A first machine learning module generates, for each pixel, a corresponding intermediate radiance feature vector, based on the radiance vectors associated with the sampling points within that pixel. A second machine learning module generates, for each pixel, a corresponding final radiance feature vector, based on an intermediate radiance feature vector for that pixel, and one or more other intermediate radiance feature vectors for one or more other pixels neighboring that pixel. One or more kernels are generated, based on the final radiance feature vectors, and applied to corresponding pixels of the image, to generate a lower noise image.Type: ApplicationFiled: November 10, 2020Publication date: May 12, 2022Applicant: Adobe Inc.Inventors: Mustafa Isik, Michael Yanis Gharbi, Matthew David Fisher, Krishna Bhargava Mullia Lakshminarayana, Jonathan Eisenmann, Federico Perazzi
-
Publication number: 20220122224Abstract: The present disclosure relates to an image retouching system that automatically retouches digital images by accurately correcting face imperfections such as skin blemishes and redness. For instance, the image retouching system automatically retouches a digital image through separating digital images into multiple frequency layers, utilizing a separate corresponding neural network to apply frequency-specific corrections at various frequency layers, and combining the retouched frequency layers into a retouched digital image. As described herein, the image retouching system efficiently utilizes different neural networks to target and correct skin features specific to each frequency layer.Type: ApplicationFiled: October 16, 2020Publication date: April 21, 2022Inventors: Federico Perazzi, Jingwan Lu
-
Patent number: 11281970Abstract: Certain embodiments involve techniques for efficiently estimating denoising kernels for generating denoised images. For instance, a neural network receives a noisy reference image to denoise. The neural network uses a kernel dictionary of base kernels and generates a coefficient vector for each pixel in the reference image such that the coefficient vector includes a coefficient value for each base kernel in the kernel dictionary, where the base kernels are combined to generate a denoising kernel and each coefficient value indicates a contribution of a given base kernel to a denoising kernel. The neural network calculates the denoising kernel for a given pixel by applying the coefficient vector for that pixel to the kernel dictionary. The neural network applies each denoising kernel to the respective pixel to generate a denoised output image.Type: GrantFiled: November 18, 2019Date of Patent: March 22, 2022Assignee: Adobe Inc.Inventors: Federico Perazzi, Zhihao Xia, Michael Gharbi, Kalyan Sunkavalli
-
Patent number: 11244204Abstract: In implementations of determining video cuts in video clips, a video cut detection system can receive a video clip that includes a sequence of digital video frames that depict one or more scenes. The video cut detection system can determine scene characteristics for the digital video frames. The video cut detection system can determine, from the scene characteristics, a probability of a video cut between two adjacent digital video frames having a boundary between the two adjacent digital video frames that is centered in the sequence of digital video frames. The video cut detection system can then compare the probability of the video cut to a cut threshold to determine whether the video cut exists between the two adjacent digital video frames.Type: GrantFiled: May 20, 2020Date of Patent: February 8, 2022Assignee: Adobe Inc.Inventors: Oliver Wang, Nico Alexander Becherer, Markus Woodson, Federico Perazzi, Nikhil Kalra
-
Publication number: 20220012885Abstract: The present disclosure relates to utilizing a neural network having a two-stream encoder architecture to accurately generate composite digital images that realistically portray a foreground object from one digital image against a scene from another digital image. For example, the disclosed systems can utilize a foreground encoder of the neural network to identify features from a foreground image and further utilize a background encoder to identify features from a background image. The disclosed systems can then utilize a decoder to fuse the features together and generate a composite digital image. The disclosed systems can train the neural network utilizing an easy-to-hard data augmentation scheme implemented via self-teaching. The disclosed systems can further incorporate the neural network within an end-to-end framework for automation of the image composition process.Type: ApplicationFiled: September 23, 2021Publication date: January 13, 2022Inventors: Zhe Lin, Jianming Zhang, He Zhang, Federico Perazzi
-
Publication number: 20210365742Abstract: In implementations of determining video cuts in video clips, a video cut detection system can receive a video clip that includes a sequence of digital video frames that depict one or more scenes. The video cut detection system can determine scene characteristics for the digital video frames. The video cut detection system can determine, from the scene characteristics, a probability of a video cut between two adjacent digital video frames having a boundary between the two adjacent digital video frames that is centered in the sequence of digital video frames. The video cut detection system can then compare the probability of the video cut to a cut threshold to determine whether the video cut exists between the two adjacent digital video frames.Type: ApplicationFiled: May 20, 2020Publication date: November 25, 2021Applicant: Adobe Inc.Inventors: Oliver Wang, Nico Alexander Becherer, Markus Woodson, Federico Perazzi, Nikhil Kalra
-
Patent number: 11158055Abstract: The present disclosure relates to utilizing a neural network having a two-stream encoder architecture to accurately generate composite digital images that realistically portray a foreground object from one digital image against a scene from another digital image. For example, the disclosed systems can utilize a foreground encoder of the neural network to identify features from a foreground image and further utilize a background encoder to identify features from a background image. The disclosed systems can then utilize a decoder to fuse the features together and generate a composite digital image. The disclosed systems can train the neural network utilizing an easy-to-hard data augmentation scheme implemented via self-teaching. The disclosed systems can further incorporate the neural network within an end-to-end framework for automation of the image composition process.Type: GrantFiled: July 26, 2019Date of Patent: October 26, 2021Assignee: ADOBE INC.Inventors: Zhe Lin, Jianming Zhang, He Zhang, Federico Perazzi
-
Publication number: 20210319232Abstract: A Video Semantic Segmentation System (VSSS) is disclosed that performs accurate and fast semantic segmentation of videos using a set of temporally distributed neural networks. The VSSS receives as input a video signal comprising a contiguous sequence of temporally-related video frames. The VSSS extracts features from the video frames in the contiguous sequence and based upon the extracted features, selects, from a set of labels, a label to be associated with each pixel of each video frame in the video signal. In certain embodiments, a set of multiple neural networks are used to extract the features to be used for video segmentation and the extraction of features is distributed among the multiple neural networks in the set. A strong feature representation representing the entirety of the features is produced for each video frame in the sequence of video frames by aggregating the output features extracted by the multiple neural networks.Type: ApplicationFiled: April 13, 2020Publication date: October 14, 2021Inventors: Federico Perazzi, Zhe Lin, Ping Hu, Oliver Wang, Fabian David Caba Heilbron
-
Patent number: 11126890Abstract: Systems and methods are described for object detection within a digital image using a hierarchical softmax function. The method may include applying a first softmax function of a softmax hierarchy on a digital image based on a first set of object classes that are children of a root node of a class hierarchy, then apply a second (and subsequent) softmax functions to the digital image based on a second (and subsequent) set of object classes, where the second (and subsequent) object classes are children nodes of an object class from the first (or parent) object classes. The methods may then include generating an object recognition output using a convolutional neural network (CNN) based at least in part on applying the first and second (and subsequent) softmax functions. In some cases, the hierarchical softmax function is the loss function for the CNN.Type: GrantFiled: April 18, 2019Date of Patent: September 21, 2021Assignee: ADOBE INC.Inventors: Zhe Lin, Mingyang Ling, Jianming Zhang, Jason Kuen, Federico Perazzi, Brett Butterfield, Baldo Faieta