Patents by Inventor Richard Zhang

Richard Zhang has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20230360376
    Abstract: Semantic fill techniques are described that support generating fill and editing images from semantic inputs. A user input, for example, is received by a semantic fill system that indicates a selection of a first region of a digital image and a corresponding semantic label. The user input is utilized by the semantic fill system to generate a guidance attention map of the digital image. The semantic fill system leverages the guidance attention map to generate a sparse attention map of a second region of the digital image. A semantic fill of pixels is generated for the first region based on the semantic label and the sparse attention map. The edited digital image is displayed in a user interface.
    Type: Application
    Filed: May 16, 2022
    Publication date: November 9, 2023
    Applicant: Adobe Inc.
    Inventors: Tobias Hinz, Taesung Park, Richard Zhang, Matthew David Fisher, Difan Liu, Evangelos Kalogerakis
  • Publication number: 20230342893
    Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for combining digital images. In particular, in one or more embodiments, the disclosed systems combine latent codes of a source digital image and a target digital image utilizing a blending network to determine a combined latent encoding and generate a combined digital image from the combined latent encoding utilizing a generative neural network. In some embodiments, the disclosed systems determine an intersection face mask between the source digital image and the combined digital image utilizing a face segmentation network and combine the source digital image and the combined digital image utilizing the intersection face mask to generate a blended digital image.
    Type: Application
    Filed: April 21, 2022
    Publication date: October 26, 2023
    Inventors: Tobias Hinz, Shabnam Ghadar, Richard Zhang, Ratheesh Kalarot, Jingwan Lu, Elya Shechtman
  • Publication number: 20230316591
    Abstract: Techniques for identity preserved controllable facial image manipulation are described that support generation of a manipulated digital image based on a facial image and a render image. For instance, a facial image depicting a facial representation of an individual is received as input. A feature space including an identity parameter and at least one other visual parameter is extracted from the facial image. An editing module edits one or more of the visual parameters and preserves the identity parameter. A renderer generates a render image depicting a morphable model reconstruction of the facial image based on the edit. The render image and facial image are encoded, and a generator of a neural network is implemented to generate a manipulated digital image based on the encoded facial image and the encoded render image.
    Type: Application
    Filed: March 31, 2022
    Publication date: October 5, 2023
    Applicant: Adobe Inc.
    Inventors: Zhixin Shu, Zhe Lin, Yuchen Liu, Yijun Li, Richard Zhang
  • Publication number: 20230316606
    Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for latent-based editing of digital images using a generative neural network. In particular, in one or more embodiments, the disclosed systems perform latent-based editing of a digital image by mapping a feature tensor and a set of style vectors for the digital image into a joint feature style space. In one or more implementations, the disclosed systems apply a joint feature style perturbation and/or modification vectors within the joint feature style space to determine modified style vectors and a modified feature tensor. Moreover, in one or more embodiments the disclosed systems generate a modified digital image utilizing a generative neural network from the modified style vectors and the modified feature tensor.
    Type: Application
    Filed: March 21, 2022
    Publication date: October 5, 2023
    Inventors: Hui Qu, Baldo Faieta, Cameron Smith, Elya Shechtman, Jingwan Lu, Ratheesh Kalarot, Richard Zhang, Saeid Motiian, Shabnam Ghadar, Wei-An Lin
  • Patent number: 11763495
    Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately and efficiently modifying a generative adversarial neural network using few-shot adaptation to generate digital images corresponding to a target domain while maintaining diversity of a source domain and realism of the target domain. In particular, the disclosed systems utilize a generative adversarial neural network with parameters learned from a large source domain. The disclosed systems preserve relative similarities and differences between digital images in the source domain using a cross-domain distance consistency loss. In addition, the disclosed systems utilize an anchor-based strategy to encourage different levels or measures of realism over digital images generated from latent vectors in different regions of a latent space.
    Type: Grant
    Filed: January 29, 2021
    Date of Patent: September 19, 2023
    Assignee: Adobe Inc.
    Inventors: Utkarsh Ojha, Yijun Li, Richard Zhang, Jingwan Lu, Elya Shechtman, Alexei A. Efros
  • Patent number: 11762951
    Abstract: Embodiments are disclosed for generative image congealing which provides an unsupervised learning technique that learns transformations of real data to improve the image quality of GANs trained using that image data. In particular, in one or more embodiments, the disclosed systems and methods comprise generating, by a spatial transformer network, an aligned real image for a real image from an unaligned real dataset, providing, by the spatial transformer network, the aligned real image to an adversarial discrimination network to determine if the aligned real image resembles aligned synthetic images generated by a generator network, and training, by a training manager, the spatial transformer network to learn updated transformations based on the determination of the adversarial discrimination network.
    Type: Grant
    Filed: November 18, 2020
    Date of Patent: September 19, 2023
    Assignee: Adobe Inc.
    Inventors: Elya Shechtman, William Peebles, Richard Zhang, Jun-Yan Zhu, Alyosha Efros
  • Publication number: 20230289970
    Abstract: In implementations of systems for image inversion using multiple latent spaces, a computing device implements an inversion system to generate a segment map that segments an input digital image into a first image region and a second image region and assigns the first image region to a first latent space and the second image region to a second latent space that corresponds to a layer of a convolutional neural network. An inverted latent representation of the input digital image is computed using a binary mask for the second image region. The inversion system modifies the inverted latent representation of the input digital image using an edit direction vector that corresponds to a visual feature. An output digital image is generated that depicts a reconstruction of the input digital image having the visual feature based on the modified inverted latent representation of the input digital image.
    Type: Application
    Filed: March 14, 2022
    Publication date: September 14, 2023
    Applicant: Adobe Inc.
    Inventors: Gaurav Parmar, Krishna Kumar Singh, Yijun Li, Richard Zhang, Jingwan Lu
  • Publication number: 20230274535
    Abstract: An image generation system enables user input during the process of training a generative model to influence the model's ability to generate new images with desired visual features. A source generative model for a source domain is fine-tuned using training images in a target domain to provide an adapted generative model for the target domain. Interpretable factors are determined for the source generative model and the adapted generative model. A user interface is provided that enables a user to select one or more interpretable factors. The user-selected interpretable factor(s) are used to generate a user-adapted generative model, for instance, by using a loss function based on the user-selected interpretable factor(s). The user-adapted generative model can be used to create new images in the target domain.
    Type: Application
    Filed: February 25, 2022
    Publication date: August 31, 2023
    Inventors: Yijun Li, Utkarsh Ojha, Richard Zhang, Jingwan Lu, Elya Shechtman, Alexei A. Efros
  • Publication number: 20230260175
    Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for generating digital images depicting photorealistic scenes utilizing a digital image collaging neural network. For example, the disclosed systems utilize a digital image collaging neural network having a particular architecture for disentangling generation of scene layouts and pixel colors for different regions of a digital image. In some cases, the disclosed systems break down the process of generating a collage digital into generating images representing different regions such as a background and a foreground to be collaged into a final result. For example, utilizing the digital image collaging neural network, the disclosed systems determine scene layouts and pixel colors for both foreground digital images and background digital images to ultimately collage the foreground and background together into a collage digital image depicting a real-world scene.
    Type: Application
    Filed: February 14, 2022
    Publication date: August 17, 2023
    Inventors: Nadav Epstein, Alexei A. Efros, Taesung Park, Richard Zhang, Elya Shechtman
  • Publication number: 20230245363
    Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately and flexibly generating modified digital images utilizing a novel swapping autoencoder that incorporates scene layout. In particular, the disclosed systems can receive a scene layout map that indicates or defines locations for displaying specific digital content within a digital image. In addition, the disclosed systems can utilize the scene layout map to guide combining portions of digital image latent code to generate a modified digital image with a particular textural appearance and a particular geometric structure defined by the scene layout map. Additionally, the disclosed systems can utilize a scene layout map that defines a portion of a digital image to modify by, for instance, adding new digital content to the digital image, and can generate a modified digital image depicting the new digital content.
    Type: Application
    Filed: April 10, 2023
    Publication date: August 3, 2023
    Inventors: Taesung Park, Alexei A. Efros, Elya Shechtman, Richard Zhang, Junyan Zhu
  • Patent number: 11625875
    Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately and flexibly generating modified digital images utilizing a novel swapping autoencoder that incorporates scene layout. In particular, the disclosed systems can receive a scene layout map that indicates or defines locations for displaying specific digital content within a digital image. In addition, the disclosed systems can utilize the scene layout map to guide combining portions of digital image latent code to generate a modified digital image with a particular textural appearance and a particular geometric structure defined by the scene layout map. Additionally, the disclosed systems can utilize a scene layout map that defines a portion of a digital image to modify by, for instance, adding new digital content to the digital image, and can generate a modified digital image depicting the new digital content.
    Type: Grant
    Filed: November 6, 2020
    Date of Patent: April 11, 2023
    Assignee: Adobe Inc.
    Inventors: Taesung Park, Alexei A. Efros, Elya Shechtman, Richard Zhang, Junyan Zhu
  • Publication number: 20230102055
    Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for generating a modified digital image from extracted spatial and global codes. For example, the disclosed systems can utilize a global and spatial autoencoder to extract spatial codes and global codes from digital images. The disclosed systems can further utilize the global and spatial autoencoder to generate a modified digital image by combining extracted spatial and global codes in various ways for various applications such as style swapping, style blending, and attribute editing.
    Type: Application
    Filed: November 22, 2022
    Publication date: March 30, 2023
    Inventors: Taesung Park, Richard Zhang, Oliver Wang, Junyan Zhu, Jingwan Lu, Elya Shechtman, Alexei A. Efros
  • Patent number: 11615292
    Abstract: A target image is projected into a latent space of generative model by determining a latent vector by applying a gradient-free technique and a class vector by applying a gradient-based technique. An image is generated from the latent and class vectors, and a loss function is used to determine a loss between the target image and the generated image. This determining of the latent vector and the class vector, generating an image, and using the loss function is repeated until a loss condition is satisfied. In response to the loss condition being satisfied, the latent and class vectors that resulted in the loss condition being satisfied are identified as the final latent and class vectors, respectively. The final latent and class vectors are provided to the generative model and multiple weights of the generative model are adjusted to fine-tune the generative model.
    Type: Grant
    Filed: August 31, 2022
    Date of Patent: March 28, 2023
    Assignee: Adobe Inc.
    Inventors: Richard Zhang, Sylvain Philippe Paris, Junyan Zhu, Aaron Phillip Hertzmann, Jacob Minyoung Huh
  • Publication number: 20230055204
    Abstract: This disclosure describes methods, non-transitory computer readable storage media, and systems that utilize one or more stages of a two-stage image colorization neural network to colorize or re-colorize digital images. In one or more embodiments, the disclosed system generates a color digital image from a grayscale digital image by utilizing a colorization neural network. Additionally, the disclosed system receives one or more inputs indicating local hints comprising one or more color selections to apply to one or more objects of the color digital image. The disclosed system then utilizes a re-colorization neural network to generate a modified digital image from the color digital image by modifying one or more colors of the object(s) based on the luminance channel, color channels, and selected color(s).
    Type: Application
    Filed: August 18, 2021
    Publication date: February 23, 2023
    Inventors: Adrian-Stefan Ungureanu, Ionut Mironica, Richard Zhang
  • Patent number: 11562166
    Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for generating shift-resilient neural network outputs based on utilizing a dense pooling layer, a low-pass filter layer, and a downsampling layer of a neural network. For example, the disclosed systems can generate a pooled feature map utilizing a dense pooling layer to densely pool feature values extracted from an input. The disclosed systems can further apply a low-pass filter to the pooled feature map to generate a shift-adaptive feature map. In addition, the disclosed systems can downsample the shift-adaptive feature map utilizing a downsampling layer. Based on the downsampled, shift-adaptive feature map, the disclosed systems can generate shift-resilient neural network outputs such as digital image classifications.
    Type: Grant
    Filed: May 21, 2021
    Date of Patent: January 24, 2023
    Assignee: Adobe Inc.
    Inventor: Richard Zhang
  • Patent number: 11544880
    Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for generating a modified digital image from extracted spatial and global codes. For example, the disclosed systems can utilize a global and spatial autoencoder to extract spatial codes and global codes from digital images. The disclosed systems can further utilize the global and spatial autoencoder to generate a modified digital image by combining extracted spatial and global codes in various ways for various applications such as style swapping, style blending, and attribute editing.
    Type: Grant
    Filed: May 14, 2020
    Date of Patent: January 3, 2023
    Assignee: Adobe Inc.
    Inventors: Taesung Park, Richard Zhang, Oliver Wang, Junyan Zhu, Jingwan Lu, Elya Shechtman, Alexei A Efros
  • Publication number: 20220414431
    Abstract: A target image is projected into a latent space of generative model by determining a latent vector by applying a gradient-free technique and a class vector by applying a gradient-based technique. An image is generated from the latent and class vectors, and a loss function is used to determine a loss between the target image and the generated image. This determining of the latent vector and the class vector, generating an image, and using the loss function is repeated until a loss condition is satisfied. In response to the loss condition being satisfied, the latent and class vectors that resulted in the loss condition being satisfied are identified as the final latent and class vectors, respectively. The final latent and class vectors are provided to the generative model and multiple weights of the generative model are adjusted to fine-tune the generative model.
    Type: Application
    Filed: August 31, 2022
    Publication date: December 29, 2022
    Applicant: Adobe Inc.
    Inventors: Richard Zhang, Sylvain Philippe Paris, Junyan Zhu, Aaron Phillip Hertzmann, Jacob Minyoung Huh
  • Patent number: 11514632
    Abstract: This disclosure describes methods, non-transitory computer readable storage media, and systems that utilize a contrastive perceptual loss to modify neural networks for generating synthetic digital content items. For example, the disclosed systems generate a synthetic digital content item based on a guide input to a generative neural network. The disclosed systems utilize an encoder neural network to generate encoded representations of the synthetic digital content item and a corresponding ground-truth digital content item. Additionally, the disclosed systems sample patches from the encoded representations of the encoded digital content items and then determine a contrastive loss based on the perceptual distances between the patches in the encoded representations. Furthermore, the disclosed systems jointly update the parameters of the generative neural network and the encoder neural network utilizing the contrastive loss.
    Type: Grant
    Filed: November 6, 2020
    Date of Patent: November 29, 2022
    Assignee: Adobe Inc.
    Inventors: Bryan Russell, Taesung Park, Richard Zhang, Junyan Zhu, Alexander Andonian
  • Patent number: 11508148
    Abstract: The present disclosure relates to systems, computer-implemented methods, and non-transitory computer readable medium for automatically transferring makeup from a reference face image to a target face image using a neural network trained using semi-supervised learning. For example, the disclosed systems can receive, at a neural network, a target face image and a reference face image, where the target face image is selected by a user via a graphical user interface (GUI) and the reference face image has makeup. The systems transfer, by the neural network, the makeup from the reference face image to the target face image, where the neural network is trained to transfer the makeup from the reference face image to the target face image using semi-supervised learning. The systems output for display the makeup on the target face image.
    Type: Grant
    Filed: March 18, 2020
    Date of Patent: November 22, 2022
    Assignee: ADOBE INC.
    Inventors: Yijun Li, Zhifei Zhang, Richard Zhang, Jingwan Lu
  • Patent number: 11478804
    Abstract: A sprinkler assembly including a pressure regulator and a flow stop element combined in a single assembly. In one embodiment, the sprinkler includes a sprinkler body in fluid communication with a water supply to provide water to the sprinkler, a riser movably mounted in the sprinkler body and in fluid communication with the sprinkler body such that the riser rises up and out of the sprinkler body when water is provided to the sprinkler, a nozzle assembly mounted on a top end of the riser and in fluid communication with the riser; and a regulation element mounted in the riser and in contact with a portion of the nozzle assembly. The nozzle assembly holds the regulation member in an open position to allow flow of water to the nozzle assembly and to maintain a substantially constant water pressure in the nozzle assembly.
    Type: Grant
    Filed: January 9, 2019
    Date of Patent: October 25, 2022
    Assignee: K-Rain Manufacturing Corp.
    Inventors: Carl L. C. Kah, Jr., Carl L. C. Kah, III, Weiming Feng, Richard Zhang