Patents by Inventor Sergey Tulyakov

Sergey Tulyakov has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Patent number: 12776075
    Abstract: Methods and systems are disclosed for enhancing or modifying an image by a machine learning model. The methods and systems receive an image depicting a real-world object. The methods and systems analyze the image using a machine learning model to generate a modified image that depicts one or more augmented reality stylizations overlaid on the real-world object, the machine learning model trained in multiple stages having different training data sets and different conditions applied in each of the multiple stages. The methods and systems present the modified image on a device.
    Type: Grant
    Filed: August 9, 2023
    Date of Patent: October 27, 2026
    Assignee: Snap Inc.
    Inventors: Ekaterina Deyneka, Andrey Alejandrovich Gomez Zharkov, Sergey Tulyakov, Aleksei Stoliar, Konstantin Gudkov
  • Publication number: 20260332736
    Abstract: A method for generating photorealistic 4D scenes from text inputs is disclosed. The method utilizes a text-to-video diffusion model to generate a reference video and a freeze-time video. A canonical 3D representation is reconstructed using deformable 3D Gaussian Splats (D-3DGS) based on the freeze-time video. Temporal deformations are learned to capture dynamic interactions in the reference video. The method employs a novel Score Distillation Sampling strategy combining multi-view and temporal aspects to enhance consistency and robustness. The resulting 4D scenes feature multiple objects interacting with detailed background environments, viewable from different angles and times. The method enables flexible camera control and integration with augmented and virtual reality applications. Some examples include features such as image-to-4D generation.
    Type: Application
    Filed: June 26, 2026
    Publication date: October 22, 2026
    Inventors: Hsin-Ying Lee, Willi Menapace, Aliaksandr Siarohin, Sergey Tulyakov, Chaoyang Wang, Heng Yu, Pieye Zhuang
  • Publication number: 20260320974
    Abstract: The present disclosure seeks to address technical problems arising in the field of artificial intelligence (AI) by providing for a multiple-input, fixed-size representation that is used for generative tasks, such as image generation. The representation combines identity preserving features for a subject from the multiple inputs to facilitate generation of high quality images of the subject in a variety of poses, states, conditions, and environments. To generate the representation, an encoder is trained using a multi-decoder training strategy to facilitate the generation of a representation that can be used by various decoders with various objectives.
    Type: Application
    Filed: April 9, 2025
    Publication date: October 15, 2026
    Inventors: Guocheng Qian, Kuan-Chieh Wang, Or Patashnik, Negin Heravi, Daniil Ostashev, Sergey Tulyakov, Daniel Cohen-Or, Kfir Aberman
  • Publication number: 20260320886
    Abstract: Examples relate to systems and methods for cross-modal audiovisual generation using diffusion features. A multi-modal diffusion model includes a first modality path comprising diffusion transformer (DiT) blocks pretrained to generate outputs of a first modality, a second modality path comprising DiT blocks pretrained to generate outputs of a second modality, and fusion blocks connecting the paths. Each fusion block receives activations from both modality paths, establishes temporal correspondence between them using self-attention mechanisms with rotary position embeddings, and provides modified activations back to subsequent DiT blocks in both paths. The system processes an input in one modality to generate temporally-aligned output in the other modality, with the fusion blocks enabling bidirectional information exchange while maintaining temporal alignment. The approach eliminates the need for specialized feature extractors by leveraging the activations of frozen pretrained diffusion models.
    Type: Application
    Filed: April 11, 2025
    Publication date: October 15, 2026
    Inventors: Moayed Haji Ali, Alper Canberk, Kwot Sin Lee, Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Sergey Tulyakov
  • Publication number: 20260311479
    Abstract: Described is a system for improving machine learning models by accessing a first latent diffusion machine learning model, accessing a second latent diffusion machine learning model that was derived from the first latent diffusion machine learning model, the second latent diffusion machine learning model trained to perform a second number of denoising steps, generating noise data, processing the noise data via the first latent diffusion machine learning model to generate one or more first latent features, processing the noise data via the second latent diffusion machine learning model to generate one or more second latent features, and inputting the one or more first latent features and the one or more second latent features into a loss function. The system then modifies a parameter of the second latent diffusion machine learning model based on the output of the loss function.
    Type: Application
    Filed: June 16, 2026
    Publication date: October 8, 2026
    Inventors: Pavlo Chemerys, Colin Eles, Ju Hu, Qing Jin, Yanyu Li, Ergeta Muca, Jian Ren, Dhritiman Sagar, Aleksei Stoliar, Sergey Tulyakov, Huan Wang
  • Publication number: 20260292114
    Abstract: Examples relate to computer-implemented systems and methods for generating four-dimensional (4D) video content. A 4D video generation model receives a freeze-time video showing a scene varying in viewpoint and a fixed-view video showing the scene varying in time. The model processes these inputs through parallel pathways—a freeze-time pathway and a fixed-view pathway—each comprising pretrained diffusion transformer blocks. The pathways are synchronized via interleaved synchronization layers to generate a grid of video frames varying consistently in both time and viewpoint. The generated frame grid can be used to reconstruct a three-dimensional representation of the scene for real-time rendering of views from arbitrary viewpoints.
    Type: Application
    Filed: March 24, 2025
    Publication date: September 24, 2026
    Inventors: Hsin-Ying Lee, Willi Menapace, Tuan Duc Ngo, Aliaksandr Siarohin, Ivan Skorokhodov, Sergey Tulyakov, Mikhail Vasilkovskii, Chaoyang Wang, Peter Wonka, Peiye Zhuang
  • Publication number: 20260289238
    Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and method for video synthesis. The program and method provide for accessing a primary generative adversarial network (GAN) comprising a pre-trained image generator, a motion generator comprising a plurality of neural networks, and a video discriminator; generating an updated GAN based on the primary GAN, by performing operations comprising identifying input data of the updated GAN, the input data comprising an initial latent code and a motion domain dataset, training the motion generator based on the input data, and adjusting weights of the plurality of neural networks of the primary GAN based on an output of the video discriminator; and generating a synthesized video based on the primary GAN and the input data.
    Type: Application
    Filed: June 2, 2026
    Publication date: September 24, 2026
    Inventors: Menglei Chai, Kyle Olszewski, Jian Ren, Yu Tian, Sergey Tulyakov
  • Publication number: 20260289307
    Abstract: Techniques for training a neural network having a plurality of computational layers with associated weights and activations for computational layers in fixed-point formats include determining an optimal fractional length for weights and activations for the computational layers; training a learned clipping-level with fixed-point quantization using a PACT process for the computational layers; and quantizing on effective weights that fuses a weight of a convolution layer with a weight and running variance from a batch normalization layer. A fractional length for weights of the computational layers is determined from current values of weights using the determined optimal fractional length for the weights of the computational layers. A fixed-point activation between adjacent computational layers is related using PACT quantization of the clipping-level and an activation fractional length from a node in a following computational layer.
    Type: Application
    Filed: May 21, 2026
    Publication date: September 24, 2026
    Inventors: Sumant Milind Hanumante, Qing Jin, Sergei Korolev, Denys Makoviichuk, Jian Ren, Dhritiman Sagar, Patrick Timothy McSweeney Simons, Sergey Tulyakov, Yang Wen, Richard Zhuang
  • Publication number: 20260279039
    Abstract: Described is a system for improving machine learning models. In some cases, the system improves such models by identifying an autoencoder for a latent diffusion machine learning model, the latent diffusion machine learning model is trained to receive text as input and output an image based on the received text. The system identifies a number of channels in a decoder of the autoencoder, the decoder being configured to receive latent features as input and output images. The system further identifies a performance characteristic of the decoder and changes the node topology of the decoder based on the performance characteristic to generate an updated decoder. The system retrains the latent diffusion machine learning model using the updated decoder by inputting latent features to the updated decoder, receiving an outputted image from the updated decoder, and updating one or more weights of the decoder based on an assessment of the outputted image.
    Type: Application
    Filed: April 9, 2026
    Publication date: September 17, 2026
    Inventors: Pavlo Chemerys, Colin Eles, Ju Hu, Qing Jin, Yanyu Li, Ergeta Muca, Jian Ren, Dhritiman Sagar, Aleksei Stoliar, Sergey Tulyakov, Huan Wang
  • Publication number: 20260278905
    Abstract: Example systems take as input a text prompt describing events and generate a multi-event video, audio, or data sequence based on the text prompt. Example operations include accessing a sequence of events, the sequence of events comprising time ranges for the events of the sequence of events and text descriptions for the events, accessing a global caption comprising a text description for the sequence of events, and inputting the sequence of events and the global caption simultaneously into a trained neural network to generate a video, audio, or data sequence, where frames of the video, sound of the audio, or data of the data sequence during a time range of the time ranges are in accordance with a corresponding text description of the text descriptions. Additionally, systems are disclosed that train a neural network to generate the multi-event video, audio, or a data sequence from the text prompt.
    Type: Application
    Filed: March 17, 2025
    Publication date: September 17, 2026
    Inventors: Varnith Uttam Chordia, Yuwei Fang, Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Sergey Tulyakov, Ziyi Wu
  • Publication number: 20260267493
    Abstract: A system of machine learning schemes can be configured to efficiently perform image processing tasks on a user device, such as a mobile phone. The system can selectively detect and transform individual regions within each frame of a live streaming video. The system can selectively partition and toggle image effects within the live streaming video.
    Type: Application
    Filed: May 4, 2026
    Publication date: September 10, 2026
    Inventors: Theresa Barton, Yanping Chen, Jaewook Chung, Christopher Yale Crutchfield, Aymeric Damien, Sergei Kotcur, Igor Kudrlashov, Sergey Tulyakov, Andrew Wan, Emre Yamangil
  • Patent number: 12726637
    Abstract: Examples relate to systems and methods for generating videos with precise camera control using a camera-conditioned video diffusion transformer (DiT) model. The model performs a denoising process through a series of pretrained video DiT blocks, where camera trajectory information is processed through a camera conditioning branch to generate camera activations. During an initial portion of denoising passes, the process is conditioned on camera activations for an initial subset of video DiT blocks, while later denoising passes proceed without camera conditioning. This approach leverages the insight that camera motion is established early in the denoising process, enabling precise camera control while maintaining high visual quality.
    Type: Grant
    Filed: May 6, 2025
    Date of Patent: September 1, 2026
    Assignee: Snap Inc.
    Inventors: Sherwin Bahmani, Willi Menapace, Guocheng Qian, Aliaksandr Siarohin, Ivan Skorokhodov, Sergey Tulyakov
  • Patent number: 12705875
    Abstract: Described is a system for improving machine learning models by accessing a first latent diffusion machine learning model, accessing a second latent diffusion machine learning model that was derived from the first latent diffusion machine learning model, the second latent diffusion machine learning model trained to perform a second number of denoising steps, generating noise data, processing the noise data via the first latent diffusion machine learning model to generate one or more first latent features, processing the noise data via the second latent diffusion machine learning model to generate one or more second latent features, and inputting the one or more first latent features and the one or more second latent features into a loss function. The system then modifies a parameter of the second latent diffusion machine learning model based on the output of the loss function.
    Type: Grant
    Filed: March 5, 2024
    Date of Patent: August 11, 2026
    Assignee: Snap Inc.
    Inventors: Pavlo Chemerys, Colin Eles, Ju Hu, Qing Jin, Yanyu Li, Ergeta Muca, Jian Ren, Dhritiman Sagar, Aleksei Stoliar, Sergey Tulyakov, Huan Wang
  • Patent number: 12700160
    Abstract: A text-to-video framework including a far-reaching interleaved transformer (FIT) block configured to learn a compressed representation of video input using a set of learnable latent tokens. The FIT block includes a diffusion framework and joint spatiotemporal modeling. The FIT block performs patchification of the video input to produce a sequence of patch tokens that are divided into groups. The FIT block instantiates the set of latent tokens and applies a sequence of computational blocks, and projects the patch tokens to generate video frames.
    Type: Grant
    Filed: January 31, 2024
    Date of Patent: August 4, 2026
    Assignee: Snap Inc.
    Inventors: Tsai-Shien Chen, Yuwei Fang, Anil Kag, Willi Menapace, Jian Ren, Aliaksandr Siarohin, Ivan Skorokhodov, Sergey Tulyakov
  • Patent number: 12701209
    Abstract: A method for generating photorealistic 4D scenes from text inputs is disclosed. The method utilizes a text-to-video diffusion model to generate a reference video and a freeze-time video. A canonical 3D representation is reconstructed using deformable 3D Gaussian Splats (D-3DGS) based on the freeze-time video. Temporal deformations are learned to capture dynamic interactions in the reference video. The method employs a novel Score Distillation Sampling strategy combining multi-view and temporal aspects to enhance consistency and robustness. The resulting 4D scenes feature multiple objects interacting with detailed background environments, viewable from different angles and times. The method enables flexible camera control and integration with augmented and virtual reality applications. Some examples include features such as image-to-4D generation.
    Type: Grant
    Filed: September 23, 2024
    Date of Patent: August 4, 2026
    Assignee: Snap Inc.
    Inventors: Hsin-Ying Lee, Willi Menapace, Aliaksandr Siarohin, Sergey Tulyakov, Chaoyang Wang, Heng Yu, Peiye Zhuang
  • Publication number: 20260205619
    Abstract: A method, computer readable medium, and system are disclosed for action video generation. The method includes the steps of generating, by a recurrent neural network, a sequence of motion vectors from a first set of random variables and receiving, by a generator neural network, the sequence of motion vectors and a content vector sample. The sequence of motion vectors and the content vector sample are sampled by the generator neural network to produce a video clip.
    Type: Application
    Filed: December 22, 2025
    Publication date: July 16, 2026
    Inventors: Ming-Yu Liu, Xiaodong Yang, Jan Kautz, Sergey Tulyakov
  • Patent number: 12670362
    Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and method for video synthesis. The program and method provide for accessing a primary generative adversarial network (GAN) comprising a pre-trained image generator, a motion generator comprising a plurality of neural networks, and a video discriminator; generating an updated GAN based on the primary GAN, by performing operations comprising identifying input data of the updated GAN, the input data comprising an initial latent code and a motion domain dataset, training the motion generator based on the input data, and adjusting weights of the plurality of neural networks of the primary GAN based on an output of the video discriminator; and generating a synthesized video based on the primary GAN and the input data.
    Type: Grant
    Filed: September 30, 2021
    Date of Patent: June 30, 2026
    Assignee: Snap Inc,.
    Inventors: Menglei Chai, Kyle Olszewski, Jian Ren, Yu Tian, Sergey Tulyakov
  • Patent number: 12663914
    Abstract: A system of machine learning schemes can be configured to efficiently perform image processing tasks on a user device, such as a mobile phone. The system can selectively detect and transform individual regions within each frame of a live streaming video. The system can selectively partition and toggle image effects within the live streaming video.
    Type: Grant
    Filed: August 9, 2023
    Date of Patent: June 23, 2026
    Inventors: Theresa Barton, Yanping Chen, Jaewook Chung, Christopher Yale Crutchfield, Aymeric Damien, Sergei Kotcur, Igor Kudriashov, Sergey Tulyakov, Andrew Wan, Emre Yamangil
  • Publication number: 20260162367
    Abstract: A system and method are described for generating 3D garments from two-dimensional (2D) scribble images drawn by users. The system includes a conditional 2D generator, a conditional 3D generator, and two intermediate media including dimension-coupling color-density pairs and flat point clouds that bridge the gap between dimensions. Given a scribble image, the 2D generator synthesizes dimension-coupling color-density pairs including the RGB projection and density map from the front and rear views of the scribble image. A density-aware sampling algorithm converts the 2D dimension-coupling color-density pairs into a 3D flat point cloud representation, where the depth information is ignored. The 3D generator predicts the depth information from the flat point cloud. Dynamic variations per garment due to deformations resulting from a wearer's pose as well as irregular wrinkles and folds may be bypassed by taking advantage of 2D generative models to bridge the dimension gap in a non-parametric way.
    Type: Application
    Filed: April 15, 2025
    Publication date: June 11, 2026
    Inventors: Panagiotis Achlioptas, Menglei Chai, Hsin-Ying Lee, Kyle Olszewski, Jian Ren, Sergey Tulyakov
  • Patent number: 12645936
    Abstract: Techniques for training a neural network having a plurality of computational layers with associated weights and activations for computational layers in fixed-point formats include determining an optimal fractional length for weights and activations for the computational layers; training a learned clipping-level with fixed-point quantization using a PACT process for the computational layers; and quantizing on effective weights that fuses a weight of a convolution layer with a weight and running variance from a batch normalization layer. A fractional length for weights of the computational layers is determined from current values of weights using the determined optimal fractional length for the weights of the computational layers. A fixed-point activation between adjacent computational layers is related using PACT quantization of the clipping-level and an activation fractional length from a node in a following computational layer.
    Type: Grant
    Filed: December 31, 2021
    Date of Patent: June 2, 2026
    Assignee: Snap Inc.
    Inventors: Sumant Milind Hanumante, Qing Jin, Sergei Korolev, Denys Makoviichuk, Jian Ren, Dhritiman Sagar, Patrick Timothy McSweeney Simons, Sergey Tulyakov, Yang Wen, Richard Zhuang