Patents by Inventor Sergey Tulyakov
Sergey Tulyakov has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Patent number: 12776075Abstract: Methods and systems are disclosed for enhancing or modifying an image by a machine learning model. The methods and systems receive an image depicting a real-world object. The methods and systems analyze the image using a machine learning model to generate a modified image that depicts one or more augmented reality stylizations overlaid on the real-world object, the machine learning model trained in multiple stages having different training data sets and different conditions applied in each of the multiple stages. The methods and systems present the modified image on a device.Type: GrantFiled: August 9, 2023Date of Patent: October 27, 2026Assignee: Snap Inc.Inventors: Ekaterina Deyneka, Andrey Alejandrovich Gomez Zharkov, Sergey Tulyakov, Aleksei Stoliar, Konstantin Gudkov
-
Publication number: 20260332736Abstract: A method for generating photorealistic 4D scenes from text inputs is disclosed. The method utilizes a text-to-video diffusion model to generate a reference video and a freeze-time video. A canonical 3D representation is reconstructed using deformable 3D Gaussian Splats (D-3DGS) based on the freeze-time video. Temporal deformations are learned to capture dynamic interactions in the reference video. The method employs a novel Score Distillation Sampling strategy combining multi-view and temporal aspects to enhance consistency and robustness. The resulting 4D scenes feature multiple objects interacting with detailed background environments, viewable from different angles and times. The method enables flexible camera control and integration with augmented and virtual reality applications. Some examples include features such as image-to-4D generation.Type: ApplicationFiled: June 26, 2026Publication date: October 22, 2026Inventors: Hsin-Ying Lee, Willi Menapace, Aliaksandr Siarohin, Sergey Tulyakov, Chaoyang Wang, Heng Yu, Pieye Zhuang
-
Publication number: 20260320974Abstract: The present disclosure seeks to address technical problems arising in the field of artificial intelligence (AI) by providing for a multiple-input, fixed-size representation that is used for generative tasks, such as image generation. The representation combines identity preserving features for a subject from the multiple inputs to facilitate generation of high quality images of the subject in a variety of poses, states, conditions, and environments. To generate the representation, an encoder is trained using a multi-decoder training strategy to facilitate the generation of a representation that can be used by various decoders with various objectives.Type: ApplicationFiled: April 9, 2025Publication date: October 15, 2026Inventors: Guocheng Qian, Kuan-Chieh Wang, Or Patashnik, Negin Heravi, Daniil Ostashev, Sergey Tulyakov, Daniel Cohen-Or, Kfir Aberman
-
Publication number: 20260320886Abstract: Examples relate to systems and methods for cross-modal audiovisual generation using diffusion features. A multi-modal diffusion model includes a first modality path comprising diffusion transformer (DiT) blocks pretrained to generate outputs of a first modality, a second modality path comprising DiT blocks pretrained to generate outputs of a second modality, and fusion blocks connecting the paths. Each fusion block receives activations from both modality paths, establishes temporal correspondence between them using self-attention mechanisms with rotary position embeddings, and provides modified activations back to subsequent DiT blocks in both paths. The system processes an input in one modality to generate temporally-aligned output in the other modality, with the fusion blocks enabling bidirectional information exchange while maintaining temporal alignment. The approach eliminates the need for specialized feature extractors by leveraging the activations of frozen pretrained diffusion models.Type: ApplicationFiled: April 11, 2025Publication date: October 15, 2026Inventors: Moayed Haji Ali, Alper Canberk, Kwot Sin Lee, Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Sergey Tulyakov
-
Publication number: 20260311479Abstract: Described is a system for improving machine learning models by accessing a first latent diffusion machine learning model, accessing a second latent diffusion machine learning model that was derived from the first latent diffusion machine learning model, the second latent diffusion machine learning model trained to perform a second number of denoising steps, generating noise data, processing the noise data via the first latent diffusion machine learning model to generate one or more first latent features, processing the noise data via the second latent diffusion machine learning model to generate one or more second latent features, and inputting the one or more first latent features and the one or more second latent features into a loss function. The system then modifies a parameter of the second latent diffusion machine learning model based on the output of the loss function.Type: ApplicationFiled: June 16, 2026Publication date: October 8, 2026Inventors: Pavlo Chemerys, Colin Eles, Ju Hu, Qing Jin, Yanyu Li, Ergeta Muca, Jian Ren, Dhritiman Sagar, Aleksei Stoliar, Sergey Tulyakov, Huan Wang
-
Publication number: 20260292114Abstract: Examples relate to computer-implemented systems and methods for generating four-dimensional (4D) video content. A 4D video generation model receives a freeze-time video showing a scene varying in viewpoint and a fixed-view video showing the scene varying in time. The model processes these inputs through parallel pathways—a freeze-time pathway and a fixed-view pathway—each comprising pretrained diffusion transformer blocks. The pathways are synchronized via interleaved synchronization layers to generate a grid of video frames varying consistently in both time and viewpoint. The generated frame grid can be used to reconstruct a three-dimensional representation of the scene for real-time rendering of views from arbitrary viewpoints.Type: ApplicationFiled: March 24, 2025Publication date: September 24, 2026Inventors: Hsin-Ying Lee, Willi Menapace, Tuan Duc Ngo, Aliaksandr Siarohin, Ivan Skorokhodov, Sergey Tulyakov, Mikhail Vasilkovskii, Chaoyang Wang, Peter Wonka, Peiye Zhuang
-
Publication number: 20260289238Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and method for video synthesis. The program and method provide for accessing a primary generative adversarial network (GAN) comprising a pre-trained image generator, a motion generator comprising a plurality of neural networks, and a video discriminator; generating an updated GAN based on the primary GAN, by performing operations comprising identifying input data of the updated GAN, the input data comprising an initial latent code and a motion domain dataset, training the motion generator based on the input data, and adjusting weights of the plurality of neural networks of the primary GAN based on an output of the video discriminator; and generating a synthesized video based on the primary GAN and the input data.Type: ApplicationFiled: June 2, 2026Publication date: September 24, 2026Inventors: Menglei Chai, Kyle Olszewski, Jian Ren, Yu Tian, Sergey Tulyakov
-
Publication number: 20260289307Abstract: Techniques for training a neural network having a plurality of computational layers with associated weights and activations for computational layers in fixed-point formats include determining an optimal fractional length for weights and activations for the computational layers; training a learned clipping-level with fixed-point quantization using a PACT process for the computational layers; and quantizing on effective weights that fuses a weight of a convolution layer with a weight and running variance from a batch normalization layer. A fractional length for weights of the computational layers is determined from current values of weights using the determined optimal fractional length for the weights of the computational layers. A fixed-point activation between adjacent computational layers is related using PACT quantization of the clipping-level and an activation fractional length from a node in a following computational layer.Type: ApplicationFiled: May 21, 2026Publication date: September 24, 2026Inventors: Sumant Milind Hanumante, Qing Jin, Sergei Korolev, Denys Makoviichuk, Jian Ren, Dhritiman Sagar, Patrick Timothy McSweeney Simons, Sergey Tulyakov, Yang Wen, Richard Zhuang
-
Publication number: 20260279039Abstract: Described is a system for improving machine learning models. In some cases, the system improves such models by identifying an autoencoder for a latent diffusion machine learning model, the latent diffusion machine learning model is trained to receive text as input and output an image based on the received text. The system identifies a number of channels in a decoder of the autoencoder, the decoder being configured to receive latent features as input and output images. The system further identifies a performance characteristic of the decoder and changes the node topology of the decoder based on the performance characteristic to generate an updated decoder. The system retrains the latent diffusion machine learning model using the updated decoder by inputting latent features to the updated decoder, receiving an outputted image from the updated decoder, and updating one or more weights of the decoder based on an assessment of the outputted image.Type: ApplicationFiled: April 9, 2026Publication date: September 17, 2026Inventors: Pavlo Chemerys, Colin Eles, Ju Hu, Qing Jin, Yanyu Li, Ergeta Muca, Jian Ren, Dhritiman Sagar, Aleksei Stoliar, Sergey Tulyakov, Huan Wang
-
Publication number: 20260278905Abstract: Example systems take as input a text prompt describing events and generate a multi-event video, audio, or data sequence based on the text prompt. Example operations include accessing a sequence of events, the sequence of events comprising time ranges for the events of the sequence of events and text descriptions for the events, accessing a global caption comprising a text description for the sequence of events, and inputting the sequence of events and the global caption simultaneously into a trained neural network to generate a video, audio, or data sequence, where frames of the video, sound of the audio, or data of the data sequence during a time range of the time ranges are in accordance with a corresponding text description of the text descriptions. Additionally, systems are disclosed that train a neural network to generate the multi-event video, audio, or a data sequence from the text prompt.Type: ApplicationFiled: March 17, 2025Publication date: September 17, 2026Inventors: Varnith Uttam Chordia, Yuwei Fang, Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Sergey Tulyakov, Ziyi Wu
-
Publication number: 20260267493Abstract: A system of machine learning schemes can be configured to efficiently perform image processing tasks on a user device, such as a mobile phone. The system can selectively detect and transform individual regions within each frame of a live streaming video. The system can selectively partition and toggle image effects within the live streaming video.Type: ApplicationFiled: May 4, 2026Publication date: September 10, 2026Inventors: Theresa Barton, Yanping Chen, Jaewook Chung, Christopher Yale Crutchfield, Aymeric Damien, Sergei Kotcur, Igor Kudrlashov, Sergey Tulyakov, Andrew Wan, Emre Yamangil
-
Patent number: 12726637Abstract: Examples relate to systems and methods for generating videos with precise camera control using a camera-conditioned video diffusion transformer (DiT) model. The model performs a denoising process through a series of pretrained video DiT blocks, where camera trajectory information is processed through a camera conditioning branch to generate camera activations. During an initial portion of denoising passes, the process is conditioned on camera activations for an initial subset of video DiT blocks, while later denoising passes proceed without camera conditioning. This approach leverages the insight that camera motion is established early in the denoising process, enabling precise camera control while maintaining high visual quality.Type: GrantFiled: May 6, 2025Date of Patent: September 1, 2026Assignee: Snap Inc.Inventors: Sherwin Bahmani, Willi Menapace, Guocheng Qian, Aliaksandr Siarohin, Ivan Skorokhodov, Sergey Tulyakov
-
Patent number: 12705875Abstract: Described is a system for improving machine learning models by accessing a first latent diffusion machine learning model, accessing a second latent diffusion machine learning model that was derived from the first latent diffusion machine learning model, the second latent diffusion machine learning model trained to perform a second number of denoising steps, generating noise data, processing the noise data via the first latent diffusion machine learning model to generate one or more first latent features, processing the noise data via the second latent diffusion machine learning model to generate one or more second latent features, and inputting the one or more first latent features and the one or more second latent features into a loss function. The system then modifies a parameter of the second latent diffusion machine learning model based on the output of the loss function.Type: GrantFiled: March 5, 2024Date of Patent: August 11, 2026Assignee: Snap Inc.Inventors: Pavlo Chemerys, Colin Eles, Ju Hu, Qing Jin, Yanyu Li, Ergeta Muca, Jian Ren, Dhritiman Sagar, Aleksei Stoliar, Sergey Tulyakov, Huan Wang
-
Patent number: 12700160Abstract: A text-to-video framework including a far-reaching interleaved transformer (FIT) block configured to learn a compressed representation of video input using a set of learnable latent tokens. The FIT block includes a diffusion framework and joint spatiotemporal modeling. The FIT block performs patchification of the video input to produce a sequence of patch tokens that are divided into groups. The FIT block instantiates the set of latent tokens and applies a sequence of computational blocks, and projects the patch tokens to generate video frames.Type: GrantFiled: January 31, 2024Date of Patent: August 4, 2026Assignee: Snap Inc.Inventors: Tsai-Shien Chen, Yuwei Fang, Anil Kag, Willi Menapace, Jian Ren, Aliaksandr Siarohin, Ivan Skorokhodov, Sergey Tulyakov
-
Patent number: 12701209Abstract: A method for generating photorealistic 4D scenes from text inputs is disclosed. The method utilizes a text-to-video diffusion model to generate a reference video and a freeze-time video. A canonical 3D representation is reconstructed using deformable 3D Gaussian Splats (D-3DGS) based on the freeze-time video. Temporal deformations are learned to capture dynamic interactions in the reference video. The method employs a novel Score Distillation Sampling strategy combining multi-view and temporal aspects to enhance consistency and robustness. The resulting 4D scenes feature multiple objects interacting with detailed background environments, viewable from different angles and times. The method enables flexible camera control and integration with augmented and virtual reality applications. Some examples include features such as image-to-4D generation.Type: GrantFiled: September 23, 2024Date of Patent: August 4, 2026Assignee: Snap Inc.Inventors: Hsin-Ying Lee, Willi Menapace, Aliaksandr Siarohin, Sergey Tulyakov, Chaoyang Wang, Heng Yu, Peiye Zhuang
-
Publication number: 20260205619Abstract: A method, computer readable medium, and system are disclosed for action video generation. The method includes the steps of generating, by a recurrent neural network, a sequence of motion vectors from a first set of random variables and receiving, by a generator neural network, the sequence of motion vectors and a content vector sample. The sequence of motion vectors and the content vector sample are sampled by the generator neural network to produce a video clip.Type: ApplicationFiled: December 22, 2025Publication date: July 16, 2026Inventors: Ming-Yu Liu, Xiaodong Yang, Jan Kautz, Sergey Tulyakov
-
Patent number: 12670362Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and method for video synthesis. The program and method provide for accessing a primary generative adversarial network (GAN) comprising a pre-trained image generator, a motion generator comprising a plurality of neural networks, and a video discriminator; generating an updated GAN based on the primary GAN, by performing operations comprising identifying input data of the updated GAN, the input data comprising an initial latent code and a motion domain dataset, training the motion generator based on the input data, and adjusting weights of the plurality of neural networks of the primary GAN based on an output of the video discriminator; and generating a synthesized video based on the primary GAN and the input data.Type: GrantFiled: September 30, 2021Date of Patent: June 30, 2026Assignee: Snap Inc,.Inventors: Menglei Chai, Kyle Olszewski, Jian Ren, Yu Tian, Sergey Tulyakov
-
Patent number: 12663914Abstract: A system of machine learning schemes can be configured to efficiently perform image processing tasks on a user device, such as a mobile phone. The system can selectively detect and transform individual regions within each frame of a live streaming video. The system can selectively partition and toggle image effects within the live streaming video.Type: GrantFiled: August 9, 2023Date of Patent: June 23, 2026Inventors: Theresa Barton, Yanping Chen, Jaewook Chung, Christopher Yale Crutchfield, Aymeric Damien, Sergei Kotcur, Igor Kudriashov, Sergey Tulyakov, Andrew Wan, Emre Yamangil
-
Publication number: 20260162367Abstract: A system and method are described for generating 3D garments from two-dimensional (2D) scribble images drawn by users. The system includes a conditional 2D generator, a conditional 3D generator, and two intermediate media including dimension-coupling color-density pairs and flat point clouds that bridge the gap between dimensions. Given a scribble image, the 2D generator synthesizes dimension-coupling color-density pairs including the RGB projection and density map from the front and rear views of the scribble image. A density-aware sampling algorithm converts the 2D dimension-coupling color-density pairs into a 3D flat point cloud representation, where the depth information is ignored. The 3D generator predicts the depth information from the flat point cloud. Dynamic variations per garment due to deformations resulting from a wearer's pose as well as irregular wrinkles and folds may be bypassed by taking advantage of 2D generative models to bridge the dimension gap in a non-parametric way.Type: ApplicationFiled: April 15, 2025Publication date: June 11, 2026Inventors: Panagiotis Achlioptas, Menglei Chai, Hsin-Ying Lee, Kyle Olszewski, Jian Ren, Sergey Tulyakov
-
Patent number: 12645936Abstract: Techniques for training a neural network having a plurality of computational layers with associated weights and activations for computational layers in fixed-point formats include determining an optimal fractional length for weights and activations for the computational layers; training a learned clipping-level with fixed-point quantization using a PACT process for the computational layers; and quantizing on effective weights that fuses a weight of a convolution layer with a weight and running variance from a batch normalization layer. A fractional length for weights of the computational layers is determined from current values of weights using the determined optimal fractional length for the weights of the computational layers. A fixed-point activation between adjacent computational layers is related using PACT quantization of the clipping-level and an activation fractional length from a node in a following computational layer.Type: GrantFiled: December 31, 2021Date of Patent: June 2, 2026Assignee: Snap Inc.Inventors: Sumant Milind Hanumante, Qing Jin, Sergei Korolev, Denys Makoviichuk, Jian Ren, Dhritiman Sagar, Patrick Timothy McSweeney Simons, Sergey Tulyakov, Yang Wen, Richard Zhuang