Patents by Inventor Tadas BALTRUSAITIS
Tadas BALTRUSAITIS has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Publication number: 20240320498Abstract: There is a region of interest of a synthetic image depicting an object from a class of objects. A trained neural image generator, having been trained to map embeddings from a latent space to photorealistic images of objects in the class, is accessed. A first embedding is computed from the latent space, the first embedding corresponding to an image which is similar to the region of interest while maintaining photorealistic appearance. A second embedding is computed from the latent space, the second embedding corresponding to an image which matches the synthetic image. Blending of the first embedding and the second embedding is done to form a blended embedding. At least one output image is generated from the blended embedding, the output image being more photorealistic than the synthetic image.Type: ApplicationFiled: May 23, 2024Publication date: September 26, 2024Inventors: Stephan Joachim GARBIN, Marek Adam KOWALSKI, Matthew Alastair JOHNSON, Tadas BALTRUSAITIS, Martin DE LA GORCE, Virginia ESTELLERS CASAS, Sebastian Karol DZIADZIO, Jamie Daniel Joseph SHOTTON
-
Patent number: 12062140Abstract: Computing an image depicting a face having an expression with wrinkles is described. A 3D polygon mesh model of a face has a non-neutral expression. A tension map is computed from the 3D polygon mesh model. A neutral texture, a compressed wrinkle texture and an expanded wrinkle texture are computed or obtained from a library. The neutral texture comprises a map of the first face with a neutral expression. The compressed wrinkle texture is a map of the first face formed by aggregating maps of the first face with different expressions using the tension map, and the expanded wrinkle texture comprises a map of the first face formed by aggregating maps of the first face with different expressions using the tension map. A graphics engine may be used to apply the wrinkle textures to the 3D model according to the tension map; and render the image from the 3D model.Type: GrantFiled: September 1, 2022Date of Patent: August 13, 2024Assignee: Microsoft Technology Licensing, LLC.Inventors: Tadas Baltrusaitis, Charles Thomas Hewitt, Erroll William Wood, Chirag Anantha Raman
-
Publication number: 20240265659Abstract: A method of updating a pose of a plurality of joints of a kinematic tree of an articulated object is described. The method comprises receiving, for each of the joints in the kinematic tree, an initial pose. A single first embedding vector is computed by encoding the initial poses in an embedding space. For each of some but not all of the joints in the kinematic tree, a target pose is received. A single second embedding vector representing the target poses is computed in the embedding space. The first embedding vector is modified using the second embedding vector to form a third embedding vector. Decoding the third embedding vector produces the updated pose of the articulated object.Type: ApplicationFiled: January 3, 2024Publication date: August 8, 2024Inventors: Mohammand Sadegh ALI AKBARIAN, Tadas BALTRUSAITIS
-
Publication number: 20240257382Abstract: To predict the pose of a person using images from egocentric cameras, at least one image from at least one egocentric camera is received, wherein the image depicts only a portion of the person. The method described herein uses a trained neural network to directly predict a distribution over rotation of a joint of the person or three-dimensional, 3D, location of a joint of the person.Type: ApplicationFiled: February 1, 2023Publication date: August 1, 2024Inventors: Charles Thomas HEWITT, Hanz CUEVAS VELASQUEZ, Mohammad Sadegh ALI AKBARIAN, Tadas BALTRUSAITIS
-
Patent number: 12045925Abstract: In various examples there is an apparatus for computing an image depicting a face of a wearer of a head mounted display (HMD), as if the wearer was not wearing the HMD. An input image depicts a partial view of the wearer's face captured from at least one face facing capture device in the HMD. A machine learning apparatus is available which has been trained to compute expression parameters from the input image. A 3D face model that has expressions parameters is accessible as well as a photorealiser being a machine learning model trained to map images rendered from the 3D face model to photorealistic images. The apparatus computes expression parameter values from the image using the machine learning apparatus. The apparatus drives the 3D face model with the expression parameter values to produce a 3D model of the face of the wearer and then renders the 3D model from a specified viewpoint to compute a rendered image. The rendered image is upgraded to a photorealistic image using the photorealiser.Type: GrantFiled: June 11, 2020Date of Patent: July 23, 2024Assignee: Microsoft Technology Licensing, LLCInventors: Matthew Alastair Johnson, Marta Malgorzata Wilczkowiak, Daniel Stephen Wilde, Paul Malcolm McIlroy, Tadas Baltrusaitis, Virginia Estellers Casas, Marek Adam Kowalski, Christopher Maurice Mei, Stephan Joachim Garbin
-
Patent number: 12033084Abstract: There is a region of interest of a synthetic image depicting an object from a class of objects. A trained neural image generator, having been trained to map embeddings from a latent space to photorealistic images of objects in the class, is accessed. A first embedding is computed from the latent space, the first embedding corresponding to an image which is similar to the region of interest while maintaining photorealistic appearance. A second embedding is computed from the latent space, the second embedding corresponding to an image which matches the synthetic image. Blending of the first embedding and the second embedding is done to form a blended embedding. At least one output image is generated from the blended embedding, the output image being more photorealistic than the synthetic image.Type: GrantFiled: May 23, 2022Date of Patent: July 9, 2024Assignee: Microsoft Technology Licensing, LLCInventors: Stephan Joachim Garbin, Marek Adam Kowalski, Matthew Alastair Johnson, Tadas Baltrusaitis, Martin De La Gorce, Virginia Estellers Casas, Sebastian Karol Dziadzio, Jamie Daniel Joseph Shotton
-
Publication number: 20240127522Abstract: Examples are disclosed that relate to generating expressive avatars using multi-modal three-dimensional face modeling and tracking. One example includes a computer system comprising a processor coupled to a storage system that stores instructions. Upon execution by the processor, the instructions cause the processor to receive initialization data describing an initial state of a facial model. The instructions further cause the processor to receive a plurality of multi-modal data signals. The instructions further cause the processor to perform a fitting process using the initialization data and the plurality of multi-modal data signals. The instructions further cause the processor to determine a set of parameters based on the fitting process, wherein the determined set of parameters describes an updated state of the facial model.Type: ApplicationFiled: December 6, 2022Publication date: April 18, 2024Applicant: Microsoft Technology Licensing, LLCInventors: Harpreet Singh SAWHNEY, Benjamin Eliot LUNDELL, Anshul Bhavesh SHAH, Calin CRISTIAN, Charles Thomas HEWITT, Tadas BALTRUSAITIS, Mladen RADOJEVIC, Kosta GRUJCIC, Ivan STOJILJKOVIC, Paul Malcolm MCILROY, John Ishola OLAFENWA, Jouya JADIDIAN, Kenneth Mitchell JAKUBZAK
-
Publication number: 20240078755Abstract: Computing an image depicting a face having an expression with wrinkles is described. A 3D polygon mesh model of a face has a non-neutral expression. A tension map is computed from the 3D polygon mesh model. A neutral texture, a compressed wrinkle texture and an expanded wrinkle texture are computed or obtained from a library. The neutral texture comprises a map of the first face with a neutral expression. The compressed wrinkle texture is a map of the first face formed by aggregating maps of the first face with different expressions using the tension map, and the expanded wrinkle texture comprises a map of the first face formed by aggregating maps of the first face with different expressions using the tension map. A graphics engine may be used to apply the wrinkle textures to the 3D model according to the tension map; and render the image from the 3D model.Type: ApplicationFiled: September 1, 2022Publication date: March 7, 2024Inventors: Tadas BALTRUSAITIS, Charles Thomas HEWITT, Erroll William WOOD, Chirag Anantha RAMAN
-
Publication number: 20230419581Abstract: Systems and methods are provided that are directed to generating video sequences including physio-realistic avatars. In examples, an albedo for an avatar is received, a sub-surface skin color associated with the albedo is modified based on physiological data associated with physiologic characteristic, and an avatar based on the albedo and the modified sub-surface skin color is rendered. The rendered avatar may then be synthesized in a frame of video. In some examples, a video including the synthesized avatar may be used to train a machine learning model to detect a physiological characteristic. The machine learning model may receive a plurality of video segments, where one or more of the video segments includes a synthetic physio-realistic avatar generated with the physiological characteristic. The machine learning model may be trained using the plurality of video segments. The trained model may be provided to a requesting entity.Type: ApplicationFiled: September 11, 2023Publication date: December 28, 2023Applicant: Microsoft Technology Licensing, LLCInventors: Daniel J. MCDUFF, Javier HERNANDEZ RIVERA, Tadas BALTRUSAITIS, Erroll William WOOD
-
Publication number: 20230360309Abstract: In various examples there is a method of image processing comprising: storing a real image of an object in memory, the object being a specified type of object. The method involves computing, using a first encoder, a factorized embedding of the real image. The method receives a value of at least one parameter of a synthetic image rendering apparatus for rendering synthetic images of objects of the specified type. The parameter controls an attribute of synthetic images of objects rendered by the rendering apparatus. The method computes an embedding factor of the received value using a second encoder. The factorized embedding is modified with the computed embedding factor. The method computes, using a decoder with the modified embedding as input, an output image of an object which is substantially the same as the real image except for the attribute controlled by the parameter.Type: ApplicationFiled: July 18, 2023Publication date: November 9, 2023Inventors: Marek Adam KOWALSKI, Stephan Joachim GARBIN, Matthew Alastair JOHNSON, Tadas BALTRUSAITIS, Martin DE LA GORCE, Virginia ESTELLERS CASAS, Sebastian Karol DZIADZIO
-
Patent number: 11790586Abstract: Systems and methods are provided that are directed to generating video sequences including physio-realistic avatars. In examples, an albedo for an avatar is received, a sub-surface skin color associated with the albedo is modified based on physiological data associated with physiologic characteristic, and an avatar based on the albedo and the modified sub-surface skin color is rendered. The rendered avatar may then be synthesized in a frame of video. In some examples, a video including the synthesized avatar may be used to train a machine learning model to detect a physiological characteristic. The machine learning model may receive a plurality of video segments, where one or more of the video segments includes a synthetic physio-realistic avatar generated with the physiological characteristic. The machine learning model may be trained using the plurality of video segments. The trained model may be provided to a requesting entity.Type: GrantFiled: June 19, 2020Date of Patent: October 17, 2023Assignee: Microsoft Technology Licensing, LLCInventors: Daniel J. McDuff, Javier Hernandez Rivera, Tadas Baltrusaitis, Erroll William Wood
-
Publication number: 20230316552Abstract: The techniques described herein disclose a system that is configured to detect and track the three-dimensional pose of an object (e.g., a head-mounted display device) in a color image using an accessible three-dimensional model of the object. The system uses the three-dimensional pose of the object to repair pixel depth values associated with a region (e.g., a surface) of the object that is composed of material that absorbs light emitted by a time-of-flight depth sensor to determine depth. Consequently, a color-depth image (e.g., a Red-Green-Blue-Depth image or RGB-D image) can be produced that does not include dark holes on and around the region of the object that is composed of material that absorbs light emitted by the time-of-flight depth sensor.Type: ApplicationFiled: April 4, 2022Publication date: October 5, 2023Inventors: JingJing SHEN, Erroll William WOOD, Toby SHARP, Ivan RAZUMENIC, Tadas BALTRUSAITIS, Julien Pascal Christophe VALENTIN, Predrag JOVANOVIC
-
Publication number: 20230281945Abstract: Keypoints are predicted in an image. A neural network is executed that is configured to predict each of the keypoints as a 2D random variable, normally distributed with a 2D position and 2×2 covariance matrix. The neural network is trained to maximize a log-likelihood that samples from each of the predicted keypoints equal a ground truth. The trained neural network is used to predict keypoints of an image without generating a heatmap.Type: ApplicationFiled: June 28, 2022Publication date: September 7, 2023Inventors: Thomas Joseph CASHMAN, Erroll William WOOD, Martin DE LA GORCE, Tadas BALTRUSAITIS, Daniel Stephen WILDE, Jingjing SHEN, Matthew Alastair JOHNSON, Julien Pascal Christophe VALENTIN
-
Publication number: 20230281863Abstract: Keypoints are predicted in an image. Predictions are generated for each of the keypoints of an image as a 2D random variable, normally distributed with location (x, y) and standard deviation sigma. A neural network is trained to maximize a log-likelihood that samples from each of the predicted keypoints equal a ground truth. The trained neural network is used to predict keypoints of an image without generating a heatmap.Type: ApplicationFiled: June 28, 2022Publication date: September 7, 2023Inventors: Julien Pascal Christophe VALENTIN, Erroll William WOOD, Thomas Joseph CASHMAN, Martin de LA GORCE, Tadas BALTRUSAITIS, Daniel Stephen WILDE, Jingjing SHEN, Matthew Alastair JOHNSON, Charles Thomas HEWITT, Nikola MILOSAVLJEVIC, Stephan Joachim GARBIN, Toby SHARP, Ivan STOJILJKOVIC
-
Patent number: 11748932Abstract: In various examples there is a method of image processing comprising: storing a real image of an object in memory, the object being a specified type of object. The method involves computing, using a first encoder, a factorized embedding of the real image. The method receives a value of at least one parameter of a synthetic image rendering apparatus for rendering synthetic images of objects of the specified type. The parameter controls an attribute of synthetic images of objects rendered by the rendering apparatus. The method computes an embedding factor of the received value using a second encoder. The factorized embedding is modified with the computed embedding factor. The method computes, using a decoder with the modified embedding as input, an output image of an object which is substantially the same as the real image except for the attribute controlled by the parameter.Type: GrantFiled: June 29, 2020Date of Patent: September 5, 2023Assignee: Microsoft Technology Licensing, LLCInventors: Marek Adam Kowalski, Stephan Joachim Garbin, Matthew Alastair Johnson, Tadas Baltrusaitis, Martin De la Gorce, Virginia Estellers Casas, Sebastian Karol Dziadzio
-
Publication number: 20220284655Abstract: There is a region of interest of a synthetic image depicting an object from a class of objects. A trained neural image generator, having been trained to map embeddings from a latent space to photorealistic images of objects in the class, is accessed. A first embedding is computed from the latent space, the first embedding corresponding to an image which is similar to the region of interest while maintaining photorealistic appearance. A second embedding is computed from the latent space, the second embedding corresponding to an image which matches the synthetic image. Blending of the first embedding and the second embedding is done to form a blended embedding. At least one output image is generated from the blended embedding, the output image being more photorealistic than the synthetic image.Type: ApplicationFiled: May 23, 2022Publication date: September 8, 2022Inventors: Stephan Joachim GARBIN, Marek Adam KOWALSKI, Matthew Alastair JOHNSON, Tadas BALTRUSAITIS, Martin DE LA GORCE, Virginia ESTELLERS CASAS, Sebastian Karol DZIADZIO, Jamie Daniel Joseph SHOTTON
-
Patent number: 11354846Abstract: There is a region of interest of a synthetic image depicting an object from a class of objects. A trained neural image generator, having been trained to map embeddings from a latent space to photorealistic images of objects in the class, is accessed. A first embedding is computed from the latent space, the first embedding corresponding to an image which is similar to the region of interest while maintaining photorealistic appearance. A second embedding is computed from the latent space, the second embedding corresponding to an image which matches the synthetic image. Blending of the first embedding and the second embedding is done to form a blended embedding. At least one output image is generated from the blended embedding, the output image being more photorealistic than the synthetic image.Type: GrantFiled: June 29, 2020Date of Patent: June 7, 2022Assignee: Microsoft Technology Licensing, LLCInventors: Stephan Joachim Garbin, Marek Adam Kowalski, Matthew Alastair Johnson, Tadas Baltrusaitis, Martin De La Gorce, Virginia Estellers Casas, Sebastian Karol Dziadzio, Jamie Daniel Joseph Shotton
-
Publication number: 20210398337Abstract: Systems and methods are provided that are directed to generating video sequences including physio-realistic avatars. In examples, an albedo for an avatar is received, a sub-surface skin color associated with the albedo is modified based on physiological data associated with physiologic characteristic, and an avatar based on the albedo and the modified sub-surface skin color is rendered. The rendered avatar may then be synthesized in a frame of video. In some examples, a video including the synthesized avatar may be used to train a machine learning model to detect a physiological characteristic. The machine learning model may receive a plurality of video segments, where one or more of the video segments includes a synthetic physio-realistic avatar generated with the physiological characteristic. The machine learning model may be trained using the plurality of video segments. The trained model may be provided to a requesting entity.Type: ApplicationFiled: June 19, 2020Publication date: December 23, 2021Applicant: Microsoft Technology Licensing, LLCInventors: Daniel J. MCDUFF, Javier HERNANDEZ RIVERA, Tadas BALTRUSAITIS, Erroll William WOOD
-
Publication number: 20210390767Abstract: In various examples there is an apparatus for computing an image depicting a face of a wearer of a head mounted display (HMD), as if the wearer was not wearing the HMD. An input image depicts a partial view of the wearer's face captured from at least one face facing capture device in the HMD. A machine learning apparatus is available which has been trained to compute expression parameters from the input image. A 3D face model that has expressions parameters is accessible as well as a photorealiser being a machine learning model trained to map images rendered from the 3D face model to photorealistic images. The apparatus computes expression parameter values from the image using the machine learning apparatus. The apparatus drives the 3D face model with the expression parameter values to produce a 3D model of the face of the wearer and then renders the 3D model from a specified viewpoint to compute a rendered image. The rendered image is upgraded to a photorealistic image using the photorealiser.Type: ApplicationFiled: June 11, 2020Publication date: December 16, 2021Inventors: Matthew Alastair JOHNSON, Marta Malgorzata WILCZKOWIAK, Daniel Stephen WILDE, Paul Malcolm MCILROY, Tadas BALTRUSAITIS, Virginia ESTELLERS CASAS, Marek Adam KOWALSKI, Christopher Maurice MEI, Stephan Joachim GARBIN
-
Publication number: 20210343063Abstract: There is a region of interest of a synthetic image depicting an object from a class of objects. A trained neural image generator, having been trained to map embeddings from a latent space to photorealistic images of objects in the class, is accessed. A first embedding is computed from the latent space, the first embedding corresponding to an image which is similar to the region of interest while maintaining photorealistic appearance. A second embedding is computed from the latent space, the second embedding corresponding to an image which matches the synthetic image. Blending of the first embedding and the second embedding is done to form a blended embedding. At least one output image is generated from the blended embedding, the output image being more photorealistic than the synthetic image.Type: ApplicationFiled: June 29, 2020Publication date: November 4, 2021Inventors: Stephan Joachim GARBIN, Marek Adam KOWALSKI, Matthew Alastair JOHNSON, Tadas BALTRUSAITIS, Martin DE LA GORCE, Virginia ESTELLERS CASAS, Sebastian Karol DZIADZIO, Jamie Daniel Joseph SHOTTON