Patents by Inventor Adrien GAIDON

Adrien GAIDON has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20230377180
    Abstract: In accordance with one embodiment of the present disclosure, a method includes receiving a set of images, each image depicting a view of a scene, generating sparse depth data from each image of the set of images, training a monocular depth estimation model with the sparse depth data, generating, with the trained monocular depth estimation model, depth data and uncertainty data for each image, training a NeRF model with the set of images, wherein the training is constrained by the depth data and uncertainty data, and rendering, with the trained NeRF model, a new image having a new view of the scene.
    Type: Application
    Filed: May 18, 2022
    Publication date: November 23, 2023
    Applicant: Toyota Research Institute Inc.
    Inventors: Rares Ambrus, Sergey Zakharov, Vitor C. Guizilini, Adrien Gaidon
  • Publication number: 20230351768
    Abstract: A method for self-calibrating alignment between image data and point cloud data utilizing a machine learning model includes receiving, with an electronic control unit, image data from a vision sensor and point cloud data from a depth sensor, implementing, with the electronic control unit, a machine learning model trained to: align the point cloud data and the image data based on a current calibration, detect a difference in alignment of the point cloud data and the image data, adjust the current calibration based on the difference in alignment, and output a calibrated embedding feature map based on adjustments to the current calibration.
    Type: Application
    Filed: April 29, 2022
    Publication date: November 2, 2023
    Applicants: Toyota Research Institute, Inc., Toyota Jidosha Kabushiki Kaisha
    Inventors: Jie Li, Vitor Guizilini, Adrien Gaidon
  • Publication number: 20230237856
    Abstract: A method includes receiving first driving data associated with a first vehicle, receiving second driving data associated with one or more vehicles around the first vehicle, creating training data by labeling the first driving data as positive data and treating the second driving data as unlabeled, and using the training data to train a classifier to predict whether driving data input to the classifier is positive or unlabeled.
    Type: Application
    Filed: January 27, 2022
    Publication date: July 27, 2023
    Applicant: Toyota Research Institute
    Inventors: Blake Wulfe, Adrien Gaidon
  • Patent number: 11704822
    Abstract: Systems and methods for self-supervised depth estimation using image frames captured from a camera mounted on a vehicle comprise: receiving a first image from the camera mounted at a first location on the vehicle; receiving a second image from the camera mounted at a second location on the vehicle; predicting a depth map for the first image; warping the first image to a perspective of the camera mounted at the second location on the vehicle to arrive at a warped first image; projecting the warped first image onto the second image; determining a loss based on the projection; and updating the predicted depth values for the first image.
    Type: Grant
    Filed: January 13, 2022
    Date of Patent: July 18, 2023
    Assignee: TOYOTA RESEARCH INSTITUTE, INC.
    Inventors: Vitor Guizilini, Igor Vasiljevic, Rares A. Ambrus, Adrien Gaidon
  • Publication number: 20230154145
    Abstract: In accordance with one embodiment of the present disclosure, a method includes receiving an input image having an object and a background, intrinsically decomposing the object and the background into an input image data having a set of features, augmenting the input image data with a 2.5D differentiable renderer for each feature of the set of features to create a set of augmented images, and compiling the input image and the set of augmented images into a training data set for training a downstream task network.
    Type: Application
    Filed: January 19, 2022
    Publication date: May 18, 2023
    Applicant: Toyota Research Institute, Inc.
    Inventors: Sergey Zakharov, Rares Ambrus, Vitor Guizilini, Adrien Gaidon
  • Patent number: 11615544
    Abstract: Systems and methods for map construction using a video sequence captured on a camera of a vehicle in an environment, comprising: receiving a video sequence from the camera, the video sequence including a plurality of image frames capturing a scene of the environment of the vehicle; using a neural camera model to predict a depth map and a ray surface for the plurality of image frames in the received video sequence; and constructing a map of the scene of the environment based on image data captured in the plurality of frames and depth information in the predicted depth maps.
    Type: Grant
    Filed: September 15, 2020
    Date of Patent: March 28, 2023
    Assignee: TOYOTA RESEARCH INSTITUTE, INC.
    Inventors: Vitor Guizilini, Igor Vasiljevic, Rares A. Ambrus, Sudeep Pillai, Adrien Gaidon
  • Publication number: 20230037731
    Abstract: Systems and methods for self-supervised depth estimation using image frames captured from cameras, may include: receiving a first image captured by a first camera while the camera is mounted at a first location, the first image comprising pixels representing a first scene of an environment of a vehicle; receiving a reference image captured by a second camera while the second camera is mounted at a second location, the reference image comprising pixels representing a second scene of the environment; warping the first image to a perspective of the second camera at the second location on the vehicle to arrive at a warped first image; projecting the warped first image onto the reference image; determining a loss based on the projection; and updating predicted depth values for the first image.
    Type: Application
    Filed: October 13, 2022
    Publication date: February 9, 2023
    Inventors: VITOR GUIZILINI, IGOR VASILJEVIC, RARES A. AMBRUS, ADRIEN GAIDON
  • Patent number: 11508080
    Abstract: Systems and methods for self-supervised learning for visual odometry using camera images captured on a camera, may include: using a key point network to learn a keypoint matrix for a target image and a context image captured by the camera; using the learned descriptors to estimate correspondences between the target image and the context image; based on the keypoint correspondences, lifting a set of 2D keypoints to 3D, using a learned neural camera model; estimating a transformation between the target image and the context image using 3D-2D keypoint correspondences; and projecting the 3D keypoints into the context image using the learned neural camera model.
    Type: Grant
    Filed: September 15, 2020
    Date of Patent: November 22, 2022
    Assignee: TOYOTA RESEARCH INSTITUTE, INC.
    Inventors: Vitor Guizilini, Igor Vasiljevic, Rares A. Ambrus, Sudeep Pillai, Adrien Gaidon
  • Patent number: 11494927
    Abstract: Systems and methods for self-supervised depth estimation using image frames captured from a vehicle-mounted camera, may include: receiving a first image captured by the camera while the camera is mounted at a first location on the vehicle, the source image comprising pixels representing a scene of the environment of the vehicle; receiving a reference image captured by the camera while the camera is mounted at a second location on the vehicle, the reference image comprising pixels representing a scene of the environment; predicting a depth map for the first image comprising predicted depth values for pixels of the first image; warping the first image to a perspective of the camera at the second location on the vehicle to arrive at a warped first image; projecting the warped first image onto the source image; determining a loss based on the projection; and updating predicted depth values for the first image.
    Type: Grant
    Filed: September 15, 2020
    Date of Patent: November 8, 2022
    Assignee: TOYOTA RESEARCH INSTITUTE, INC.
    Inventors: Vitor Guizilini, Igor Vasiljevic, Rares A. Ambrus, Adrien Gaidon
  • Publication number: 20220245843
    Abstract: Systems and methods for self-supervised learning for visual odometry using camera images, may include: estimating correspondences between keypoints of a target camera image and keypoints of a context camera image; based on the keypoint correspondences, lifting a set of 2D keypoints to 3D, using a neural camera model; and projecting the 3D keypoints into the context camera image using the neural camera model. Some embodiments may use the neural camera model to achieve the lifting and projecting of keypoints without a known or calibrated camera model.
    Type: Application
    Filed: April 17, 2022
    Publication date: August 4, 2022
    Inventors: VITOR GUIZILINI, Igor Vasiljevic, Rares A. Ambrus, Sudeep Pillai, Adrien Gaidon
  • Patent number: 11361557
    Abstract: A method for performing vehicle taillight recognition is described. The method includes extracting spatial features from a sequence of images of a real-world traffic scene during operation of an ego vehicle. The method includes selectively focusing a convolutional neural network (CNN) of a CNN-long short-term memory (CNN-LSTM) framework on a selected region of the sequence of images according to a spatial attention model for a vehicle taillight recognition task. The method includes selecting, by an LSTM network of the CNN-LSTM framework, frames within the selected region of the sequence of images according to a temporal attention model for the vehicle taillight recognition task. The method includes inferring, according to the selected frames within the selected region of the sequence of images, an intent of an ado vehicle according to a taillight state. The method includes planning a trajectory of the ego vehicle from the intent inferred from the ado vehicle.
    Type: Grant
    Filed: April 19, 2019
    Date of Patent: June 14, 2022
    Assignee: TOYOTA RESEARCH INSTITUTE, INC.
    Inventors: Kuan-Hui Lee, Takaaki Tagawa, Jia-En M. Pan, Adrien Gaidon, Bertrand Douillard
  • Publication number: 20220180101
    Abstract: Systems and methods for multi-view cooperative contrastive self-supervised learning, may include receiving a plurality of video sequences, the video sequences comprising a plurality of image frames; applying selected images of a first and second video sequence of the plurality of video sequences to a plurality of different encoders to derive a plurality of embeddings for different views of the selected images of the first and second video sequences; determining distances of the derived plurality of embeddings for the selected images of the first and second video sequences; detecting inconsistencies in the determined distances; and predicting semantics of a future image based on the determined distances.
    Type: Application
    Filed: December 3, 2020
    Publication date: June 9, 2022
    Inventors: Nishant Rai, Ehsan Adeli Mosabbeb, Kuan-Hui Lee, Adrien Gaidon, Juan Carlos Niebles
  • Publication number: 20220138975
    Abstract: Systems and methods for self-supervised depth estimation using image frames captured from a camera mounted on a vehicle comprise: receiving a first image from the camera mounted at a first location on the vehicle; receiving a second image from the camera mounted at a second location on the vehicle; predicting a depth map for the first image; warping the first image to a perspective of the camera mounted at the second location on the vehicle to arrive at a warped first image; projecting the warped first image onto the second image; determining a loss based on the projection; and updating the predicted depth values for the first image.
    Type: Application
    Filed: January 13, 2022
    Publication date: May 5, 2022
    Inventors: Vitor Guizilini, Igor Vasiljevic, Rares A. Ambrus, Adrien Gaidon
  • Patent number: 11321862
    Abstract: Systems and methods for self-supervised depth estimation using image frames captured from a plurality of cameras mounted on a vehicle, may include: receiving a first image from a camera mounted at a first location on the vehicle, the source image comprising pixels representing a scene of the environment of the vehicle; receiving a reference image from a camera mounted at a second location on the vehicle, the reference image comprising pixels representing a scene of the environment; predicting a depth map for the first image, the depth map comprising predicted depth values for pixels of the first image; warping the first image to a perspective of the camera mounted at the second location on the vehicle to arrive at a warped first image; projecting the warped first image onto the source image; determining a loss based on the projection; and updating the predicted depth values for the first image.
    Type: Grant
    Filed: September 15, 2020
    Date of Patent: May 3, 2022
    Assignee: TOYOTA RESEARCH INSTITUTE, INC.
    Inventors: Vitor Guizilini, Igor Vasiljevic, Rares A. Ambrus, Adrien Gaidon
  • Patent number: 11302028
    Abstract: A method for monocular 3D object perception is described. The method includes sampling multiple, stochastic latent variables from a learned latent feature distribution of an RGB image for a 2D object detected in the RGB image. The method also includes lifting a 3D proposal for each stochastic latent variable sampled for the detected 2D object. The method further includes selecting a 3D proposal for the detected 2D object using a proposal selection algorithm to reduce 3D proposal lifting overlap. The method also includes planning a trajectory of an ego vehicle according to a 3D location and pose of the 2D object according to the selected 3D proposal.
    Type: Grant
    Filed: January 24, 2020
    Date of Patent: April 12, 2022
    Assignee: TOYOTA RESEARCH INSTITUTE, INC.
    Inventors: Yu Yao, Wadim Kehl, Adrien Gaidon
  • Publication number: 20220084232
    Abstract: Systems and methods for map construction using a video sequence captured on a camera of a vehicle in an environment, comprising: receiving a video sequence from the camera, the video sequence including a plurality of image frames capturing a scene of the environment of the vehicle; using a neural camera model to predict a depth map and a ray surface for the plurality of image frames in the received video sequence; and constructing a map of the scene of the environment based on image data captured in the plurality of frames and depth information in the predicted depth maps.
    Type: Application
    Filed: September 15, 2020
    Publication date: March 17, 2022
    Inventors: VITOR GUIZILINI, IGOR VASILJEVIC, RARES A. AMBRUS, SUDEEP PILLAI, ADRIEN GAIDON
  • Publication number: 20220084231
    Abstract: Systems and methods for self-supervised learning for visual odometry using camera images captured on a camera, may include: using a key point network to learn a keypoint matrix for a target image and a context image captured by the camera; using the learned descriptors to estimate correspondences between the target image and the context image; based on the keypoint correspondences, lifting a set of 2D keypoints to 3D, using a learned neural camera model; estimating a transformation between the target image and the context image using 3D-2D keypoint correspondences; and projecting the 3D keypoints into the context image using the learned neural camera model.
    Type: Application
    Filed: September 15, 2020
    Publication date: March 17, 2022
    Inventors: VITOR GUIZILINI, Igor Vasiljevic, Rares A. Ambrus, Sudeep Pillai, Adrien Gaidon
  • Publication number: 20220084230
    Abstract: Systems and methods for self-supervised depth estimation using image frames captured from a vehicle-mounted camera, may include: receiving a first image captured by the camera while the camera is mounted at a first location on the vehicle, the source image comprising pixels representing a scene of the environment of the vehicle; receiving a reference image captured by the camera while the camera is mounted at a second location on the vehicle, the reference image comprising pixels representing a scene of the environment; predicting a depth map for the first image comprising predicted depth values for pixels of the first image; warping the first image to a perspective of the camera at the second location on the vehicle to arrive at a warped first image; projecting the warped first image onto the source image; determining a loss based on the projection; and updating predicted depth values for the first image.
    Type: Application
    Filed: September 15, 2020
    Publication date: March 17, 2022
    Inventors: Vitor Guizilini, Igor Vasiljevic, Rares A. Ambrus, Adrien Gaidon
  • Publication number: 20220084229
    Abstract: Systems and methods for self-supervised depth estimation using image frames captured from a plurality of cameras mounted on a vehicle, may include: receiving a first image from a camera mounted at a first location on the vehicle, the source image comprising pixels representing a scene of the environment of the vehicle; receiving a reference image from a camera mounted at a second location on the vehicle, the reference image comprising pixels representing a scene of the environment; predicting a depth map for the first image, the depth map comprising predicted depth values for pixels of the first image; warping the first image to a perspective of the camera mounted at the second location on the vehicle to arrive at a warped first image; projecting the warped first image onto the source image; determining a loss based on the projection; and updating the predicted depth values for the first image.
    Type: Application
    Filed: September 15, 2020
    Publication date: March 17, 2022
    Inventors: VITOR GUIZILINI, IGOR VASILJEVIC, RARES A. AMBRUS, ADRIEN GAIDON
  • Publication number: 20220026918
    Abstract: A method for controlling an ego agent includes capturing a two-dimensional (2D) image of an environment adjacent to the ego agent. The method also includes generating a semantically segmented image of the environment based on the 2D image. The method further includes generating a depth map of the environment based on the semantically segmented image. The method additionally includes generating a three-dimensional (3D) estimate of the environment based on the depth map. The method also includes controlling an action of the ego agent based on the identified location.
    Type: Application
    Filed: July 23, 2020
    Publication date: January 27, 2022
    Applicant: TOYOTA RESEARCH INSTITUTE, INC.
    Inventors: Vitor GUIZILINI, Jie LI, Rares A. AMBRUS, Sudeep PILLAI, Adrien GAIDON