Patents by Inventor Adrien David GAIDON

Adrien David GAIDON has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20220204030
    Abstract: System, methods, and other embodiments described herein relate to improving controls in a device according to risk. In one embodiment, a method includes, in response to receiving sensor data about a surrounding environment of the device, identifying objects from the sensor data that are present in the surrounding environment. The method includes generating a control sequence for controlling the device according to a risk-sensitivity parameter to navigate toward a destination while considering risk associated with encountering the objects defined by the risk-sensitivity parameter. The method includes controlling the device according to the control sequence.
    Type: Application
    Filed: December 31, 2020
    Publication date: June 30, 2022
    Inventors: Haruki Nishimura, Boris Ivanovic, Adrien David Gaidon, Marco Pavone, Mac Schwager
  • Publication number: 20220207270
    Abstract: A bird's eye view feature map, augmented with semantic information, can be used to detect an object in an environment. A point cloud data set augmented with the semantic information that is associated with identities of classes of objects can be obtained. Features can be extracted from the point cloud data set. Based on the features, an initial bird's eye view feature map can be produced. Because operations performed on the point cloud data set to extract the features or to produce the initial bird's eye view feature map can have an effect of diminishing an ability to distinguish the semantic information in the initial bird's eye view feature map, the initial bird's eye view feature map can be augmented with the semantic information to produce an augmented bird's eye view feature map. Based on the augmented bird's eye view feature map, the object in the environment can be detected.
    Type: Application
    Filed: December 31, 2020
    Publication date: June 30, 2022
    Inventors: Jie Li, Rares A. Ambrus, Vitor Guizilini, Adrien David Gaidon, Jia-En Pan
  • Publication number: 20220180170
    Abstract: System, methods, and other embodiments described herein relate to improving trajectory forecasting in a device. In one embodiment, a method includes, in response to receiving sensor data about a surrounding environment of the device, identifying an object from the sensor data that is present in the surrounding environment. The method includes determining category probabilities for the object, the category probabilities indicating semantic classes for classifying the object and probabilities that the object belongs to the semantic classes. The method includes forecasting trajectories for the object based, at least in part, on the category probabilities and the sensor data. The method includes controlling the device according to the trajectories.
    Type: Application
    Filed: December 4, 2020
    Publication date: June 9, 2022
    Inventors: Boris Ivanovic, Kuan-Hui Lee, Jie Li, Adrien David Gaidon, Pavel Tokmakov
  • Patent number: 11341719
    Abstract: A method is presented. The method includes estimating an ego-motion of an agent based on a current image from a sequence of images and at least one previous image from the sequence of images. Each image in the sequence of images may be a two-dimensional (2D) image. The method also includes estimating a depth of the current image based the at least one previous image. The estimated depth accounts for a depth uncertainty measurement in the current image and the at least one previous image. The method further includes generating a three-dimensional (3D) reconstruction of the current image based on the estimated ego-motion and the estimated depth. The method still further includes controlling an action of the agent based on the three-dimensional reconstruction.
    Type: Grant
    Filed: May 7, 2020
    Date of Patent: May 24, 2022
    Assignee: TOYOTA RESEARCH INSTITUTE, INC.
    Inventors: Vitor Guizilini, Adrien David Gaidon
  • Publication number: 20220156525
    Abstract: System, methods, and other embodiments described herein relate to training a multi-task network using real and virtual data. In one embodiment, a method includes acquiring training data that includes real data and virtual data for training a multi-task network that performs at least depth prediction and semantic segmentation. The method includes generating a first output from the multi-task network using the real data and second output from the multi-task network using the virtual data. The method includes generating a mixed loss by analyzing the first output to produce a real loss and the second output to produce a virtual loss. The method includes updating the multi-task network using the mixed loss.
    Type: Application
    Filed: March 29, 2021
    Publication date: May 19, 2022
    Inventors: Vitor Guizilini, Adrien David Gaidon, Jie Li, Rares A. Ambrus
  • Publication number: 20220156971
    Abstract: Systems and methods described herein relate to training a machine-learning-based monocular depth estimator.
    Type: Application
    Filed: March 31, 2021
    Publication date: May 19, 2022
    Inventors: Vitor Guizilini, Rares A. Ambrus, Adrien David Gaidon, Jie Li
  • Publication number: 20220148204
    Abstract: System, methods, and other embodiments described herein relate to determining depths of a scene from a monocular image. In one embodiment, a method includes generating depth features from sensor data according to whether the sensor data includes sparse depth data. The method includes selectively injecting the depth features into a depth model. The method includes generating a depth map from at least a monocular image using the depth model that is guided by the depth features when injected. The method includes providing the depth map as depth estimates of objects represented in the monocular image.
    Type: Application
    Filed: February 16, 2021
    Publication date: May 12, 2022
    Inventors: Vitor Guizilini, Rares A. Ambrus, Adrien David Gaidon
  • Publication number: 20220148206
    Abstract: A method for monocular depth/pose estimation in a camera agnostic network is described. The method includes projecting lifted 3D points onto an image plane according to a predicted ray vector based on a monocular depth model, a monocular pose model, and a camera center of a camera agnostic network. The method also includes predicting a warped target image from a predicted depth map of the monocular depth model, a ray surface of the predicted ray vector, and a projection of the lifted 3D points according to the camera agnostic network.
    Type: Application
    Filed: January 21, 2022
    Publication date: May 12, 2022
    Applicant: TOYOTA RESEARCH INSTITUTE, INC.
    Inventors: Vitor GUIZILINI, Sudeep PILLAI, Adrien David GAIDON, Rares A. AMBRUS, Igor VASILJEVIC
  • Publication number: 20220148203
    Abstract: System, methods, and other embodiments described herein relate to training a depth model for joint depth completion and prediction. In one arrangement, a method includes generating depth features from sparse depth data according to a sparse auxiliary network (SAN) of a depth model. The method includes generating a first depth map from a monocular image and a second depth map from the monocular image and the depth features using the depth model. The method includes generating a depth loss from the second depth map and the sparse depth data and an image loss from the first depth map and the sparse depth data. The method includes updating the depth model including the SAN using the depth loss and the image loss.
    Type: Application
    Filed: January 21, 2021
    Publication date: May 12, 2022
    Inventors: Vitor Guizilini, Rares A. Ambrus, Adrien David Gaidon
  • Publication number: 20220148202
    Abstract: System, methods, and other embodiments described herein relate to determining depths of a scene from a monocular image. In one embodiment, a method includes generating depth features from depth data using a sparse auxiliary network (SAN) by i) sparsifying the depth data, ii) applying sparse residual blocks of the SAN to the depth data, and iii) densifying the depth features. The method includes generating a depth map from the depth features and a monocular image that corresponds with the depth data according to a depth model that includes the SAN. The method includes providing the depth map as depth estimates of objects represented in the monocular image.
    Type: Application
    Filed: January 7, 2021
    Publication date: May 12, 2022
    Inventors: Vitor Guizilini, Rares A. Ambrus, Adrien David Gaidon
  • Patent number: 11328517
    Abstract: A system and method generate feature space data that may be used for object detection. The system includes one or more processors and a memory. The memory may include one or more modules having instructions that, when executed by the one or more processors, cause the one or more processors to obtain a two-dimension image of a scene, generate an output depth map based on the two-dimension image of the scene, generate a pseudo-LIDAR point cloud based on the output depth map, generate a bird's eye view (BEV) feature space based on the pseudo-LIDAR point cloud, and modify the BEV feature space to generate an improved BEV feature space using feature space neural network that was trained by using a training LIDAR feature space as a ground truth based on a LIDAR point cloud.
    Type: Grant
    Filed: May 20, 2020
    Date of Patent: May 10, 2022
    Assignee: Toyota Research Institute, Inc.
    Inventors: Victor Vaquero Gomez, Rares A. Ambrus, Vitor Guizilini, Adrien David Gaidon
  • Patent number: 11321863
    Abstract: Systems, methods, and other embodiments described herein relate to generating depth estimates of an environment depicted in a monocular image. In one embodiment, a method includes identifying semantic features in the monocular image according to a semantic model. The method includes injecting the semantic features into a depth model using pixel-adaptive convolutions. The method includes generating a depth map from the monocular image using the depth model that is guided by the semantic features. The pixel-adaptive convolutions are integrated into a decoder of the depth model. The method includes providing the depth map as the depth estimates for the monocular image.
    Type: Grant
    Filed: February 28, 2020
    Date of Patent: May 3, 2022
    Assignee: Toyota Research Institute, Inc.
    Inventors: Vitor Guizilini, Rares A. Ambrus, Jie Li, Adrien David Gaidon
  • Patent number: 11321859
    Abstract: A method for scene reconstruction includes generating a depth estimate and a first pose estimate from a current image. The method also includes generating a second pose estimate based on the current image and one or more previous images in a sequence of images. The method further includes generating a warped image by warping each pixel in the current image based on the depth estimate, the first pose estimate, and the second pose estimate. The method still further includes controlling an action of an agent based on the second warped image.
    Type: Grant
    Filed: June 22, 2020
    Date of Patent: May 3, 2022
    Assignee: TOYOTA RESEARCH INSTITUTE, INC.
    Inventors: Vitor Guizilini, Adrien David Gaidon
  • Patent number: 11315269
    Abstract: A system for generating point clouds having surface normal information includes one or more processors and a memory having a depth map generating module, a point cloud generating module, and surface normal generating module. The depth map generating module causes the one or more processors to generate a depth map from one or more images of a scene. The point cloud causes the one or more processors to generate a point cloud from the depth map having a plurality of points corresponding to one or more pixels of the depth map. The surface normal generating module causes the one or more processors to generate surface normal information for at least a portion of the one or more pixels of the depth map and inject the surface normal information into the point cloud such that the plurality of points of the point cloud include three-dimensional location information and surface normal information.
    Type: Grant
    Filed: August 24, 2020
    Date of Patent: April 26, 2022
    Assignee: Toyota Research Institute, Inc.
    Inventors: Victor Vaquero Gomez, Rares A. Ambrus, Vitor Guizilini, Adrien David Gaidon
  • Publication number: 20220108463
    Abstract: A method for using an artificial neural network associated with an agent to estimate depth, includes receiving, at the artificial neural network, an input image captured via a sensor associated with the agent. The method also includes upsampling, at each decoding layer of a plurality of decoding layers of the artificial neural network, decoded features associated with the input image to a resolution associated with a final output of the artificial neural network. The method further includes concatenating, at each decoding layer, the upsampled decoded features with features obtained at a convolution layer associated with a respective decoding layer. The method still further includes estimating, at a recurrent module of the artificial neural network, a depth of the input image based on receiving the concatenated upsampled decoded features from each decoding layer. The method also includes controlling an action of an agent based on the depth estimate.
    Type: Application
    Filed: December 17, 2021
    Publication date: April 7, 2022
    Applicant: TOYOTA RESEARCH INSTITUTE, INC.
    Inventors: Vitor GUIZILINI, Adrien David GAIDON
  • Publication number: 20220080585
    Abstract: Systems and methods described herein relate to controlling a robot. One embodiment receives an initial state of the robot, an initial nominal control trajectory of the robot, and a Kullback-Leibler (KL) divergence bound between a modeled probability distribution for a stochastic disturbance and an unknown actual probability distribution for the stochastic disturbance; solves a bilevel optimization problem subject to the modeled probability distribution and the KL divergence bound using an iterative Linear-Exponential-Quadratic-Gaussian (iLEQG) algorithm and a cross-entropy process, the iLEQG algorithm outputting an updated nominal control trajectory, the cross-entropy process outputting a risk-sensitivity parameter; and controls operation of the robot based, at least in part, on the updated nominal control trajectory and the risk-sensitivity parameter.
    Type: Application
    Filed: February 12, 2021
    Publication date: March 17, 2022
    Inventors: Haruki Nishimura, Negar Zahedi Mehr, Adrien David Gaidon, Mac Schwager
  • Publication number: 20220066460
    Abstract: A mobile robot can be caused to move according to a planned trajectory. The mobile robot can be a vehicle. Information about agents in an environment of the mobile robot can be received from sensors. At a first time, a spatiotemporal graph can be produced. The spatiotemporal graph can represent relationships among the agents in the environment. The mobile robot can be one of the agents in the environment. Information from the spatiotemporal graph can be input to neural networks to produce information for a mixture of affine time-varying systems. The mixture of affine time-varying systems can represent an evolution of agent states of the agents. Using the mixture of affine time-varying systems and information associated with the first time, a prediction of the agent states at a second time can be calculated. The mobile robot can be caused to move according to the planned trajectory determined from the prediction.
    Type: Application
    Filed: April 12, 2021
    Publication date: March 3, 2022
    Inventors: Boris Ivanovic, Amine Elhafsi, Guy Rosman, Adrien David Gaidon, Marco Pavone
  • Publication number: 20220058817
    Abstract: A system for generating point clouds having surface normal information includes one or more processors and a memory having a depth map generating module, a point cloud generating module, and surface normal generating module. The depth map generating module causes the one or more processors to generate a depth map from one or more images of a scene. The point cloud causes the one or more processors to generate a point cloud from the depth map having a plurality of points corresponding to one or more pixels of the depth map. The surface normal generating module causes the one or more processors to generate surface normal information for at least a portion of the one or more pixels of the depth map and inject the surface normal information into the point cloud such that the plurality of points of the point cloud include three-dimensional location information and surface normal information.
    Type: Application
    Filed: August 24, 2020
    Publication date: February 24, 2022
    Inventors: Victor Vaquero Gomez, Rares A. Ambrus, Vitor Guizilini, Adrien David Gaidon
  • Publication number: 20220055663
    Abstract: A method for behavior cloned vehicle trajectory planning is described. The method includes perceiving vehicles proximate an ego vehicle in a driving environment, including a scalar confidence value of each perceived vehicle. The method also includes generating a bird's-eye-view (BEV) grid showing the ego vehicle and each perceived vehicle based on each of the scalar confidence value. The method further includes ignoring at least one of the perceived vehicles when the scalar confidence value of the at least one of the perceived vehicles is less than a predetermined value. The method also includes selecting an ego vehicle trajectory based on a cloned expert vehicle behavior policy according to remaining perceived vehicles.
    Type: Application
    Filed: August 21, 2020
    Publication date: February 24, 2022
    Applicant: TOYOTA RESEARCH INSTITUTE, INC.
    Inventors: Andreas BUEHLER, Adrien David GAIDON, Rares A. AMBRUS, Guy ROSMAN, Wolfram BURGARD
  • Patent number: 11257231
    Abstract: A method for monocular depth/pose estimation in a camera agnostic network is described. The method includes training a monocular depth model and a monocular pose model to learn monocular depth estimation and monocular pose estimation based on a target image and context images from monocular video captured by the camera agnostic network. The method also includes lifting 3D points from image pixels of the target image according to the context images. The method further includes projecting the lifted 3D points onto an image plane according to a predicted ray vector based on the monocular depth model, the monocular pose model, and a camera center of the camera agnostic network. The method also includes predicting a warped target image from a predicted depth map of the monocular depth model, a ray surface of the predicted ray vector, and a projection of the lifted 3D points according to the camera agnostic network.
    Type: Grant
    Filed: June 17, 2020
    Date of Patent: February 22, 2022
    Assignee: TOYOTA RESEARCH INSTITUTE, INC.
    Inventors: Vitor Guizilini, Sudeep Pillai, Adrien David Gaidon, Rares A. Ambrus, Igor Vasiljevic