Patents by Inventor Raquel Urtasun

Raquel Urtasun has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Patent number: 10859384
    Abstract: Systems and methods for autonomous vehicle localization are provided. In one example embodiment, a computer-implemented method includes obtaining, by a computing system that includes one or more computing devices onboard an autonomous vehicle, sensor data indicative of one or more geographic cues within the surrounding environment of the autonomous vehicle. The method includes obtaining, by the computing system, sparse geographic data associated with the surrounding environment of the autonomous vehicle. The sparse geographic data is indicative of the one or more geographic cues. The method includes determining, by the computing system, a location of the autonomous vehicle within the surrounding environment based at least in part on the sensor data indicative of the one or more geographic cues and the sparse geographic data. The method includes outputting, by the computing system, data indicative of the location of the autonomous vehicle within the surrounding environment.
    Type: Grant
    Filed: September 6, 2018
    Date of Patent: December 8, 2020
    Assignee: UATC, LLC
    Inventors: Wei-Chiu Ma, Shenlong Wang, Namdar Homayounfar, Shrinidhi Kowshika Lakshmikanth, Raquel Urtasun
  • Patent number: 10803325
    Abstract: Systems and methods for facilitating communication with autonomous vehicles are provided. In one example embodiment, a computing system can obtain rasterized LIDAR data associated with a surrounding environment of an autonomous vehicle. The rasterized LIDAR data can include LIDAR image data that is rasterized from a LIDAR point cloud. The computing system can access data indicative of a machine-learned lane boundary detection model. The computing system can input the rasterized LIDAR data associated with the surrounding environment of the autonomous vehicle into the machine-learned lane boundary detection model. The computing system can obtain an output from the machine-learned lane boundary detection model. The output can be indicative of one or more lane boundaries within the surrounding environment of the autonomous vehicle.
    Type: Grant
    Filed: September 5, 2018
    Date of Patent: October 13, 2020
    Assignee: UATC, LLC
    Inventors: Min Bai, Gellert Sandor Mattyus, Namdar Homayounfar, Shenlong Wang, Shrindihi Kowshika Lakshmikanth, Raquel Urtasun, Wei-Chiu Ma
  • Patent number: 10803328
    Abstract: Systems and methods for detecting objects are provided. In one example, a computer-implemented method includes receiving sensor data from one or more sensors configured to generate sensor data. The method includes inputting the sensor data to a machine-learned model that generates a class prediction and an instance prediction for each of a plurality of portions of the sensor data. The instance prediction includes an energy value based on a distance to at least one object boundary. The machine learned model can be trained to generate a common energy value to represent the at least one object boundary. The method includes generating as outputs of the machine-learned model, an instance prediction and a class prediction corresponding to each of the plurality of portions of the sensor data. The method includes generating one or more object segments based at least in part on the instance predictions and the class predictions.
    Type: Grant
    Filed: September 6, 2018
    Date of Patent: October 13, 2020
    Assignee: UATC, LLC
    Inventors: Min Bai, Raquel Urtasun
  • Publication number: 20200298891
    Abstract: Systems, methods, tangible non-transitory computer-readable media, and devices associated with object perception and prediction of object motion are provided. For example, a plurality of temporal instance representations can be generated. Each temporal instance representation can be associated with differences in the appearance and motion of objects over past time intervals. Past paths and candidate paths of a set of objects can be determined based on the temporal instance representations and current detections of objects. Predicted paths of the set of objects using a machine-learned model trained that uses the past paths and candidate paths to determine the predicted paths. Past path data that includes information associated with the predicted paths can be generated for each object of the set of objects respectively.
    Type: Application
    Filed: March 23, 2020
    Publication date: September 24, 2020
    Inventors: Ming Liang, Bin Yang, Yun Chen, Raquel Urtasun
  • Publication number: 20200304835
    Abstract: A machine-learned image compression model includes a first encoder configured to generate a first image code based at least in part on first image data. The first encoder includes a first series of convolutional layers configured to generate a first series of respective feature maps based at least in part on the first image. A second encoder is configured to generate a second image code based at least in part on second image data and includes a second series of convolutional layers configured to generate a second series of respective feature maps based at least in part on the second image and disparity-warped feature data. Respective parametric skip functions associated convolutional layers of the second series are configured to generate disparity-warped feature data based at least in part on disparity associated with the first series of respective feature maps and the second series of respective feature maps.
    Type: Application
    Filed: March 20, 2020
    Publication date: September 24, 2020
    Inventors: Jerry Junkai Liu, Shenlong Wang, Raquel Urtasun
  • Publication number: 20200302627
    Abstract: Systems, methods, tangible non-transitory computer-readable media, and devices associated with depth estimation are provided. For example, a feature representation associated with stereo images including a first and second plurality of points can be accessed. Sparse disparity estimates associated with disparities between the first and second plurality of points can be determined. The sparse disparity estimates can be based on machine-learned models that estimate disparities based on comparisons of the first plurality of points to the second plurality of points. Confidence ranges associated with the disparities between the first and second plurality of points can be determined based on the sparse disparity estimates and the machine-learned models. A disparity map for the stereo images can be generated based on using the confidence ranges and machine-learned models to prune the disparities outside the confidence ranges.
    Type: Application
    Filed: March 23, 2020
    Publication date: September 24, 2020
    Inventors: Shivam Duggal, Shenlong Wang, Wei-Chiu Ma, Raquel Urtasun
  • Publication number: 20200302662
    Abstract: The present disclosure is directed to generating high quality map data using obtained sensor data. In particular a computing system comprising one or more computing devices can obtain sensor data associated with a portion of a travel way. The computing system can identify, using a machine-learned model, feature data associated with one or more lane boundaries in the portion of the travel way based on the obtained sensor data. The computing system can generate a graph representing lane boundaries associated with the portion of the travel way by identifying a respective node location for the respective lane boundary based in part on identified feature data associated with lane boundary information, determining, for the respective node location, an estimated direction value and an estimated lane state, and generating, based on the respective node location, the estimated direction value, and the estimated lane state, a predicted next node location.
    Type: Application
    Filed: March 20, 2020
    Publication date: September 24, 2020
    Inventors: Namdar Homayounfar, Justin Liang, Wei-Chiu Ma, Raquel Urtasun
  • Publication number: 20200301799
    Abstract: The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned model can predict one or more dropout probabilities for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.
    Type: Application
    Filed: March 23, 2020
    Publication date: September 24, 2020
    Inventors: Sivabalan Manivasagam, Shenlong Wang, Wei-Chiu Ma, Kelvin Ka Wing Wong, Wenyuan Zeng, Raquel Urtasun
  • Publication number: 20200160104
    Abstract: Systems, methods, tangible non-transitory computer-readable media, and devices associated with object localization and generation of compressed feature representations are provided. For example, a computing system can access training data including a source feature representation and a target feature representation. An encoded target feature representation can be generated based on the target feature representation and a machine-learned encoding model. A binarized target feature representation can be generated based on the encoded target feature representation and lossless binarization operations. A reconstructed target feature representation can be generated based on the binarized target feature representation and a machine-learned decoding model. A matching score for the source feature representation and the reconstructed target feature representation can be determined. A loss associated with the matching score can be determined.
    Type: Application
    Filed: October 10, 2019
    Publication date: May 21, 2020
    Inventors: Raquel Urtasun, Xinkai Wei, Ioan Andrei Barsan, Julieta Covarrubias Martinez, Shenlong Wang
  • Publication number: 20200160532
    Abstract: Systems and methods for identifying travel way features in real time are provided. A method can include receiving two-dimensional and three-dimensional data associated with the surrounding environment of a vehicle. The method can include providing the two-dimensional data as one or more input into a machine-learned segmentation model to output a two-dimensional segmentation. The method can include fusing the two-dimensional segmentation with the three-dimensional data to generate a three-dimensional segmentation. The method can include storing the three-dimensional segmentation in a classification database with data indicative of one or more previously generated three-dimensional segmentations. The method can include providing one or more datapoint sets from the classification database as one or more inputs into a machine-learned enhancing model to obtain an enhanced three-dimensional segmentation.
    Type: Application
    Filed: November 15, 2019
    Publication date: May 21, 2020
    Inventors: Raquel Urtasun, Min Bai, Shenlong Wang
  • Publication number: 20200160537
    Abstract: Systems, methods, tangible non-transitory computer-readable media, and devices associated with motion flow estimation are provided. For example, scene data including representations of an environment over a first set of time intervals can be accessed. Extracted visual cues can be generated based on the representations and machine-learned feature extraction models. At least one of the machine-learned feature extraction models can be configured to generate a portion of the extracted visual cues based on a first set of the representations of the environment from a first perspective and a second set of the representations of the environment from a second perspective. The extracted visual cues can be encoded using energy functions. Three-dimensional motion estimates of object instances at time intervals subsequent to the first set of time intervals can be determined based on the energy functions and machine-learned inference models.
    Type: Application
    Filed: August 5, 2019
    Publication date: May 21, 2020
    Inventors: Raquel Urtasun, Wei-Chiu Ma, Shenlong Wang, Yuwen Xiong, Rui Hu
  • Publication number: 20200160559
    Abstract: Provided are systems and methods that perform multi-task and/or multi-sensor fusion for three-dimensional object detection in furtherance of, for example, autonomous vehicle perception and control. In particular, according to one aspect of the present disclosure, example systems and methods described herein exploit simultaneous training of a machine-learned model ensemble relative to multiple related tasks to learn to perform more accurate multi-sensor 3D object detection. For example, the present disclosure provides an end-to-end learnable architecture with multiple machine-learned models that interoperate to reason about 2D and/or 3D object detection as well as one or more auxiliary tasks. According to another aspect of the present disclosure, example systems and methods described herein can perform multi-sensor fusion (e.g.
    Type: Application
    Filed: October 16, 2019
    Publication date: May 21, 2020
    Inventors: Raquel Urtasun, Bin Yang, Ming Liang
  • Publication number: 20200160117
    Abstract: Systems, methods, tangible non-transitory computer-readable media, and devices associated with object localization and generation of compressed feature representations are provided. For example, a computing system can access training data including a target feature representation and a source feature representation. An attention feature representation can be generated based on the target feature representation and a machine-learned attention model. An attended target feature representation can be generated based on masking the target feature representation with the attention feature representation. A matching score for the source feature representation and the target feature representation can be determined. A loss associated with the matching score and a ground-truth matching score for the source feature representation and the target feature representation can be determined. Furthermore, parameters of the machine-learned attention model can be adjusted based on the loss.
    Type: Application
    Filed: October 10, 2019
    Publication date: May 21, 2020
    Inventors: Raquel Urtasun, Xinkai Wei, Ioan Andrei Barsan, Julieta Covarrubias Martinez, Shenlong Wang
  • Publication number: 20200160558
    Abstract: Systems and methods for determining a location based on image data are provided. A method can include receiving, by a computing system, a query image depicting a surrounding environment of a vehicle. The query image can be input into a machine-learned image embedding model and a machine-learned feature extraction model to obtain a query embedding and a query feature representation, respectively. The method can include identifying a subset of candidate embeddings that have embeddings similar to the query embedding. The method can include obtaining a respective feature representation for each image associated with the subset of candidate embeddings. The method can include determining a set of relative displacements between each image associated with the subset of candidate embeddings and the query image and determining a localized state of a vehicle based at least in part on the set of relative displacements.
    Type: Application
    Filed: September 17, 2019
    Publication date: May 21, 2020
    Inventors: Raquel Urtasun, Julieta Martinez Covarrubias, Shenlong Wang, Hongbo Fan
  • Publication number: 20200159225
    Abstract: Systems and methods for generating motion plans including target trajectories for autonomous vehicles are provided. An autonomous vehicle may include or access a machine-learned motion planning model including a backbone network configured to generate a cost volume including data indicative of a cost associated with future locations of the autonomous vehicle. The cost volume can be generated from raw sensor data as part of motion planning for the autonomous vehicle. The backbone network can generate intermediate representations associated with object detections and objection predictions. The motion planning model can include a trajectory generator configured to evaluate one or more potential trajectories for the autonomous vehicle and to select a target trajectory based at least in part on the cost volume generate by the backbone network.
    Type: Application
    Filed: August 15, 2019
    Publication date: May 21, 2020
    Inventors: Wenyuan Zeng, Wenjie Luo, Abbas Sadat, Bin Yang, Raquel Urtasun
  • Publication number: 20200160598
    Abstract: The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned geometry model can predict one or more adjusted depths for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.
    Type: Application
    Filed: September 11, 2019
    Publication date: May 21, 2020
    Inventors: Sivabalan Manivasagam, Shenlong Wang, Wei-Chiu Ma, Raquel Urtasun
  • Publication number: 20200160151
    Abstract: Systems, methods, tangible non-transitory computer-readable media, and devices associated with object localization and generation of compressed feature representations are provided. For example, a computing system can access source data and target data. The source data can include a source representation of an environment including a source object. The target data can include a compressed target feature representation of the environment. The compressed target feature representation can be based on compression of a target feature representation of the environment produced by machine-learned models. A source feature representation can be generated based on the source representation and the machine-learned models. The machine-learned models can include machine-learned feature extraction models or machine-learned attention models. A localized state of the source object with respect to the environment can be determined based on the source feature representation and the compressed target feature representation.
    Type: Application
    Filed: October 10, 2019
    Publication date: May 21, 2020
    Inventors: Raquel Urtasun, Xinkai Wei, Ioan Andrei Barsan, Julieta Covarrubias Martinez, Shenlong Wang
  • Publication number: 20200025935
    Abstract: Generally, the disclosed systems and methods implement improved detection of objects in three-dimensional (3D) space. More particularly, an improved 3D object detection system can exploit continuous fusion of multiple sensors and/or integrated geographic prior map data to enhance effectiveness and robustness of object detection in applications such as autonomous driving. In some implementations, geographic prior data (e.g., geometric ground and/or semantic road features) can be exploited to enhance three-dimensional object detection for autonomous vehicle applications. In some implementations, object detection systems and methods can be improved based on dynamic utilization of multiple sensor modalities. More particularly, an improved 3D object detection system can exploit both LIDAR systems and cameras to perform very accurate localization of objects within three-dimensional space relative to an autonomous vehicle.
    Type: Application
    Filed: March 14, 2019
    Publication date: January 23, 2020
    Inventors: Ming Liang, Bin Yang, Shenlong Wang, Wei-Chiu Ma, Raquel Urtasun
  • Publication number: 20200025931
    Abstract: Generally, the disclosed systems and methods implement improved detection of objects in three-dimensional (3D) space. More particularly, an improved 3D object detection system can exploit continuous fusion of multiple sensors and/or integrated geographic prior map data to enhance effectiveness and robustness of object detection in applications such as autonomous driving. In some implementations, geographic prior data (e.g., geometric ground and/or semantic road features) can be exploited to enhance three-dimensional object detection for autonomous vehicle applications. In some implementations, object detection systems and methods can be improved based on dynamic utilization of multiple sensor modalities. More particularly, an improved 3D object detection system can exploit both LIDAR systems and cameras to perform very accurate localization of objects within three-dimensional space relative to an autonomous vehicle.
    Type: Application
    Filed: March 14, 2019
    Publication date: January 23, 2020
    Inventors: Ming Liang, Bin Yang, Shenlong Wang, Wei-Chiu Ma, Raquel Urtasun
  • Publication number: 20190383945
    Abstract: Aspects of the present disclosure involve systems, methods, and devices for autonomous vehicle localization using a Lidar intensity map. A system is configured to generate a map embedding using a first neural network and to generate an online Lidar intensity embedding using a second neural network. The map embedding is based on input map data comprising a Lidar intensity map, and the Lidar sweep embedding is based on online Lidar sweep data. The system is further configured to generate multiple pose candidates based on the online Lidar intensity embedding and compute a three-dimensional (3D) score map comprising a match score for each pose candidate that indicates a similarity between the pose candidate and the map embedding. The system is further configured to determine a pose of a vehicle based on the 3D score map and to control one or more operations of the vehicle based on the determined pose.
    Type: Application
    Filed: June 11, 2019
    Publication date: December 19, 2019
    Inventors: Shenlong Wang, Andrei Pokrovsky, Raquel Urtasun Sotil, Ioan Andrei Bârsan