Patents by Inventor Mengye Ren

Mengye Ren has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20210325882
    Abstract: The present disclosure provides systems and methods that apply neural networks such as, for example, convolutional neural networks, to sparse imagery in an improved manner. For example, the systems and methods of the present disclosure can be included in or otherwise leveraged by an autonomous vehicle. In one example, a computing system can extract one or more relevant portions from imagery, where the relevant portions are less than an entirety of the imagery. The computing system can provide the relevant portions of the imagery to a machine-learned convolutional neural network and receive at least one prediction from the machine-learned convolutional neural network based at least in part on the one or more relevant portions of the imagery. Thus, the computing system can skip performing convolutions over regions of the imagery where the imagery is sparse and/or regions of the imagery that are not relevant to the prediction being sought.
    Type: Application
    Filed: June 30, 2021
    Publication date: October 21, 2021
    Inventors: Raquel Urtasun, Mengye Ren, Andrei Pokrovsky, Bin Yang
  • Publication number: 20210303922
    Abstract: Systems and methods for training object detection models using adversarial examples are provided. A method includes obtaining a training scene and identifying a target object within the training scene. The method includes obtaining an adversarial object and generating a modified training scene based on the adversarial object, the target object, and the training scene. The modified training scene includes the training scene modified to include the adversarial object placed on the target object. The modified training scene is input to a machine-learned model configured to detect the training object. A detection score is determined based on whether the training object is detected, and the machine-learned model and the parameters of the adversarial object are trained based on the detection output. The machine-learned model is trained to maximize the detection output. The parameters of the adversarial object are trained to minimize the detection output.
    Type: Application
    Filed: August 31, 2020
    Publication date: September 30, 2021
    Inventors: James Tu, Sivabalan Manivasagam, Mengye Ren, Ming Liang, Bin Yang, Raquel Urtasun
  • Publication number: 20210278852
    Abstract: Systems and methods for generating attention masks are provided. In particular, a computing system can access sensor data and map data for an area around an autonomous vehicle. The computing system can generate a voxel grid representation of the sensor data and map data. The computing system can generate an attention mask based on the voxel grid representation. The computing system can generate, by using the voxel grid representation and the attention mask as input to a machine-learned model, an attention weighted feature map. The computing system can determine using the attention weighted feature map, a planning cost volume for an area around the autonomous vehicle. The computing system can select a trajectory for the autonomous vehicle based, at least in part, on the planning cost volume.
    Type: Application
    Filed: January 15, 2021
    Publication date: September 9, 2021
    Inventors: Raquel Urtasun, Bob Qingyuan Wei, Mengye Ren, Wenyuan Zeng, Ming Liang, Bin Yang
  • Publication number: 20210279640
    Abstract: Systems and methods for vehicle-to-vehicle communications are provided. An adverse system can obtain sensor data representative of an environment proximate to a targeted system. The adverse system can generate an intermediate representation of the environment and a representation deviation for the intermediate representation. The representation deviation can be designed to disrupt a machine-learned model associated with the target system. The adverse system can communicate the intermediate representation modified by the representation deviation to the target system. The target system can train the machine-learned model associated with the target system to detect the modified intermediate representation. Detected modified intermediate representations can be discarded before disrupting the machine-learned model.
    Type: Application
    Filed: January 15, 2021
    Publication date: September 9, 2021
    Inventors: Xuanyuan Tu, Raquel Urtasun, Tsu-shuan Wang, Sivabalan Manivasagam, Jingkang Wang, Mengye Ren
  • Publication number: 20210276591
    Abstract: Systems and methods for generating semantic occupancy maps are provided. In particular, a computing system can obtain map data for a geographic area and sensor data obtained by the autonomous vehicle. The computer system can identify feature data included in the map data and sensor data. The computer system can, for a respective semantic object type from a plurality of semantic object types, determine, by the computing system and using feature data as input to a respective machine-learned model from a plurality of machine-learned models, one or more occupancy maps for one or more timesteps in the future, and wherein the respective machine-learned model is trained to determine occupancy for the respective semantic object type. The computer system can select a trajectory for the autonomous vehicle based on a plurality of occupancy maps associated with the plurality of semantic object types.
    Type: Application
    Filed: January 15, 2021
    Publication date: September 9, 2021
    Inventors: Raquel Urtasun, Abbas Sadat, Sergio Casas, Mengye Ren
  • Publication number: 20210248460
    Abstract: A computing system can be configured to generate, for an autonomous vehicle, a route through a transportation network comprising a plurality of segments. The method can include receiving sets of agent attention data from additional autonomous vehicles that are respectively currently located at one or more other segments of the transportation network. The method can include inputting the sets of agent attention data into a value iteration graph neural network that comprises a plurality of nodes that respectively correspond to the plurality of segments of the transportation network. The method can include receiving node values respectively for the segments as an output of the value iteration graph neural network. The method can include selecting a next segment to include in the route for the autonomous vehicle based at least in part on the node values.
    Type: Application
    Filed: September 25, 2020
    Publication date: August 12, 2021
    Inventors: Quinlan Sykora, Mengye Ren, Raquel Urtasun
  • Patent number: 11061402
    Abstract: The present disclosure provides systems and methods that apply neural networks such as, for example, convolutional neural networks, to sparse imagery in an improved manner. For example, the systems and methods of the present disclosure can be included in or otherwise leveraged by an autonomous vehicle. In one example, a computing system can extract one or more relevant portions from imagery, where the relevant portions are less than an entirety of the imagery. The computing system can provide the relevant portions of the imagery to a machine-learned convolutional neural network and receive at least one prediction from the machine-learned convolutional neural network based at least in part on the one or more relevant portions of the imagery. Thus, the computing system can skip performing convolutions over regions of the imagery where the imagery is sparse and/or regions of the imagery that are not relevant to the prediction being sought.
    Type: Grant
    Filed: February 7, 2018
    Date of Patent: July 13, 2021
    Assignee: UATC, LLC
    Inventors: Raquel Urtasun, Mengye Ren, Andrei Pokrovsky, Bin Yang
  • Publication number: 20210200212
    Abstract: Systems and methods for generating motion plans for autonomous vehicles are provided. An autonomous vehicle can include a machine-learned motion planning system including one or more machine-learned models configured to generate target trajectories for the autonomous vehicle. The model(s) include a behavioral planning stage configured to receive situational data based at least in part on the one or more outputs of the set of sensors and to generate behavioral planning data based at least in part on the situational data and a unified cost function. The model(s) includes a trajectory planning stage configured to receive the behavioral planning data from the behavioral planning stage and to generate target trajectory data for the autonomous vehicle based at least in part on the behavioral planning data and the unified cost function.
    Type: Application
    Filed: March 20, 2020
    Publication date: July 1, 2021
    Inventors: Raquel Urtasun, Abbas Sadat, Mengye Ren, Andrei Pokrovsky, Yen-Chen Lin, Ersin Yumer
  • Publication number: 20210146963
    Abstract: A computing system can input first relative location embedding data into an interaction transformer model and receive, as an output of the interaction transformer model, motion forecast data for actors relative to a vehicle. The computing system can input the motion forecast data into a prediction model to receive respective trajectories for the actors for a current time step and respective projected trajectories for the actors for a subsequent time step. The computing system can generate second relative location embedding data based on the respective projected trajectories from the second time step. The computing system can produce second motion forecast data using the interaction transformer model based on the second relative location embedding. The computing system can determine second respective trajectories for the actors using the prediction model based on the second forecast data.
    Type: Application
    Filed: September 2, 2020
    Publication date: May 20, 2021
    Inventors: Lingyun Li, Bin Yang, Wenyuan Zeng, Ming Liang, Mengye Ren, Sean Segal, Raquel Urtasun
  • Publication number: 20210012116
    Abstract: Systems and methods of the present disclosure provide an improved approach for open-set instance segmentation by identifying both known and unknown instances in an environment. For example, a method can include receiving sensor point cloud input data including a plurality of three-dimensional points. The method can include determining a feature embedding and at least one of an instance embedding, class embedding, and/or background embedding for each of the plurality of three-dimensional points. The method can include determining a first subset of points associated with one or more known instances within the environment based on the class embedding and the background embedding associated with each point in the plurality of points. The method can include determining a second subset of points associated with one or more unknown instances within the environment based on the first subset of points. The method can include segmenting the input data into known and unknown instances.
    Type: Application
    Filed: March 20, 2020
    Publication date: January 14, 2021
    Inventors: Raquel Urtasun, Kelvin Ka Wing Wong, Shenlong Wang, Mengye Ren, Ming Liang
  • Publication number: 20210009166
    Abstract: A computing system can be configured to input data that describes sensor data into an object detection model and receive, as an output of the object detection model, object detection data describing features of the plurality of the actors relative to the autonomous vehicle. The computing system can generate an input sequence that describes the object detection data. The computing system can analyze the input sequence using an interaction model to produce, as an output of the interaction model, an attention embedding with respect to the plurality of actors. The computing system can be configured to input the attention embedding into a recurrent model and determine respective trajectories for the plurality of actors based on motion forecast data received as an output of the recurrent model.
    Type: Application
    Filed: February 26, 2020
    Publication date: January 14, 2021
    Inventors: Lingyun Li, Bin Yang, Ming Liang, Wenyuan Zeng, Mengye Ren, Sean Segal, Raquel Urtasun
  • Publication number: 20190146497
    Abstract: The present disclosure provides systems and methods that apply neural networks such as, for example, convolutional neural networks, to sparse imagery in an improved manner. For example, the systems and methods of the present disclosure can be included in or otherwise leveraged by an autonomous vehicle. In one example, a computing system can extract one or more relevant portions from imagery, where the relevant portions are less than an entirety of the imagery. The computing system can provide the relevant portions of the imagery to a machine-learned convolutional neural network and receive at least one prediction from the machine-learned convolutional neural network based at least in part on the one or more relevant portions of the imagery. Thus, the computing system can skip performing convolutions over regions of the imagery where the imagery is sparse and/or regions of the imagery that are not relevant to the prediction being sought.
    Type: Application
    Filed: February 7, 2018
    Publication date: May 16, 2019
    Inventors: Raquel Urtasun, Mengye Ren, Andrei Pokrovsky, Bin Yang