Patents by Inventor Shenlong Wang

Shenlong Wang has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Patent number: 10672136
    Abstract: An active depth detection system can generate a depth map from an image and user interaction data, such as a pair of clicks. The active depth detection system can be implemented as a recurrent neural network that can receive the user interaction data as runtime inputs after training. The active depth detection system can store the generated depth map for further processing, such as image manipulation or real-world object detection.
    Type: Grant
    Filed: August 31, 2018
    Date of Patent: June 2, 2020
    Assignee: Snap Inc.
    Inventors: Kun Duan, Daniel Ron, Chongyang Ma, Ning Xu, Shenlong Wang, Sumant Milind Hanumante, Dhritiman Sagar
  • Publication number: 20200160598
    Abstract: The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned geometry model can predict one or more adjusted depths for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.
    Type: Application
    Filed: September 11, 2019
    Publication date: May 21, 2020
    Inventors: Sivabalan Manivasagam, Shenlong Wang, Wei-Chiu Ma, Raquel Urtasun
  • Publication number: 20200160558
    Abstract: Systems and methods for determining a location based on image data are provided. A method can include receiving, by a computing system, a query image depicting a surrounding environment of a vehicle. The query image can be input into a machine-learned image embedding model and a machine-learned feature extraction model to obtain a query embedding and a query feature representation, respectively. The method can include identifying a subset of candidate embeddings that have embeddings similar to the query embedding. The method can include obtaining a respective feature representation for each image associated with the subset of candidate embeddings. The method can include determining a set of relative displacements between each image associated with the subset of candidate embeddings and the query image and determining a localized state of a vehicle based at least in part on the set of relative displacements.
    Type: Application
    Filed: September 17, 2019
    Publication date: May 21, 2020
    Inventors: Raquel Urtasun, Julieta Martinez Covarrubias, Shenlong Wang, Hongbo Fan
  • Publication number: 20200160117
    Abstract: Systems, methods, tangible non-transitory computer-readable media, and devices associated with object localization and generation of compressed feature representations are provided. For example, a computing system can access training data including a target feature representation and a source feature representation. An attention feature representation can be generated based on the target feature representation and a machine-learned attention model. An attended target feature representation can be generated based on masking the target feature representation with the attention feature representation. A matching score for the source feature representation and the target feature representation can be determined. A loss associated with the matching score and a ground-truth matching score for the source feature representation and the target feature representation can be determined. Furthermore, parameters of the machine-learned attention model can be adjusted based on the loss.
    Type: Application
    Filed: October 10, 2019
    Publication date: May 21, 2020
    Inventors: Raquel Urtasun, Xinkai Wei, Ioan Andrei Barsan, Julieta Covarrubias Martinez, Shenlong Wang
  • Publication number: 20200160532
    Abstract: Systems and methods for identifying travel way features in real time are provided. A method can include receiving two-dimensional and three-dimensional data associated with the surrounding environment of a vehicle. The method can include providing the two-dimensional data as one or more input into a machine-learned segmentation model to output a two-dimensional segmentation. The method can include fusing the two-dimensional segmentation with the three-dimensional data to generate a three-dimensional segmentation. The method can include storing the three-dimensional segmentation in a classification database with data indicative of one or more previously generated three-dimensional segmentations. The method can include providing one or more datapoint sets from the classification database as one or more inputs into a machine-learned enhancing model to obtain an enhanced three-dimensional segmentation.
    Type: Application
    Filed: November 15, 2019
    Publication date: May 21, 2020
    Inventors: Raquel Urtasun, Min Bai, Shenlong Wang
  • Publication number: 20200160537
    Abstract: Systems, methods, tangible non-transitory computer-readable media, and devices associated with motion flow estimation are provided. For example, scene data including representations of an environment over a first set of time intervals can be accessed. Extracted visual cues can be generated based on the representations and machine-learned feature extraction models. At least one of the machine-learned feature extraction models can be configured to generate a portion of the extracted visual cues based on a first set of the representations of the environment from a first perspective and a second set of the representations of the environment from a second perspective. The extracted visual cues can be encoded using energy functions. Three-dimensional motion estimates of object instances at time intervals subsequent to the first set of time intervals can be determined based on the energy functions and machine-learned inference models.
    Type: Application
    Filed: August 5, 2019
    Publication date: May 21, 2020
    Inventors: Raquel Urtasun, Wei-Chiu Ma, Shenlong Wang, Yuwen Xiong, Rui Hu
  • Publication number: 20200160104
    Abstract: Systems, methods, tangible non-transitory computer-readable media, and devices associated with object localization and generation of compressed feature representations are provided. For example, a computing system can access training data including a source feature representation and a target feature representation. An encoded target feature representation can be generated based on the target feature representation and a machine-learned encoding model. A binarized target feature representation can be generated based on the encoded target feature representation and lossless binarization operations. A reconstructed target feature representation can be generated based on the binarized target feature representation and a machine-learned decoding model. A matching score for the source feature representation and the reconstructed target feature representation can be determined. A loss associated with the matching score can be determined.
    Type: Application
    Filed: October 10, 2019
    Publication date: May 21, 2020
    Inventors: Raquel Urtasun, Xinkai Wei, Ioan Andrei Barsan, Julieta Covarrubias Martinez, Shenlong Wang
  • Publication number: 20200160151
    Abstract: Systems, methods, tangible non-transitory computer-readable media, and devices associated with object localization and generation of compressed feature representations are provided. For example, a computing system can access source data and target data. The source data can include a source representation of an environment including a source object. The target data can include a compressed target feature representation of the environment. The compressed target feature representation can be based on compression of a target feature representation of the environment produced by machine-learned models. A source feature representation can be generated based on the source representation and the machine-learned models. The machine-learned models can include machine-learned feature extraction models or machine-learned attention models. A localized state of the source object with respect to the environment can be determined based on the source feature representation and the compressed target feature representation.
    Type: Application
    Filed: October 10, 2019
    Publication date: May 21, 2020
    Inventors: Raquel Urtasun, Xinkai Wei, Ioan Andrei Barsan, Julieta Covarrubias Martinez, Shenlong Wang
  • Publication number: 20200074653
    Abstract: An active depth detection system can generate a depth map from an image and user interaction data, such as a pair of clicks. The active depth detection system can be implemented as a recurrent neural network that can receive the user interaction data as runtime inputs after training. The active depth detection system can store the generated depth map for further processing, such as image manipulation or real-world object detection.
    Type: Application
    Filed: August 31, 2018
    Publication date: March 5, 2020
    Inventors: Kun Duan, Daniel Ron, Chongyang Ma, Ning Xu, Shenlong Wang, Sumant Milind Hanumante, Dhritiman Sagar
  • Patent number: 10552968
    Abstract: Dense feature scale detection can be implemented using multiple convolutional neural networks trained on scale data to more accurately and efficiently match pixels between images. An input image can be used to generate multiple scaled images. The multiple scaled images are input into a feature net, which outputs feature data for the multiple scaled images. An attention net is used to generate an attention map from the input image. The attention map assigns emphasis as a soft distribution to different scales based on texture analysis. The feature data and the attention data can be combined through a multiplication process and then summed to generate dense features for comparison.
    Type: Grant
    Filed: September 22, 2017
    Date of Patent: February 4, 2020
    Assignee: Snap Inc.
    Inventors: Shenlong Wang, Linjie Luo, Ning Zhang, Jia Li
  • Publication number: 20200025931
    Abstract: Generally, the disclosed systems and methods implement improved detection of objects in three-dimensional (3D) space. More particularly, an improved 3D object detection system can exploit continuous fusion of multiple sensors and/or integrated geographic prior map data to enhance effectiveness and robustness of object detection in applications such as autonomous driving. In some implementations, geographic prior data (e.g., geometric ground and/or semantic road features) can be exploited to enhance three-dimensional object detection for autonomous vehicle applications. In some implementations, object detection systems and methods can be improved based on dynamic utilization of multiple sensor modalities. More particularly, an improved 3D object detection system can exploit both LIDAR systems and cameras to perform very accurate localization of objects within three-dimensional space relative to an autonomous vehicle.
    Type: Application
    Filed: March 14, 2019
    Publication date: January 23, 2020
    Inventors: Ming Liang, Bin Yang, Shenlong Wang, Wei-Chiu Ma, Raquel Urtasun
  • Publication number: 20200025935
    Abstract: Generally, the disclosed systems and methods implement improved detection of objects in three-dimensional (3D) space. More particularly, an improved 3D object detection system can exploit continuous fusion of multiple sensors and/or integrated geographic prior map data to enhance effectiveness and robustness of object detection in applications such as autonomous driving. In some implementations, geographic prior data (e.g., geometric ground and/or semantic road features) can be exploited to enhance three-dimensional object detection for autonomous vehicle applications. In some implementations, object detection systems and methods can be improved based on dynamic utilization of multiple sensor modalities. More particularly, an improved 3D object detection system can exploit both LIDAR systems and cameras to perform very accurate localization of objects within three-dimensional space relative to an autonomous vehicle.
    Type: Application
    Filed: March 14, 2019
    Publication date: January 23, 2020
    Inventors: Ming Liang, Bin Yang, Shenlong Wang, Wei-Chiu Ma, Raquel Urtasun
  • Publication number: 20190383945
    Abstract: Aspects of the present disclosure involve systems, methods, and devices for autonomous vehicle localization using a Lidar intensity map. A system is configured to generate a map embedding using a first neural network and to generate an online Lidar intensity embedding using a second neural network. The map embedding is based on input map data comprising a Lidar intensity map, and the Lidar sweep embedding is based on online Lidar sweep data. The system is further configured to generate multiple pose candidates based on the online Lidar intensity embedding and compute a three-dimensional (3D) score map comprising a match score for each pose candidate that indicates a similarity between the pose candidate and the map embedding. The system is further configured to determine a pose of a vehicle based on the 3D score map and to control one or more operations of the vehicle based on the determined pose.
    Type: Application
    Filed: June 11, 2019
    Publication date: December 19, 2019
    Inventors: Shenlong Wang, Andrei Pokrovsky, Raquel Urtasun Sotil, Ioan Andrei Bârsan
  • Publication number: 20190147253
    Abstract: Systems and methods for facilitating communication with autonomous vehicles are provided. In one example embodiment, a computing system can obtain rasterized LIDAR data associated with a surrounding environment of an autonomous vehicle. The rasterized LIDAR data can include LIDAR image data that is rasterized from a LIDAR point cloud. The computing system can access data indicative of a machine-learned lane boundary detection model. The computing system can input the rasterized LIDAR data associated with the surrounding environment of the autonomous vehicle into the machine-learned lane boundary detection model. The computing system can obtain an output from the machine-learned lane boundary detection model. The output can be indicative of one or more lane boundaries within the surrounding environment of the autonomous vehicle.
    Type: Application
    Filed: September 5, 2018
    Publication date: May 16, 2019
    Inventors: Min Bai, Gellert Sandor Mattyus, Namdar Homayounfar, Shenlong Wang, Shrindihi Kowshika Lakshmikanth, Raquel Urtasun, Wei-Chiu Ma
  • Publication number: 20190145784
    Abstract: Systems and methods for autonomous vehicle localization are provided. In one example embodiment, a computer-implemented method includes obtaining, by a computing system that includes one or more computing devices onboard an autonomous vehicle, sensor data indicative of one or more geographic cues within the surrounding environment of the autonomous vehicle. The method includes obtaining, by the computing system, sparse geographic data associated with the surrounding environment of the autonomous vehicle. The sparse geographic data is indicative of the one or more geographic cues. The method includes determining, by the computing system, a location of the autonomous vehicle within the surrounding environment based at least in part on the sensor data indicative of the one or more geographic cues and the sparse geographic data. The method includes outputting, by the computing system, data indicative of the location of the autonomous vehicle within the surrounding environment.
    Type: Application
    Filed: September 6, 2018
    Publication date: May 16, 2019
    Inventors: Wei-Chiu Ma, Shenlong Wang, Namdar Homayounfar, Shrinidhi Kowshika Lakshmikanth, Raquel Urtasun
  • Publication number: 20190147335
    Abstract: Systems and methods are provided for machine-learned models including convolutional neural networks that generate predictions using continuous convolution techniques. For example, the systems and methods of the present disclosure can be included in or otherwise leveraged by an autonomous vehicle. In one example, a computing system can perform, with a machine-learned convolutional neural network, one or more convolutions over input data using a continuous filter relative to a support domain associated with the input data, and receive a prediction from the machine-learned convolutional neural network. A machine-learned convolutional neural network in some examples includes at least one continuous convolution layer configured to perform convolutions over input data with a parametric continuous kernel.
    Type: Application
    Filed: October 30, 2018
    Publication date: May 16, 2019
    Inventors: Shenlong Wang, Wei-Chiu Ma, Shun Da Suo, Raquel Urtasun, Ming Liang
  • Publication number: 20190147254
    Abstract: Systems and methods for facilitating communication with autonomous vehicles are provided. In one example embodiment, a computing system can obtain a first type of sensor data (e.g., camera image data) associated with a surrounding environment of an autonomous vehicle and/or a second type of sensor data (e.g., LIDAR data) associated with the surrounding environment of the autonomous vehicle. The computing system can generate overhead image data indicative of at least a portion of the surrounding environment of the autonomous vehicle based at least in part on the first and/or second types of sensor data. The computing system can determine one or more lane boundaries within the surrounding environment of the autonomous vehicle based at least in part on the overhead image data indicative of at least the portion of the surrounding environment of the autonomous vehicle and a machine-learned lane boundary detection model.
    Type: Application
    Filed: September 5, 2018
    Publication date: May 16, 2019
    Inventors: Min Bai, Gellert Sandor Mattyus, Namdar Homayounfar, Shenlong Wang, Shrindihi Kowshika Lakshmikanth, Raquel Urtasun
  • Patent number: 9886652
    Abstract: Correspondences in content items may be determined using a trained decision tree to detect distinctive matches between portions of content items. The techniques described include determining a first group of patches associated with a first content item and processing a first patch based at least partly on causing the first patch to move through a decision tree, and determining a second group of patches associated with a second content item and processing a second patch based at least partly on causing the second patch to move through the decision tree. The techniques described include determining that the first patch and the second patch are associated with a same leaf node of the decision tree and determining that the first patch and the second patch are corresponding patches based at least partly on determining that the first patch and the second patch are associated with the same leaf node.
    Type: Grant
    Filed: March 15, 2016
    Date of Patent: February 6, 2018
    Assignee: Microsoft Technology Licensing, LLC
    Inventors: Sean Ryan Francesco Fanello, Shahram Izadi, Pushmeet Kohli, Christoph Rhemann, Shenlong Wang
  • Publication number: 20170270390
    Abstract: Correspondences in content items may be determined using a trained decision tree to detect distinctive matches between portions of content items. The techniques described include determining a first group of patches associated with a first content item and processing a first patch based at least partly on causing the first patch to move through a decision tree, and determining a second group of patches associated with a second content item and processing a second patch based at least partly on causing the second patch to move through the decision tree. The techniques described include determining that the first patch and the second patch are associated with a same leaf node of the decision tree and determining that the first patch and the second patch are corresponding patches based at least partly on determining that the first patch and the second patch are associated with the same leaf node.
    Type: Application
    Filed: March 15, 2016
    Publication date: September 21, 2017
    Inventors: Sean Ryan Francesco Fanello, Shahram Izadi, Pushmeet Kohli, Christoph Rhemann, Shenlong Wang