Patents by Inventor Junhua Mao
Junhua Mao has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Patent number: 11967103Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for estimating a 3-D pose of an object of interest from image and point cloud data. In one aspect, a method includes obtaining an image of an environment; obtaining a point cloud of a three-dimensional region of the environment; generating a fused representation of the image and the point cloud; and processing the fused representation using a pose estimation neural network and in accordance with current values of a plurality of pose estimation network parameters to generate a pose estimation network output that specifies, for each of multiple keypoints, a respective estimated position in the three-dimensional region of the environment.Type: GrantFiled: October 20, 2021Date of Patent: April 23, 2024Assignee: Waymo LLCInventors: Jingxiao Zheng, Xinwei Shi, Alexander Gorban, Junhua Mao, Andre Liang Cornman, Yang Song, Ting Liu, Ruizhongtai Qi, Yin Zhou, Congcong Li, Dragomir Anguelov
-
Patent number: 11861481Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for searching an autonomous vehicle sensor data repository. One of the methods includes maintaining a collection of sensor samples and, for each sensor sample, an embedding of the sensor sample; receiving a request specifying a query sensor sample, wherein the query sensor sample characterizes a query environment region; and identifying, from the collection of sensor samples, a plurality of relevant sensor samples that characterize similar environment regions to the query environment region, comprising: processing the query sensor sample through the embedding neural network to generate a query embedding; and identifying, from sensor samples in a subset of the sensor samples in the collection, a plurality of sensor samples that have embeddings that are closest to the query embedding.Type: GrantFiled: December 23, 2019Date of Patent: January 2, 2024Assignee: Waymo LLCInventors: Zijian Guo, Nichola Abdo, Junhua Mao, Congcong Li, Edward Stephen Walker, Jr.
-
Patent number: 11842282Abstract: Aspects of the subject matter disclosed herein include methods, systems, and other techniques for training, in a first phase, an object classifier neural network with a first set of training data, the first set of training data including a first plurality of training examples, each training example in the first set of training data being labeled with a coarse-object classification; and training, in a second phase after completion of the first phase, the object classifier neural network with a second set of training data, the second set of training data including a second plurality of training examples, each training example in the second set of training data being labeled with a fine-object classification.Type: GrantFiled: June 9, 2022Date of Patent: December 12, 2023Assignee: Waymo LLCInventors: Junhua Mao, Congcong Li, Yang Song
-
Patent number: 11790038Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating rare pose data. One of the methods includes obtaining a three-dimensional model of a dynamic object, wherein the dynamic object has multiple movable elements that define a plurality of poses of the dynamic object. A plurality of template poses of the dynamic object are used to generate additional poses for the dynamic object including varying angles of one or more key joints of the dynamic object according to the three-dimensional model. Point cloud data is generated for the additional poses generated for the dynamic object.Type: GrantFiled: November 16, 2021Date of Patent: October 17, 2023Assignee: Waymo LLCInventors: Xiaohan Jin, Junhua Mao, Ruizhongtai Qi, Congcong Li, Huayi Zeng
-
Patent number: 11783568Abstract: Some aspects of the subject matter disclosed herein include a system implemented on one or more data processing apparatuses. The system can include an interface configured to obtain, from one or more sensor subsystems, sensor data describing an environment of a vehicle, and to generate, using the sensor data, (i) one or more first neural network inputs representing sensor measurements for a particular object in the environment and (ii) a second neural network input representing sensor measurements for at least a portion of the environment that encompasses the particular object and additional portions of the environment that are not represented by the one or more first neural network inputs; and a convolutional neural network configured to process the second neural network input to generate an output, the output including a plurality of feature vectors that each correspond to a different one a plurality of regions of the environment.Type: GrantFiled: April 7, 2021Date of Patent: October 10, 2023Assignee: Waymo LLCInventors: Junhua Mao, Qian Yu, Congcong Li
-
Patent number: 11774596Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing data generated by a sensing system that rotationally senses an environment. In one aspect, a method comprises partitioning a predetermined period of time into a plurality of sub-periods, wherein the predetermined period of time is a period of time for which data generated by the sensing system constitutes a complete rotational sensing of the environment; for each sub-period: receiving current data generated by the sensing system during the sub-period and characterizing a respective partial scene of the environment; processing the current data using an object detection neural network to generate a current object detection output that is specific to the respective partial scene of the environment.Type: GrantFiled: September 1, 2022Date of Patent: October 3, 2023Assignee: Google LLCInventors: Jonathon Shlens, Vijay Vasudevan, Jiquan Ngiam, Wei Han, Zhifeng Chen, Brandon Chauloon Yang, Benjamin James Caine, Zhengdong Zhang, Christoph Sprunk, Ouais Alsharif, Junhua Mao, Chen Wu
-
Publication number: 20230150550Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent behavior prediction using keypoint data. One of the methods includes obtaining data characterizing a scene in an environment, the data comprising: (i) context data comprising data characterizing historical trajectories of a plurality of agents up to the current time point; and (ii) keypoint data for a target agent; processing the context data using a context data encoder neural network to generate a context embedding for the target agent; processing the keypoint data using a keypoint encoder neural network to generate a keypoint embedding for the target agent; generating a combined embedding for the target agent from the context embedding and the keypoint embedding; and processing the combined embedding using a decoder neural network to generate a behavior prediction output for the target agent that characterizes predicted behavior of the target agent after the current time point.Type: ApplicationFiled: November 16, 2022Publication date: May 18, 2023Inventors: Xinwei Shi, Tian Lan, Jonathan Chandler Stroud, Zhishuai Zhang, Junhua Mao, Jeonhyung Kang, Khaled Refaat, Jiachen Li
-
Publication number: 20230124460Abstract: A one time programmable OTP memory array and a read and write method thereof are provided. The OTP memory array according to the present disclosure includes M×N OTP memories, the OTP memories each include a storage MOS transistor, a first MOS transistor, a second MOS transistor and a detection MOS transistor, an isolation module is disposed between a control terminal of the detection MOS transistor and the storage MOS transistor; the isolation module includes at least one isolation MOS transistor; and in the array, a gate of each storage MOS transistor is connected to a same storage control point, each isolation MOS transistor is distinguished based on a distance from the storage MOS transistor, and gates of isolation MOS transistors with a same distance from the storage MOS transistor are connected to a same isolation control point.Type: ApplicationFiled: July 19, 2022Publication date: April 20, 2023Inventors: Jack Zezhong Peng, Junhua Mao
-
Publication number: 20230099920Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a classifier to detect open vehicle doors. One of the methods includes obtaining a plurality of initial training examples, each initial training example comprising (i) a sensor sample from a collection of sensor samples and (ii) data classifying the sensor sample as characterizing a vehicle that has an open door; generating a plurality of additional training examples, comprising, for each initial training example: identifying, from the collection of sensor samples, one or more additional sensor samples that were captured less than a threshold amount of time before the sensor sample in the initial training example was captured; and training the machine learning classifier on first training data that includes the initial training examples and the additional training examples to generate updated weights for the machine learning classifier.Type: ApplicationFiled: November 28, 2022Publication date: March 30, 2023Inventors: Junhua Mao, Lo Po Tsui, Congcong Li, Edward Stephen Walker, JR.
-
Patent number: 11610423Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing point cloud data using spatio-temporal-interactive networks.Type: GrantFiled: November 16, 2020Date of Patent: March 21, 2023Assignee: Waymo LLCInventors: Junhua Mao, Jiyang Gao, Yukai Liu, Congcong Li, Zhishuai Zhang, Dragomir Anguelov
-
Publication number: 20230062158Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium that determine yield behavior for an autonomous vehicle, and can include identifying an agent that is in a vicinity of an autonomous vehicle navigating through a scene at a current time point. Scene features can be obtained and can include features of (i) the agent and (ii) the autonomous vehicle. An input that can include the scene features can be processed using a first machine learning model that is configured to generate (i) a crossing intent prediction that includes a crossing intent score that represents a likelihood that the agent intends to cross a roadway in a future time window after the current time, and (ii) a crossing action prediction that includes a crossing action score that represents a likelihood that the agent will cross the roadway in the future time window after the current time.Type: ApplicationFiled: September 2, 2022Publication date: March 2, 2023Inventors: Xinwei Shi, Junhua Mao, Khaled Refaat, Tian Lan, Jeonhyung Kang, Zhishuai Zhang, Jonathan Chandler Stroud
-
Patent number: 11593612Abstract: Presented herein are embodiments of a multimodal Recurrent Neural Network (m-RNN) model for generating novel image captions. In embodiments, it directly models the probability distribution of generating a word given a previous word or words and an image, and image captions are generated according to this distribution. In embodiments, the model comprises two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. In embodiments, these two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of an embodiment of model was validated on four benchmark datasets, and it outperformed the state-of-the-art methods. In embodiments, the m-RNN model may also be applied to retrieval tasks for retrieving images or captions.Type: GrantFiled: August 19, 2019Date of Patent: February 28, 2023Assignee: BAIDU USA LLCInventors: Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang
-
Publication number: 20230059370Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for predicting gaze and awareness using a neural network model. One of the methods includes obtaining sensor data (i) that is captured by one or more sensors of an autonomous vehicle and (ii) that characterizes an agent that is in a vicinity of the autonomous vehicle in an environment at a current time point. The sensor data is processed using a gaze prediction neural network to generate a gaze prediction that predicts a gaze of the agent at the current time point. The gaze prediction neural network includes an embedding subnetwork that is configured to process the sensor data to generate an embedding characterizing the agent, and a gaze subnetwork that is configured to process the embedding to generate the gaze prediction.Type: ApplicationFiled: August 12, 2022Publication date: February 23, 2023Inventors: Junhua Mao, Xinwei Shi, Anne Hobbs Dorsey, Rui Yan, Chi Yeung Jonathan Ng
-
Publication number: 20220415042Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing data generated by a sensing system that rotationally senses an environment. In one aspect, a method comprises partitioning a predetermined period of time into a plurality of sub-periods, wherein the predetermined period of time is a period of time for which data generated by the sensing system constitutes a complete rotational sensing of the environment; for each sub-period: receiving current data generated by the sensing system during the sub-period and characterizing a respective partial scene of the environment; processing the current data using an object detection neural network to generate a current object detection output that is specific to the respective partial scene of the environment.Type: ApplicationFiled: September 1, 2022Publication date: December 29, 2022Inventors: Jonathon Shlens, Vijay Vasudevan, Jiquan Ngiam, Wei Han, Zhifeng Chen, Brandon Chauloon Yang, Benjamin James Caine, Zhengdong Zhang, Christoph Sprunk, Ouais Alsharif, Junhua Mao, Chen Wu
-
Patent number: 11514310Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a classifier to detect open vehicle doors. One of the methods includes obtaining a plurality of initial training examples, each initial training example comprising (i) a sensor sample from a collection of sensor samples and (ii) data classifying the sensor sample as characterizing a vehicle that has an open door; generating a plurality of additional training examples, comprising, for each initial training example: identifying, from the collection of sensor samples, one or more additional sensor samples that were captured less than a threshold amount of time before the sensor sample in the initial training example was captured; and training the machine learning classifier on first training data that includes the initial training examples and the additional training examples to generate updated weights for the machine learning classifier.Type: GrantFiled: December 21, 2018Date of Patent: November 29, 2022Assignee: Waymo LLCInventors: Junhua Mao, Lo Po Tsui, Congcong Li, Edward Stephen Walker, Jr.
-
Publication number: 20220374650Abstract: Aspects of the subject matter disclosed herein include methods, systems, and other techniques for training, in a first phase, an object classifier neural network with a first set of training data, the first set of training data including a first plurality of training examples, each training example in the first set of training data being labeled with a coarse-object classification; and training, in a second phase after completion of the first phase, the object classifier neural network with a second set of training data, the second set of training data including a second plurality of training examples, each training example in the second set of training data being labeled with a fine-object classification.Type: ApplicationFiled: June 9, 2022Publication date: November 24, 2022Inventors: Junhua Mao, Congcong Li, Yang Song
-
Patent number: 11508147Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing data generated by a sensing system that rotationally senses an environment. In one aspect, a method comprises partitioning a predetermined period of time into a plurality of sub-periods, wherein the predetermined period of time is a period of time for which data generated by the sensing system constitutes a complete rotational sensing of the environment; for each sub-period: receiving current data generated by the sensing system during the sub-period and characterizing a respective partial scene of the environment; processing the current data using an object detection neural network to generate a current object detection output that is specific to the respective partial scene of the environment.Type: GrantFiled: March 6, 2020Date of Patent: November 22, 2022Assignee: Google LLCInventors: Jonathon Shlens, Vijay Vasudevan, Jiquan Ngiam, Wei Han, Zhifeng Chen, Brandon Chauloon Yang, Benjamin James Caine, Zhengdong Zhang, Christoph Sprunk, Ouais Alsharif, Junhua Mao, Chen Wu
-
Patent number: 11480963Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating vehicle intent predictions using a neural network. One of the methods includes obtaining an input characterizing one or more vehicles in an environment; generating, from the input, features of each of the vehicles; and for each of the vehicles: processing the features of the vehicle using each of a plurality of intent-specific neural networks, wherein each of the intent-specific neural networks corresponds to a respective intent from a set of intents, and wherein each intent-specific neural network is configured to process the features of the vehicle to generate an output for the corresponding intent.Type: GrantFiled: December 20, 2019Date of Patent: October 25, 2022Assignee: Waymo LLCInventors: Jiyang Gao, Junhua Mao, Yi Shen, Congcong Li, Chen Sun
-
Patent number: 11361187Abstract: Aspects of the subject matter disclosed herein include methods, systems, and other techniques for training, in a first phase, an object classifier neural network with a first set of training data, the first set of training data including a first plurality of training examples, each training example in the first set of training data being labeled with a coarse-object classification; and training, in a second phase after completion of the first phase, the object classifier neural network with a second set of training data, the second set of training data including a second plurality of training examples, each training example in the second set of training data being labeled with a fine-object classification.Type: GrantFiled: December 11, 2020Date of Patent: June 14, 2022Assignee: Waymo LLCInventors: Junhua Mao, Congcong Li, Yang Song
-
Publication number: 20220156511Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating rare pose data. One of the methods includes obtaining a three-dimensional model of a dynamic object, wherein the dynamic object has multiple movable elements that define a plurality of poses of the dynamic object. A plurality of template poses of the dynamic object are used to generate additional poses for the dynamic object including varying angles of one or more key joints of the dynamic object according to the three-dimensional model. Point cloud data is generated for the additional poses generated for the dynamic object.Type: ApplicationFiled: November 16, 2021Publication date: May 19, 2022Inventors: Xiaohan Jin, Junhua Mao, Ruizhongtai Qi, Congcong Li, Huayi Zeng