Patents by Inventor Henrik Kretzschmar

Henrik Kretzschmar has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20230334842
    Abstract: Methods, systems, and apparatus for processing inputs that include video frames using neural networks. In one aspect, a system comprises one or more computers configured to obtain a set of one or more training images and, for each training image, ground truth instance data that identifies, for each of one or more object instances, a corresponding region of the training image that depicts the object instance. For each training image in the set, the one or more computers process the training image using an instance segmentation neural network to generate an embedding output comprising a respective embedding for each of a plurality of output pixels. The one or more computers then train the instance segmentation neural network to minimize a loss function.
    Type: Application
    Filed: April 18, 2023
    Publication date: October 19, 2023
    Inventors: Alex Zihao Zhu, Vincent Michael Casser, Henrik Kretzschmar, Reza Mahjourian, Soeren Pirk
  • Publication number: 20230281824
    Abstract: Methods, systems, and apparatus for generating a panoptic segmentation label for a sensor data sample. In one aspect, a system comprises one or more computers configured to obtain a sensor data sample characterizing a scene in an environment. The one or more computers obtain a 3D bounding box annotation at each time point for a point cloud characterizing the scene at the time point. The one or more computers obtain, for each camera image and each time point, annotation data identifying object instances depicted in the camera image, and the one or more computers generate a panoptic segmentation label for the sensor data sample characterizing the scene in the environment.
    Type: Application
    Filed: March 7, 2023
    Publication date: September 7, 2023
    Inventors: Jieru Mei, Hang Yan, Liang-Chieh Chen, Siyuan Qiao, Yukun Zhu, Alex Zihao Zhu, Xinchen Yan, Henrik Kretzschmar
  • Publication number: 20230213643
    Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for processing sensor data. In one aspect, a method includes obtaining image data representing a camera sensor measurement of a scene; obtaining radar data representing a radar sensor measurement of the scene; generating a feature representation of the image data; generating a respective initial depth estimate for each of a subset of the plurality of pixels; generating a feature representation of the radar data; for each of the subset of the plurality of pixels, generating a respective adjusted depth estimate for the pixel using the initial depth estimate for the pixel and the radar feature vectors for a corresponding subset of the plurality of radar reflection points; generating a fused point cloud that includes a plurality of three-dimensional data points; and processing the fused point cloud to generate an output that characterizes the scene.
    Type: Application
    Filed: December 7, 2022
    Publication date: July 6, 2023
    Inventors: Jyh-Jing Hwang, Henrik Kretzschmar, Dragomir Anguelov
  • Publication number: 20230177822
    Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for rendering a new image that depicts a scene from a perspective of a camera at a new camera viewpoint.
    Type: Application
    Filed: December 2, 2022
    Publication date: June 8, 2023
    Inventors: Vincent Michael Casser, Henrik Kretzschmar, Matthew Justin Tancik, Sabeek Mani Pradhan, Benjamin Joseph Mildenhall, Pratul Preeti Srinivasan, Jonathan Tilton Barron
  • Patent number: 11562573
    Abstract: Aspects of the disclosure relate to training and using a phrase recognition model to identify phrases in images. As an example, a selected phrase list may include a plurality of phrases is received. Each phrase of the plurality of phrases includes text. An initial plurality of images may be received. A training image set may be selected from the initial plurality of images by identifying the phrase-containing images that include one or more phrases from the selected phrase list. Each given phrase-containing image of the training image set may be labeled with information identifying the one or more phrases from the selected phrase list included in the given phrase-containing images. The model may be trained based on the training image set such that the model is configured to, in response to receiving an input image, output data indicating whether a phrase of the plurality of phrases is included in the input image.
    Type: Grant
    Filed: December 16, 2020
    Date of Patent: January 24, 2023
    Assignee: Waymo LLC
    Inventors: Victoria Dean, Abhijit S Ogale, Henrik Kretzschmar, David Harrison Silver, Carl Kershaw, Pankaj Chaudhari, Chen Wu, Congcong Li
  • Publication number: 20220366175
    Abstract: Aspects of the disclosure relate to controlling a vehicle. For instance, using a camera, a first camera image including a first object may be captured. A first bounding box for the first object and a distance to the first object may be identified. A second camera image including a second object may be captured. A second bounding box for the second image and a distance to the second object may be identified. Whether the first object is the second object may be determined using a plurality of models to compare visual similarity of the two bounding boxes, to compare a three-dimensional location based on the distance to the first object and a three-dimensional location based on the distance to the second object, and to compare results from the first and second models. The vehicle may be controlled in an autonomous driving mode based on a result of the third model.
    Type: Application
    Filed: May 13, 2021
    Publication date: November 17, 2022
    Applicant: WAYMO LLC
    Inventors: Ruichi Yu, Kang Li, Tao Han, Robert Cosgriff, Henrik Kretzschmar
  • Publication number: 20220358314
    Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for generating and editing object track labels for objects detected in video data. One of the methods includes obtaining a video segment comprising multiple image frames associated with multiple time points; obtaining object track data specifying a set of object tracks; providing, for presentation to a user, a user interface for modifying the object track data, the user interface displaying object timeline representations of the object tracks; receiving one or more user inputs that indicate one or more modifications to the object timeline representations; updating the object timeline representations displayed in the timeline display area; and updating the object track data according to the updated object timeline representations.
    Type: Application
    Filed: May 7, 2021
    Publication date: November 10, 2022
    Inventors: Yulai Shen, Henrik Kretzschmar, Jeffrey Sham, Jeffrey Carlson, Lo Po Tsui, Dragomir Anguelov
  • Publication number: 20220180549
    Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting three-dimensional object locations from images. One of the methods includes obtaining a sequence of images that comprises, at each of a plurality of time steps, a respective image that was captured by a camera at the time step; generating, for each image in the sequence, respective pseudo-lidar features of a respective pseudo-lidar representation of a region in the image that has been determined to depict a first object; generating, for a particular image at a particular time step in the sequence, image patch features of the region in the particular image that has been determined to depict the first object; and generating, from the respective pseudo-lidar features and the image patch features, a prediction that characterizes a location of the first object in a three-dimensional coordinate system at the particular time step in the sequence.
    Type: Application
    Filed: December 8, 2021
    Publication date: June 9, 2022
    Inventors: Longlong Jing, Ruichi Yu, Jiyang Gao, Henrik Kretzschmar, Kang Li, Ruizhongtai Qi, Hang Zhao, Alper Ayvaci, Xu Chen, Dillon Cower, Congcong Li
  • Publication number: 20210390407
    Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for training a perspective computer vision model. The model is configured to receive input data characterizing an input scene in an environment from an input viewpoint and to process the input data in accordance with a set of model parameters to generate an output perspective representation of the scene from the input viewpoint. The system trains the model based on first data characterizing a scene in the environment from a first viewpoint and second data characterizing the scene in the environment from a second, different viewpoint.
    Type: Application
    Filed: June 10, 2021
    Publication date: December 16, 2021
    Inventors: Vincent Michael Casser, Yuning Chai, Dragomir Anguelov, Hang Zhao, Henrik Kretzschmar, Reza Mahjourian, Anelia Angelova, Ariel Gordon, Soeren Pirk
  • Publication number: 20210192238
    Abstract: Aspects of the disclosure relate to training and using a phrase recognition model to identify phrases in images. As an example, a selected phrase list may include a plurality of phrases is received. Each phrase of the plurality of phrases includes text. An initial plurality of images may be received. A training image set may be selected from the initial plurality of images by identifying the phrase-containing images that include one or more phrases from the selected phrase list. Each given phrase-containing image of the training image set may be labeled with information identifying the one or more phrases from the selected phrase list included in the given phrase-containing images. The model may be trained based on the training image set such that the model is configured to, in response to receiving an input image, output data indicating whether a phrase of the plurality of phrases is included in the input image.
    Type: Application
    Filed: December 16, 2020
    Publication date: June 24, 2021
    Applicant: WAYMO LLC
    Inventors: Victoria Dean, Abhijit S. Ogale, Henrik Kretzschmar, David Harrison Silver, Carl Kershaw, Pankaj Chaudhari, Chen Wu, Congcong Li
  • Publication number: 20210150799
    Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generated simulated sensor data. One of the methods includes obtaining a surfel map generated from sensor observations of a real-world environment and generating, for each surfel in the surfel map, a respective grid having a plurality of grid cells, wherein each grid has an orientation matching an orientation of a corresponding surfel, and wherein each grid cell within each grid is assigned a respective color value. For a simulated location within a simulated representation of the real-world environment, a textured surfel rendering is generated, including combining color information from grid cells visible from the simulated location within the simulated representation of the real-world environment.
    Type: Application
    Filed: November 16, 2020
    Publication date: May 20, 2021
    Inventors: Zhenpei Yang, Yuning Chai, Yin Zhou, Pei Sun, Henrik Kretzschmar, Sean Rafferty, Dumitru Erhan, Dragomir Anguelov
  • Publication number: 20210150349
    Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for multi object tracking using memory attention.
    Type: Application
    Filed: November 16, 2020
    Publication date: May 20, 2021
    Inventors: Wei-Chih Hung, Henrik Kretzschmar, Yuning Chai, Dragomir Anguelov
  • Patent number: 10902272
    Abstract: Aspects of the disclosure relate to training and using a phrase recognition model to identify phrases in images. As an example, a selected phrase list may include a plurality of phrases is received. Each phrase of the plurality of phrases includes text. An initial plurality of images may be received. A training image set may be selected from the initial plurality of images by identifying the phrase-containing images that include one or more phrases from the selected phrase list. Each given phrase-containing image of the training image set may be labeled with information identifying the one or more phrases from the selected phrase list included in the given phrase-containing images. The model may be trained based on the training image set such that the model is configured to, in response to receiving an input image, output data indicating whether a phrase of the plurality of phrases is included in the input image.
    Type: Grant
    Filed: May 20, 2020
    Date of Patent: January 26, 2021
    Assignee: WAYMO LLC
    Inventors: Victoria Dean, Abhijit S. Ogale, Henrik Kretzschmar, David Harrison Silver, Carl Kershaw, Pankaj Chaudhari, Chen Wu, Congcong Li
  • Publication number: 20200356794
    Abstract: Aspects of the disclosure relate to training and using a phrase recognition model to identify phrases in images. As an example, a selected phrase list may include a plurality of phrases is received. Each phrase of the plurality of phrases includes text. An initial plurality of images may be received. A training image set may be selected from the initial plurality of images by identifying the phrase-containing images that include one or more phrases from the selected phrase list. Each given phrase-containing image of the training image set may be labeled with information identifying the one or more phrases from the selected phrase list included in the given phrase-containing images. The model may be trained based on the training image set such that the model is configured to, in response to receiving an input image, output data indicating whether a phrase of the plurality of phrases is included in the input image.
    Type: Application
    Filed: May 20, 2020
    Publication date: November 12, 2020
    Inventors: Victoria Dean, Abhijit S. Ogale, Henrik Kretzschmar, David Harrison Silver, Carl Kershaw, Pankaj Chaudhari, Chen Wu, Congcong Li
  • Patent number: 10699141
    Abstract: Aspects of the disclosure relate to training and using a phrase recognition model to identify phrases in images. As an example, a selected phrase list may include a plurality of phrases is received. Each phrase of the plurality of phrases includes text. An initial plurality of images may be received. A training image set may be selected from the initial plurality of images by identifying the phrase-containing images that include one or more phrases from the selected phrase list. Each given phrase-containing image of the training image set may be labeled with information identifying the one or more phrases from the selected phrase list included in the given phrase-containing images. The model may be trained based on the training image set such that the model is configured to, in response to receiving an input image, output data indicating whether a phrase of the plurality of phrases is included in the input image.
    Type: Grant
    Filed: June 26, 2018
    Date of Patent: June 30, 2020
    Assignee: Waymo LLC
    Inventors: Victoria Dean, Abhijit S. Ogale, Henrik Kretzschmar, David Harrison Silver, Carl Kershaw, Pankaj Chaudhari, Chen Wu, Congcong Li
  • Publication number: 20190392231
    Abstract: Aspects of the disclosure relate to training and using a phrase recognition model to identify phrases in images. As an example, a selected phrase list may include a plurality of phrases is received. Each phrase of the plurality of phrases includes text. An initial plurality of images may be received. A training image set may be selected from the initial plurality of images by identifying the phrase-containing images that include one or more phrases from the selected phrase list. Each given phrase-containing image of the training image set may be labeled with information identifying the one or more phrases from the selected phrase list included in the given phrase-containing images. The model may be trained based on the training image set such that the model is configured to, in response to receiving an input image, output data indicating whether a phrase of the plurality of phrases is included in the input image.
    Type: Application
    Filed: June 26, 2018
    Publication date: December 26, 2019
    Inventors: Victoria Dean, Abhijit S. Ogale, Henrik Kretzschmar, David Harrison Silver, Carl Kershaw, Pankaj Chaudhari, Chen Wu, Congcong Li
  • Patent number: 9014905
    Abstract: Methods and systems for detecting hand signals of a cyclist by an autonomous vehicle are described. An example method may involve a computing device receiving a plurality of data points corresponding to an environment of an autonomous vehicle. The computing device may then determine one or more subsets of data points from the plurality of data points indicative of at least a body region of a cyclist. Further, based on an output of a comparison of the one or more subsets with one or more predetermined sets of cycling signals, the computing device may determine an expected adjustment of one or more of a speed of the cyclist and a direction of movement of the cyclist. Still further, based on the expected adjustment, the computing device may provide instructions to adjust one or more of a speed of the autonomous vehicle and a direction of movement of the autonomous vehicle.
    Type: Grant
    Filed: January 28, 2014
    Date of Patent: April 21, 2015
    Assignee: Google Inc.
    Inventors: Henrik Kretzschmar, Jiajun Zhu