Patents by Inventor Buyu Liu

Buyu Liu has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20250118009
    Abstract: A computer-implemented method for synthesizing an image includes capturing data from a scene and fusing grid-based representations of the scene from different encodings to inherit beneficial properties of the different encodings, The encodings include Lidar encoding and a high definition map encoding. Rays are rendered from fused grid-based representations. A density and color are determined for points in the rays. A volume rendering is employed for the rays with the density and color. An image is synthesized from the volume rendered rays with the density and the color.
    Type: Application
    Filed: October 1, 2024
    Publication date: April 10, 2025
    Inventors: Bingbing Zhuang, Ziyu Jiang, Buyu Liu, Manmohan Chandraker, Shanlin Sun
  • Publication number: 20250115254
    Abstract: Systems and methods for a hybrid motion planner for autonomous vehicles. A multi-lane intelligent driver model (MIDM) can predict trajectory predictions from collected data by considering adjacent lanes of an ego vehicle. A multi-lane hybrid planning driver model (MPDM) can be trained using open-loop ground truth data and close-loop simulations to obtain a trained MPDM. The trained MPDM can predict planned trajectories with collected data and the trajectory predictions to generate final trajectories for the autonomous vehicles. The final trajectories can be employed to control the autonomous vehicles.
    Type: Application
    Filed: October 3, 2024
    Publication date: April 10, 2025
    Inventors: Buyu Liu, Francesco Pittaluga, Bingbing Zhuang, Manmohan Chandraker, Samuel Sohn
  • Publication number: 20250115278
    Abstract: Systems and methods for generating adversarial driving scenarios for autonomous vehicles. An artificial intelligence model can compute an adversarial loss function by minimizing the distance between predicted adversarial perturbed trajectories and corresponding generated neighbor future trajectories from input data. A traffic violation loss function can be computed based on observed adversarial agents adhering to driving rules from the input data. A comfort loss function can be computed based on the predicted driving characteristics of adversarial vehicles relevant to comfort of hypothetical passengers from the input data. A planner module can be trained for autonomous vehicles based on a combined loss function of the adversarial loss function, the traffic violation loss function and the comfort loss function to generate adversarial driving scenarios. An autonomous vehicle can be controlled based on trajectories generated in the adversarial driving scenarios.
    Type: Application
    Filed: October 3, 2024
    Publication date: April 10, 2025
    Inventors: Francesco Pittaluga, Buyu Liu, Manmohan Chandraker, Kaiyuan Zhang
  • Patent number: 12254681
    Abstract: Systems and methods are provided for multi-modal test-time adaptation. The method includes inputting a digital image into a pre-trained Camera Intra-modal Pseudo-label Generator, and inputting a point cloud set into a pre-trained Lidar Intra-modal Pseudo-label Generator. The method further includes applying a fast 2-dimension (2D) model, and a slow 2D model, to the inputted digital image to apply pseudo-labels, and applying a fast 3-dimension (3D) model, and a slow 3D model, to the inputted point cloud set to apply pseudo-labels. The method further includes fusing pseudo-label predictions from the fast models and the slow models through an Inter-modal Pseudo-label Refinement module to obtain robust pseudo labels, and measuring a prediction consistency for the pseudo-labels.
    Type: Grant
    Filed: September 6, 2022
    Date of Patent: March 18, 2025
    Assignee: NEC Corporation
    Inventors: Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, Ramin Moslemi, Inkyu Shin
  • Patent number: 12131422
    Abstract: A method for achieving high-fidelity novel view synthesis and 3D reconstruction for large-scale scenes is presented. The method includes obtaining images from a video stream received from a plurality of video image capturing devices, grouping the images into different image clusters representing a large-scale 3D scene, training a neural radiance field (NeRF) and an uncertainty multilayer perceptron (MLP) for each of the image clusters to generate a plurality of NeRFs and a plurality of uncertainty MLPs for the large-scale 3D scene, applying a rendering loss and an entropy loss to the plurality of NeRFs, performing uncertainty-based fusion to the plurality of NeRFs to define a fused NeRF, and jointly fine-tuning the plurality of NeRFs and the plurality of uncertainty MLPs, and during inference, applying the fused NeRF for novel view synthesis of the large-scale 3D scene.
    Type: Grant
    Filed: October 11, 2022
    Date of Patent: October 29, 2024
    Assignee: NEC Corporation
    Inventors: Bingbing Zhuang, Samuel Schulter, Yi-Hsuan Tsai, Buyu Liu, Nanbo Li
  • Patent number: 12131557
    Abstract: A computer-implemented method for road layout prediction is provided. The method includes segmenting, by a first processor-based element, an RGB image to output pixel-level semantic segmentation results for the RGB image in a perspective view for both visible and occluded pixels in the perspective view based on contextual clues. The method further includes learning, by a second processor-based element, a mapping from the pixel-level semantic segmentation results for the RGB image in the perspective view to a top view of the RGB image using a road plane assumption. The method also includes generating, by a third processor-based element, an occlusion-aware parametric road layout prediction for road layout related attributes in the top view.
    Type: Grant
    Filed: November 8, 2021
    Date of Patent: October 29, 2024
    Assignee: NEC Corporation
    Inventors: Buyu Liu, Bingbing Zhuang, Manmohan Chandraker
  • Publication number: 20240351582
    Abstract: Methods and systems for trajectory prediction include encoding trajectories of agents in a scene from past images of the scene. Lane centerlines are encoded for agents in the scene. The agents in the scene are encoded using the encoded trajectories and the encoded lane centerlines. A hypercolumn trajectory is decoded from the encoded agents to generate predicted trajectories for the agents. A vehicle is automatically operated responsive to the predicted trajectories.
    Type: Application
    Filed: April 18, 2024
    Publication date: October 24, 2024
    Inventors: Buyu Liu, Sriram Nochur Narayanan, Bingbing Zhuang, Yumin Suh
  • Patent number: 11987236
    Abstract: A method provided for 3D object localization predicts pairs of 2D bounding boxes. Each pair corresponds to a detected object in each of the two consecutive input monocular images. The method generates, for each detected object, a relative motion estimation specifying a relative motion between the two images. The method constructs an object cost volume by aggregating temporal features from the two images using the pairs of 2D bounding boxes and the relative motion estimation to predict a range of object depth candidates and a confidence score for each object depth candidate and an object depth from the object depth candidates. The method updates the relative motion estimation based on the object cost volume and the object depth to provide a refined object motion and a refined object depth. The method reconstructs a 3D bounding box for each detected object based on the refined object motion and refined object depth.
    Type: Grant
    Filed: August 23, 2021
    Date of Patent: May 21, 2024
    Assignee: NEC Corporation
    Inventors: Pan Ji, Buyu Liu, Bingbing Zhuang, Manmohan Chandraker, Xiangyu Chen
  • Publication number: 20240153250
    Abstract: Methods and systems for training a model include training a size estimation model to generate an estimated object size using a training dataset with differing levels of annotation. Two-dimensional object detection is performed on a training image to identify an object. The training image is cropped around the object. A category-level shape reconstruction is generated using a neural radiance field model. A normalized coordinate model is trained using the training image and ground truth information from the category-level shape reconstruction.
    Type: Application
    Filed: November 1, 2023
    Publication date: May 9, 2024
    Inventors: Bingbing Zhuang, Samuel Schulter, Buyu Liu, Zhixiang Min
  • Publication number: 20240153251
    Abstract: Methods and systems for training a model include performing two-dimensional object detection on a training image to identify an object. The training image is cropped around the object. A category-level shape reconstruction is generated using a neural radiance field model. A normalized coordinate model is trained using the training image and ground truth information from the category-level shape reconstruction.
    Type: Application
    Filed: November 1, 2023
    Publication date: May 9, 2024
    Inventors: Bingbing Zhuang, Samuel Schulter, Buyu Liu, Zhixiang Min
  • Patent number: 11816901
    Abstract: Methods and systems for training a trajectory prediction model and performing a vehicle maneuver include encoding a set of training data to generate encoded training vectors, where the training data includes trajectory information for agents over time. Trajectory scenarios are simulated based on the encoded training vectors, with each simulated trajectory scenario representing one or more agents with respective agent trajectories, to generate simulated training data. A predictive neural network model is trained using the simulated training data to generate predicted trajectory scenarios based on a detected scene.
    Type: Grant
    Filed: February 26, 2021
    Date of Patent: November 14, 2023
    Inventors: Sriram Nochur Narayanan, Buyu Liu, Ramin Moslemi, Francesco Pittaluga, Manmohan Chandraker
  • Publication number: 20230154104
    Abstract: A method for achieving high-fidelity novel view synthesis and 3D reconstruction for large-scale scenes is presented. The method includes obtaining images from a video stream received from a plurality of video image capturing devices, grouping the images into different image clusters representing a large-scale 3D scene, training a neural radiance field (NeRF) and an uncertainty multilayer perceptron (MLP) for each of the image clusters to generate a plurality of NeRFs and a plurality of uncertainty MLPs for the large-scale 3D scene, applying a rendering loss and an entropy loss to the plurality of NeRFs, performing uncertainty-based fusion to the plurality of NeRFs to define a fused NeRF, and jointly fine-tuning the plurality of NeRFs and the plurality of uncertainty MLPs, and during inference, applying the fused NeRF for novel view synthesis of the large-scale 3D scene.
    Type: Application
    Filed: October 11, 2022
    Publication date: May 18, 2023
    Inventors: Bingbing Zhuang, Samuel Schulter, Yi-Hsuan Tsai, Buyu Liu, Nanbo Li
  • Patent number: 11610420
    Abstract: Systems and methods for human detection are provided. The system aligns image level features between a source domain and a target domain based on an adversarial learning process while training a domain discriminator. The target domain includes humans in one or more different scenes. The system selects, using the domain discriminator, unlabeled samples from the target domain that are far away from existing annotated samples from the target domain. The system selects, based on a prediction score of each of the unlabeled samples, samples with lower prediction scores. The system annotates the samples with the lower prediction scores.
    Type: Grant
    Filed: December 21, 2020
    Date of Patent: March 21, 2023
    Inventors: Yi-Hsuan Tsai, Kihyuk Sohn, Buyu Liu, Manmohan Chandraker, Jong-Chyi Su
  • Publication number: 20230081913
    Abstract: Systems and methods are provided for multi-modal test-time adaptation. The method includes inputting a digital image into a pre-trained Camera Intra-modal Pseudo-label Generator, and inputting a point cloud set into a pre-trained Lidar Intra-modal Pseudo-label Generator. The method further includes applying a fast 2-dimension (2D) model, and a slow 2D model, to the inputted digital image to apply pseudo-labels, and applying a fast 3-dimension (3D) model, and a slow 3D model, to the inputted point cloud set to apply pseudo-labels. The method further includes fusing pseudo-label predictions from the fast models and the slow models through an Inter-modal Pseudo-label Refinement module to obtain robust pseudo labels, and measuring a prediction consistency for the pseudo-labels.
    Type: Application
    Filed: September 6, 2022
    Publication date: March 16, 2023
    Inventors: Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, Ramin Moslemi, Inkyu Shin
  • Patent number: 11604945
    Abstract: Systems and methods for lane marking and road sign recognition are provided. The system aligns image level features between a source domain and a target domain based on an adversarial learning process while training a domain discriminator. The target domain includes one or more road scenes having lane markings and road signs. The system selects, using the domain discriminator, unlabeled samples from the target domain that are far away from existing annotated samples from the target domain. The system selects, based on a prediction score of each of the unlabeled samples, samples with lower prediction scores. The system annotates the samples with the lower prediction scores.
    Type: Grant
    Filed: December 21, 2020
    Date of Patent: March 14, 2023
    Inventors: Yi-Hsuan Tsai, Kihyuk Sohn, Buyu Liu, Manmohan Chandraker, Jong-Chyi Su
  • Patent number: 11594041
    Abstract: Systems and methods for obstacle detection are provided. The system aligns image level features between a source domain and a target domain based on an adversarial learning process while training a domain discriminator. The target domain includes one or more road scenes having obstacles. The system selects, using the domain discriminator, unlabeled samples from the target domain that are far away from existing annotated samples from the target domain. The system selects, based on a prediction score of each of the unlabeled samples, samples with lower prediction scores. The system annotates the samples with the lower prediction scores.
    Type: Grant
    Filed: December 21, 2020
    Date of Patent: February 28, 2023
    Inventors: Yi-Hsuan Tsai, Kihyuk Sohn, Buyu Liu, Manmohan Chandraker, Jong-Chyi Su
  • Patent number: 11580334
    Abstract: Systems and methods for construction zone segmentation are provided. The system aligns image level features between a source domain and a target domain based on an adversarial learning process while training a domain discriminator. The target domain includes construction zones scenes having various objects. The system selects, using the domain discriminator, unlabeled samples from the target domain that are far away from existing annotated samples from the target domain. The system selects, based on a prediction score of each of the unlabeled samples, samples with lower prediction scores. The system annotates the samples with the lower prediction scores.
    Type: Grant
    Filed: December 21, 2020
    Date of Patent: February 14, 2023
    Inventors: Yi-Hsuan Tsai, Kihyuk Sohn, Buyu Liu, Manmohan Chandraker, Jong-Chyi Su
  • Patent number: 11462112
    Abstract: A method is provided in an Advanced Driver-Assistance System (ADAS). The method extracts, from an input video stream including a plurality of images using a multi-task Convolutional Neural Network (CNN), shared features across different perception tasks. The perception tasks include object detection and other perception tasks. The method concurrently solves, using the multi-task CNN, the different perception tasks in a single pass by concurrently processing corresponding ones of the shared features by respective different branches of the multi-task CNN to provide a plurality of different perception task outputs. Each respective different branch corresponds to a respective one of the different perception tasks. The method forms a parametric representation of a driving scene as at least one top-view map responsive to the plurality of different perception task outputs.
    Type: Grant
    Filed: February 11, 2020
    Date of Patent: October 4, 2022
    Inventors: Quoc-Huy Tran, Samuel Schulter, Paul Vernaza, Buyu Liu, Pan Ji, Yi-Hsuan Tsai, Manmohan Chandraker
  • Patent number: 11455813
    Abstract: Systems and methods are provided for producing a road layout model. The method includes capturing digital images having a perspective view, converting each of the digital images into top-down images, and conveying a top-down image of time t to a neural network that performs a feature transform to form a feature map of time t. The method also includes transferring the feature map of the top-down image of time t to a feature transform module to warp the feature map to a time t+1, and conveying a top-down image of time t+1 to form a feature map of time t+1. The method also includes combining the warped feature map of time t with the feature map of time t+1 to form a combined feature map, transferring the combined feature map to a long short-term memory (LSTM) module to generate the road layout model, and displaying the road layout model.
    Type: Grant
    Filed: November 12, 2020
    Date of Patent: September 27, 2022
    Inventors: Buyu Liu, Bingbing Zhuang, Samuel Schulter, Manmohan Chandraker
  • Patent number: 11373067
    Abstract: A method for implementing parametric models for scene representation to improve autonomous task performance includes generating an initial map of a scene based on at least one image corresponding to a perspective view of the scene, the initial map including a non-parametric top-view representation of the scene, implementing a parametric model to obtain a scene element representation based on the initial map, the scene element representation providing a description of one or more scene elements of the scene and corresponding to an estimated semantic layout of the scene, identifying one or more predicted locations of the one or more scene elements by performing three-dimensional localization based on the at least one image, and obtaining an overlay for performing an autonomous task by placing the one or more scene elements with the one or more respective predicted locations onto the scene element representation.
    Type: Grant
    Filed: July 30, 2019
    Date of Patent: June 28, 2022
    Inventors: Samuel Schulter, Ziyan Wang, Buyu Liu, Manmohan Chandraker