Patents by Inventor Sparsh Garg

Sparsh Garg has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20250148757
    Abstract: Systems and methods for a self-improving data engine for autonomous vehicles is presented. To train the self-improving data engine for autonomous vehicles (SIDE), multi-modality dense captioning (MMDC) models can detect unrecognized classes from diversified descriptions for input images. A vision-language-model (VLM) can generate textual features from the diversified descriptions and image features from corresponding images to the diversified descriptions. Curated features, including curated textual features and curated image features, can be obtained by comparing similarity scores between the textual features and top-ranked image features based on their likelihood scores. Generate annotations, including bounding boxes and labels, can be generated for the curated features by comparing the similarity scores of labels generated by a zero-shot classifier and the curated textual features. The SIDE can be trained using the curated features, annotations, and feedback.
    Type: Application
    Filed: October 30, 2024
    Publication date: May 8, 2025
    Inventors: Jong-Chyi Su, Sparsh Garg, Samuel Schulter, Manmohan Chandraker, Mingfu Liang
  • Publication number: 20250118067
    Abstract: Systems and methods include generating a detection output for an image over multiple iterations by applying a dropout randomly to a different convolutional layer of a learning model for each iteration. The detection outputs are clustered, on labels, for each iteration. A total surface area for the clusters is computed over the iteration. A confidence is computed for the image using the total surface area for the clusters as an uncertainty score. A system is disabled if the confidence is below a threshold.
    Type: Application
    Filed: September 17, 2024
    Publication date: April 10, 2025
    Inventors: Sparsh Garg, Samuel Schulter, Yumin Suh
  • Publication number: 20250118044
    Abstract: Systems and methods for identifying novel objects in an image include detecting one or more objects in an image and generating one or more captions for the image. One or more predicted categories of the one or more objects detected in the image and the one or more captions are matched to identify, from the one or more predicted categories, a category of a novel object in the image. An image feature and a text description feature are generated using a description of the novel object. A relevant image is selected using a similarity score between the image feature and the text description feature. A model is updated using the relevant image and associated description of the novel object.
    Type: Application
    Filed: September 20, 2024
    Publication date: April 10, 2025
    Inventors: Jong-Chyi Su, Samuel Schulter, Sparsh Garg, Manmohan Chandraker, Mingfu Liang
  • Publication number: 20250118063
    Abstract: Systems and methods include detecting one or more objects in an image and generating one or more captions for the image. One or more predicted categories of the one or more objects detected in the image and the one or more captions are matched. From the one or more predicted categories, a category that is not successfully predicted in the image is identified. Data is curated to improve the category that is not successfully predicted in the image. A perception model is finetuned using data curated.
    Type: Application
    Filed: September 20, 2024
    Publication date: April 10, 2025
    Inventors: Jong-Chyi Su, Samuel Schulter, Sparsh Garg, Manmohan Chandraker, Mingfu Liang
  • Patent number: 12254681
    Abstract: Systems and methods are provided for multi-modal test-time adaptation. The method includes inputting a digital image into a pre-trained Camera Intra-modal Pseudo-label Generator, and inputting a point cloud set into a pre-trained Lidar Intra-modal Pseudo-label Generator. The method further includes applying a fast 2-dimension (2D) model, and a slow 2D model, to the inputted digital image to apply pseudo-labels, and applying a fast 3-dimension (3D) model, and a slow 3D model, to the inputted point cloud set to apply pseudo-labels. The method further includes fusing pseudo-label predictions from the fast models and the slow models through an Inter-modal Pseudo-label Refinement module to obtain robust pseudo labels, and measuring a prediction consistency for the pseudo-labels.
    Type: Grant
    Filed: September 6, 2022
    Date of Patent: March 18, 2025
    Assignee: NEC Corporation
    Inventors: Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, Ramin Moslemi, Inkyu Shin
  • Patent number: 12205356
    Abstract: Methods and systems for detecting faults include capturing an image of a scene using a camera. The image is embedded using a segmentation model that includes an image branch having an image embedding layer that embeds images into a joint latent space and a text branch having a text embedding layer that embeds text into the joint latent space. Semantic information is generated for a region of the image corresponding to a predetermined static object using the embedded image. A fault of the camera is identified based on a discrepancy between the semantic information and semantic information of the predetermined static image. The fault of the camera is corrected.
    Type: Grant
    Filed: March 23, 2023
    Date of Patent: January 21, 2025
    Assignee: NEC Corporation
    Inventors: Samuel Schulter, Sparsh Garg, Manmohan Chandraker
  • Publication number: 20240354921
    Abstract: Systems and methods for road defect level prediction. A depth map is obtained from an image dataset received from input peripherals by employing a vision transformer model. A plurality of semantic maps is obtained from the image dataset by employing a semantic segmentation model to give pixel-wise segmentation results of road scenes to detect road pixels. Regions of interest (ROI) are detected by utilizing the road pixels. Road defect levels are predicted by fitting the ROI and the depth map into a road surface model to generate road points classified into road defect levels. The predicted road defect levels are visualized on a road map.
    Type: Application
    Filed: March 26, 2024
    Publication date: October 24, 2024
    Inventors: Sparsh Garg, Bingbing Zhuang, Samuel Schulter, Manmohan Chandraker
  • Publication number: 20240355102
    Abstract: Systems and methods for traffic violation prediction. The systems and methods include obtaining a plurality of bounding boxes of road scene categories from an input dataset by employing a pre-trained detection model. A plurality of pseudo-labels of road scene categories for the plurality of bounding boxes can be obtained by employing the pre-trained detection model. A labeled dataset can be obtained by filtering the input dataset for images having the plurality of pseudo-labels and the plurality of bounding boxes. A traffic violation prediction model can be trained with both unlabeled and labeled dataset including the road scene categories obtained from the pre-trained detection model to predict simultaneous traffic violations of one or more riders in a road scene.
    Type: Application
    Filed: March 19, 2024
    Publication date: October 24, 2024
    Inventors: Sparsh Garg, Samuel Schulter
  • Publication number: 20240354583
    Abstract: Methods and systems for training a model include annotating a subset of an unlabeled training dataset, that includes images of road scenes, with labels. A road defect detection model is iteratively trained, including adding pseudo-labels to a remainder of examples from the unlabeled training dataset and training the road defect detection model based on the labels and the pseudo-labels.
    Type: Application
    Filed: March 25, 2024
    Publication date: October 24, 2024
    Inventors: Sparsh Garg, Samuel Schulter, Bingbing Zhuang, Manmohan Chandraker
  • Patent number: 12045992
    Abstract: Methods and systems for training a model include combining data from multiple datasets, the datasets having different respective label spaces. Relationships between labels in the different label spaces are identified. A unified neural network model is trained, using the combined data and the identified relationships to generate a unified model, with a class relational binary cross-entropy loss.
    Type: Grant
    Filed: November 5, 2021
    Date of Patent: July 23, 2024
    Assignee: NEC Corporation
    Inventors: Yi-Hsuan Tsai, Masoud Faraki, Yumin Suh, Sparsh Garg, Manmohan Chandraker, Dongwan Kim
  • Publication number: 20240071105
    Abstract: Methods and systems for training a model include pre-training a backbone model with a pre-training decoder, using an unlabeled dataset with multiple distinct sensor data modalities that derive from different sensor types. The backbone model is fine-tuned with an output decoder after pre-training, using a labeled dataset with the multiple modalities.
    Type: Application
    Filed: August 22, 2023
    Publication date: February 29, 2024
    Inventors: Samuel Schulter, Bingbing Zhuang, Vijay Kumar Baikampady Gopalkrishna, Sparsh Garg, Zhixing Zhang
  • Publication number: 20230281999
    Abstract: Methods and systems identifying road hazards include capturing an image of a road scene using a camera. The image is embedded using a segmentation model that includes an image branch having an image embedding layer that embeds images into a joint latent space and a text branch having a text embedding layer that embeds text into the joint latent space. A mask is generated for an object within the image using the segmentation model. A probability is determined that the object matches a road hazard using the segmentation mode. A signal is generated responsive to the probability to ameliorate a danger posed by the road hazard.
    Type: Application
    Filed: March 23, 2023
    Publication date: September 7, 2023
    Inventors: Samuel Schulter, Sparsh Garg
  • Publication number: 20230281977
    Abstract: Methods and systems for detecting faults include capturing an image of a scene using a camera. The image is embedded using a segmentation model that includes an image branch having an image embedding layer that embeds images into a joint latent space and a text branch having a text embedding layer that embeds text into the joint latent space. Semantic information is generated for a region of the image corresponding to a predetermined static object using the embedded image. A fault of the camera is identified based on a discrepancy between the semantic information and semantic information of the predetermined static image. The fault of the camera is corrected.
    Type: Application
    Filed: March 23, 2023
    Publication date: September 7, 2023
    Inventors: Samuel Schulter, Sparsh Garg, Manmohan Chandraker
  • Publication number: 20230088335
    Abstract: Systems and methods is provided for road hazard analysis. The method includes obtaining sensor data of a road environment including a road and observable surroundings, and applying labels to the sensor data. The method further includes training a first neural network model to identify road hazards, training a second neural network model to identify faded lane markings, and training a third neural network model to identify overhanging trees and blocking foliage. The method further includes implementing the trained neural network models to detect road hazards in a real road setting.
    Type: Application
    Filed: September 9, 2022
    Publication date: March 23, 2023
    Inventors: Sparsh Garg, Samuel Schulter, Vijay Kumar Baikampady Gopalkrishna
  • Publication number: 20230081913
    Abstract: Systems and methods are provided for multi-modal test-time adaptation. The method includes inputting a digital image into a pre-trained Camera Intra-modal Pseudo-label Generator, and inputting a point cloud set into a pre-trained Lidar Intra-modal Pseudo-label Generator. The method further includes applying a fast 2-dimension (2D) model, and a slow 2D model, to the inputted digital image to apply pseudo-labels, and applying a fast 3-dimension (3D) model, and a slow 3D model, to the inputted point cloud set to apply pseudo-labels. The method further includes fusing pseudo-label predictions from the fast models and the slow models through an Inter-modal Pseudo-label Refinement module to obtain robust pseudo labels, and measuring a prediction consistency for the pseudo-labels.
    Type: Application
    Filed: September 6, 2022
    Publication date: March 16, 2023
    Inventors: Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, Ramin Moslemi, Inkyu Shin
  • Publication number: 20230073055
    Abstract: A computer-implemented method for rut detection is provided. The method includes detecting, by a rut detection system, areas in a road-scene image that include ruts with pixel-wise probability values, wherein a higher value indicates a better chance of being a rut. The method further includes performing at least one of rut repair and vehicle rut avoidance responsive to the pixel-wise probability values. The detecting step includes performing neural network-based, pixel-wise semantic segmentation with context information on the road-scene image to distinguish rut pixels from non-rut pixels on a road depicted in the road-scene image.
    Type: Application
    Filed: September 6, 2022
    Publication date: March 9, 2023
    Inventors: Yi-Hsuan Tsai, Sparsh Garg, Manmohan Chandraker, Samuel Shulter, Vijay Kumar Baikampady Gopalkrishna
  • Publication number: 20220148189
    Abstract: Methods and systems for training a model include combining data from multiple datasets, the datasets having different respective label spaces. Relationships between labels in the different label spaces are identified. A unified neural network model is trained, using the combined data and the identified relationships to generate a unified model, with a class relational binary cross-entropy loss.
    Type: Application
    Filed: November 5, 2021
    Publication date: May 12, 2022
    Inventors: Yi-Hsuan Tsai, Masoud Faraki, Yumin Suh, Sparsh Garg, Manmohan Chandraker, Dongwan Kim