Patents by Inventor Sparsh Garg
Sparsh Garg has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Publication number: 20250148757Abstract: Systems and methods for a self-improving data engine for autonomous vehicles is presented. To train the self-improving data engine for autonomous vehicles (SIDE), multi-modality dense captioning (MMDC) models can detect unrecognized classes from diversified descriptions for input images. A vision-language-model (VLM) can generate textual features from the diversified descriptions and image features from corresponding images to the diversified descriptions. Curated features, including curated textual features and curated image features, can be obtained by comparing similarity scores between the textual features and top-ranked image features based on their likelihood scores. Generate annotations, including bounding boxes and labels, can be generated for the curated features by comparing the similarity scores of labels generated by a zero-shot classifier and the curated textual features. The SIDE can be trained using the curated features, annotations, and feedback.Type: ApplicationFiled: October 30, 2024Publication date: May 8, 2025Inventors: Jong-Chyi Su, Sparsh Garg, Samuel Schulter, Manmohan Chandraker, Mingfu Liang
-
Publication number: 20250118067Abstract: Systems and methods include generating a detection output for an image over multiple iterations by applying a dropout randomly to a different convolutional layer of a learning model for each iteration. The detection outputs are clustered, on labels, for each iteration. A total surface area for the clusters is computed over the iteration. A confidence is computed for the image using the total surface area for the clusters as an uncertainty score. A system is disabled if the confidence is below a threshold.Type: ApplicationFiled: September 17, 2024Publication date: April 10, 2025Inventors: Sparsh Garg, Samuel Schulter, Yumin Suh
-
Publication number: 20250118044Abstract: Systems and methods for identifying novel objects in an image include detecting one or more objects in an image and generating one or more captions for the image. One or more predicted categories of the one or more objects detected in the image and the one or more captions are matched to identify, from the one or more predicted categories, a category of a novel object in the image. An image feature and a text description feature are generated using a description of the novel object. A relevant image is selected using a similarity score between the image feature and the text description feature. A model is updated using the relevant image and associated description of the novel object.Type: ApplicationFiled: September 20, 2024Publication date: April 10, 2025Inventors: Jong-Chyi Su, Samuel Schulter, Sparsh Garg, Manmohan Chandraker, Mingfu Liang
-
Publication number: 20250118063Abstract: Systems and methods include detecting one or more objects in an image and generating one or more captions for the image. One or more predicted categories of the one or more objects detected in the image and the one or more captions are matched. From the one or more predicted categories, a category that is not successfully predicted in the image is identified. Data is curated to improve the category that is not successfully predicted in the image. A perception model is finetuned using data curated.Type: ApplicationFiled: September 20, 2024Publication date: April 10, 2025Inventors: Jong-Chyi Su, Samuel Schulter, Sparsh Garg, Manmohan Chandraker, Mingfu Liang
-
Patent number: 12254681Abstract: Systems and methods are provided for multi-modal test-time adaptation. The method includes inputting a digital image into a pre-trained Camera Intra-modal Pseudo-label Generator, and inputting a point cloud set into a pre-trained Lidar Intra-modal Pseudo-label Generator. The method further includes applying a fast 2-dimension (2D) model, and a slow 2D model, to the inputted digital image to apply pseudo-labels, and applying a fast 3-dimension (3D) model, and a slow 3D model, to the inputted point cloud set to apply pseudo-labels. The method further includes fusing pseudo-label predictions from the fast models and the slow models through an Inter-modal Pseudo-label Refinement module to obtain robust pseudo labels, and measuring a prediction consistency for the pseudo-labels.Type: GrantFiled: September 6, 2022Date of Patent: March 18, 2025Assignee: NEC CorporationInventors: Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, Ramin Moslemi, Inkyu Shin
-
Patent number: 12205356Abstract: Methods and systems for detecting faults include capturing an image of a scene using a camera. The image is embedded using a segmentation model that includes an image branch having an image embedding layer that embeds images into a joint latent space and a text branch having a text embedding layer that embeds text into the joint latent space. Semantic information is generated for a region of the image corresponding to a predetermined static object using the embedded image. A fault of the camera is identified based on a discrepancy between the semantic information and semantic information of the predetermined static image. The fault of the camera is corrected.Type: GrantFiled: March 23, 2023Date of Patent: January 21, 2025Assignee: NEC CorporationInventors: Samuel Schulter, Sparsh Garg, Manmohan Chandraker
-
Publication number: 20240354921Abstract: Systems and methods for road defect level prediction. A depth map is obtained from an image dataset received from input peripherals by employing a vision transformer model. A plurality of semantic maps is obtained from the image dataset by employing a semantic segmentation model to give pixel-wise segmentation results of road scenes to detect road pixels. Regions of interest (ROI) are detected by utilizing the road pixels. Road defect levels are predicted by fitting the ROI and the depth map into a road surface model to generate road points classified into road defect levels. The predicted road defect levels are visualized on a road map.Type: ApplicationFiled: March 26, 2024Publication date: October 24, 2024Inventors: Sparsh Garg, Bingbing Zhuang, Samuel Schulter, Manmohan Chandraker
-
Publication number: 20240355102Abstract: Systems and methods for traffic violation prediction. The systems and methods include obtaining a plurality of bounding boxes of road scene categories from an input dataset by employing a pre-trained detection model. A plurality of pseudo-labels of road scene categories for the plurality of bounding boxes can be obtained by employing the pre-trained detection model. A labeled dataset can be obtained by filtering the input dataset for images having the plurality of pseudo-labels and the plurality of bounding boxes. A traffic violation prediction model can be trained with both unlabeled and labeled dataset including the road scene categories obtained from the pre-trained detection model to predict simultaneous traffic violations of one or more riders in a road scene.Type: ApplicationFiled: March 19, 2024Publication date: October 24, 2024Inventors: Sparsh Garg, Samuel Schulter
-
Publication number: 20240354583Abstract: Methods and systems for training a model include annotating a subset of an unlabeled training dataset, that includes images of road scenes, with labels. A road defect detection model is iteratively trained, including adding pseudo-labels to a remainder of examples from the unlabeled training dataset and training the road defect detection model based on the labels and the pseudo-labels.Type: ApplicationFiled: March 25, 2024Publication date: October 24, 2024Inventors: Sparsh Garg, Samuel Schulter, Bingbing Zhuang, Manmohan Chandraker
-
Patent number: 12045992Abstract: Methods and systems for training a model include combining data from multiple datasets, the datasets having different respective label spaces. Relationships between labels in the different label spaces are identified. A unified neural network model is trained, using the combined data and the identified relationships to generate a unified model, with a class relational binary cross-entropy loss.Type: GrantFiled: November 5, 2021Date of Patent: July 23, 2024Assignee: NEC CorporationInventors: Yi-Hsuan Tsai, Masoud Faraki, Yumin Suh, Sparsh Garg, Manmohan Chandraker, Dongwan Kim
-
Publication number: 20240071105Abstract: Methods and systems for training a model include pre-training a backbone model with a pre-training decoder, using an unlabeled dataset with multiple distinct sensor data modalities that derive from different sensor types. The backbone model is fine-tuned with an output decoder after pre-training, using a labeled dataset with the multiple modalities.Type: ApplicationFiled: August 22, 2023Publication date: February 29, 2024Inventors: Samuel Schulter, Bingbing Zhuang, Vijay Kumar Baikampady Gopalkrishna, Sparsh Garg, Zhixing Zhang
-
Publication number: 20230281999Abstract: Methods and systems identifying road hazards include capturing an image of a road scene using a camera. The image is embedded using a segmentation model that includes an image branch having an image embedding layer that embeds images into a joint latent space and a text branch having a text embedding layer that embeds text into the joint latent space. A mask is generated for an object within the image using the segmentation model. A probability is determined that the object matches a road hazard using the segmentation mode. A signal is generated responsive to the probability to ameliorate a danger posed by the road hazard.Type: ApplicationFiled: March 23, 2023Publication date: September 7, 2023Inventors: Samuel Schulter, Sparsh Garg
-
Publication number: 20230281977Abstract: Methods and systems for detecting faults include capturing an image of a scene using a camera. The image is embedded using a segmentation model that includes an image branch having an image embedding layer that embeds images into a joint latent space and a text branch having a text embedding layer that embeds text into the joint latent space. Semantic information is generated for a region of the image corresponding to a predetermined static object using the embedded image. A fault of the camera is identified based on a discrepancy between the semantic information and semantic information of the predetermined static image. The fault of the camera is corrected.Type: ApplicationFiled: March 23, 2023Publication date: September 7, 2023Inventors: Samuel Schulter, Sparsh Garg, Manmohan Chandraker
-
Publication number: 20230088335Abstract: Systems and methods is provided for road hazard analysis. The method includes obtaining sensor data of a road environment including a road and observable surroundings, and applying labels to the sensor data. The method further includes training a first neural network model to identify road hazards, training a second neural network model to identify faded lane markings, and training a third neural network model to identify overhanging trees and blocking foliage. The method further includes implementing the trained neural network models to detect road hazards in a real road setting.Type: ApplicationFiled: September 9, 2022Publication date: March 23, 2023Inventors: Sparsh Garg, Samuel Schulter, Vijay Kumar Baikampady Gopalkrishna
-
Publication number: 20230081913Abstract: Systems and methods are provided for multi-modal test-time adaptation. The method includes inputting a digital image into a pre-trained Camera Intra-modal Pseudo-label Generator, and inputting a point cloud set into a pre-trained Lidar Intra-modal Pseudo-label Generator. The method further includes applying a fast 2-dimension (2D) model, and a slow 2D model, to the inputted digital image to apply pseudo-labels, and applying a fast 3-dimension (3D) model, and a slow 3D model, to the inputted point cloud set to apply pseudo-labels. The method further includes fusing pseudo-label predictions from the fast models and the slow models through an Inter-modal Pseudo-label Refinement module to obtain robust pseudo labels, and measuring a prediction consistency for the pseudo-labels.Type: ApplicationFiled: September 6, 2022Publication date: March 16, 2023Inventors: Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, Ramin Moslemi, Inkyu Shin
-
Publication number: 20230073055Abstract: A computer-implemented method for rut detection is provided. The method includes detecting, by a rut detection system, areas in a road-scene image that include ruts with pixel-wise probability values, wherein a higher value indicates a better chance of being a rut. The method further includes performing at least one of rut repair and vehicle rut avoidance responsive to the pixel-wise probability values. The detecting step includes performing neural network-based, pixel-wise semantic segmentation with context information on the road-scene image to distinguish rut pixels from non-rut pixels on a road depicted in the road-scene image.Type: ApplicationFiled: September 6, 2022Publication date: March 9, 2023Inventors: Yi-Hsuan Tsai, Sparsh Garg, Manmohan Chandraker, Samuel Shulter, Vijay Kumar Baikampady Gopalkrishna
-
Publication number: 20220148189Abstract: Methods and systems for training a model include combining data from multiple datasets, the datasets having different respective label spaces. Relationships between labels in the different label spaces are identified. A unified neural network model is trained, using the combined data and the identified relationships to generate a unified model, with a class relational binary cross-entropy loss.Type: ApplicationFiled: November 5, 2021Publication date: May 12, 2022Inventors: Yi-Hsuan Tsai, Masoud Faraki, Yumin Suh, Sparsh Garg, Manmohan Chandraker, Dongwan Kim