Patents by Inventor Nakul Agarwal
Nakul Agarwal has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Patent number: 12094214Abstract: A system and method for providing an agent action anticipative transformer that include receiving image data associated with a video of a surrounding environment of an ego agent. The system and method additionally include analyzing the image data and extracting short range clips from the image data. The system and method also include analyzing the short range clips and extracting clip-level features associated with each of the short range clips. The system and method further include executing self-supervision using causal masking with respect to the extracted clip-level features to output action predictions and feature predictions to enable ego-centric action anticipation with respect to at least one target agent to autonomously control the ego agent.Type: GrantFiled: April 29, 2022Date of Patent: September 17, 2024Assignee: HONDA MOTOR CO., LTD.Inventors: Harshayu Girase, Nakul Agarwal, Chiho Choi
-
Publication number: 20240161447Abstract: According to one aspect, spatial action localization in the future (SALF) may include feeding a frame from a time step of a video clip through an encoder to generate a latent feature, feeding the latent feature and one or more latent features from one or more previous time steps of the video clip through a future feature predictor to generate a cumulative information for the time step, feeding the cumulative information through a decoder to generate a predicted action area and a predicted action classification associated with the predicted action area, and implementing an action based on the predicted action area and the predicted action classification. The encoder may include a 2D convolutional neural network (CNN) and/or a 3D-CNN. The future feature predictor may be based on an ordinary differential equation (ODE) function.Type: ApplicationFiled: April 14, 2023Publication date: May 16, 2024Inventors: Hyung-gun CHI, Kwonjoon LEE, Nakul AGARWAL, Yi XU, Chiho CHOI
-
Patent number: 11845464Abstract: Driver behavior risk assessment and pedestrian awareness may include an receiving an input stream of images of an environment including one or more objects within the environment, estimating an intention of an ego vehicle based on the input stream of images and a temporal recurrent network (TRN), generating a scene representation based on the input stream of images and a graph neural network (GNN), generating a prediction of a situation based on the scene representation and the intention of the ego vehicle, and generating an influenced or non-influenced action determination based on the prediction of the situation and the scene representation.Type: GrantFiled: January 29, 2021Date of Patent: December 19, 2023Assignee: HONDA MOTOR CO., LTD.Inventors: Nakul Agarwal, Yi-Ting Chen
-
Publication number: 20230351759Abstract: A system and method for providing an agent action anticipative transformer that include receiving image data associated with a video of a surrounding environment of an ego agent. The system and method additionally include analyzing the image data and extracting short range clips from the image data. The system and method also include analyzing the short range clips and extracting clip-level features associated with each of the short range clips. The system and method further include executing self-supervision using causal masking with respect to the extracted clip-level features to output action predictions and feature predictions to enable ego-centric action anticipation with respect to at least one target agent to autonomously control the ego agent.Type: ApplicationFiled: April 29, 2022Publication date: November 2, 2023Inventors: Harshayu GIRASE, Nakul AGARWAL, Chiho CHOI
-
Publication number: 20230311942Abstract: Driver behavior risk assessment and pedestrian awareness may include an receiving an input stream of images of an environment including one or more objects within the environment, estimating an intention of an ego vehicle based on the input stream of images and a temporal recurrent network (TRN), generating a scene representation based on the input stream of images and a graph neural network (GNN), generating a prediction of a situation based on the scene representation and the intention of the ego vehicle, and generating an influenced or non-influenced action determination based on the prediction of the situation and the scene representation.Type: ApplicationFiled: June 8, 2023Publication date: October 5, 2023Inventors: Nakul AGARWAL, Yi-Ting CHEN
-
Patent number: 11741723Abstract: A system and method for performing intersection scenario retrieval that includes receiving a video stream of a surrounding environment of an ego vehicle. The system and method also include analyzing the video stream to trim the video stream into video clips of an intersection scene associated with the travel of the ego vehicle. The system and method additionally include annotating the ego vehicle, dynamic objects, and their motion paths that are included within the intersection scene with action units that describe an intersection scenario. The system and method further include retrieving at least one intersection scenario based on a query of an electronic dataset that stores a combination of action units to operably control a presentation of at least one intersection scenario video clip that includes the at least one intersection scenario.Type: GrantFiled: June 29, 2020Date of Patent: August 29, 2023Assignee: HONDA MOTOR CO., LTD.Inventors: Yi-Ting Chen, Nakul Agarwal, Behzad Dariush, Ahmed Taha
-
Publication number: 20230154195Abstract: According to one aspect, intersection scenario description may be implemented by receiving a video stream of a surrounding environment of an ego-vehicle, extracting tracklets and appearance features associated with dynamic objects from the surrounding environment, extracting motion features associated with dynamic objects from the surrounding environment based on the corresponding tracklets, passing the appearance features through an appearance neural network to generate an appearance model, passing the motion features through a motion neural network to generate a motion model, passing the appearance model and the motion model through a fusion network to generate a fusion output, passing the fusion output through a classifier to generate a classifier output, and passing the classifier output through a loss function to generate a multi-label classification output associated with the ego-vehicle, dynamic objects, and corresponding motion paths.Type: ApplicationFiled: June 30, 2022Publication date: May 18, 2023Inventors: Nakul AGARWAL, Yi-Ting CHEN
-
Publication number: 20230141037Abstract: A system and method for providing weakly-supervised online action segmentation that include receiving image data associated with multi-view videos of a procedure, wherein the procedure involves a plurality of atomic actions. The system and method also include analyzing the image data using weakly-supervised action segmentation to identify each of the plurality of atomic actions by using an ordered sequence of action labels. The system and method additionally include training a neural network with data pertaining to the plurality of atomic actions based on the weakly-supervised action segmentation. The system and method further include executing online action segmentation to label atomic actions that are occurring in real-time based on the plurality of atomic actions trained to the neural network.Type: ApplicationFiled: February 1, 2022Publication date: May 11, 2023Inventors: Reza GHODDOOSIAN, Isht DWIVEDI, Nakul AGARWAL, Chiho CHOI, Behzad DARIUSH
-
Patent number: 11580743Abstract: A system and method for providing unsupervised domain adaption for spatio-temporal action localization that includes receiving video data associated with a source domain and a target domain that are associated with a surrounding environment of a vehicle. The system and method also include analyzing the video data associated with the source domain and the target domain and determining a key frame of the source domain and a key frame of the target domain. The system and method additionally include completing an action localization model to model a temporal context of actions occurring within the key frame of the source domain and the key frame of the target domain and completing an action adaption model to localize individuals and their actions and to classify the actions based on the video data. The system and method further include combining losses to complete spatio-temporal action localization of individuals and actions.Type: GrantFiled: March 25, 2022Date of Patent: February 14, 2023Assignee: HONDA MOTOR CO., LTD.Inventors: Yi-Ting Chen, Behzad Dariush, Nakul Agarwal, Ming-Hsuan Yang
-
Patent number: 11403850Abstract: A system and method for providing unsupervised domain adaption for spatio-temporal action localization that includes receiving video data associated with a surrounding environment of a vehicle. The system and method also include completing an action localization model to model a temporal context of actions occurring within the surrounding environment of the vehicle based on the video data and completing an action adaption model to localize individuals and their actions and to classify the actions based on the video data. The system and method further include combining losses from the action localization model and the action adaption model to complete spatio-temporal action localization of individuals and actions that occur within the surrounding environment of the vehicle.Type: GrantFiled: February 28, 2020Date of Patent: August 2, 2022Assignee: Honda Motor Co., Ltd.Inventors: Yi-Ting Chen, Behzad Dariush, Nakul Agarwal, Ming-Hsuan Yang
-
Publication number: 20220215661Abstract: A system and method for providing unsupervised domain adaption for spatio-temporal action localization that includes receiving video data associated with a source domain and a target domain that are associated with a surrounding environment of a vehicle. The system and method also include analyzing the video data associated with the source domain and the target domain and determining a key frame of the source domain and a key frame of the target domain. The system and method additionally include completing an action localization model to model a temporal context of actions occurring within the key frame of the source domain and the key frame of the target domain and completing an action adaption model to localize individuals and their actions and to classify the actions based on the video data. The system and method further include combining losses to complete spatio-temporal action localization of individuals and actions.Type: ApplicationFiled: March 25, 2022Publication date: July 7, 2022Inventors: Yi-Ting CHEN, Behzad DARIUSH, Nakul AGARWAL, Ming-Hsuan YANG
-
Publication number: 20220144303Abstract: Driver behavior risk assessment and pedestrian awareness may include an receiving an input stream of images of an environment including one or more objects within the environment, estimating an intention of an ego vehicle based on the input stream of images and a temporal recurrent network (TRN), generating a scene representation based on the input stream of images and a graph neural network (GNN), generating a prediction of a situation based on the scene representation and the intention of the ego vehicle, and generating an influenced or non-influenced action determination based on the prediction of the situation and the scene representation.Type: ApplicationFiled: January 29, 2021Publication date: May 12, 2022Inventors: Nakul Agarwal, Yi-Ting Chen
-
Publication number: 20210271898Abstract: A system and method for performing intersection scenario retrieval that includes receiving a video stream of a surrounding environment of an ego vehicle. The system and method also include analyzing the video stream to trim the video stream into video clips of an intersection scene associated with the travel of the ego vehicle. The system and method additionally include annotating the ego vehicle, dynamic objects, and their motion paths that are included within the intersection scene with action units that describe an intersection scenario. The system and method further include retrieving at least one intersection scenario based on a query of an electronic dataset that stores a combination of action units to operably control a presentation of at least one intersection scenario video clip that includes the at least one intersection scenario.Type: ApplicationFiled: June 29, 2020Publication date: September 2, 2021Inventors: Yi-Ting Chen, Nakul Agarwal, Behzad Dariush, Ahmed Taha
-
Publication number: 20210027066Abstract: A system and method for providing unsupervised domain adaption for spatio-temporal action localization that includes receiving video data associated with a surrounding environment of a vehicle. The system and method also include completing an action localization model to model a temporal context of actions occurring within the surrounding environment of the vehicle based on the video data and completing an action adaption model to localize individuals and their actions and to classify the actions based on the video data. The system and method further include combining losses from the action localization model and the action adaption model to complete spatio-temporal action localization of individuals and actions that occur within the surrounding environment of the vehicle.Type: ApplicationFiled: February 28, 2020Publication date: January 28, 2021Inventors: Yi-Ting Chen, Behzad Dariush, Nakul Agarwal, Ming-Hsuan Yang