Patents by Inventor Haoye CAI

Haoye CAI has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Spatio-temporal graph for video captioning with knowledge distillation
    Patent number: 11604936
    Abstract: A method for scene perception using video captioning based on a spatio-temporal graph model is described. The method includes decomposing the spatio-temporal graph model of a scene in input video into a spatial graph and a temporal graph. The method also includes modeling a two branch framework having an object branch and a scene branch according to the spatial graph and the temporal graph to learn object interactions between the object branch and the scene branch. The method further includes transferring the learned object interactions from the object branch to the scene branch as privileged information. The method also includes captioning the scene by aligning language logits from the object branch and the scene branch according to the learned object interactions.
    Type: Grant
    Filed: March 23, 2020
    Date of Patent: March 14, 2023
    Assignees: TOYOTA RESEARCH INSTITUTE, INC., THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIOR UNIVERSITY
    Inventors: Boxiao Pan, Haoye Cai, De-An Huang, Kuan-Hui Lee, Adrien David Gaidon, Ehsan Adeli-Mosabbeb, Juan Carlos Niebles Duque
  • SPATIO-TEMPORAL GRAPH FOR VIDEO CAPTIONING WITH KNOWLEDGE DISTILLATION
    Publication number: 20210295093
    Abstract: A method for scene perception using video captioning based on a spatio-temporal graph model is described. The method includes decomposing the spatio-temporal graph model of a scene in input video into a spatial graph and a temporal graph. The method also includes modeling a two branch framework having an object branch and a scene branch according to the spatial graph and the temporal graph to learn object interactions between the object branch and the scene branch. The method further includes transferring the learned object interactions from the object branch to the scene branch as privileged information. The method also includes captioning the scene by aligning language logits from the object branch and the scene branch according to the learned object interactions.
    Type: Application
    Filed: March 23, 2020
    Publication date: September 23, 2021
    Applicants: TOYOTA RESEARCH INSTITUTE, INC., THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIOR UNIVERSITY
    Inventors: Boxiao PAN, Haoye CAI, De-An HUANG, Kuan-Hui LEE, Adrien David GAIDON, Ehsan ADELI-MOSABBEB, Juan Carlos NIEBLES DUQUE