Abstract: What is disclosed is a system and method for generating a respiration gating signal from a video of a subject for gating diagnostic imaging and therapeutic delivery applications which require respiration phase and/or respiration amplitude gating. One embodiment involves receiving a video of a subject and generating a plurality of time-series signals from the video image frames. A set of features are extracted from the time-series signals and multi-dimensional feature vectors are formed. The feature vectors are clustered. Time-series signals corresponding in each of the clusters are averaged in a temporal direction to obtain a representative signal for each cluster. One cluster is selected and a respiration gating signal is generated from that cluster's representative signal. Thereafter, the respiration gating signal is used to gate diagnostic imaging and therapeutic delivery applications which requires gating based on a threshold set with respect to either respiration phase or respiration amplitude.