Abstract: Disclosed is an activity detection system (“ADS”) that detects, classifies, and isolates previously unseen activities in unlabeled videos based on different previously seen activities in labeled videos and semantic similarity between the unseen and seen activities. The ADS receive a first set of videos that are labeled with a first activity, and may determine a feature set within frames of the first set of videos that represents the first activity. The ADS may receive a second set of videos that are not labeled, and a query for videos of a second activity that is determined to be semantically similar to the first activity. The ADS may provide, in response to the query for the second activity, a particular video from the second set of videos containing the feature set representing the first activity that is semantically similar to the queried for second activity.