Abstract: A system and method for determining sub-activities in videos and segmenting the videos is disclosed. The method includes extracting one or more batches from one or more videos and extracting one or more features from set of frames associated with the one or more batches. The method further includes generating a set of predicted codes and determining a cross-entropy loss, temporal coherence loss and a final loss. Further, the method includes categorizing the set of frames into one or more predefined clusters and generating one or more segmented videos based on the categorized set of frames, the determined final loss, and the set of predicted codes by using s activity determination-based ML model. The method includes outputting the generated one or more segmented videos on user interface screen of one or more electronic devices associated with one or more users.