Abstract: A system and method for classification of moving objects and user authoring of new object classes is disclosed. In one embodiment, in a method of classification of moving objects, a moving object is inputted. Then, an object descriptor and a motion descriptor are extracted from the inputted moving object. Multiple initial candidate library object descriptors are identified from an object library and a motion library using the extracted object descriptor and the extracted motion descriptor. An initial object class estimate is identified based on the identified multiple initial candidate library object descriptors. Then, an initial residue is computed based on the extracted object descriptor and the identified multiple initial candidate library object descriptors associated with the initial object class estimate. The object class estimates are iteratively identified and it is determined whether the object class estimates converge based on a stopping criterion.