Abstract: A method and system for generating a classifier to classify sub-objects of an object based on a relationship between sub-objects is provided. The classification system provides training sub-objects along with the actual classification of each training sub-object. The classification system may iteratively train sub-classifiers based on feature vectors representing the features of each sub-object, the actual classification of the sub-object, and a weight associated with the sub-object. After a sub-classifier is trained, the classification system classifies the training sub-objects using the trained sub-classifier. The classification system then adjusts the classifications based on relationships between training sub-objects. The classification system assigns a weight for the sub-classifier and weight for each sub-object based on the accuracy of the adjusted classifications.