Abstract: A system and method for generating gold questions for labeling tasks are disclosed. The method includes sampling a positive class from a predefined set of classes to be used in labeling documents, based on a computed measure of class popularity. A set of negative classes is identified from the set of classes based on a distance measure between the positive class and other classes in the set of classes. A gold question is generated which includes a document representative of the positive class and a set of candidate answers. The candidate answers include a label for the positive class and a label for each of the negative classes in the identified set of negative classes. A task may be generated which includes the gold question and a plurality of standard questions which each include a document to be labeled. A computer processor may implement all or part of the method.