Abstract: Systems and methods for the crowdsourced clustering of data items in accordance embodiments of the invention are disclosed. In one embodiment of the invention, a method for determining categories for a set of source data includes obtaining a set of source data, determining a plurality of subsets of the source data, where a subset of the source data includes a plurality of pieces of source data in the set of source data, generating a set of pairwise annotations for the pieces of source data in each subset of source data, clustering the set of source data into related subsets of source data based on the sets of pairwise labels for each subset of source data, and identifying a category for each related subset of source data based on the clusterings of source data and the source data metadata for the pieces of source data in the group of source data.

Abstract: Systems and methods for determining annotator performance in the distributed annotation of source data in accordance embodiments of the invention are disclosed. In one embodiment of the invention, a method for clustering annotators includes obtaining a set of source data, determining a training data set representative of the set of source data, obtaining sets of annotations from a set of annotators for a portion of the training data set, for each annotator determining annotator recall metadata based on the set of annotations provided by the annotator for the training data set and determining annotator precision metadata based on the set of annotations provided by the annotator for the training data set, and grouping the annotators into annotator groups based on the annotator recall metadata and the annotator precision metadata.

Abstract: Similar faces may be determined within images based on human perception of facial similarity. The user may provide an image including a query face to which the user wishes to find faces that are similar. Similar faces may be determined based on similarity information. Similarity information may be generated from information related to a human perception of facial similarity. Images that include faces determined to be similar, based on the similarity information, may be provided to the user as search result images. The user then may provide feedback to indicate the user's perception of similarity between the query face and the search result images.

Abstract: A system and method of sorting graphics files with a graphic server and image search engine is disclosed. The method preferably comprises the steps of: receiving a plurality of search results where each of the search results comprising one or more associated graphics; using a general purpose computer to identifying one or more groups of said graphics that depict or otherwise possess similar visual features; and returning the plurality of search results to a user in accordance with said identified groups. The preferred embodiment effectively clusters graphics files, particularly image files, based on upon shared scale-invariant features in order to enable the user to readily identify collections of images relevant to the user while skip past collections of less relevant images.

Abstract: Systems and methods for the crowdsourced clustering of data items in accordance embodiments of the invention are disclosed. In one embodiment of the invention, a method for determining categories for a set of source data includes obtaining a set of source data, determining a plurality of subsets of the source data, where a subset of the source data includes a plurality of pieces of source data in the set of source data, generating a set of pairwise annotations for the pieces of source data in each subset of source data, clustering the set of source data into related subsets of source data based on the sets of pairwise labels for each subset of source data, and identifying a category for each related subset of source data based on the clusterings of source data and the source data metadata for the pieces of source data in the group of source data.

Abstract: Systems and methods for distributed data annotation in accordance embodiments of the invention are disclosed. In one embodiment of the invention, a distributed data annotation server system includes a storage device configured to store source data, one or more annotators, annotation tasks and a processor, wherein a distributed data annotation application configures the processor to receive source data including one or more pieces of source data, select one or more annotators, create one or more annotation tasks for the selected annotators and source data, request one or more annotations for the source data using the annotation tasks, receive annotations, determine source data metadata for at least one piece of source data using the received annotations, generate annotator metadata for at least one annotator using the received annotations and the source data, and estimate the ground truth for the source data using the source data metadata and the annotator metadata.

Abstract: Systems and methods for distributed data annotation in accordance embodiments of the invention are disclosed. In one embodiment of the invention, a distributed data annotation server system includes a storage device configured to store source data, one or more annotators, annotation tasks and a processor, wherein a distributed data annotation application configures the processor to receive source data including one or more pieces of source data, select one or more annotators, create one or more annotation tasks for the selected annotators and source data, request one or more annotations for the source data using the annotation tasks, receive annotations, determine source data metadata for at least one piece of source data using the received annotations, generate annotator metadata for at least one annotator using the received annotations and the source data, and estimate the ground truth for the source data using the source data metadata and the annotator metadata.

Abstract: Images are searched to locate faces that are the same as a query face. Images that include a face that is the same as the query face may be presented to a user as search result images. Images also may be sorted by the faces included in the images and presented to the user as sorted search result images. The user may provide explicit or implicit feedback regarding the search result images. Additional feedback may be inferred regarding the search result images based on the user-provided feedback, and the results may be updated based on the user-provided and inferred feedback.

Abstract: Systems and methods for determining annotator performance in the distributed annotation of source data in accordance embodiments of the invention are disclosed. In one embodiment of the invention, a method for clustering annotators includes obtaining a set of source data, determining a training data set representative of the set of source data, obtaining sets of annotations from a set of annotators for a portion of the training data set, for each annotator determining annotator recall metadata based on the set of annotations provided by the annotator for the training data set and determining annotator precision metadata based on the set of annotations provided by the annotator for the training data set, and grouping the annotators into annotator groups based on the annotator recall metadata and the annotator precision metadata.

Abstract: Systems and methods for the annotation of source data using confidence labels in accordance embodiments of the invention are disclosed. In one embodiment of the invention, a method for determining confidence labels for crowdsourced annotations includes obtaining a set of source data, obtaining a set of training data representative of the set of source data, determining the ground truth for each piece of training data, obtaining a set of training data annotations including a confidence label, measuring annotator accuracy data for at least one piece of training data, and automatically generating a set of confidence labels for the set of unlabeled data based on the measured annotator accuracy data and the set of annotator labels used.

Abstract: Similar faces may be determined within images based on human perception of facial similarity. The user may provide an image including a query face to which the user wishes to find faces that are similar. Similar faces may be determined based on similarity information. Similarity information may be generated from information related to a human perception of facial similarity. Images that include faces determined to be similar, based on the similarity information, may be provided to the user as search result images. The user then may provide feedback to indicate the user's perception of similarity between the query face and the search result images.

Abstract: Images are searched to locate faces that are the same as a query face. Images that include a face that is the same as the query face may be presented to a user as search result images. Images also may be sorted by the faces included in the images and presented to the user as sorted search result images. The user may provide explicit or implicit feedback regarding the search result images. Additional feedback may be inferred regarding the search result images based on the user-provided feedback, and the results may be updated based on the user-provided and inferred feedback.

Abstract: Similar faces may be determined within images based on human perception of facial similarity. The user may provide an image including a query face to which the user wishes to find faces that are similar. Similar faces may be determined based on similarity information. Similarity information may be generated from information related to a human perception of facial similarity. Images that include faces determined to be similar, based on the similarity information, may be provided to the user as search result images. The user then may provide feedback to indicate the user's perception of similarity between the query face and the search result images.

Abstract: Similar faces may be determined within images based on human perception of facial similarity. The user may provide an image including a query face to which the user wishes to find faces that are similar. Similar faces may be determined based on similarity information. Similarity information may be generated from information related to a human perception of facial similarity. Images that include faces determined to be similar, based on the similarity information, may be provided to the user as search result images. The user then may provide feedback to indicate the user's perception of similarity between the query face and the search result images.

Abstract: Images are searched to locate faces that are the same as a query face. Images that include a face that is the same as the query face may be presented to a user as search result images. Images also may be sorted by the faces included in the images and presented to the user as sorted search result images. The user may provide explicit or implicit feedback regarding the search result images. Additional feedback may be inferred regarding the search result images based on the user-provided feedback, and the results may be updated based on the user-provided and inferred feedback.

Abstract: Similar faces may be determined within images based on human perception of facial similarity. The user may provide an image including a query face to which the user wishes to find faces that are similar. Similar faces may be determined based on similarity information. Similarity information may be generated from information related to a human perception of facial similarity. Images that include faces determined to be similar, based on the similarity information, may be provided to the user as search result images. The user then may provide feedback to indicate the user's perception of similarity between the query face and the search result images.

Abstract: Images are searched to locate faces that are the same as a query face. Images that include a face that is the same as the query face may be presented to a user as search result images. Images also may be sorted by the faces included in the images and presented to the user as sorted search result images. The user may provide explicit or implicit feedback regarding the search result images. Additional feedback may be inferred regarding the search result images based on the user-provided feedback, and the results may be updated based on the user-provided and inferred feedback.

Abstract: Similar faces may be determined within images based on human perception of facial similarity. The user may provide an image including a query face to which the user wishes to find faces that are similar. Similar faces may be determined based on similarity information. Similarity information may be generated from information related to a human perception of facial similarity. Images that include faces determined to be similar, based on the similarity information, may be provided to the user as search result images. The user then may provide feedback to indicate the user's perception of similarity between the query face and the search result images.

Abstract: Images are searched to locate faces that are the same as a query face. Images that include a face that is the same as the query face may be presented to a user as search result images. Images also may be sorted by the faces included in the images and presented to the user as sorted search result images. The user may provide explicit or implicit feedback regarding the search result images. Additional feedback may be inferred regarding the search result images based on the user-provided feedback, and the results may be updated based on the user-provided and inferred feedback.

Abstract: Images are searched to locate faces that are the same as a query face. Images that include a face that is the same as the query face may be presented to a user as search result images. Images also may be sorted by the faces included in the images and presented to the user as sorted search result images. The user may provide explicit or implicit feedback regarding the search result images. Additional feedback may be inferred regarding the search result images based on the user-provided feedback, and the results may be updated based on the user-provided and inferred feedback.