Abstract: Systems, apparatuses, and methods for more efficiently and effectively determining the accuracy with which a human evaluates a set of data, as this may reduce the error in the assessment of a model's performance. This can be helpful in situations where human inputs are used to confirm the output of a machine learning generated classification and in situations where it is desired to evaluate the accuracy of data that may have been labeled or annotated by a human curator. This may assist in reducing the need for new validation/test data when evaluating a new model. The system and methods described can be used to evaluate the accuracy of a trained Machine Learning (ML) model, and as a result, allow a comparison between models based on different ML algorithms.

Abstract: Performance in a multi-classification system having multiple component classifiers can be based on a combination of the true positive rate (TPR) and false positive rate (FPR) of the component classifiers. Each component classifier can be configured with a decision threshold, and its TPR and FPR determined from a training set presented to the component classifier so configured. A system TPR and system FPR can be determined from the component TPRs and FPRs. A set of system TPRs and FPRs can be determined from additional sets of decision thresholds.

Abstract: Embodiments are directed to systems, apparatuses, and methods for efficiently evaluating the performance of a machine learning classifier. Embodiments improve the efficiency of techniques used to evaluate a classifier by substituting labels which are inexpensive to generate or collect for labels which are relatively more expensive to generate or collect, reducing the number of samples that need to be labeled, and reducing the number of data samples that need to be input to a machine learning classifier to evaluate the classifier.

Abstract: Configuring a multi-classification system having multiple component classifiers includes storing of data records that represent different levels of performance of the system. The component classifiers are configured with corresponding decision threshold values contained in a selected one of the data records. Performance of the multi-classification system subsequent to configuring the component classifiers is approximated by the performance level associated with the selected data record.

Abstract: Systems, apparatuses, and methods for more efficiently and effectively determining the accuracy with which a human evaluates a set of data, as this may reduce the error in the assessment of a model's performance. This can be helpful in situations where human inputs are used to confirm the output of a machine learning generated classification and in situations where it is desired to evaluate the accuracy of data that may have been labeled or annotated by a human curator. This may assist in reducing the need for new validation/test data when evaluating a new model. The system and methods described can be used to evaluate the accuracy of a trained Machine Learning (ML) model, and as a result, allow a comparison between models based on different ML algorithms.

Abstract: Configuring a multi-classification system having multiple component classifiers includes storing of data records that represent different levels of performance of the system. The component classifiers are configured with corresponding decision threshold values contained in a selected one of the data records. Performance of the multi-classification system subsequent to configuring the component classifiers is approximated by the performance level associated with the selected data record.

Abstract: Performance in a multi-classification system having multiple component classifiers can be based on a combination of the true positive rate (TPR) and false positive rate (FPR) of the component classifiers. Each component classifier can be configured with a decision threshold, and its TPR and FPR determined from a training set presented to the component classifier so configured. A system TPR and system FPR can be determined from the component TPRs and FPRs. A set of system TPRs and FPRs can be determined from additional sets of decision thresholds.