Abstract: A method and system for applying a task to a crowdsourcing platform for processing are disclosed. For example, a method forwards the task having a question to the crowdsourcing platform, and receives a plurality of answers to the question, wherein the plurality of answers is provided by a plurality of workers associated with the crowdsourcing platform. The method analyzes the plurality of answers on a per-worker basis to determine an adjustment, and applies the adjustment to the plurality of answers.

Abstract: Systems and computer-implemented methods include a validation phase in which validation projects are submitted to an online crowdsourcing system. The validation projects each have different combinations of the two validation control parameters, payment per human intelligence task (“HIT”) and number of solutions per HIT. By analyzing the solutions for the validation projects, final control parameters can be selected from the validation project having the best combination of high quality, low cost, and short time to completion. The final control parameters can be submitted along with a final project to the online crowdsourcing system. In addition, a novel outlier detection algorithm enables the system to detect outlier workers based on the distribution of non-experts' agreement probabilities, reject outlier workers' results, and thus save labeling cost.

Abstract: A method and system for applying a task to a crowdsourcing platform for processing are disclosed. For example, a method forwards the task having a question to the crowdsourcing platform, and receives a plurality of answers to the question, wherein the plurality of answers is provided by a plurality of workers associated with the crowdsourcing platform. The method analyzes the plurality of answers on a per-worker basis to determine an adjustment, and applies the adjustment to the plurality of answers.

Abstract: Systems and computer-implemented methods include a validation phase in which validation projects are submitted to an online crowdsourcing system. The validation projects each have different combinations of the two validation control parameters, payment per human intelligence task (“HIT”) and number of solutions per HIT. By analyzing the solutions for the validation projects, final control parameters can be selected from the validation project having the best combination of high quality, low cost, and short time to completion. The final control parameters can be submitted along with a final project to the online crowdsourcing system. In addition, a novel outlier detection algorithm enables the system to detect outlier workers based on the distribution of non-experts' agreement probabilities, reject outlier workers' results, and thus save labeling cost.