Abstract: Workers are trained and assisted to improve quality of task results generated by them. Initially, workers seeking to perform intelligence tasks within the context of a crowdsourcing system are provided with offline training. Subsequently, workers are qualified, with different nuances of the performance of intelligence tasks being assessed independently. A task owner specifies minimum quality thresholds, including separately specified thresholds for different nuances. Qualified workers are assigned to perform intelligence tasks, thereby producing useful work. While doing so, their quality is monitored, and online training is provided to aid workers with specific nuances of the intelligence tasks where the workers' performance is suboptimal. Workers whose quality drops below minimum quality thresholds are aided in improving their quality, including being prevented from performing intelligence tasks for a predefined period of time, being mandated to perform offline training, or combinations thereof.

Abstract: Different options associated with the performance of intelligence tasks are flighted. Different versions of applications that provide the context within which intelligence tasks are performed are sourced for each of the different combinations of options. Subsets of the human workers are selected and provided such different versions of such applications. Correlations are made between those applications that were selected by individual workers and the options that such applications represented, and further between intelligence task results, including, optionally, an evaluation of the quality of such task results, and such options. Options, and the settings thereof, which affect and optimize the intelligence task results generated by workers, are, thereby, more efficiently identified.

Abstract: Results, generated by human workers in response to HITs assigned to them, are evaluated based upon the behavior of the human workers in generating such results. Workers receive, together with an intelligence task to be performed, a behavior logger by which the worker's behavior is monitored while the worker performs the intelligence task. Machine learning is utilized to identify behavioral factors upon which the evaluation can be based and then to learn how to utilize such behavioral factors to evaluate the HIT results generated by workers, as well as the workers themselves. The identification of behavioral factors, and the subsequent utilization thereof, is informed by the behavior of, and corresponding results generated by, a trusted set of workers. Results evaluated to have been improperly generated can be discarded or simply downweighted. Workers evaluated to be operating improperly can be removed or retrained.

Abstract: Reference intelligence tasks are automatically generated for subsequent utilization in crowdsourced processing of intelligence tasks. Reference intelligence tasks are categorized into predetermined categories, including categories defined by an intended utilization of such intelligence tasks. A trusted set of workers are provided with intelligence tasks and, if a specified number of those trusted workers reach consensus as to what is an appropriate answer, then such an answer is a definitive answer. Conversely, if no consensus is initially reached, then additional trusted workers can be utilized to determine if, in combination, consensus can be reached. To frame categorization considerations by the trusted set of workers, they are provided with statistics relevant to categorizations. An automatic category assignment, or reassignment, can be performed if necessary.