Abstract: A facility for identifying workers in a crowdsourcing or micro-tasking platform who perform low-quality work and/or are really automated bots is described. To identify users who perform low-quality work and/or are really bots, the facility (1) measures the quality of at least a portion of the work done by each user, and (2) tracks the pattern of behavior performed by each user on the platform—such as which work projects they select, the content of the responses, and the timing of each user interface interaction. The facility uses this information to build and maintain a model, such as a statistical model, that uses the pattern of a user's behavior to predict the level of quality of the user's work. Users for which the model predicts a low level of quality are flagged for manual review, or automatically suspended from working or from receiving payment.

Abstract: Systems and methods that use machine learning to optimize the execution of micro-tasks, by partially automating the generation and validation actions are disclosed. The system uses a combination of automatic intelligent model-based decision systems and human-in-the-loop for generating annotated task instances with respect to an identified task. Before a task is executed, the system can compute the crowd effort to generate data for each task instance, as well as the effort to validate and/or correct them. These computations can occur multiple times during the execution of task. The generation, validation and correction effort are measures that allow the system to design more efficient workflows that combine machine learning models and human input because the system can decide automatically what is the most efficient next step to obtain the best results.

Abstract: A facility for providing a workflow tailored to defining a project for collecting multimodal data from each of a set of crowdsourcing or microtasking platform workers is described. The facility enables customers of a crowdsourcing or microtasking platform to easily define multimodal data collection projects. The facility enables customers to define any of the following types of information associated with multimodal data collection projects: worker requirements, project environment parameters, video data, audio data, physiological data, and/or location-related data. Some of this data is collected using different kinds of sensors in one or more devices (e.g., smart phones, fitness wearables, etc.) associated with the crowdsourcing or micro-tasking platforms' workers. Prior to computing data results generated by executing a multimodal data collection project, the facility an align at least a first portion of the collected data with a second portion of the second data.

Abstract: A facility to crowdsource training of virtual assistants and other textual natural language understanding systems is described. The facility first specifies a set of possible user intents (e.g., a kind of question asked by users). As part of specifying an intent, entities, that represent salient items of information associated with the intent are identified. Then, for each of the intents, the facility directs users of a crowdsourcing platform to input a number of different textual queries they might use to express this intent. Then, additional crowdsourcing platform users are asked to perform semantic annotation of the cleaned queries, for each selecting its intent and entities from predefined lists. Next, still other crowdsourcing platform users are asked whether the selection of intents and entities during semantic annotation was correct for each query. Once validated, the annotated queries are used to train the assistant.

Abstract: A facility to crowdsource training of virtual assistants and other textual natural language understanding systems is described. The facility first specifies a set of possible user intents (e.g., a kind of question asked by users). As part of specifying an intent, entities, that represent salient items of information associated with the intent are identified. Then, for each of the intents, the facility directs users of a crowdsourcing platform to input a number of different textual queries they might use to express this intent. Then, additional crowdsourcing platform users are asked to perform semantic annotation of the cleaned queries, for each selecting its intent and entities from predefined lists. Next, still other crowdsourcing platform users are asked whether the selection of intents and entities during semantic annotation was correct for each query. Once validated, the annotated queries are used to train the assistant.