Abstract: A facility for identifying workers in a crowdsourcing or micro-tasking platform who perform low-quality work and/or are really automated bots is described. To identify users who perform low-quality work and/or are really bots, the facility (1) measures the quality of at least a portion of the work done by each user, and (2) tracks the pattern of behavior performed by each user on the platform—such as which work projects they select, the content of the responses, and the timing of each user interface interaction. The facility uses this information to build and maintain a model, such as a statistical model, that uses the pattern of a user's behavior to predict the level of quality of the user's work. Users for which the model predicts a low level of quality are flagged for manual review, or automatically suspended from working or from receiving payment.

Abstract: A facility for providing a workflow tailored to defining a project for collecting multimodal data from each of a set of crowdsourcing or microtasking platform workers is described. The facility enables customers of a crowdsourcing or microtasking platform to easily define multimodal data collection projects. The facility enables customers to define any of the following types of information associated with multimodal data collection projects: worker requirements, project environment parameters, video data, audio data, physiological data, and/or location-related data. Some of this data is collected using different kinds of sensors in one or more devices (e.g., smart phones, fitness wearables, etc.) associated with the crowdsourcing or micro-tasking platforms' workers. Prior to computing data results generated by executing a multimodal data collection project, the facility an align at least a first portion of the collected data with a second portion of the second data.

Abstract: A method includes causing a first crowdsourced validation job to be provided to one or more first validation devices, the first crowdsourced validation job comprising first instructions for a crowd user to provide an indication of an accuracy of a transcription of natural language content, receiving a plurality of responses from the one or more first validation devices, wherein the plurality of responses include at least a first response from at least a first validation device from among the one or more first validation devices, the first response including a first indication of an accuracy of the transcription of the natural language content, and determining a first confidence score of the first validation device based, at least in part, on the plurality of responses received from the one or more first validation devices and the first response received from the first validation device.

Abstract: A facility to crowdsource training of virtual assistants and other textual natural language understanding systems is described. The facility first specifies a set of possible user intents (e.g., a kind of question asked by users). As part of specifying an intent, entities, that represent salient items of information associated with the intent are identified. Then, for each of the intents, the facility directs users of a crowdsourcing platform to input a number of different textual queries they might use to express this intent. Then, additional crowdsourcing platform users are asked to perform semantic annotation of the cleaned queries, for each selecting its intent and entities from predefined lists. Next, still other crowdsourcing platform users are asked whether the selection of intents and entities during semantic annotation was correct for each query. Once validated, the annotated queries are used to train the assistant.

Abstract: Systems and methods of validating transcriptions of natural language content using crowdsourced validation jobs are provided herein. In various implementations, a transcription pair comprising natural language content and text corresponding to a transcription of the natural language content may be gathered. A group of validation devices may be selected for reviewing the transcription pair. A crowdsourced validation job may be created for the group of validation devices. The crowdsourced validation job may be provided to the group of validation devices. One or more votes representing whether or not the text accurately represents the natural language content may be received from the group of validation devices. Based on the one or more votes received, the transcription pair may be stored in a validated transcription library, which may be used to process end-user voice data.

Abstract: A facility to crowdsource training of virtual assistants and other textual natural language understanding systems is described. The facility first specifies a set of possible user intents (e.g., a kind of question asked by users). As part of specifying an intent, entities, that represent salient items of information associated with the intent are identified. Then, for each of the intents, the facility directs users of a crowdsourcing platform to input a number of different textual queries they might use to express this intent. Then, additional crowdsourcing platform users are asked to perform semantic annotation of the cleaned queries, for each selecting its intent and entities from predefined lists. Next, still other crowdsourcing platform users are asked whether the selection of intents and entities during semantic annotation was correct for each query. Once validated, the annotated queries are used to train the assistant.

Abstract: A facility to crowdsource training of virtual assistants and other textual natural language understanding systems is described. The facility first specifies a set of possible user intents (e.g., a kind of question asked by users). As part of specifying an intent, entities, that represent salient items of information associated with the intent are identified. Then, for each of the intents, the facility directs users of a crowdsourcing platform to input a number of different textual queries they might use to express this intent. Then, additional crowdsourcing platform users are asked to perform semantic annotation of the cleaned queries, for each selecting its intent and entities from predefined lists. Next, still other crowdsourcing platform users are asked whether the selection of intents and entities during semantic annotation was correct for each query. Once validated, the annotated queries are used to train the assistant.

Abstract: Systems and methods of validating transcriptions of natural language content using crowdsourced validation jobs are provided herein. In various implementations, a transcription pair comprising natural language content and text corresponding to a transcription of the natural language content may be gathered. A group of validation devices may be selected for reviewing the transcription pair. A crowdsourced validation job may be created for the group of validation devices. The crowdsourced validation job may be provided to the group of validation devices. One or more votes representing whether or not the text accurately represents the natural language content may be received from the group of validation devices. Based on the one or more votes received, the transcription pair may be stored in a validated transcription library, which may be used to process end-user voice data.

Abstract: A system and method of tagging utterances with Named Entity Recognition (“NER”) labels using unmanaged crowds is provided. The system may generate various annotation jobs in which a user, among a crowd, is asked to tag which parts of an utterance, if any, relate to various entities associated with a domain. For a given domain that is associated with a number of entities that exceeds a threshold N value, multiple batches of jobs (each batch having jobs that have a limited number of entities for tagging) may be used to tag a given utterance from that domain. This reduces the cognitive load imposed on a user, and prevents the user from having to tag more than N entities. As such, a domain with a large number of entities may be tagged efficiently by crowd participants without overloading each crowd participant with too many entities to tag.

Abstract: A facility for providing a workflow tailored to defining a project for collecting multimodal data from each of a set of crowdsourcing or microtasking platform workers is described. The facility enables customers of a crowdsourcing or microtasking platform to easily define multimodal data collection projects. The facility enables customers to define any of the following types of information associated with multimodal data collection projects: worker requirements, project environment parameters, video data, audio data, physiological data, and/or location-related data. Some of this data is collected using different kinds of sensors in one or more devices (e.g., smart phones, fitness wearables, etc.) associated with the crowdsourcing or micro-tasking platforms' workers. Prior to computing data results generated by executing a multimodal data collection project, the facility an align at least a first portion of the collected data with a second portion of the second data.

Abstract: Systems and methods of validating transcriptions of natural language content using crowdsourced validation jobs are provided herein. In various implementations, a transcription pair comprising natural language content and text corresponding to a transcription of the natural language content may be gathered. A group of validation devices may be selected for reviewing the transcription pair. A crowdsourced validation job may be created for the group of validation devices. The crowdsourced validation job may be provided to the group of validation devices. One or more votes representing whether or not the text accurately represents the natural language content may be received from the group of validation devices. Based on the one or more votes received, the transcription pair may be stored in a validated transcription library, which may be used to process end-user voice data.

Abstract: A facility for identifying workers in a crowdsourcing or micro-tasking platform who perform low-quality work and/or are really automated bots is described. To identify users who perform low-quality work and/or are really bots, the facility (1) measures the quality of at least a portion of the work done by each user, and (2) tracks the pattern of behavior performed by each user on the platform—such as which work projects they select, the content of the responses, and the timing of each user interface interaction. The facility uses this information to build and maintain a model, such as a statistical model, that uses the pattern of a user's behavior to predict the level of quality of the user's work. Users for which the model predicts a low level of quality are flagged for manual review, or automatically suspended from working or from receiving payment.

Abstract: A facility to crowdsource training of virtual assistants and other textual natural language understanding systems is described. The facility first specifies a set of possible user intents (e.g., a kind of question asked by users). As part of specifying an intent, entities, that represent salient items of information associated with the intent are identified. Then, for each of the intents, the facility directs users of a crowdsourcing platform to input a number of different textual queries they might use to express this intent. Then, additional crowdsourcing platform users are asked to perform semantic annotation of the cleaned queries, for each selecting its intent and entities from predefined lists. Next, still other crowdsourcing platform users are asked whether the selection of intents and entities during semantic annotation was correct for each query. Once validated, the annotated queries are used to train the assistant.

Abstract: A system and method of tagging utterances with Named Entity Recognition (“NER”) labels using unmanaged crowds is provided. The system may generate various annotation jobs in which a user, among a crowd, is asked to tag which parts of an utterance, if any, relate to various entities associated with a domain. For a given domain that is associated with a number of entities that exceeds a threshold N value, multiple batches of jobs (each batch having jobs that have a limited number of entities for tagging) may be used to tag a given utterance from that domain. This reduces the cognitive load imposed on a user, and prevents the user from having to tag more than N entities. As such, a domain with a large number of entities may be tagged efficiently by crowd participants without overloading each crowd participant with too many entities to tag.

Abstract: Systems and methods of validating transcriptions of natural language content using crowdsourced validation jobs are provided herein. In various implementations, a transcription pair comprising natural language content and text corresponding to a transcription of the natural language content may be gathered. A group of validation devices may be selected for reviewing the transcription pair. A crowdsourced validation job may be created for the group of validation devices. The crowdsourced validation job may be provided to the group of validation devices. One or more votes representing whether or not the text accurately represents the natural language content may be received from the group of validation devices. Based on the one or more votes received, the transcription pair may be stored in a validated transcription library, which may be used to process end-user voice data.

Abstract: Systems and methods gathering text commands in response to a command context using a first crowdsourced are discussed herein. A command context for a natural language processing system may be identified, where the command context is associated with a command context condition to provide commands to the natural language processing system. One or more command creators associated with one or more command creation devices may be selected. A first application one the one or more command creation devices may be configured to display command creation instructions for each of the one or more command creators to provide text commands that satisfy the command context, and to display a field for capturing a user-generated text entry to satisfy the command creation condition in accordance with the command creation instructions. Systems and methods for reviewing the text commands using second and crowdsourced jobs are also presented herein.

Abstract: Systems and methods gathering text commands in response to a command context using a first crowdsourced are discussed herein. A command context for a natural language processing system may be identified, where the command context is associated with a command context condition to provide commands to the natural language processing system. One or more command creators associated with one or more command creation devices may be selected. A first application one the one or more command creation devices may be configured to display command creation instructions for each of the one or more command creators to provide text commands that satisfy the command context, and to display a field for capturing a user-generated text entry to satisfy the command creation condition in accordance with the command creation instructions. Systems and methods for reviewing the text commands using second and crowdsourced jobs are also presented herein.

Abstract: A system and method of recording utterances for building Named Entity Recognition (“NER”) models, which are used to build dialog systems in which a computer listens and responds to human voice dialog. Utterances to be uttered may be provided to users through their mobile devices, which may record the user uttering (e.g., verbalizing, speaking, etc.) the utterances and upload the recording to a computer for processing. The use of the user's mobile device, which is programmed with an utterance collection application (e.g., configured as a mobile app), facilitates the use of crowd-sourcing human intelligence tasking for widespread collection of utterances from a population of users. As such, obtaining large datasets for building NER models may be facilitated by the system and method disclosed herein.

Abstract: A system and method of tagging utterances with Named Entity Recognition (“NER”) labels using unmanaged crowds is provided. The system may generate various annotation jobs in which a user, among a crowd, is asked to tag which parts of an utterance, if any, relate to various entities associated with a domain. For a given domain that is associated with a number of entities that exceeds a threshold N value, multiple batches of jobs (each batch having jobs that have a limited number of entities for tagging) may be used to tag a given utterance from that domain. This reduces the cognitive load imposed on a user, and prevents the user from having to tag more than N entities. As such, a domain with a large number of entities may be tagged efficiently by crowd participants without overloading each crowd participant with too many entities to tag.

Abstract: Systems and methods of validating transcriptions of natural language content using crowdsourced validation jobs are provided herein. In various implementations, a transcription pair comprising natural language content and text corresponding to a transcription of the natural language content may be gathered. A first group of validation devices may be selected for reviewing the transcription pair. A first crowdsourced validation job may be created for the first group of validation devices. The first crowdsourced validation job may be provided to the first group of validation devices. A vote representing whether or not the text accurately represents the natural language content may be received from each of the first group of validation devices. A validation score may be assigned to the transcription pair based, at least in part, on the votes from each of the first group of validation devices.