Abstract: Systems and methods of adaptive workflow definition of crowd sourced tasks and quality control mechanisms for multiple business applications are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, for generating and distributing tasks to workers to optimize product categorization in an online marketplace, generating and distributing tasks to workers for search engine optimization of a website, and/or generating and distributing tasks to workers for business data verification and enhancement. Results can be aggregated from multiple workers to enhance or improve site information, site organization, or other business information to improve site placement in search results and/or to enhance/improve user experience.

Abstract: In an embodiment, process for providing a secure remote workspace includes accessing, via a first client application, a remote desktop application. The process includes activating, within the remote desktop, a second client application to provide access to a task. The process includes obtaining user input in connection with executing the task, and transmitting user input information associated with the execution of the task to a server.

Abstract: Managing annotation jobs is disclosed, including: receiving an annotation job, including: obtaining an input text to be annotated; and obtaining an ontology specifying a plurality of labels to be used to annotate the input text; distributing the annotation job to a plurality of annotator devices via an annotation platform; receiving a plurality of annotation results with respect to the input text from the plurality of annotator devices; and providing an aggregate annotation report corresponding to the annotation job, wherein the aggregate annotation report combines the plurality of annotation results.

Abstract: Image annotation includes: accessing initial object prediction information associated with an image, wherein the initial object prediction information includes a plurality of initial predictions associated with a plurality of objects in the image, including bounding box information associated with the plurality of objects; presenting the image and at least a portion of the initial object prediction information to be displayed; receiving adjusted object prediction information pertaining to at least some of the plurality of objects, wherein the adjusted object prediction information is obtained from a user input made via a user interface configured for a user to make annotation adjustments to at least some of the initial object prediction information; and outputting updated object prediction information, wherein the updated object prediction information is based at least in part on the adjusted object prediction information.

Abstract: A technique is disclosed for automating tracking of annotated objects and improves the throughput and efficiency of existing methods while maintaining a degree of accuracy comparable to a human annotator. In particular, the disclosed technique provides an automated annotated object tracking tool that allows machine-learning teams to annotate an object within a frame and have that annotation persist across frames as the annotated object is tracked within a series of frames, still ensuring that every frame is accurately reviewed by a human where high quality annotation is required. This technique incorporates human feedback via a user adjustment that allows the tool to adapt and improve its accuracy in tracking an annotated object across a sequence of frames.

Abstract: Systems and methods of dynamic optimization for data quality control in crowd sourcing tasks to crowd labor are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, for dynamically monitoring results received from workers for a task distributed for evaluation via a job distribution platform, incrementally assigning additional workers to the task using the results and continuously monitoring additional results to assign any additional workers if needed to meet a quality metric for the task.

Abstract: A set of utterances collected from a plurality of contributors is received. Semantically irrelevant utterances are removed from the set of utterances to obtain a processed set of utterances, including by applying a machine learning model to the set of utterances. An annotation user interface is provided to a plurality of human annotators to perform annotation on the processed set of utterances to obtain an annotated set of utterances. A curation user interface is provided to one or more domain experts to perform curation of the annotated set of utterances to obtain a curated set of utterances. The curated set of utterances is outputted as a training set for an automated dialogue agent.

Abstract: Image annotation includes: accessing an image and a plurality of annotation data sets for the image, wherein the plurality of annotation data sets are made by a plurality of contributors, and the image has a plurality of original image channels; aggregating the plurality of annotation data sets to obtain an aggregated annotation data set for the image; and outputting the aggregated annotation data set. Aggregating the plurality of annotation data sets to obtain an aggregated annotation data set for the image includes: generating an additional image channel based at least in part on weight averages of confidence measures of the plurality of contributors; and applying an object detection model to at least a part of the plurality of original image channels and at least a part of the additional image channel to generate the aggregated annotation data set.

Abstract: Systems and methods of crowd sourcing by distributing a task over a network to multiple workers for completion while providing quality control are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, of identifying an attribute of the task, the task including multiple unit jobs, determining target accuracy in completion of the task by the multiple workers, computing a number of results to be collected for a unit job of the task based on the target accuracy, setting a wage amount that is offered to a worker for completing a unit job of the task based on the attribute and the target accuracy, and/or collecting and aggregating statistics of the workers completing the select unit jobs for quality control.

Abstract: A technique is disclosed for automating tracking of annotated objects and improves the throughput and efficiency of existing methods while maintaining a degree of accuracy comparable to a human annotator. In particular, the disclosed technique provides an automated annotated object tracking tool that allows machine-learning teams to annotate an object within a frame and have that annotation persist across frames as the annotated object is tracked within a series of frames, still ensuring that every frame is accurately reviewed by a human where high quality annotation is required. This technique incorporates human feedback via a user adjustment that allows the tool to adapt and improve its accuracy in tracking an annotated object across a sequence of frames.