Abstract: A method and system for determining a result for a task executed in a crowd-sourced environment is disclosed. The method comprises receiving, a plurality of results of the task having been submitted by a plurality of human assessors; receiving a quality score for each human assessor of the plurality of human assessors; generating a plurality of vector representations comprising a vector representation for each of the results; mapping, the plurality of vector representations into a vector space; clustering the plurality of vector representations into at least a first cluster and a second cluster; executing a machine learning algorithm configured to generate a first confidence parameter and a second confidence parameter; in response to a given one of the first confidence parameter and the second confidence parameter meeting a predetermined condition, generating, an aggregated vector representation; and selecting the aggregated vector representation as the result of the task.

Abstract: Disclosed are a method and a system for determining a response to a digital task in a computer-implemented crowd-sourced environment. The method comprises determining if a number of the plurality of responses to the digital task received meets a pre-determined minimum answer threshold; in response to the number of the plurality of responses to the digital task meeting the pre-determined minimum answer threshold, executing: for each of the plurality of responses generating, by the server, a confidence parameter representing a probability of an associated one of the plurality of responses being correct; ranking the plurality of responses based on the confidence parameter to determine a top response being associated with a highest confidence parameter; and in response to the highest confidence parameter being above a pre-determined minimum confidence threshold, assigning a value of the top response as a label for the digital task and terminating the digital task execution.

Abstract: Non-limiting embodiments of the present technology are directed to a method and system for generating a training dataset. The method comprises: accessing data associated with a plurality of assessors executing digital tasks of a first type and digital tasks of a second type; generating, a first ranked list of assessors and a second ranked list of assessors based on their past performance; for a given one of the plurality of assessors: generating, a class score for the common class of digital tasks; acquiring a request for executing a digital task of a third type; ranking, the plurality of assessors based on respective class scores, the given one from the plurality of assessors being one of top ranked ones from the plurality of assessors; transmitting the digital task of the third type to the given one; generating the training data for the MLA based on a response from the given one.

Abstract: There is disclosed a method system for selecting a label for a task, the method comprising: receiving a plurality of labels, each of the label included within the plurality of labels being indicative of a given assessor's perceived preference of a first object of over a second object; analyzing the comparison task to determine a set of latent biasing features; executing a MLA configured to generating a respective latent score parameter for the first object and the second object, the respective latent score parameter indicative of a probable offset between the given assessor's perceived preference and an unbiased preference parameter of the first object over the second object; generating a predicted bias degree parameter for the given assessor; generating the unbiased preference parameter; using, by the server, the unbiased preference parameter as the label for the comparison task for the given assessor.

Abstract: A method and a system for generating training data for an MLA are provided. The method comprises: retrieving assessor data including data indicative of a plurality of results responsive to the given digital task having been submitted to a set of assessors; determining, for a given result, a number of instances thereof within the plurality of results; determining, a respective value of an aggerate quality metric associated with the given result; identifying a reliable result of the plurality of results as being associated with a maximum value of the aggregate quality metric; determining, based on the reliable result, updated quality scores for each one of the current set of assessors; generating, based on the respective updated quality score, an updated set of assessors; and generating the training data for the MLA including data generated in response to respective ones of the updated set of assessors completing a subsequent digital task.

Abstract: A method and a system for generating training data for an MLA are provided. The method comprises: retrieving assessor data associated with a plurality of assessors, the assessor data including data indicative of a plurality of results responsive to a given digital task having been submitted to the plurality of assessors; based on the plurality of results, determining at least one set of assessors in the plurality of assessors, such that a consistency metric amongst results provided by the at least one set of assessors for the given digital task is maximized, transmitting a subsequent digital task to respective electronic devices associated with the at least one set of assessors; and generating the training data for the computer-executable MLA including data generated in response to respective ones of the at least one set of assessors completing the subsequent digital task.

Abstract: A method of completing a task, the task being of a given type of task. The method includes receiving, by a server, an indication of a first result of the task having been completed by a human assessor, executing by the server a machine learning algorithm (MLA) to complete the task by the MLA to generate a second result of the task, determining, by the server, a confidence level parameter indicative of a probability of the first result being correct, and determining, by the server, whether the probability indicated by the determined confidence level parameter exceeds a pre-defined threshold probability. In response to determining that the probability indicated by the determined confidence level parameter exceeds the pre-defined threshold probability, the server processes the task as having been completed with the first result.

Abstract: There is disclosed a method system for selecting a label for a task, the method comprising: receiving a plurality of labels, each of the label included within the plurality of labels being indicative of a given assessor's perceived preference of a first object of over a second object; analyzing the comparison task to determine a set of latent biasing features; executing a MLA configured to generating a respective latent score parameter for the first object and the second object, the respective latent score parameter indicative of a probable offset between the given assessor's perceived preference and an unbiased preference parameter of the first object over the second object; generating a predicted bias degree parameter for the given assessor; generating the unbiased preference parameter; using, by the server, the unbiased preference parameter as the label for the comparison task for the given assessor.

Abstract: Disclosed are a method and a system for determining a response to a digital task in a computer-implemented crowd-sourced environment. The method comprises determining if a number of the plurality of responses to the digital task received meets a pre-determined minimum answer threshold; in response to the number of the plurality of responses to the digital task meeting the pre-determined minimum answer threshold, executing: for each of the plurality of responses generating, by the server, a confidence parameter representing a probability of an associated one of the plurality of responses being correct; ranking the plurality of responses based on the confidence parameter to determine a top response being associated with a highest confidence parameter; and in response to the highest confidence parameter being above a pre-determined minimum confidence threshold, assigning a value of the top response as a label for the digital task and terminating the digital task execution.

Abstract: A method and system for determining a result for a task executed in a crowd-sourced environment is disclosed. The method comprises receiving, a plurality of results of the task having been submitted by a plurality of human assessors; receiving a quality score for each human assessor of the plurality of human assessors; generating a plurality of vector representations comprising a vector representation for each of the results; mapping, the plurality of vector representations into a vector space; clustering the plurality of vector representations into at least a first cluster and a second cluster; executing a machine learning algorithm configured to generate a first confidence parameter and a second confidence parameter; in response to a given one of the first confidence parameter and the second confidence parameter meeting a predetermined condition, generating, an aggregated vector representation; and selecting the aggregated vector representation as the result of the task.

Abstract: A method of completing a task, the task being of a given type of task. The method includes receiving, by a server, an indication of a first result of the task having been completed by a human assessor, executing by the server a machine learning algorithm (MLA) to complete the task by the MLA to generate a second result of the task, determining, by the server, a confidence level parameter indicative of a probability of the first result being correct, and determining, by the server, whether the probability indicated by the determined confidence level parameter exceeds a pre-defined threshold probability. In response to determining that the probability indicated by the determined confidence level parameter exceeds the pre-defined threshold probability, the server processes the task as having been completed with the first result.

Abstract: There is disclosed a computer implemented method of generating a training object for training a machine learning algorithm (MLA). The method comprises: acquiring a digital training document to be used in the training; transmitting the digital training document to a plurality of assessors, transmitting further including indicating a range of possible labels for the assessors to assess from, the range of possible labels including at least a first possible label and a second possible label; obtaining from each of the plurality of assessors a selected label to form a pool of selected labels; generating a consensus label distribution based on the pool of selected labels, the consensus label distribution representing a range of perceived labels for the digital training document and an associated probability score for each of the perceived labels; and training the machine learning algorithm using the digital training document and the consensus label distribution.

Abstract: There is disclosed a computer implemented method of generating a training object for training a machine learning algorithm (MLA). The method comprises: acquiring a digital training document to be used in the training; transmitting the digital training document to a plurality of assessors, transmitting further including indicating a range of possible labels for the assessors to assess from, the range of possible labels including at least a first possible label and a second possible label; obtaining from each of the plurality of assessors a selected label to form a pool of selected labels; generating a consensus label distribution based on the pool of selected labels, the consensus label distribution representing a range of perceived labels for the digital training document and an associated probability score for each of the perceived labels; and training the machine learning algorithm using the digital training document and the consensus label distribution.

Abstract: There is disclosed a computer implemented method for training a search ranker, the search ranker being configured to ranking search results. The method comprises: retrieving, by the server, a training dataset including a plurality of training objects; for each training object, based on the corresponding associated object feature vector: determining a weight parameter, the weight parameter being indicative of a quality of the label; determining a relevance parameter, the relevance parameter being indicative of a moderated value of the labels relative to other labels within the training dataset; training the search ranker using the plurality of training objects of the training dataset, the determined relevance parameter for each training object of the plurality of training objects of the training dataset, and the determined weight parameter for each object of the plurality of training objects of the training dataset to rank a new document.