Abstract: A reputation system to evaluate work is disclosed. Answer data for each of one or more unsupervised workers is received. The answer data includes one or more answers each representing a judgment by the worker that reflects on the quality of a given work product. The reputation system determines programmatically, based at least in part on a reputation data of an owner of the work product, the answer data, and a respective reputation data of the respective unsupervised workers, which answers are correct.
Abstract: Using a distributed set of unsupervised workers to produce a work product is disclosed. In some embodiments, a work product is received. A review task to review the work product is provided to a reviewing worker included in the set of unsupervised workers. A result of the review task is received. A determination is made, based at least in part on the review result, whether the work product satisfies an acceptance criteria.