Abstract: Crowdsourcing systems with machine learning are described. Specifically, item-label inference methods and systems are presented, for example, to provide aggregated answers to a crowdsourced task in a manner achieving good accuracy even where observed data about past behavior of crowd members is sparse. In various examples, an item-label inference system infers variables describing characteristics of both individual crowd workers and communities of the workers. In various examples, an item-label inference system provides aggregated labels while considering the inferred worker characteristics and the inferred characteristics of the worker communities. In examples the item-label inference system provides uncertainty information associated with the inference results for selecting workers and generating future tasks.

Abstract: Crowdsourcing systems with machine learning are described. Specifically, item-label inference methods and systems are presented, for example, to provide aggregated answers to a crowdsourced task in a manner achieving good accuracy even where observed data about past behavior of crowd members is sparse. In various examples, an item-label inference system infers variables describing characteristics of both individual crowd workers and communities of the workers. In various examples, an item-label inference system provides aggregated labels while considering the inferred worker characteristics and the inferred characteristics of the worker communities. In examples the item-label inference system provides uncertainty information associated with the inference results for selecting workers and generating future tasks.

Abstract: Crowdsourcing systems with machine learning are described, for example, to aggregate answers to a crowdsourced task in a manner achieving good accuracy even where observed data about past behavior of crowd members is sparse. In various examples a machine learning system jointly learns variables describing characteristics of both individual crowd workers and communities of the workers. In various examples, the machine learning system learns aggregated labels. In examples learnt variables describing characteristics of an individual crowd worker are related, by addition of noise, to learnt variables describing characteristics of a community of which the individual is a member. In examples the crowdsourcing system uses the learnt variables describing characteristics of individual workers and of communities of workers for any one or more of: active learning, targeted training of workers, targeted issuance of tasks, calculating and issuing rewards.

Abstract: Results, generated by human workers in response to HITs assigned to them, are evaluated based upon the behavior of the human workers in generating such results. Workers receive, together with an intelligence task to be performed, a behavior logger by which the worker's behavior is monitored while the worker performs the intelligence task. Machine learning is utilized to identify behavioral factors upon which the evaluation can be based and then to learn how to utilize such behavioral factors to evaluate the HIT results generated by workers, as well as the workers themselves. The identification of behavioral factors, and the subsequent utilization thereof, is informed by the behavior of, and corresponding results generated by, a trusted set of workers. Results evaluated to have been improperly generated can be discarded or simply downweighted. Workers evaluated to be operating improperly can be removed or retrained.

Abstract: Crowdsourcing systems with machine learning are described, for example, to aggregate answers to a crowdsourced task in a manner achieving good accuracy even where observed data about past behavior of crowd members is sparse. In various examples a machine learning system jointly learns variables describing characteristics of both individual crowd workers and communities of the workers. In various examples, the machine learning system learns aggregated labels. In examples learnt variables describing characteristics of an individual crowd worker are related, by addition of noise, to learnt variables describing characteristics of a community of which the individual is a member. In examples the crowdsourcing system uses the learnt variables describing characteristics of individual workers and of communities of workers for any one or more of: active learning, targeted training of workers, targeted issuance of tasks, calculating and issuing rewards.

Abstract: Methods of dynamic control of an electronic message system are described. In an embodiment, a system which is separate from a messaging service within the electronic message system generates an event signal which relates to an event external to the messaging service. The event signal is received by the messaging service and this triggers a dynamic update in one or more electronic messages which are identified based on the event signal received. For example, the dynamic update may result in the message content being displayed in a different manner or the message being delivered or deleted. Examples of external events include a community response to a particular message, based on a subset of information about the message which has been shared, the location of one or more users and a change in membership of a group.

Abstract: Tagging of resources in order to associate them is described. In an embodiment it is possible to assign tags to resources or switch between resources with different tags, in the context of current work being undertaken by a user. For example, from a single application window that is currently used, a user is able to switch to other resources by tag in an embodiment. Different embodiments illustrate how tags may have multiple purposes and those purposes may evolve thus enabling associated resources to be exposed in different ways. In some embodiments rich metadata about tag usage and/or the use of resource associated with a tag is stored and used to provide additional functionality. Examples are described in which persistence of resources associated with tags is provided and may involve representation of resources such as by duplication of files or by creating alternative representations of resources.

Abstract: A global currency for crowdsourcing which comprises stored credibility values for every buyer (of human intelligence) and seller (of human intelligence) in a crowdsourcing system is described, creating an ecosystem where buyers and sellers are interdependent on each other. This dependence is the property of a global currency of credibility, where a buyer's credibility is a function of the credibility of the sellers who engaged with HITs published by the buyer, while the credibility of a seller is a function of the credibility scores associated with the HITs, which in turn is dependent on the buyer's credibility. The credibility scores are updated with every HIT completion and propagated through a network that connects HITs with buyers, sellers and platforms, as well as sellers with other sellers and buyers with other buyers. Buyers and sellers can bid, auction and refer HITs as a function of their credibility scores.

Abstract: Methods of dynamic control of an electronic message system are described. In an embodiment, a system which is separate from a messaging service within the electronic message system generates an event signal which relates to an event external to the messaging service. The event signal is received by the messaging service and this triggers a dynamic update in one or more electronic messages which are identified based on the event signal received. For example, the dynamic update may result in the message content being displayed in a different manner or the message being delivered or deleted. Examples of external events include a community response to a particular message, based on a subset of information about the message which has been shared, the location of one or more users and a change in membership of a group.

Abstract: Tagging of resources in order to associate them is described. In an embodiment it is possible to assign tags to resources or switch between resources with different tags, in the context of current work being undertaken by a user. For example, from a single application window that is currently used, a user is able to switch to other resources by tag in an embodiment. Different embodiments illustrate how tags may have multiple purposes and those purposes may evolve thus enabling associated resources to be exposed in different ways. In some embodiments rich metadata about tag usage and/or the use of resource associated with a tag is stored and used to provide additional functionality. Examples are described in which persistence of resources associated with tags is provided and may involve representation of resources such as by duplication of files or by creating alternative representations of resources.