Abstract: A manager agent access a grain hierarchy that represents a state for a computation that is to performed by a distributed processing system, wherein the grain hierarchy includes manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system, and worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task. A manager agent performs processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent, and worker agents perform tasks assigned to the worker agents based on an assignment of a respective worker agent grain to the worker agent.

Abstract: A manager agent access a grain hierarchy that represents a state for a computation that is to performed by a distributed processing system, wherein the grain hierarchy includes manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system, and worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task. A manager agent performs processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent, and worker agents perform tasks assigned to the worker agents based on an assignment of a respective worker agent grain to the worker agent.

Abstract: A method and a system for variable aggregation among large-scale data processing workers is discussed. Workers join groups and add variables to the groups. One worker is elected as the group leader. Workers post values to group variables, which are aggregated by the leader. The leader also announces the aggregated values. This variable aggregation allows for asynchronous communication among data processing workers.

Abstract: Aspects of the disclosure relate to learning the most useful backup tasks when processing a job in a distributed computing application. For example, training data may be generated by running a job and running backup tasks for some of the tasks for that job. List of features may be generated for each backup task at different times. A representative list of features may be labeled according to the backup tasks usefulness to the job. The lists and usefulness labels may be used to generate a model of usefulness. The model may then be used to evaluate the usefulness of the tasks for the next job. This information may then be used to schedule backup tasks for execution. In addition, lists of features for these backup tasks may be generated and then used to update the model in a continuous learning loop.

Abstract: Parallel data processing may include map and reduce processes. Map processes may include at least one input thread and at least one output thread. Input threads may apply map operations to produce key/value pairs from input data blocks. These pairs may be sent to internal shuffle units which distribute the pairs, sending specific pairs to particular output threads. Output threads may include multiblock accumulators to accumulate and/or multiblock combiners to combine values associated with common keys in the key/value pairs. Output threads can output intermediate pairs of keys and combined values. Likewise, reduce processes may access intermediate key/value pairs using multiple input threads. Reduce operations may be applied to the combined values associated with each key. Reduce processes may contain internal shuffle units that distributes key/value pairs and sends specific pairs to particular output threads. These threads then produce the final output.