Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for a task centric resource scheduling framework. A scheduler executing on a master node computer of a distributed parallel computing system allocates computing resources of the parallel computing system to a program according to one or more policies associated with the program. Each policy includes a set of pre-determined computing resource constraints. Allocation of the computing resources includes performing multiple iterations of negotiation between the scheduler and a resource manager of the parallel computing system. In each iteration, a policy engine of the scheduler submits requests to get more resources from, or requests to release already acquired resources to, the resource manager. The policy engine generates the requests by balancing suggestions provided by analyzer components of the policy engine and a corresponding policy.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for a distributed parallel computing system to adapt a foreign program to execute on the distributed parallel computing system. The foreign program is a program written for a computing framework that is different from a computing framework of the parallel computing system. The distributed parallel computing system includes a master node computer and one or more worker node computers. A scheduler executing on the master node computer acts as an intermediary between the foreign program and the parallel computing system. The scheduler negotiates with a resource manager of the parallel computing system to acquire computing resources. The scheduler then allocates the computing resources to the worker node computers as containers. The foreign program executes in the containers on the worker node computers in parallel.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for a distributed parallel computing system to adapt a foreign program to execute on the distributed parallel computing system. The foreign program is a program written for a computing framework that is different from a computing framework of the parallel computing system. The distributed parallel computing system includes a master node computer and one or more worker node computers. A scheduler executing on the master node computer acts as an intermediary between the foreign program and the parallel computing system. The scheduler negotiates with a resource manager of the parallel computing system to acquire computing resources. The scheduler then allocates the computing resources to the worker node computers as containers. The foreign program executes in the containers on the worker node computers in parallel.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for a task centric resource scheduling framework. A scheduler executing on a master node computer of a distributed parallel computing system allocates computing resources of the parallel computing system to a program according to one or more policies associated with the program. Each policy includes a set of pre-determined computing resource constraints. Allocation of the computing resources includes performing multiple iterations of negotiation between the scheduler and a resource manager of the parallel computing system. In each iteration, a policy engine of the scheduler submits requests to get more resources from, or requests to release already acquired resources to, the resource manager. The policy engine generates the requests by balancing suggestions provided by analyzer components of the policy engine and a corresponding policy.