Abstract: A technique for multi-layer quality of service (QoS) management in a distributed computing environment includes receiving a workload to run in a distributed computing environment. A workload quality of service (QoS) class for the workload is identified, and the workload QoS class is translated to a storage level QoS class. The workload is scheduled to run on a compute node of the environment, and the storage level QoS class is communicated to a workload execution manager of the compute node. The storage level QoS class is communicated to one or more storage managers of the environment where the storage managers manage storage resources in the environment. The storage managers extend the storage level QoS class to the storage resources to support the workload QoS class.

Abstract: A system for multi-layer quality of service (QoS) management in a distributed computing environment includes a management node hosting a workload scheduler operable to receive a workload and identify a workload QoS class for the workload. The system also includes a plurality of distributed compute nodes. A workload scheduler is operable to schedule running of the workload on the compute nodes. The workload scheduler is operable to translate the workload QoS class to a storage level QoS class and communicate the storage level QoS class to a workload execution manager of the compute nodes. The workload scheduler communicates the storage level QoS class to one or more storage managers where the storage managers manage storage resources, and the storage managers are operable to extend the storage level QoS class to the storage resources to support the workload QoS class.

Abstract: In some examples, a print cartridge includes a memory device comprising quantized coefficients derived from a compression of a difference table including a plurality of difference nodes in which each difference node represents a value that is a difference of a value of a node of a color table and a value of a corresponding node of a reference table, the quantized coefficients useable to produce a reconstructed difference table, and a plurality of residue nodes in which each residue node represents a value that is a difference of a value of a node of the color table and a value of a corresponding node of the reconstructed difference table.

Abstract: In some examples, a print cartridge includes a memory device comprising quantized coefficients derived from a compression of a difference table including a plurality of difference nodes in which each difference node represents a value that is a difference of a value of a node of a color table and a value of a corresponding node of a reference table, the quantized coefficients useable to produce a reconstructed difference table, and a plurality of residue nodes in which each residue node represents a value that is a difference of a value of a node of the color table and a value of a corresponding node of the reconstructed difference table.

Abstract: Methods and arrangements for assembling tasks in a progressive queue. At least one job is received, each job comprising a dependee set of tasks and a depender set of at least one task. The dependee tasks are assembled in a progressive queue for execution, and the dependee tasks are executed. Other variants and embodiments are broadly contemplated herein.

Abstract: At a cache manager of a directed acyclic graph-based data analytic platform, memory usage statistics are obtained from each of a plurality of monitor components on a plurality of worker nodes. The worker nodes have a plurality of tasks executing thereon, and each of the tasks has at least one distributed dataset associated therewith. Each of the worker nodes has a distributed dataset cache. At least one of the following is carried out: increasing a size of a given one of the distributed dataset caches if the memory usage statistics indicate that corresponding ones of the tasks are using too little memory; and decreasing a size of another given one of the distributed dataset caches if the memory usage statistics indicate contention between corresponding ones of the tasks and a corresponding one of the distributed datasets.

Abstract: Methods and arrangements for managing data segments. At least one job is received, each job comprising a dependee set of tasks and a depender set of at least one task, and the at least one of the dependee set of tasks is executed. There is extracted, from the at least one of the dependee set of tasks, at least one service common to at least another of the dependee set of tasks. Other variants and embodiments are broadly contemplated herein.

Abstract: Methods and arrangements for yielding resources in data processing. At least one job is received, each job comprising a dependee set of tasks and a depender set of at least one task, and the at least one of the dependee set of tasks is executed. At least one resource of the at least one of the dependee set of tasks is yielded upon detection of resource underutilization in at least one other location. Other variants and embodiments are broadly contemplated herein. Other variants and embodiments are broadly contemplated herein.

Abstract: Embodiments of the present invention provide systems and methods for allocating multiple resources. In one embodiment, a configured resource plan is used to construct a hierarchical tree. The system then identifies a set of unowned resources from the configured resource plan and sends the set of unowned resource to a share pool. The share pool is either a global or local pool and can be accessed by one or more consumers. In response to changes in workload demands, a set of unused resources are lent to a global or local pool.

Abstract: Embodiments of the present invention provide systems and methods for allocating multiple resources. In one embodiment, a configured resource plan is used to construct a hierarchical tree. The system then identifies a set of unowned resources from the configured resource plan and sends the set of unowned resource to a share pool. The share pool is either a global or local pool and can be accessed by one or more consumers. In response to changes in workload demands, a set of unused resources are lent to a global or local pool.

Abstract: A technique for managing pooled resources in a distributed computing environment includes a resource manager executable to receive resource allocation requests from a plurality of workload managers for running one or more applications. The applications are initiated by a consumer through the workload managers. Each workload manager is configured to translate demand requests from the applications into resource allocation requests. A determination is made whether there are available resources to satisfy the resource requests according to a resource allocation policy. Resources are allocated to the workload managers where the applications can continue running on the allocated resources if the resource manager suffers downtime or is taken offline. The resource allocation policy is determined by a consumer profile of the consumer specifying ownership and share rights to the requested resources.

Abstract: A single workload scheduler schedules sessions and tasks having a tree structure to resources, wherein the single workload scheduler has scheduling control of the resources and the tasks of the parent-child workload sessions and tasks. The single workload scheduler receives a request to schedule a child session created by a scheduled parent task that when executed results in a child task; the scheduled parent task is dependent on a result of the child task. The single workload scheduler receives a message from the scheduled parent task yielding a resource based on the resource not being used by the scheduled parent task, schedules tasks to backfill the resource, and returns the resource yielded by the scheduled parent task to the scheduled parent task based on receiving a resume request from the scheduled parent task or determining dependencies of the scheduled parent task have been met.

Abstract: In an approach to adaptively pipeline a MapReduce job, a processor receives one or more data records from a storage and inserts the one or more data records into a map queue, wherein a size of the map queue is adaptive to one or more utilizations of one or more resources in the processor. One or more processors apply a map function to the one or more data records in the first buffer and sort the records that are output from the map function and store the sorted records. One or more processors receive and insert the sorted records into a reduce queue, wherein a size of the reduce queue is adaptive to one or more utilizations of resources in the one or more processors. One or more processors apply a reduce function to the sorted records in the reduce queue and store a result in a storage.

Abstract: In an approach to adaptively pipeline a MapReduce job, a processor receives one or more data records from a storage and inserts the one or more data records into a map queue, wherein a size of the map queue is adaptive to one or more utilizations of one or more resources in the processor. One or more processors apply a map function to the one or more data records in the first buffer and sort the records that are output from the map function and store the sorted records. One or more processors receive and insert the sorted records into a reduce queue, wherein a size of the reduce queue is adaptive to one or more utilizations of resources in the one or more processors. One or more processors apply a reduce function to the sorted records in the reduce queue and store a result in a storage.

Abstract: A shuffler receives information associated with partition segments of map task outputs and a pipeline policy for a job running on a computing device. The shuffler transmits to an operating system of the computing device a request to lock partition segments of the map task outputs and transmits an advisement to keep or load partition segments of map task outputs in the memory of the computing device. The shuffler creates a pipeline based on the pipeline policy, wherein the pipeline includes partition segments locked in the memory and partition segments advised to keep or load in the memory, of the computing device for the job, and the shuffler selects the partition segments locked in the memory, followed by partition segments advised to keep or load in the memory, as a preferential order of partition segments to shuffle.

Abstract: A single workload scheduler schedules sessions and tasks having a tree structure to resources, wherein the single workload scheduler has scheduling control of the resources and the tasks of the parent-child workload sessions and tasks. The single workload scheduler receives a request to schedule a child session created by a scheduled parent task that when executed results in a child task; the scheduled parent task is dependent on a result of the child task. The single workload scheduler receives a message from the scheduled parent task yielding a resource based on the resource not being used by the scheduled parent task, schedules tasks to backfill the resource, and returns the resource yielded by the scheduled parent task to the scheduled parent task based on receiving a resume request from the scheduled parent task or determining dependencies of the scheduled parent task have been met.

Abstract: A resource manager for managing the sharing of resources among multiple workloads in a distributed computing environment. The resource manager comprises an allocation service component which applies a resource allocation policy to determine whether the requested resources may be allocated. The resource manager also comprises a dynamic provisioning service component which applies a reprovisioning policy to modify the make-up of the resources in response to unsatisfied resource requests. The allocation of resources to consumers may be determined in accordance with the resource ownership and share rights of the consumer on behalf of which the application or workload is being demanded.

Abstract: A method to determine a hierarchical fair share of resources among one or more scheduling peers. The method determines a greatest fairness difference between the usage percentage difference of a most-served competing peer and that of a least-served competing peer. The method determines a greatest accumulated fairness difference between the accumulated usage percentage difference of the most-served competing peer and that of the least-served competing peer. The method adjusts a resource allocation such that resources from the most-served competing peer are allocated to the least-served competing peer if an adjustment condition is met, wherein the adjustment condition is based on one or both of i) the greatest fairness difference, and ii) whether the greatest accumulated fairness difference exceeds at least one threshold.

Abstract: A method to determine a hierarchical fair share of resources among one or more scheduling peers. The method determines a greatest fairness difference between the usage percentage difference of a most-served competing peer and that of a least-served competing peer. The method determines a greatest accumulated fairness difference between the accumulated usage percentage difference of the most-served competing peer and that of the least-served competing peer. The method adjusts a resource allocation such that resources from the most-served competing peer are allocated to the least-served competing peer if an adjustment condition is met, wherein the adjustment condition is based on one or both of i) the greatest fairness difference, and ii) whether the greatest accumulated fairness difference exceeds at least one threshold.

Abstract: A technique for multi-layer quality of service (QoS) management in a distributed computing environment includes receiving a workload to run in a distributed computing environment. A workload quality of service (QoS) class for the workload is identified, and the workload QoS class is translated to a storage level QoS class. The workload is scheduled to run on a compute node of the environment, and the storage level QoS class is communicated to a workload execution manager of the compute node. The storage level QoS class is communicated to one or more storage managers of the environment where the storage managers manage storage resources in the environment. The storage managers extend the storage level QoS class to the storage resources to support the workload QoS class.