Abstract: Example implementations relate to a role-based autoscaling approach for scaling of nodes of a stateful application in a large scale virtual data processing (LSVDP) environment. Information is received regarding a role performed by the nodes of a virtual cluster of an LSVDP environment on which a stateful application is or will be deployed. Role-based autoscaling policies are maintained defining conditions under which the roles are to be scaled. A policy for a first role upon which a second role is dependent specifies a condition for scaling out the first role by a first step and a second step by which the second role is to be scaled out in tandem. When load information for the first role meets the condition, nodes in the virtual cluster that perform the first role are increased by the first step and nodes that perform the second role are increased by the second step.

Abstract: Described herein are systems, methods, and software to enhance the management and deployment of data processing clusters in a computing environment. In one example, a management system may monitor data processing efficiency information for a cluster and determine when the efficiency meets efficiency criteria. When the efficiency criteria are met, the management system may identify a new configuration for the cluster and initiate an operation to implement the new configuration for the cluster.

Abstract: Described herein are systems, methods, and software to enhance the management and deployment of data processing clusters in a computing environment. In one example, a management system may monitor data processing efficiency information for a cluster and determine when the efficiency meets efficiency criteria. When the efficiency criteria are met, the management system may identify a new configuration for the cluster and initiate an operation to implement the new configuration for the cluster.

Abstract: Example implementations relate to a role-based autoscaling approach for scaling of nodes of a stateful application in a large scale virtual data processing (LSVDP) environment. Information is received regarding a role performed by the nodes of a virtual cluster of an LSVDP environment on which a stateful application is or will be deployed. Role-based autoscaling policies are maintained defining conditions under which the roles are to be scaled. A policy for a first role upon which a second role is dependent specifies a condition for scaling out the first role by a first step and a second step by which the second role is to be scaled out in tandem. When load information for the first role meets the condition, nodes in the virtual cluster that perform the first role are increased by the first step and nodes that perform the second role are increased by the second step.

Abstract: Described herein are systems, methods, and software to enhance the management and deployment of data processing clusters in a computing environment. In one example, a management system may monitor data processing efficiency information for a cluster and determine when the efficiency meets efficiency criteria. When the efficiency criteria are met, the management system may identify a new configuration for the cluster and initiate an operation to implement the new configuration for the cluster.

Abstract: Described herein are systems, methods, and software to manage configurations between dependent clusters. In one implementation, a management system maintains a data structure that indicates relationships between clusters in a computing environment. The management system further identifies a configuration modification to a first cluster and identifies other clusters associated with the first cluster based on the data structure. Once the other clusters are identified, the management system may determine configuration modifications for the other clusters based on the data structure and initiate deployment of the configuration modifications.

Abstract: Described herein are systems, methods, and software to enhance the management and deployment of data processing clusters in a computing environment. In one example, a management system may monitor data processing efficiency information for a cluster and determine when the efficiency meets efficiency criteria. When the efficiency criteria are met, the management system may identify a new configuration for the cluster and initiate an operation to implement the new configuration for the cluster.

Abstract: Example implementations relate to a role-based autoscaling approach for scaling of nodes of a stateful application in a large scale virtual data processing (LSVDP) environment. Information is received regarding a role performed by the nodes of a virtual cluster of an LSVDP environment on which a stateful application is or will be deployed. Role-based autoscaling policies are maintained defining conditions under which the roles are to be scaled. A policy for a first role upon which a second role is dependent specifies a condition for scaling out the first role by a first step and a second step by which the second role is to be scaled out in tandem. When load information for the first role meets the condition, nodes in the virtual cluster that perform the first role are increased by the first step and nodes that perform the second role are increased by the second step.

Abstract: Systems, methods, and software described herein to provide data to large-scale processing framework (LSPF) nodes in LSPF clusters. In one example, a method to provide data includes receiving an access request from a LSPF node to access data in accordance with a version of a distributed file system. The method further includes, responsive to the access request, accessing the data for the LSPF node in accordance with a different version of the distributed file system, and presenting the data to the LSPF node in accordance with the version of the distributed file system used by the LSPF node.

Abstract: Systems, methods, and software described herein facilitate interfacing between processing nodes and a plurality of data repositories. In one example, a method of interfacing between a processing node and a plurality of data repositories includes identifying, for the processing node, a data access request using a first data access format, wherein the data access request includes a data connector identifier. The method further includes translating the access request to a second data access format based on the data connector identifier, and identifying a data repository in the plurality of data repositories to service the data access request based on the data connector identifier. The method also provides accessing data for the data access request in the data repository via the second data access format.

Abstract: Systems, methods, and software described herein facilitate servicing of data requests based on quality of service assigned to processing jobs. In one example, a method of prioritizing data requests in a computing system based on quality of service includes identifying a plurality of data requests from a plurality of processing jobs. The method further includes prioritizing the plurality of data requests based on a quality of service assessed to each of the plurality of processing jobs, and assigning cache memory in the computing system to each of the plurality of data requests based on the prioritization.

Abstract: The technology disclosed herein enables optimized managing of cluster deployment on a plurality of host nodes. In a particular embodiment, a method includes defining parameters of a cluster for executing a process that will execute in a plurality of containers distributed across one or more of the plurality of host nodes. The method further provides adding a first container portion of the plurality of containers to a first host node portion of the plurality of host nodes. After adding the first container portion, the method includes determining that a remaining host node portion of the plurality of host nodes will not support more of the plurality of containers and adjusting the parameters of the cluster to allow the process to execute on the first host node portion.

Abstract: Described herein are systems, methods, and software to manage configurations between dependent clusters. In one implementation, a management system maintains a data structure that indicates relationships between clusters in a computing environment. The management system further identifies a configuration modification to a first cluster and identifies other clusters associated with the first cluster based on the data structure. Once the other clusters are identified, the management system may determine configuration modifications for the other clusters based on the data structure and initiate deployment of the configuration modifications.

Abstract: Described herein are systems, methods, and software to manage the allocation of large-scale data processing clusters in a computing environment. In one implementation, a management system obtains a request for a new data processing cluster. In response to the request, the management system may determine a tenant associated with the request and determine computing systems available to the tenant. Once identified, the management system may select at least one of the computing systems to support the request and deploy one or more virtual nodes to the at least one computing system.

Abstract: Embodiments disclosed herein provide systems, methods, and computer readable media for managing data consumption rate in a virtual data processing environment. In a particular embodiment, a method provides, in a cache node of a host system, identifying read completions for one or more virtual machines instantiated in the host system, with the one or more virtual machines processing one or more processing jobs. The method further provides allocating the read completions to individual processing jobs of the one or more processing jobs and accumulating the read completions on a per-job basis, with the cache node determining a data consumption rate for each processing job of the one or more processing jobs.

Abstract: Described herein are systems, methods, and software to enhance the management and deployment of data processing clusters in a computing environment. In one example, a management system may monitor data processing efficiency information for a cluster and determine when the efficiency meets efficiency criteria. When the efficiency criteria are met, the management system may identify a new configuration for the cluster and initiate an operation to implement the new configuration for the cluster.

Abstract: Systems, methods, and software described herein facilitate the management of virtual clusters in a large-scale processing environment. In one implementation, a method of operating a control node includes receiving a request to configure a virtual cluster with one or more data processing nodes, and identifying a tenant associated with the request. The method further provides identifying a namespace for the tenant, and identifying internet protocol (IP) addresses for the one or more data processing nodes. The method also includes generating namespace to IP address pairs for the one or more data processing nodes based on the namespace and the IP addresses, and configuring a domain name system (DNS) for the virtual cluster with the namespace to IP address pairs.

Abstract: Systems, methods, and software described herein provide enhancements for initiating application services across a virtual environment. In one implementation, a method of deploying application services includes initiating configuration process for a first service and a second service across virtual nodes, wherein the first service is dependent on the availability of the second service. The method further includes holding the configuration of the first service, completing the configuration of the second service, and responsively transferring a notification from the virtual node or nodes of the second service to the virtual node or nodes of the first service, permitting the configuration of the first service to be completed.

Abstract: Systems, methods, and software described herein provide for enhancements to large scale data processing architectures. In one implementation, a service architecture for large scale data processing includes a host computing system, and a virtual machine executing on the host computing system. The service architecture further includes a plurality of application containers executing on the virtual machine, wherein each of the application containers comprises a large scale processing node running one or more java virtual machines.

Abstract: Systems, methods, and software described herein allocate cache memory to job processes executing on a processing node. In one example, a method of allocating cache memory to a plurality of job process includes identifying the plurality of job processes executing on a processing node, and identifying a data object to be accessed by the plurality of job processes. The method further provides allocating a portion of the cache memory to each job process in the plurality of job processes and, for each job process in the plurality of job processes, identifying a segment of data from the data object, wherein the segment of data comprises a requested portion of data and a predicted portion of data. The method also includes providing the segments of data to the allocated portions of the cache memory.