Abstract: Systems, methods, and software described herein facilitate an enhanced service architecture for large-scale data processing. In one implementation, a method of providing data to a large-scale data processing architecture includes identifying a data request from a container in a plurality of containers executing on a host system, wherein the plurality of containers each run an instance of a large-scale processing framework. The method further provides identifying a storage repository for the data request, and accessing data associated with the data request from the storage repository. The method also includes caching the data in a portion of a cache memory on the host system allocated to the container, wherein the cache memory comprises a plurality of portions each allocated to one of the plurality of containers.

Abstract: Systems, methods, and software described herein enhance the generation of large-scale processing framework clusters and corresponding edge services. In one implementation, a method includes identifying a request for a large-scale processing cluster, and identifying one or more edge services based on the type and version of the large-scale processing cluster. The method further provides generating a user interface that indicates the one or more edge services available, receiving a selection of at least one edge service in the one or more edge services, and initiating execution of the large-scale processing framework cluster and the at least one edge service, wherein the at least one edge service is provided with configuration information for the large-scale processing framework cluster.

Abstract: Systems, methods, and software described herein facilitate the allocation of large scale processing jobs to host computing systems. In one example, a method of operating an administration node to allocate processes to a plurality of host computing systems includes identifying a job process for a large scale processing environment (LSPE), and identifying a data repository associated with the job process. The method further includes obtaining data retrieval performance information related to the data repository and the host systems in the LSPE. The method also provides identifying a host system in the host systems for the job process based on the data retrieval performance information, and initiating a virtual node for the job process on the identified host system.

Abstract: Systems, methods, and software described herein facilitate the allocation of large scale processing jobs to host computing systems. In one example, a method of allocating job processes to a plurality of host computing systems in a large scale processing environment includes identifying a job process for the large scale processing environment, and obtaining accommodation data for a plurality of host computing systems in the large scale processing environment. The method further provides identifying a host computing system in the plurality of host computing systems for the job process based on the accommodation data, and initiating a virtual node on the host computing system for the job process.

Abstract: Systems, methods, and software described herein facilitate the management of virtual clusters in a large-scale processing environment. In one implementation, a method of operating a control node includes receiving a request to configure a virtual cluster with one or more data processing nodes, and identifying a tenant associated with the request. The method further provides identifying a namespace for the tenant, and identifying internet protocol (IP) addresses for the one or more data processing nodes. The method also includes generating namespace to IP address pairs for the one or more data processing nodes based on the namespace and the IP addresses, and configuring a domain name system (DNS) for the virtual cluster with the namespace to IP address pairs.

Abstract: Systems, methods, and software described herein facilitate the management of virtual clusters in a large-scale processing environment. In one implementation, a method of operating a control node includes receiving a request to configure a virtual cluster with one or more data processing nodes, and identifying a tenant associated with the request. The method further provides identifying a namespace for the tenant, and identifying internet protocol (IP) addresses for the one or more data processing nodes. The method also includes generating namespace to IP address pairs for the one or more data processing nodes based on the namespace and the IP addresses, and configuring a domain name system (DNS) for the virtual cluster with the namespace to IP address pairs.

Abstract: A method for migrating a virtual machine disk (VM disk) from first physical storage to second physical storage while the virtual machine (VM) is running, the method comprising: (a) creating a first child VM disk to which writes are redirected from a first parent VM disk, the first parent VM disk being on the first physical storage; (b) copying the first parent VM disk to the second physical storage as a second parent VM disk; (c) re-parenting the first VM child disk to the second parent VM disk; and (d) consolidating the first child VM disk and the second parent VM disk.

Abstract: Systems, methods, and software described herein provide for enhancements to large scale data processing architectures. In one implementation, a service architecture for large scale data processing includes a host computing system, and a virtual machine executing on the host computing system. The service architecture further includes a plurality of application containers executing on the virtual machine, wherein each of the application containers comprises a large scale processing node running one or more java virtual machines.

Abstract: Systems, methods, and software described herein provide for enhancements to large scale data processing architectures. In one implementation, a service architecture for large scale data processing includes a host computing system, and a virtual machine executing on the host computing system. The service architecture further includes a plurality of application containers executing on the virtual machine, wherein each of the application containers comprises a large scale processing node running one or more java virtual machines.

Abstract: Systems, methods, and software described herein provide enhancements for initiating application services across a virtual environment. In one implementation, a method of deploying application services includes initiating configuration process for a first service and a second service across virtual nodes, wherein the first service is dependent on the availability of the second service. The method further includes holding the configuration of the first service, completing the configuration of the second service, and responsively transferring a notification from the virtual node or nodes of the second service to the virtual node or nodes of the first service, permitting the configuration of the first service to be completed.

Abstract: Described herein are systems, methods, and software for translating data requests in a data processing cluster. In one example, a method of operating a cache service to interface between a virtual machine cluster and job data associated with a job executed by the virtual machine cluster includes identifying a request initiated by the virtual machine cluster to access at least a portion of the job data in accordance with a first distributed object access protocol. The method further includes in response to the request, accessing at least the portion of the job data in accordance with a second distributed object access protocol, and presenting at least the portion of the job data to the virtual machine cluster in accordance with the first distributed object access protocol.

Abstract: Systems, methods, and software described herein enhance the generation of large-scale processing framework clusters and corresponding edge services. In one implementation, a method includes identifying a request for a large-scale processing cluster, and identifying one or more edge services based on the type and version of the large-scale processing cluster. The method further provides generating a user interface that indicates the one or more edge services available, receiving a selection of at least one edge service in the one or more edge services, and initiating execution of the large-scale processing framework cluster and the at least one edge service, wherein the at least one edge service is provided with configuration information for the large-scale processing framework cluster.

Abstract: Systems, methods, and software described herein enhance addressing of services in a large-scale processing environment. In one implementation, a method of operating a control node of a large-scale processing environment includes receiving a request to configure a virtual cluster with data processing nodes on one or more hosts, and identifying services associated with the data processing nodes. The method further provides generating port addresses for each service in the data processing nodes, wherein services on a shared host of the one or more hosts are each provided a different port address. The method also includes allocating the port addresses to the services in the virtual cluster.

Abstract: Systems, methods, and software described herein facilitate the management of virtual clusters in a large-scale processing environment. In one implementation, a method of operating a control node includes receiving a request to configure a virtual cluster with one or more data processing nodes, and identifying a tenant associated with the request. The method further provides identifying a namespace for the tenant, and identifying internet protocol (IP) addresses for the one or more data processing nodes. The method also includes generating namespace to IP address pairs for the one or more data processing nodes based on the namespace and the IP addresses, and configuring a domain name system (DNS) for the virtual cluster with the namespace to IP address pairs.

Abstract: Systems, methods, and software described herein facilitate an enhanced service architecture for large-scale data processing. In one implementation, a method of providing data to a large-scale data processing architecture includes identifying a data request from a container in a plurality of containers executing on a host system, wherein the plurality of containers each run an instance of a large-scale processing framework. The method further provides identifying a storage repository for the data request, and accessing data associated with the data request from the storage repository. The method also includes caching the data in a portion of a cache memory on the host system allocated to the container, wherein the cache memory comprises a plurality of portions each allocated to one of the plurality of containers.

Abstract: Described herein are systems, methods, and software for translating data requests in a data processing cluster. In one example, a method of operating a cache service to interface between a virtual machine cluster and job data associated with a job executed by the virtual machine cluster includes identifying a request initiated by the virtual machine cluster to access at least a portion of the job data in accordance with a first distributed object access protocol. The method further includes in response to the request, accessing at least the portion of the job data in accordance with a second distributed object access protocol, and presenting at least the portion of the job data to the virtual machine cluster in accordance with the first distributed object access protocol.

Abstract: Described herein are systems, methods, and software for translating data requests in a data processing cluster. In one example, a method of operating a cache service to interface between a virtual machine cluster and job data associated with a job executed by the virtual machine cluster includes identifying a request initiated by the virtual machine cluster to access at least a portion of the job data in accordance with a first distributed object access protocol. The method further includes in response to the request, accessing at least the portion of the job data in accordance with a second distributed object access protocol, and presenting at least the portion of the job data to the virtual machine cluster in accordance with the first distributed object access protocol.

Abstract: Systems, methods, and software described herein provide for enhancements to large scale data processing architectures. In one implementation, a service architecture for large scale data processing includes a host computing system, and a virtual machine executing on the host computing system. The service architecture further includes a plurality of application containers executing on the virtual machine, wherein each of the application containers comprises a large scale processing node running one or more java virtual machines.

Abstract: Systems, methods, and software described herein facilitate the management of virtual clusters in a large-scale processing environment. In one implementation, a method of operating a control node to provide virtual local area networks (VLANs) to virtual data processing clusters includes receiving a request to configure a cluster with data processing nodes. The method further provides identifying a tenant associated with the request, and a VLAN tag for the tenant. The method further provides generating internet protocol (IP) address to VLAN tag pairs for the processing nodes, and communicating the IP address to VLAN tag pairs to the large-scale processing environment.

Abstract: Systems, methods, and software described herein provide for enhancements to large scale data processing architectures. In one implementation, a service architecture for large scale data processing includes a host computing system, and a virtual machine executing on the host computing system. The service architecture further includes a plurality of application containers executing on the virtual machine, wherein each of the application containers comprises a large scale processing node running one or more java virtual machines.