Abstract: Systems and techniques are described for using virtual machines to write parallel and distributed applications. One of the techniques includes receiving a job request, wherein the job request specifies a first job to be performed by a plurality of a special purpose virtual machines, wherein the first job includes a plurality of tasks; selecting a parent special purpose virtual machine from a plurality of parent special purpose virtual machines to perform the first job; instantiating a plurality of child special purpose virtual machines from the selected parent special purpose virtual machine; partitioning the plurality of tasks among the plurality of child special purpose virtual machines by assigning one or more of the plurality of tasks to each of the child special purpose virtual machines; and performing the first job by causing each of the child special purpose virtual machines to execute the tasks assigned to the child special purpose virtual machine.

Abstract: A control system authorizes access to a networked resource. The control system includes a client agent associated with a client resource running at a user device, and a destination agent associated the networked resource. The client agent transparently injects one or more identity tokens associated with the client resource and one or more access tokens associated with the networked resource into a network request issued by the client resource and directed to the networked resource. The destination agent intercepts the network request and uses the access tokens to selectively route the network request in accordance with one or more security policies associated with the access tokens.

Abstract: Embodiments include an infrastructure shared among cloud services that supports fast provisioning of virtual machines (VMs). A set of powered-on parent VM templates and a set of powered-off child VMs are maintained by the infrastructure in a hierarchy. The child VMs are instantiated from the parent VM templates, and pre-registered to a cloud operating system in some embodiments. In response to requests from the cloud services for the child VMs, where the requests specify child VM configurations, child VMs from the set of powered-off child VMs are selected and customized based on the child VM configurations, and then deployed for use by the cloud services. In some embodiments, the fast provisioning of VMs is supported by forking operations in the infrastructure.

Abstract: A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Deployments of a distributed computing application, such as Hadoop, may be executed concurrently with a distributed database application, such as HBase, using a shared instance of a distributed filesystem, or in other cases, multiple instances of the distributed filesystem. Computing resources allocated to region server nodes executing as VMs may be isolated from compute VMs of the distributed computing application, as well as from data nodes executing as VMs of the distributed filesystem.

Abstract: A control system authorizes access to a networked resource. The control system includes a client agent associated with a client resource running at a user device, and a destination agent associated the networked resource. The client agent transparently injects one or more identity tokens associated with the client resource and one or more access tokens associated with the networked resource into a network request issued by the client resource and directed to the networked resource. The destination agent intercepts the network request and uses the access tokens to selectively route the network request in accordance with one or more security policies associated with the access tokens.

Abstract: A control system facilitates communication between a plurality of networked services. The control system includes a client agent associated with a first service of the networked services, and a destination agent associated with a second service of the networked services. The client agent includes an injection mechanism that intercepts a network request issued by the first service, transparently injects a token into the network request while the network request is in transit, and automatically transmits the network request to the second service in accordance with one or more security policies associated with the second service. The destination agent includes an interception mechanism that intercepts the network request, extracts the tokens from the network request, and determines whether to forward the network request to the second service.

Abstract: A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Production, test, and development deployments of a Hadoop application may be executed using multiple compute clusters and a shared instance of a distributed filesystem, or in other cases, multiple instances of the distributed filesystem. Data nodes executing as virtual machines (VMs) for test and development deployments can be linked clones of data nodes executing as VMs for a production deployment to reduce duplicated data and provide a shared storage space.

Abstract: A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Data and compute nodes are separated into different virtual machines (VM). Compute VMs are used to launch containers from different tenants. Compute VMs are organized in pools of hot spare VMs that are immediately available for launching a container and executing a task, and pools of cold spare VMs. Each compute VM may include a mounted network filesystem provided by a node manager to share intermediate outputs across VMs executing on the same host.

Abstract: Techniques for handling inheritance of disk state when forking virtual machines (VMs) are provided. In one embodiment, a computer system can receive a request to fork a child VM from a parent VM. In response, the computer system can take a disk snapshot of the parent VM, where the disk snapshot results in a child disk for the child VM, where the child disk is a delta disk that points to a parent disk of the parent VM, and where the parent disk serves as the parent VM's current running point. The computer system can then determine whether the parent disk is a delta disk. If so, the computer system can copy the content of the parent disk to the child disk, traverse a disk hierarchy associated with the parent disk to identify a base disk above the parent disk in the hierarchy, and cause the child disk to point directly to the base disk.

Abstract: A control system facilitates communication between a plurality of networked services. The control system includes a client agent associated with a first service of the networked services, and a destination agent associated with a second service of the networked services. The client agent includes an injection mechanism that intercepts a network request issued by the first service, transparently injects a token into the network request while the network request is in transit, and automatically transmits the network request to the second service in accordance with one or more security policies associated with the second service. The destination agent includes an interception mechanism that intercepts the network request, extracts the tokens from the network request, and determines whether to forward the network request to the second service.

Abstract: A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Multiple instances of a distributed computing framework, such as Hadoop, may be executed concurrently. A centralized manager detects when contention for computing resources, such as memory and CPU, causes tasks to run slower on VMs executing on a given host, and scales up or scales down a cluster based on the detected resource contention.

Abstract: Techniques for handling inheritance of disk state when forking virtual machines (VMs) are provided. In one embodiment, a computer system can receive a request to fork a child VM from a parent VM. In response, the computer system can take a disk snapshot of the parent VM, where the disk snapshot results in a child disk for the child VM, where the child disk is a delta disk that points to a parent disk of the parent VM, and where the parent disk serves as the parent VM's current running point. The computer system can then determine whether the parent disk is a delta disk. If so, the computer system can copy the content of the parent disk to the child disk, traverse a disk hierarchy associated with the parent disk to identify a base disk above the parent disk in the hierarchy, and cause the child disk to point directly to the base disk.

Abstract: Systems and techniques are described for using virtual machines to write parallel and distributed applications. One of the techniques includes receiving a job request, wherein the job request specifies a first job to be performed by a plurality of a special purpose virtual machines, wherein the first job includes a plurality of tasks; selecting a parent special purpose virtual machine from a plurality of parent special purpose virtual machines to perform the first job; instantiating a plurality of child special purpose virtual machines from the selected parent special purpose virtual machine; partitioning the plurality of tasks among the plurality of child special purpose virtual machines by assigning one or more of the plurality of tasks to each of the child special purpose virtual machines; and performing the first job by causing each of the child special purpose virtual machines to execute the tasks assigned to the child special purpose virtual machine.

Abstract: Techniques for handling inheritance of disk state when forking virtual machines (VMs) are provided. In one embodiment, a computer system can receive a request to fork a child VM from a parent VM. In response, the computer system can take a disk snapshot of the parent VM, where the disk snapshot results in a child disk for the child VM, where the child disk is a delta disk that points to a parent disk of the parent VM, and where the parent disk serves as the parent VM's current running point. The computer system can then determine whether the parent disk is a delta disk. If so, the computer system can copy the content of the parent disk to the child disk, traverse a disk hierarchy associated with the parent disk to identify a base disk above the parent disk in the hierarchy, and cause the child disk to point directly to the base disk.

Abstract: Embodiments support instant forking of virtual machines (VMs) and state customization. A computing device initiates execution of a first group of services (e.g., identity-independent) in a first VM. A second VM is instantiated from the first VM. The second VM shares memory and storage with the first VM. The computing device customizes the second VM based on configuration data associated with the second VM. A second group of services (e.g., identity-dependent) starts executing on the second VM after configuring the identity of the second VM. Customizing the second VM includes configuring one or more identities of the second VM. In some embodiments, a domain identity is selected from a pool of previously-created identities and applied to the second VM, before bootup completes on the second VM.

Abstract: Techniques for handling inheritance of disk state when forking virtual machines (VMs) are provided. In one embodiment, a computer system can receive a request to fork a child VM from a parent VM. In response, the computer system can take a disk snapshot of the parent VM, where the disk snapshot results in a child disk for the child VM, where the child disk is a delta disk that points to a parent disk of the parent VM, and where the parent disk serves as the parent VM's current running point. The computer system can then determine whether the parent disk is a delta disk. If so, the computer system can copy the content of the parent disk to the child disk, traverse a disk hierarchy associated with the parent disk to identify a base disk above the parent disk in the hierarchy, and cause the child disk to point directly to the base disk.

Abstract: A method for executing a plurality of data-parallel threads of a processing logic on a processor core includes grouping the plurality of data-parallel threads into one or more workgroups, associating a first workgroup from the one or more workgroups with an operating system thread on the processor core, and configuring threads from the first workgroup as user-level threads within the operating system thread. In an example, a method enables the execution of GPU-kernels that has been previously configured for a GPU, to execute on a CPU such as a multi-core CPU. The mapping of the numerous data-parallel threads to the CPU is done in such a manner as to reduce the number of costly operating system threads instantiated on the CPU, and to enable efficient debugging.

Abstract: Application resource scheduler module is provided to achieve cooperative application workload scheduling for a consolidated virtual environment. The application resource scheduler aids an application workload scheduler that is part of a distributed computing application, such as Hadoop, to achieve a specified relative priority of the application workload virtual machines to other virtual machines in the virtual environment. The application resource scheduler assists in achieving cooperative workload scheduling by revising the amount of resources that the application workload scheduler sees as available and by setting resource controls for the virtual machines of the distributed computing application to influence the resources the virtual machines receive from the underlying consolidated virtual environment.

Abstract: Embodiments include an infrastructure shared among cloud services that supports fast provisioning of virtual machines (VMs). A set of powered-on parent VM templates and a set of powered-off child VMs are maintained by the infrastructure in a hierarchy. The child VMs are instantiated from the parent VM templates, and pre-registered to a cloud operating system in some embodiments. In response to requests from the cloud services for the child VMs, where the requests specify child VM configurations, child VMs from the set of powered-off child VMs are selected and customized based on the child VM configurations, and then deployed for use by the cloud services. In some embodiments, the fast provisioning of VMs is supported by forking operations in the infrastructure.

Abstract: Embodiments support instant forking of virtual machines (VMs) and state customization. Virtual device state and persistent storage of a child VM are defined based on virtual device state and persistent storage of parent VMs. After forking, a state of the child VM is customized based on configuration data. Customizing the state includes configuring one or more identities of the child VM, before bootup completes on the child VM.