Abstract: Embodiments provide migration of services across different clusters to balance utilization and meet customer demands. Different service migration options may be performed with or without downtime. The artifacts of the service are moved to a new destination cluster. The service is created on the new destination cluster and staged so that the service is almost ready to start. In one embodiment, the service is stopped on the old cluster and started on the new cluster. After stopping the service, DNS is updated to point to the service on the new cluster. In another embodiment, the service is stopped on the old cluster and started on the new cluster with the same IP address to avoid DNS reprogramming and associated delays. In a further embodiment, the migration is performed without downtime by moving the service part by part from one cluster to another.

Abstract: Embodiments provide migration of services across different clusters to balance utilization and meet customer demands. Different service migration options may be performed with or without downtime. The artifacts of the service are moved to a new destination cluster. The service is created on the new destination cluster and staged so that the service is almost ready to start. In one embodiment, the service is stopped on the old cluster and started on the new cluster. After stopping the service, DNS is updated to point to the service on the new cluster. In another embodiment, the service is stopped on the old cluster and started on the new cluster with the same IP address to avoid DNS reprogramming and associated delays. In a further embodiment, the migration is performed without downtime by moving the service part by part from one cluster to another.

Abstract: Tenant and server platform coordination of the performance of atomic tasks that are to be performed by the platform, but which have at least a potential impact on the tenant. The server platform allows a particular tenant a timeframe within which the tenant itself may approve of initiation of each of one or more tasks. The server platform may protect itself from delayed action by imposing time constraints on the timeframe such that after the timeframe is elapsed, the tenant no longer has control over triggering the initiation of the particular task. Accordingly, the tenant is provided with flexibility on controlling initiation of a task performed by the server platform, while the server platform maintains a level of control to protect its resources and balances the needs of the various tenants.

Abstract: Embodiments are directed to implementing rate controls to limit faults detected by timeout and to learning and adjusting an optimal timeout value. In one scenario, a computer system identifies cloud components that have the potential to fail within a time frame that is specified by a timeout value. The computer system establishes a number of components that are allowed to fail during the time frame specified by the timeout value and further determines that the number of component failures within the time frame specified by the timeout value has exceeded the established number of components that are allowed to fail. In response, the computer system increases the timeout value by a specified amount of time to ensure that fewer than or equal to the established number of components fail within the time frame specified by the timeout value.

Abstract: In various embodiments, methods and systems for coordinating, between a host and a tenant, fault recovery of tenant infrastructure in a distributed system is provided. A fault occurrence is determined for a tenant infrastructure in the distributed system. The fault occurrence may be a software failure or hardware failure of the tenant infrastructure supporting a service application of the tenant. A fault recovery plan is communicated to the tenant to notify the tenant of the fault occurrence and actions taken to restore the tenant infrastructure. It is determined whether a fault recovery plan response is received from the tenant; the fault recovery plan response is an acknowledgement from the tenant of the fault recovery plan. Upon receiving the fault recovery plan response or at the expiration of a predefined time limit, the fault recovery plan is executed to restore the tenant infrastructure.

Abstract: Embodiments described herein are directed to migrating affected services away from a faulted cloud node and to handling faults during an upgrade. In one scenario, a computer system determines that virtual machines running on a first cloud node are in a faulted state. The computer system determines which cloud resources on the first cloud node were allocated to the faulted virtual machine, allocates the determined cloud resources of the first cloud node to a second, different cloud node and re-instantiates the faulted virtual machine on the second, different cloud node using the allocated cloud resources.

Abstract: Embodiments described herein are directed to migrating affected services away from a faulted cloud node and to handling faults during an upgrade. In one scenario, a computer system determines that virtual machines running on a first cloud node are in a faulted state. The computer system determines which cloud resources on the first cloud node were allocated to the faulted virtual machine, allocates the determined cloud resources of the first cloud node to a second, different cloud node and re-instantiates the faulted virtual machine on the second, different cloud node using the allocated cloud resources.

Abstract: Embodiments are directed to establishing a model for testing cloud components and to preventing cascading failures in cloud components. In one scenario, a computer system models identified cloud components (including cloud hardware components and/or cloud software components) as health entities. Each health entity is configured to provide state information about the cloud component. The computer system establishes declarative safety conditions which declaratively describe cloud computing conditions that are to be maintained at the identified cloud components. The computer system then tests against the declarative safety conditions to determine which cloud components are or are becoming problematic. Upon determining that an error has occurred, the computer system notifies users of the error and the component at which the error occurred. Guarded interfaces are established to ensure that actions taken to fix the error do not cause further failures.

Abstract: In various embodiments, methods and systems for coordinating, between a host and a tenant, fault recovery of tenant infrastructure in a distributed system is provided. A fault occurrence is determined for a tenant infrastructure in the distributed system. The fault occurrence may be a software failure or hardware failure of the tenant infrastructure supporting a service application of the tenant. A fault recovery plan is communicated to the tenant to notify the tenant of the fault occurrence and actions taken to restore the tenant infrastructure. It is determined whether a fault recovery plan response is received from the tenant; the fault recovery plan response is an acknowledgement from the tenant of the fault recovery plan. Upon receiving the fault recovery plan response or at the expiration of a predefined time limit, the fault recovery plan is executed to restore the tenant infrastructure.

Abstract: Embodiments described herein are directed to migrating affected services away from a faulted cloud node and to handling faults during an upgrade. In one scenario, a computer system determines that virtual machines running on a first cloud node are in a faulted state. The computer system determines which cloud resources on the first cloud node were allocated to the faulted virtual machine, allocates the determined cloud resources of the first cloud node to a second, different cloud node and re-instantiates the faulted virtual machine on the second, different cloud node using the allocated cloud resources.

Abstract: Embodiments are directed to isolating a cloud computing node using network- or some other type of isolation. In one scenario, a computer system determines that a cloud computing node is no longer responding to monitoring requests. The computer system isolates the determined cloud computing node to ensure that software programs running on the determined cloud computing node are no longer effectual (either the programs no longer produce outputs, or those outputs are not allowed to be transmitted). The computer system also notifies various entities that the determined cloud computing node has been isolated. The node may be isolated by powering the node down, by preventing the node from transmitting and/or receiving data, and by manually isolating the node. In some cases, isolating the node by preventing the node from transmitting and/or receiving data includes deactivating network switch ports used by the determined cloud computing node for data communication.

Abstract: Embodiments are directed to implementing rate controls to limit faults detected by timeout and to learning and adjusting an optimal timeout value. In one scenario, a computer system identifies cloud components that have the potential to fail within a time frame that is specified by a timeout value. The computer system establishes a number of components that are allowed to fail during the time frame specified by the timeout value and further determines that the number of component failures within the time frame specified by the timeout value has exceeded the established number of components that are allowed to fail. In response, the computer system increases the timeout value by a specified amount of time to ensure that fewer than or equal to the established number of components fail within the time frame specified by the timeout value.

Abstract: Embodiments are directed to establishing a model for testing cloud components and to preventing cascading failures in cloud components. In one scenario, a computer system models identified cloud components (including cloud hardware components and/or cloud software components) as health entities. Each health entity is configured to provide state information about the cloud component. The computer system establishes declarative safety conditions which declaratively describe cloud computing conditions that are to be maintained at the identified cloud components. The computer system then tests against the declarative safety conditions to determine which cloud components are or are becoming problematic. Upon determining that an error has occurred, the computer system notifies users of the error and the component at which the error occurred. Guarded interfaces are established to ensure that actions taken to fix the error do not cause further failures.

Abstract: Embodiments provide migration of services across different clusters to balance utilization and meet customer demands. Different service migration options may be performed with or without downtime. The artifacts of the service are moved to a new destination cluster. The service is created on the new destination cluster and staged so that the service is almost ready to start. In one embodiment, the service is stopped on the old cluster and started on the new cluster. After stopping the service, DNS is updated to point to the service on the new cluster. In another embodiment, the service is stopped on the old cluster and started on the new cluster with the same IP address to avoid DNS reprogramming and associated delays. In a further embodiment, the migration is performed without downtime by moving the service part by part from one cluster to another.

Abstract: A distributed application may be updated by using information provided by the application to determine grouping, sequencing, and whether or not to advance an update sequence. The application may provide input to an update sequence, and may cause the update sequence to repair one or more replicas prior to updating. The update mechanism may be used to change the distributed application topology as well as make changes to the application executable code, application configuration, quest operating system, virtual machine, and host systems, among others. In some cases, the distributed application may use a quorum to determine which version is current, and the quorum may change during the update process.