Abstract: One example provides a method of migrating a data partition from a first storage cluster to a second storage cluster, the method including determining that the data partition meets a migration criteria for migrating from the first storage cluster to the second storage cluster, on the first storage cluster, preparing partition metadata to be transferred, the partition metadata describing one or more streams within the data partition and one or more extents within each stream, transferring the partition metadata from the first storage cluster to the second storage cluster, directing new transactions associated with the data partition to the second storage cluster, including while the one or more extents reside at the first storage cluster, on the first storage cluster, changing an access attribute of the one or more extents within the data partition to read-only, and on the second storage cluster, performing new ingress for the data partition.

Abstract: One example provides a method of migrating a data partition from a first storage cluster to a second storage cluster, the method including determining that the data partition meets a migration criteria for migrating from the first storage cluster to the second storage cluster, on the first storage cluster, preparing partition metadata to be transferred, the partition metadata describing one or more streams within the data partition and one or more extents within each stream, transferring the partition metadata from the first storage cluster to the second storage cluster, directing new transactions associated with the data partition to the second storage cluster, including while the one or more extents reside at the first storage cluster, on the first storage cluster, changing an access attribute of the one or more extents within the data partition to read-only, and on the second storage cluster, performing new ingress for the data partition.

Abstract: One example provides a method of migrating a data partition from a first storage cluster to a second storage cluster, the method including determining that the data partition meets a migration criteria for migrating from the first storage cluster to the second storage cluster, on the first storage cluster, preparing partition metadata to be transferred, the partition metadata describing one or more streams within the data partition and one or more extents within each stream, transferring the partition metadata from the first storage cluster to the second storage cluster, directing new transactions associated with the data partition to the second storage cluster, including while the one or more extents reside at the first storage cluster, on the first storage cluster, changing an access attribute of the one or more extents within the data partition to read-only, and on the second storage cluster, performing new ingress for the data partition.

Abstract: Live migration of partitions can be achieved using server-to-server communications to coordinate and control partition migration and rebuild a persistent state of the partition on a destination server while the partition is served on the source server. The live migration approach breaks the heavy dependency on the partition master by eliminating most of the server-to-master communications, delegating coordination and control of the migration to the source and destination servers. After the migration is initiated by the partition master, the source server sends in-memory cache entries indicating the persistent state of the partition to the destination server. Once transferred, the two servers can cooperate to complete the handover, which loads the partition on the destination server using the cache entries received from the source server. Once the source server offloads the partition, it can redirect traffic for the partition to the destination server by triggering an update to the partition map.

Abstract: A traffic manager for a distributed data storage system includes an iterative spike identifier to identify N levels of traffic spikes in traffic data on an account basis in a distributed data storage system, where N is greater than zero. A traffic cycle identifier selectively identifies cyclic traffic spikes in at least one of the N levels of traffic spikes for each of a plurality of accounts using autocorrelation and peak detection. A partition manager communicates with the traffic cycle identifier and selectively partitions one of the plurality of accounts based on based on the autocorrelation and the peak detection.

Abstract: A traffic manager for a distributed data storage system includes an iterative spike identifier to identify N levels of traffic spikes in traffic data on an account basis in a distributed data storage system, where N is greater than zero. A traffic cycle identifier selectively identifies cyclic traffic spikes in at least one of the N levels of traffic spikes for each of a plurality of accounts using autocorrelation and peak detection. A partition manager communicates with the traffic cycle identifier and selectively partitions one of the plurality of accounts based on based on the autocorrelation and the peak detection.

Abstract: Live migration of partitions can be achieved using server-to-server communications to coordinate and control partition migration and rebuild a persistent state of the partition on a destination server while the partition is served on the source server. The live migration approach breaks the heavy dependency on the partition master by eliminating most of the server-to-master communications, delegating coordination and control of the migration to the source and destination servers. After the migration is initiated by the partition master, the source server sends in-memory cache entries indicating the persistent state of the partition to the destination server. Once transferred, the two servers can cooperate to complete the handover, which loads the partition on the destination server using the cache entries received from the source server. Once the source server offloads the partition, it can redirect traffic for the partition to the destination server by triggering an update to the partition map.

Abstract: Embodiments of the present invention relate to invoking and managing a failover of a storage account between partitions within a distributed computing environment, where each partition represents a key range of data for the storage account. The partitions affected by the failover include source partitions hosted on a primary storage stamp and destination partitions hosted on a secondary storage stamp, where the storage account's data is being actively replicated from the primary to the secondary storage stamp. Upon receiving a manual or automatic indication to perform the failover, configuring the source partitions to independently perform flush-send operations (e.g., distributing pending messages as a group) and then configuring the destination partitions to independently perform flush-replay operations (e.g., aggressively replaying currently pending transactions).

Abstract: Embodiments of the present invention relate to invoking and managing load-balancing operation(s) applied to partitions within a distributed computing environment, where each partition represents a key range of data for a storage account. The partitions affected by the load-balancing operation(s) are source partitions hosted on a primary storage stamp and/or destination partitions hosted on a secondary storage stamp, where the primary and secondary storage stamps are located in geographically distinct areas and are equipped to replicate the storage account's data therebetween. The load-balancing operation(s) include splitting partitions into child partitions upon detecting an increased workload as a result of active replication, merging partitions to form parent partitions upon detecting a reduction in workload as a result of decreased processing-related resource consumption, or offloading partitions based on resource consumption.

Abstract: Embodiments of the present invention relate to invoking and managing a failover of a storage account between partitions within a distributed computing environment, where each partition represents a key range of data for the storage account. The partitions affected by the failover include source partitions hosted on a primary storage stamp and destination partitions hosted on a secondary storage stamp, where the storage account's data is being actively replicated from the primary to the secondary storage stamp. Upon receiving a manual or automatic indication to perform the failover, configuring the source partitions to independently perform flush-send operations (e.g., distributing pending messages as a group) and then configuring the destination partitions to independently perform flush-replay operations (e.g., aggressively replaying currently pending transactions).

Abstract: Embodiments of the present invention relate to invoking and managing load-balancing operation(s) applied to partitions within a distributed computing environment, where each partition represents a key range of data for a storage account. The partitions affected by the load-balancing operation(s) are source partitions hosted on a primary storage stamp and/or destination partitions hosted on a secondary storage stamp, where the primary and secondary storage stamps are located in geographically distinct areas and are equipped to replicate the storage account's data therebetween. The load-balancing operation(s) include splitting partitions into child partitions upon detecting an increased workload as a result of active replication, merging partitions to form parent partitions upon detecting a reduction in workload as a result of decreased processing-related resource consumption, or offloading partitions based on resource consumption.

Abstract: Embodiments of the present invention relate to invoking and managing a failover of a storage account between partitions within a distributed computing environment, where each partition represents a key range of data for the storage account. The partitions affected by the failover include source partitions hosted on a primary storage stamp and destination partitions hosted on a secondary storage stamp, where the storage account's data is being actively replicated from the primary to the secondary storage stamp. Upon receiving a manual or automatic indication to perform the failover, configuring the source partitions to independently perform flush-send operations (e.g., distributing pending messages as a group) and then configuring the destination partitions to independently perform flush-replay operations (e.g., aggressively replaying currently pending transactions).

Abstract: Embodiments of the present invention relate to invoking and managing a failover of a storage account between partitions within a distributed computing environment, where each partition represents a key range of data for the storage account. The partitions affected by the failover include source partitions hosted on a primary storage stamp and destination partitions hosted on a secondary storage stamp, where the storage account's data is being actively replicated from the primary to the secondary storage stamp. Upon receiving a manual or automatic indication to perform the failover, configuring the source partitions to independently perform flush-send operations (e.g., distributing pending messages as a group) and then configuring the destination partitions to independently perform flush-replay operations (e.g., aggressively replaying currently pending transactions).

Abstract: Embodiments of the present invention relate to invoking and managing migration operations applied to partitions within a distributed computing environment, where each partition represents a key range of data for a storage account. The partitions affected by the migration operations are source partitions hosted on a primary storage stamp and/or destination partitions hosted on a secondary storage stamp, where the primary and secondary storage stamps are equipped to replicate the storage account's data therebetween upon initiating a migration. Upon substantial completion of a bootstrapping phase of replication, one migration operation includes designating the secondary storage stamp as a new primary storage stamp such that the destination partitions commence processing client requests, sending resultant transactions to the source partitions, and providing read and write access thereto.

Abstract: Embodiments of the present invention relate to invoking and managing load-balancing operation(s) applied to partitions within a distributed computing environment, where each partition represents a key range of data for a storage account. The partitions affected by the load-balancing operation(s) are source partitions hosted on a primary storage stamp and/or destination partitions hosted on a secondary storage stamp, where the primary and secondary storage stamps are located in geographically distinct areas and are equipped to replicate the storage account's data therebetween. The load-balancing operation(s) include splitting partitions into child partitions upon detecting an increased workload as a result of active replication, merging partitions to form parent partitions upon detecting a reduction in workload as a result of decreased processing-related resource consumption, or offloading partitions based on resource consumption.