Abstract: A method for load balancing includes determining a reference key within a partition key range of a partition of scalable storage, the partition key range being divided into buckets that have boundaries defining sub ranges of the partition key range. The reference key is determined based on traffic values that correspond to tracked traffic within the buckets. The traffic values are updated based on additional traffic within the buckets and the boundaries are adjusted based on the updated traffic values. A reference key speed is determined that corresponds to a rate of change of a distribution of the tracked traffic with respect to the reference key. Reference key drop-off time may be determined for reference keys. Reference keys can be utilized to determine where to split the partition and reference key speed and reference key drop-off time can be utilized to determine whether or not to split the partition.

Abstract: Presented herein are methods of replicating versioned and hierarchical data structures, as well as data structures representing complex transactions. Due to interdependencies between data entities and a lack of guaranteed message ordering, simple replication methods employed for simple data types cannot be used. Operations on data structures exhibit dependencies between the messages making up the operations. This strategy can be extended to various types of complex transactions by considering certain messages to depend on other messages or on the existence of other entries at the data store. Regardless of origin, these dependencies can be enforced by suspending the processing of messages with unsatisfied dependencies until all of its dependencies have been met. Alternately, transactions can be committed immediately, creating entities that include versioned identifiers for each of their dependencies. These entities can then be garbage collected of the parent objects are not subsequently created.

Abstract: Embodiments of the present invention relate to invoking and managing a failover of a storage account between partitions within a distributed computing environment, where each partition represents a key range of data for the storage account. The partitions affected by the failover include source partitions hosted on a primary storage stamp and destination partitions hosted on a secondary storage stamp, where the storage account's data is being actively replicated from the primary to the secondary storage stamp. Upon receiving a manual or automatic indication to perform the failover, configuring the source partitions to independently perform flush-send operations (e.g., distributing pending messages as a group) and then configuring the destination partitions to independently perform flush-replay operations (e.g., aggressively replaying currently pending transactions).

Abstract: Embodiments of the present invention relate to asynchronously replicating data in a distributed computing environment. To achieve asynchronous replication, data received at a primary data store may be annotated with information, such as an identifier of the data. The annotated data may then be communicated to a secondary data store, which may then write the data and annotated information to one or more logs for eventual replay and committal at the secondary data store. The primary data store may communicate an acknowledgment of success in committing the data at the primary data store as well as of success in writing the data to the secondary data store. Additional embodiments may include committing the data at the secondary data store in response to receiving an instruction that authorizes committal of data through a identifier.

Abstract: In various embodiments, methods and systems for integrated resource allocation and loading balancing are provided. A global resource allocator receives usage information of resources in a cloud computing system. The usage information is associated with a plurality of accounts and consumer operations pairs on servers of the cloud computing system. For selected account and consumer operation pairs associated with a particular resource, allocation targets are determined and communicated to the corresponding server of the selected account and consumer operation pairs. The servers use the resource based on the allocation targets. A load balancer receives the usage information the resource and the allocation targets. The allocation targets indicate a load by the selected account and consumer operation pairs on their corresponding servers. The load balancer performs a load balancing operation to locate a server with a capacity to process the allocated target of the selected account and consumer operation pairs.

Abstract: A method for load balancing includes determining a reference key within a partition key range of a partition of scalable storage, the partition key range being divided into buckets that have boundaries defining sub ranges of the partition key range. The reference key is determined based on traffic values that correspond to tracked traffic within the buckets. The traffic values are updated based on additional traffic within the buckets and the boundaries are adjusted based on the updated traffic values. A reference key speed is determined that corresponds to a rate of change of a distribution of the tracked traffic with respect to the reference key. Reference key drop-off time may be determined for reference keys. Reference keys can be utilized to determine where to split the partition and reference key speed and reference key drop-off time can be utilized to determine whether or not to split the partition.

Abstract: In various embodiments, methods and systems for providing access to file systems are provided. An SFA-based operation having an operation modifier for a file in a file system is received. The SFA-based operation is defined based on a file-representational state transfer (REST) interface. The SFA-REST interface comprises integrated functionality of an SFA protocol and REST protocol. The SFA-based operation is executed using the operation modifier. A REST-based operation for the file in the distributed file storage is received. The REST-based operation is defined based on the SFA-REST interface. The operation modifier is referenced in order to execute the REST-based operation. The REST-based operation based on the SFA-REST interface. A schema can be implemented for accessing the file system. The schema includes tables to store files, the table comprises fields corresponding to elements of a SFA-REST interface. A snapshot component can be implemented to backup file shares in the file system.

Abstract: One method includes receiving partitions, generating assignment plans for assignment of the partitions to servers based on dimensional values of dimensions as determined by different assignment heuristics, selecting one of the assignment plans for execution based on analyzing the dimensional values in accordance with the assignment plans, and executing the selected assignment plan on scalable storage. Another method includes determining, for a triggered optimization module that a server is over utilized on a dimension, selecting candidate operations for partitions assigned to the server, for a higher priority optimization module than the triggered optimization module, removing a candidate operation from the candidate operations that would diminish a modeled state of scalable storage, determining an operation of the candidate operations that would improve the modeled state of the scalable storage with respect to a metric of the dimension on the server, and executing the operation on the scalable storage.

Abstract: A method includes determining that a trigger condition of a triggered optimization module of a plurality of optimization modules is met and optimizing scalable storage based on an optimization routine. The optimization routine includes providing a plurality of candidate operations and for a selected optimization module of the plurality of optimization modules that has a higher priority than the triggered optimization module, removing a candidate operation from the plurality of candidate operations that would diminish a modeled state of the scalable storage for the selected optimization module. The optimization routine also includes determining at least one operation of the plurality of candidate operations that would improve the modeled state of the scalable storage for the triggered optimization module and updating the modeled state of the scalable storage to model executing the at least one operation. The method further includes executing the at least one operation.

Abstract: Systems and methods are presented for providing resources by way of a platform as a service in a distributed computing environment to perform a job. Resources of the system, job performing on the system, and schedulers of the jobs performing on the system are decoupled in a manner that allows a job to easily migrate among resources. It is contemplated that the migration of jobs from a first pool of resource to a second pool of resource is performed by the system without human intervention. The migration of a job may utilize different schedulers for the different resources. Further, it is contemplated that a pool of resources may automatically allocate additional or fewer resources in response to a migration of a job.

Abstract: Presented herein are methods of replicating versioned and hierarchical data structures, as well as data structures representing complex transactions. Due to interdependencies between data entities and a lack of guaranteed message ordering, simple replication methods employed for simple data types cannot be used. Operations on data structures exhibit dependencies between the messages making up the operations. This strategy can be extended to various types of complex transactions by considering certain messages to depend on other messages or on the existence of other entries at the data store. Regardless of origin, these dependencies can be enforced by suspending the processing of messages with unsatisfied dependencies until all of its dependencies have been met. Alternately, transactions can be committed immediately, creating entities that include versioned identifiers for each of their dependencies. These entities can then be garbage collected of the parent objects are not subsequently created.

Abstract: Systems and methods are provided for assigning and associating resources in a cloud computing environment. Virtual machines in the cloud computing environment can be assigned or associated with pools corresponding to users as dedicated, standby, or preemptible machines. The various states provide users with the ability to reserve a desired level of resources while also allowing the operator of the cloud computing environment to increase resource utilization.

Abstract: Embodiments of the present invention relate to systems, methods, and computer storage media for replicating data in a distributed computing environment utilizing a combination of replication methodologies. A full-object replication may be utilized to replicate a full state of an object from a primary data store to a secondary data store. A checkpoint created after initiating the full-object replication may be parsed to identify changes to the object that have been entered since initiating the full-object replication. This replication process is referred to as a delta-checkpoint replication methodology. Additionally, in an embodiment, a log-based replication methodology may be utilized. The log-based replication may communicate data from a log of the primary data store to the secondary data store. It is also contemplated in an exemplary embodiment that when the log-based replication fails to maintain a throughput threshold, one of the other replication methodologies may be initiated, at least temporarily.

Abstract: Systems and methods are provided for assigning and associating resources in a cloud computing environment. Virtual machines in the cloud computing environment can be assigned or associated with pools corresponding to users as dedicated, standby, or preemptible machines. The various states provide users with the ability to reserve a desired level of resources while also allowing the operator of the cloud computing environment to increase resource utilization.

Abstract: Embodiments of the present invention relate to invoking and managing load-balancing operation(s) applied to partitions within a distributed computing environment, where each partition represents a key range of data for a storage account. The partitions affected by the load-balancing operation(s) are source partitions hosted on a primary storage stamp and/or destination partitions hosted on a secondary storage stamp, where the primary and secondary storage stamps are located in geographically distinct areas and are equipped to replicate the storage account's data therebetween. The load-balancing operation(s) include splitting partitions into child partitions upon detecting an increased workload as a result of active replication, merging partitions to form parent partitions upon detecting a reduction in workload as a result of decreased processing-related resource consumption, or offloading partitions based on resource consumption.

Abstract: Systems and methods are provided that enable a general framework for partitioning application-defined jobs (e.g., computation) in a scalable environment. The general framework decouples partitioning of a job from the other aspects of the job. As a result, the effort required to define the application-defined computation in a scalable environment is reduced or minimized, as the user is not required to provide a partitioning algorithm. The general framework further allows a user to provide load balancing conditions to allow for modification of how partitions are assigned.

Abstract: Embodiments of the present invention relate to systems, methods and computer storage media for facilitating the structured storage of binary large objects (Blobs) to be accessed by an application program being executed by a computing device. Generally, the manipulation of Blobs in a structured storage system includes receiving a request for a Blob, which may be located by way of a Blob pointer. The Blob pointer allows for the data, such as properties, of the Blob to be identified and located. Expired properties are garbage collected as a manipulation of the Blob data within a structured storage system. In an embodiment, the Blob is identified by a key that is utilized within a primary structured index to located the requested Blob. In another embodiment, the requested Blob is located utilizing a secondary hash index. In an additional embodiment, the Blob is locate utilizing a file table.

Abstract: Embodiments of the present invention relate to invoking and managing a failover of a storage account between partitions within a distributed computing environment, where each partition represents a key range of data for the storage account. The partitions affected by the failover include source partitions hosted on a primary storage stamp and destination partitions hosted on a secondary storage stamp, where the storage account's data is being actively replicated from the primary to the secondary storage stamp. Upon receiving a manual or automatic indication to perform the failover, configuring the source partitions to independently perform flush-send operations (e.g., distributing pending messages as a group) and then configuring the destination partitions to independently perform flush-replay operations (e.g., aggressively replaying currently pending transactions).

Abstract: Embodiments of the present invention relate to invoking and managing load-balancing operation(s) applied to partitions within a distributed computing environment, where each partition represents a key range of data for a storage account. The partitions affected by the load-balancing operation(s) are source partitions hosted on a primary storage stamp and/or destination partitions hosted on a secondary storage stamp, where the primary and secondary storage stamps are located in geographically distinct areas and are equipped to replicate the storage account's data therebetween. The load-balancing operation(s) include splitting partitions into child partitions upon detecting an increased workload as a result of active replication, merging partitions to form parent partitions upon detecting a reduction in workload as a result of decreased processing-related resource consumption, or offloading partitions based on resource consumption.

Abstract: A cloud computing platform contains a structured storage subsystem the provides scalable queues. The cloud computing platform monitors message throughput for the scalable queues and automatically increases or decreases subqueues that provide the operational functionality for each scalable queue. A visibility start time and cloud computing platform time are maintained for each message to provide an approximate first-in-first-out order for messages within each subqueue. A message in a subqueue may be available for processing when the current cloud computing time is greater than the visibility start of the message.