Abstract: Presented herein are methods of continuously verifying data and repairing errors introduced during replication. In a particular embodiment, a primary data store sends out information sufficient to create a checkpoint together with a checksum for the data being verified at that checkpoint. At the secondary data store, a checkpoint is created in accordance with the checkpointing information, and a checksum is calculated over the indicated data at the created checkpoint. If the calculated checksum disagrees with the received checksum, additional checksums are calculated over subranges of the indicated data and compared with corresponding checksums over the data at the primary data store. The checksums at the primary data store may be requested from the primary data store or calculated locally based on the received overall checksum. Once an erroneous entry is identified, it can then be re-replicated from the primary data store to restore data consistency.

Abstract: Embodiments of the present invention relate to invoking and managing a failover of a storage account between partitions within a distributed computing environment, where each partition represents a key range of data for the storage account. The partitions affected by the failover include source partitions hosted on a primary storage stamp and destination partitions hosted on a secondary storage stamp, where the storage account's data is being actively replicated from the primary to the secondary storage stamp. Upon receiving a manual or automatic indication to perform the failover, configuring the source partitions to independently perform flush-send operations (e.g., distributing pending messages as a group) and then configuring the destination partitions to independently perform flush-replay operations (e.g., aggressively replaying currently pending transactions).

Abstract: Systems, methods, and computer storage media for upgrading a domain in a distributed computing environment are provided. Upgrading of the domain includes preparing for the upgrade, upgrading, and finalizing the upgrade. The preparation of the domain includes ensuring predefined quantities of role instances are available in domains other than the upgrade domain. The preparation also includes ensuring that a predefined number of extent replicas are available in domains other than the upgrade domain. The preparation may also include checkpointing partitions within the upgrade domain to facilitate faster loading once transferred to a domain other than the upgrade domain. The finalization may include allowing nodes within the upgrade domain to resume functionality that was suspended during the upgrade.

Abstract: Embodiments of the present invention relate to systems, methods and computer storage media for facilitating the structured storage of binary large objects (Blobs) to be accessed by an application program being executed by a computing device. Generally, the structured storage of Blobs includes a primary structured storage index for indexing Blobs, a secondary hash index that is integrated into the structured storage system, a Blob log stream, and a Blob data stream for storing blocks that include the Blob data. In an embodiment, a block is created and written to a Blob store along with a block list. The block list facilitates the locating of one or more blocks that store the Blob data. In this embodiment, a primary structured storage index and a secondary hash index are updated to facilitate efficient access of the Blob in a structured storage system.

Abstract: Embodiments of the present invention relate to systems, methods and computer storage media for facilitating the structured storage of binary large objects (Blobs) to be accessed by an application program being executed by a computing device. Generally, the manipulation of Blobs in a structured storage system includes receiving a request for a Blob, which may be located by way of a Blob pointer. The Blob pointer allows for the data, such as properties, of the Blob to be identified and located. Expired properties are garbage collected as a manipulation of the Blob data within a structured storage system. In an embodiment, the Blob is identified by a key that is utilized within a primary structured index to located the requested Blob. In another embodiment, the requested Blob is located utilizing a secondary hash index. In an additional embodiment, the Blob is locate utilizing a file table.

Abstract: Systems, methods, and computer storage media for upgrading a domain in a distributed computing environment are provided. Upgrading of the domain includes preparing for the upgrade, upgrading, and finalizing the upgrade. The preparation of the domain includes ensuring predefined quantities of role instances are available in domains other than the upgrade domain. The preparation also includes ensuring that a predefined number of extent replicas are available in domains other than the upgrade domain. The preparation may also include checkpointing partitions within the upgrade domain to facilitate faster loading once transferred to a domain other than the upgrade domain. The finalization may include allowing nodes within the upgrade domain to resume functionality that was suspended during the upgrade.

Abstract: Cloud computing platforms having computer-readable media that perform methods for facilitating communications with storage. A request having a first-interface format to access storage is intercepted. The first interface format of the request supports access to a virtual hard drive (VHD). The request is translated to a blob request having a blob interface format. The blob interface format of the blob request supports access to a plurality of blobs of data in a blob store. The blob request is communicated to a blob interface such that the blob request is executed in managing the plurality of blobs.

Abstract: Cloud computing platforms having computer-readable media that perform methods to manage virtual hard drives as blobs are provided. The cloud computing platform includes fabric computers and blob stores. The fabric computers execute virtual machines that implement one or more applications that access virtual hard drives. The data in the virtual hard drives is accessed, via a blob interface, from blobs in the blob stores. The blob stores interface with a driver that translates some application input/output (I/O) requests destined to the virtual hard drives to blob commands when accessing data in the virtual hard drives.

Abstract: Systems and methods are provided that enable a general framework for partitioning application-defined computations (e.g., jobs) in a scalable environment. The general framework decouples partitioning of a computation from the other aspects of the computation. As a result, the effort required to define an application-defined job is reduced or minimized, as the user is not required to provide a partitioning algorithm. A user can optionally take advantage of a partitioning framework by providing application-defined interfaces to perform the desired job. Optionally, a user can provide additional information to allow for modification of how partitions are assigned.

Abstract: Systems and methods are provided that enable a general framework for partitioning application-defined jobs (e.g., computation) in a scalable environment. The general framework decouples partitioning of a job from the other aspects of the job. As a result, the effort required to define the application-defined computation in a scalable environment is reduced or minimized, as the user is not required to provide a partitioning algorithm. The general framework further allows a user to provide load balancing conditions to allow for modification of how partitions are assigned.

Abstract: Systems and methods are provided that enable a general framework for partitioning application-defined jobs in a scalable environment. The general framework decouples partitioning of a job from the other aspects of the job. As a result, the effort required to define the application-defined job is reduced or minimized, as the user is not required to provide a partitioning algorithm. The general framework also facilitates management of masters and servers performing computations within the distributed environment.

Abstract: Embodiments of the present invention relate to systems, methods and computer storage media for facilitating the structured storage of binary large objects (Blobs) to be accessed by an application program being executed by a computing device. Generally, the manipulation of Blobs in a structured storage system includes receiving a request for a Blob, which may be located by way of a Blob pointer. The Blob pointer allows for the data, such as properties, of the Blob to be identified and located. Expired properties are garbage collected as a manipulation of the Blob data within a structured storage system. In an embodiment, the Blob is identified by a key that is utilized within a primary structured index to located the requested Blob. In another embodiment, the requested Blob is located utilizing a secondary hash index. In an additional embodiment, the Blob is locate utilizing a file table.

Abstract: In various embodiments, systems and methods are presented for providing resources by way of a platform as a service in a distributed computing environment to perform a job. The system may be comprised of a number of components, such as a task machine, a task location service machine, and a high-level location service machines that in combination are useable to accomplish functions provided herein. It is contemplated that the system performs methods for providing resources by determining resources of the system, such as virtual machines, and applying auto-scaling rules to the system to scale those resources. Based on the determination of the auto-scaling rules, the resources may be allocated to achieve a desired result.

Abstract: Systems and methods are presented for providing resources by way of a platform as a service in a distributed computing environment to perform a job. A user may submit a work item to the system that results in a job being processed on a pool of virtual machines. The pool may be automatically established by the system in response to the work item and other information associated with the work item, the user, and/or the account. Further, it is contemplated that resources associated with the pool, such as virtual machines, may be automatically allocated based, at least in part, on information associated with the work item, the user, the account, the pool, and/or the system.

Abstract: Systems and methods are provided for assigning and associating resources in a cloud computing environment. Virtual machines in the cloud computing environment can be assigned or associated with pools corresponding to users as dedicated, standby, or preemptible machines. The various states provide users with the ability to reserve a desired level of resources while also allowing the operator of the cloud computing environment to increase resource utilization.

Abstract: Systems and methods are presented for providing resources by way of a platform as a service in a distributed computing environment to perform a job. Resources of the system, job performing on the system, and schedulers of the jobs performing on the system are decoupled in a manner that allows a job to easily migrate among resources. It is contemplated that the migration of jobs from a first pool of resource to a second pool of resource is performed by the system without human intervention. The migration of a job may utilize different schedulers for the different resources. Further, it is contemplated that a pool of resources may automatically allocate additional or fewer resources in response to a migration of a job.

Abstract: Systems and methods are provided for assigning resources in a cloud computing environment via a spot pricing process. The spot pricing process allows virtual machines to be assigned on a preemptible basis to pools based on bids associated with the pools. The bids can be used to determine a price for assignment of preemptible virtual machines. Preemptible virtual machines can then be assigned to pools based at least in part on the submitted bids in relation to the determined price.

Abstract: Embodiments of the present invention relate to systems, methods and computer storage media for erasure coding data in a distributed computing environment. A sealed extent is identified that is comprised of two or more data blocks and two or more index blocks. The sealed extent is optimized for erasure coding by grouping the two or more data blocks within the optimized sealed extent together and grouping the two or more index blocks within the optimized sealed extent together. The optimized extent may also be erasure coded, which includes creating data fragments and coding fragments. The data fragments and the coding fragments may also be stored in the distributed computing environment. Additional embodiments include monitoring statistical information to determine if replication, erasure coding or a hybrid storage plan should be utilized.

Abstract: Systems, methods, and computer storage media for storing and retrieving data from a data store in a distributed computing environment are provided. An embodiment includes receiving data at a data store comprising a sequential journal store, RAM, and a non-sequential target store. When RAM utilization is below a threshold, received data is stored to the RAM as a write cache for the target store and the journal store. But, when the utilization is above the threshold, the data is stored to the journal store without write-caching to the RAM for the target store. When the RAM utilization falls below a threshold, data committed to the journal store, but not write-cached to the RAM for the target store, is later read from the journal store and write-cached to the RAM for a target store.

Abstract: Embodiments of the present invention relate to invoking and managing migration operations applied to partitions within a distributed computing environment, where each partition represents a key range of data for a storage account. The partitions affected by the migration operations are source partitions hosted on a primary storage stamp and/or destination partitions hosted on a secondary storage stamp, where the primary and secondary storage stamps are equipped to replicate the storage account's data therebetween upon initiating a migration. Upon substantial completion of a bootstrapping phase of replication, one migration operation includes designating the secondary storage stamp as a new primary storage stamp such that the destination partitions commence processing client requests, sending resultant transactions to the source partitions, and providing read and write access thereto.