Abstract: One example provides a method of migrating a data partition from a first storage cluster to a second storage cluster, the method including determining that the data partition meets a migration criteria for migrating from the first storage cluster to the second storage cluster, on the first storage cluster, preparing partition metadata to be transferred, the partition metadata describing one or more streams within the data partition and one or more extents within each stream, transferring the partition metadata from the first storage cluster to the second storage cluster, directing new transactions associated with the data partition to the second storage cluster, including while the one or more extents reside at the first storage cluster, on the first storage cluster, changing an access attribute of the one or more extents within the data partition to read-only, and on the second storage cluster, performing new ingress for the data partition.

Abstract: One example provides a method of migrating a data partition from a first storage cluster to a second storage cluster, the method including determining that the data partition meets a migration criteria for migrating from the first storage cluster to the second storage cluster, on the first storage cluster, preparing partition metadata to be transferred, the partition metadata describing one or more streams within the data partition and one or more extents within each stream, transferring the partition metadata from the first storage cluster to the second storage cluster, directing new transactions associated with the data partition to the second storage cluster, including while the one or more extents reside at the first storage cluster, on the first storage cluster, changing an access attribute of the one or more extents within the data partition to read-only, and on the second storage cluster, performing new ingress for the data partition.

Abstract: A system receives a request to store data at a first layer of servers in a cluster, configured to authenticate and authorize the request. The system compresses the data upon authenticating and authorizing the request and encrypts the compressed data at the first layer of servers when encryption is enabled. The system sends the request and the encrypted data to a second layer of servers in the cluster, configured to store data structures used to manage data storage in a third layer of servers in the cluster, and to distribute the request and the encrypted data to the third layer of servers using the data structures. The system stores the encrypted data in the third layer of servers. Encrypting the data at the first layer of servers reduces latency associated with transferring the data between the first, second, and third layers of servers.

Abstract: One example provides a method of migrating a data partition from a first storage cluster to a second storage cluster, the method including determining that the data partition meets a migration criteria for migrating from the first storage cluster to the second storage cluster, on the first storage cluster, preparing partition metadata to be transferred, the partition metadata describing one or more streams within the data partition and one or more extents within each stream, transferring the partition metadata from the first storage cluster to the second storage cluster, directing new transactions associated with the data partition to the second storage cluster, including while the one or more extents reside at the first storage cluster, on the first storage cluster, changing an access attribute of the one or more extents within the data partition to read-only, and on the second storage cluster, performing new ingress for the data partition.

Abstract: Techniques for implementing zone redundant computing services in distributed computing systems are described herein. In one embodiment, a computing device is configured to query a database in a first availability zone for a list of computing services corresponding to a user account and provided by servers in a second availability zone different than the first availability zone. The computing device can then selects one or more computing services provided by the servers in the second availability zone from the queried list of computing services and transmit an instruction to the servers in the second availability zone for performing the requested data operation.

Abstract: A system receives data of one or more types from one or more sources having distinct identities for storing the data in a distributed storage system. The system stores metadata associated with storing the data in data structures in the distributed storage system. The system selects a portion of the data and a portion of the metadata associated with storing the data in the distributed storage system. The system compresses the selected portions of the data and the metadata, encrypts the compressed data and metadata using a single key or using a separate key for each data type, and stores the encrypted compressed data and metadata in the distributed storage system. The system also encrypts unencrypted metadata and associated data during defragmentation procedure.

Abstract: In an embodiment, a partition cost of one or more of the plurality of partitions and a data block cost for one or more data blocks that may be subjected to a garbage collection operation are determined. The partition cost and the data block cost are combined into an overall reclaim cost by specifying both the partition cost and the data block cost in terms of a computing system latency. A byte constant multiplier that is configured to modify the overall reclaim cost to account for the amount of data objects that may be rewritten during the garbage collection operation may be applied. The one or more partitions and/or one or more data blocks that have the lowest overall reclaim cost while reclaiming an acceptable amount of data block space may be determined and be included in a garbage collection schedule.

Abstract: A system stores data in data units in a cluster in a cloud computing system, the data stored in the data units being encrypted or unencrypted depending on whether encryption is enabled or disabled when storing data in the data units. The system identifies one or more data units to defragment and defragments the identified data units by writing the data from the identified data units to one or more new data units and by releasing the identified data units for storing new data. The system encrypts unencrypted data from the identified data units when writing the data from the identified data units to the one or more new data units.

Abstract: Computer systems and methods for scheduling garbage collection in a distributed environment that includes multiple partitions that reference various data blocks that store data objects. A global occupancy aggregator may access occupancy information for each of the partitions from an occupancy index of each of the partitions. This occupancy information specifies a portion of storage resources occupied by those data blocks referenced by each of the partitions. The global occupancy aggregator may aggregate the accessed occupancy information to generate a global occupancy index that combines the occupancy information of the partitions. The global occupancy aggregator may generate a global schedule for garbage collection for the partitions based on the global occupancy index. The global schedule specifies which of the data blocks included in the global occupancy index are to be subjected to garbage collection.

Abstract: Provided is a system and method for converting active data identified by a garbage collection operation into erasure coded fragments. In one example, the method may include identifying data blocks in use and interspersed among garbage data blocks not in use in cloud storage based on a garbage collection operation, extracting object data from the identified data blocks in use into a data container while leaving object data of the garbage data blocks not in use, and fragmenting a predetermined amount of extracted object data stored within the data container, the fragmenting comprising converting the predetermined amount of object data into a plurality of fragments including data fragments storing portions of the data and parity fragments for reconstructing the data, and writing the plurality of fragments in a distributed manner among a plurality of storage nodes.

Abstract: Techniques for implementing zone redundant computing services in distributed computing systems are described herein. In one embodiment, a computing device is configured to query a database in a first availability zone for a list of computing services corresponding to a user account and provided by servers in a second availability zone different than the first availability zone. The computing device can then selects one or more computing services provided by the servers in the second availability zone from the queried list of computing services and transmit an instruction to the servers in the second availability zone for performing the requested data operation.

Abstract: Provided is a system and method for converting active data identified by a garbage collection operation into erasure coded fragments. In one example, the method may include identifying data blocks in use and interspersed among garbage data blocks not in use in cloud storage based on a garbage collection operation, extracting object data from the identified data blocks in use into a data container while leaving object data of the garbage data blocks not in use, and fragmenting a predetermined amount of extracted object data stored within the data container, the fragmenting comprising converting the predetermined amount of object data into a plurality of fragments including data fragments storing portions of the data and parity fragments for reconstructing the data, and writing the plurality of fragments in a distributed manner among a plurality of storage nodes.

Abstract: A system receives a request to store data at a first layer of servers in a cluster, configured to authenticate and authorize the request. The system compresses the data upon authenticating and authorizing the request and encrypts the compressed data at the first layer of servers when encryption is enabled. The system sends the request and the encrypted data to a second layer of servers in the cluster, configured to store data structures used to manage data storage in a third layer of servers in the cluster, and to distribute the request and the encrypted data to the third layer of servers using the data structures. The system stores the encrypted data in the third layer of servers. Encrypting the data at the first layer of servers reduces latency associated with transferring the data between the first, second, and third layers of servers.

Abstract: A system receives a request to store data at a first layer of servers in a cluster, configured to authenticate and authorize the request. The system compresses the data upon authenticating and authorizing the request and encrypts the compressed data at the first layer of servers when encryption is enabled. The system sends the request and the encrypted data to a second layer of servers in the cluster, configured to store data structures used to manage data storage in a third layer of servers in the cluster, and to distribute the request and the encrypted data to the third layer of servers using the data structures. The system stores the encrypted data in the third layer of servers. Encrypting the data at the first layer of servers reduces latency associated with transferring the data between the first, second, and third layers of servers.

Abstract: In an embodiment, a partition cost of one or more of the plurality of partitions and a data block cost for one or more data blocks that may be subjected to a garbage collection operation are determined. The partition cost and the data block cost are combined into an overall reclaim cost by specifying both the partition cost and the data block cost in terms of a computing system latency. A byte constant multiplier that is configured to modify the overall reclaim cost to account for the amount of data objects that may be rewritten during the garbage collection operation may be applied. The one or more partitions and/or one or more data blocks that have the lowest overall reclaim cost while reclaiming an acceptable amount of data block space may be determined and be included in a garbage collection schedule.

Abstract: Computer systems and methods for scheduling garbage collection in a distributed environment that includes multiple partitions that reference various data blocks that store data objects. A global occupancy aggregator may access occupancy information for each of the partitions from an occupancy index of each of the partitions. This occupancy information specifies a portion of storage resources occupied by those data blocks referenced by each of the partitions. The global occupancy aggregator may aggregate the accessed occupancy information to generate a global occupancy index that combines the occupancy information of the partitions. The global occupancy aggregator may generate a global schedule for garbage collection for the partitions based on the global occupancy index. The global schedule specifies which of the data blocks included in the global occupancy index are to be subjected to garbage collection.

Abstract: In an embodiment, a partition cost of one or more of the plurality of partitions and a data block cost for one or more data blocks that may be subjected to a garbage collection operation are determined. The partition cost and the data block cost are combined into an overall reclaim cost by specifying both the partition cost and the data block cost in terms of a computing system latency. A byte constant multiplier that is configured to modify the overall reclaim cost to account for the amount of data objects that may be rewritten during the garbage collection operation may be applied. The one or more partitions and/or one or more data blocks that have the lowest overall reclaim cost while reclaiming an acceptable amount of data block space may be determined and be included in a garbage collection schedule.

Abstract: Computer systems and methods for scheduling garbage collection in a distributed environment that includes multiple partitions that reference various data blocks that store data objects. A global occupancy aggregator may access occupancy information for each of the partitions from an occupancy index of each of the partitions. This occupancy information specifies a portion of storage resources occupied by those data blocks referenced by each of the partitions. The global occupancy aggregator may aggregate the accessed occupancy information to generate a global occupancy index that combines the occupancy information of the partitions. The global occupancy aggregator may generate a global schedule for garbage collection for the partitions based on the global occupancy index. The global schedule specifies which of the data blocks included in the global occupancy index are to be subjected to garbage collection.

Abstract: A system receives data of one or more types from one or more sources having distinct identities for storing the data in a distributed storage system. The system stores metadata associated with storing the data in data structures in the distributed storage system. The system selects a portion of the data and a portion of the metadata associated with storing the data in the distributed storage system. The system compresses the selected portions of the data and the metadata, encrypts the compressed data and metadata using a single key or using a separate key for each data type, and stores the encrypted compressed data and metadata in the distributed storage system. The system also encrypts unencrypted metadata and associated data during defragmentation procedure.

Abstract: A system receives a request to store data at a first layer of servers in a cluster, configured to authenticate and authorize the request. The system compresses the data upon authenticating and authorizing the request and encrypts the compressed data at the first layer of servers when encryption is enabled. The system sends the request and the encrypted data to a second layer of servers in the cluster, configured to store data structures used to manage data storage in a third layer of servers in the cluster, and to distribute the request and the encrypted data to the third layer of servers using the data structures. The system stores the encrypted data in the third layer of servers. Encrypting the data at the first layer of servers reduces latency associated with transferring the data between the first, second, and third layers of servers.