Abstract: Conditions are enforced to prevent unintended deletion of data stored by a data storage system. For example, to delete a collection of data, a condition on the collection of data's size may be enforced. The collection may be required to be empty, for example. In addition, a condition that there not exist a pending data processing operation that can affect fulfillment of the condition on the collection of data's size is also enforced.

Abstract: Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored. The volumes may be apportioned into failure-decorrelated subsets, and archives stored thereto may be apportioned to such subsets.

Abstract: Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored.

Abstract: A system that implements a distributed lock service may include a failure detector for servers and sessions, and may track the state of sessions on a per-client-connection basis. It may include an external lock information database that stores lock state information and that supports a higher write throughput rate than a distributed state manager. Each database record may store an identifier of a session during which a lock on a respective item was obtained (if any) and a staleness indicator. A distributed state manager may maintain a session identifier and a respective staleness indicator for each established session, and may push updates to this session information to interested client processes, which may cache the information. A client process wishing to lock an item may determine whether it can do so dependent on the information in a corresponding database record and on its own cached session information.

Abstract: Reliable and efficient storage and reconstruction of secure data files is provided. Encrypted fragments are generated by exclusive-OR (XOR) based erasure-encoding and XOR encryption of data files. At least some of the encrypted fragments, and preferably at least two copies of such encrypted fragments, are stored at two or more locations, such as but not limited to two or more servers in two or more regional storage systems. Fragments are retrieved from one or more of the multiple locations and the original data file is reconstructed, even if different encryption techniques have been used. If not enough valid fragments from that original data file can be identified then hash values, checksums, seeds, and other techniques may be used to distinguish files and to identify related or identical files which may be used to reconstruct the data file.

Abstract: If none of the fragments of an erasure-coded data file have become corrupted then the original data file can be readily reconstructed. If one or more fragments of an erasure-coded data file have become corrupted it may still be possible to find a combination of fragments that reconstruct the original data file, but the number of possible combinations may be impracticably large. If an attempt with a first set of fragments fails, an efficient approach is to use an independent set of fragments for the second attempt. Then, for further attempts, the results of a current attempt are compared with previous results. If a match is found then the original data file has been reconstructed. An original data file may also be reconstructed by separately recovering each data block of the data file from corresponding fragment blocks and assembling the data file from the recovered data blocks.

Abstract: If none of the fragments of an erasure-coded data file have become corrupted then the original data file can be readily reconstructed. If one or more fragments of an erasure-coded data file have become corrupted it may still be possible to find a combination of fragments that reconstruct the original data file, but the number of possible combinations may be impracticably large. If an attempt with a first set of fragments fails, an efficient approach is to use an independent set of fragments for the second attempt. Then, for further attempts, the results of a current attempt are compared with previous results. If a match is found then the original data file has been reconstructed. An original data file may also be reconstructed by separately recovering each data block of the data file from corresponding fragment blocks and assembling the data file from the recovered data blocks.

Abstract: Cohorts may be created on storage nodes in an object-redundant storage system that uses replication and/or a redundant encoding technique. In a cohort with N nodes, M data elements (replicas or shards) of an object are stored to M of the nodes that are selected from the N nodes. Metadata for locating other data elements for an object in the cohort may be stored with one or more of the data elements in the cohort. To reconcile the nodes, common object lists are generated on each node for at least one other node from the metadata, hashes of the lists may be exchanged among the nodes, and the hashes are compared. If the hashes for two nodes differ, specific differences are determined, and a reconciliation process performs reconciliation based on the determined differences.

Abstract: Cohorts may be created on storage nodes in an object-redundant storage system that uses replication and/or a redundant encoding technique. In a cohort with N nodes, M data elements (replicas or shards) of an object are stored to M of the nodes that are randomly (or otherwise) selected from the N nodes. Metadata for locating other data elements for an object in the cohort may be stored with one or more of the data elements in the cohort. To retrieve an object from a cohort, a subset of the nodes may be queried; the subset may be randomly or otherwise selected. If enough data elements are retrieved for the object from the queried nodes, the object is provided to the requester. Otherwise, additional data elements may be retrieved according to the metadata returned with a data element or by querying additional nodes.

Abstract: Conditions are enforced to prevent unintended deletion of data stored by a data storage system. For example, to delete a collection of data, a condition on the collection of data's size may be enforced. The collection may be required to be empty, for example. In addition, a condition that there not exist a pending data processing operation that can affect fulfillment of the condition on the collection of data's size is also enforced.