Abstract: A network-based data store may implement retention-based management techniques for data stored at the network-based data store. When data is received for storage at the network-based data store, a retention time for the data may be determined. Storage locations at persistent storage devices of the network-based data store may be selected according to the retention time. The data may then be placed at the storage locations. When a request to delete data is received, retention times of co-located data may be evaluated to determine whether the deletion may be delayed. Delayed deletions may allow the data to be subsequently deleted with at least some of the co-located data. Repair operations to maintain the data according to a durability policy may be modified according to retention time for a data suffering a loss of redundancy.

Abstract: A system includes a storage system configured to store data objects as a plurality of shards according to a redundancy encoding technique at a plurality of availability zones. The system further includes a redundancy reduction manager configured to perform a shard spreading process and a shard pruning process. The shard spreading process involves identifying an underutilized availability zone for a particular data object and moving at least one shard of the particular data object from another availability zone to the underutilized availability zone. The shard pruning process involves identifying a pruning candidate availability zone and deleting a shard of a particular data object at the pruning candidate availability zone in response to determining that deleting the shard would not violate a durability model for the particular data object.

Abstract: A method for performing auto-repair of a data volume may include determining a number of unavailable shards from a plurality of shards associated with a data volume. The plurality of shards may be stored in a plurality of data centers. The data volume may be erasure-coded with a total of N shards and can be reconstructed using at least K available shards out of the N shards. The unavailable shards may be filtered to exclude shards that have been unavailable for less than a threshold amount of time. For each of the plurality of data centers, a total number of unavailable shards may be determined if the data center is disabled. The total number of unavailable shards may be based at least in part on a number of shards associated with the data volume and stored at the data center, and a number of the filtered unavailable shards stored at the remaining plurality of data centers.

Abstract: A network-based data store may implement retention-based management techniques for data stored at the network-based data store. When data is received for storage at the network-based data store, a retention time for the data may be determined. Storage locations at persistent storage devices of the network-based data store may be selected according to the retention time. The data may then be placed at the storage locations. When a request to delete data is received, retention times of co-located data may be evaluated to determine whether the deletion may be delayed. Delayed deletions may allow the data to be subsequently deleted with at least some of the co-located data. Repair operations to maintain the data according to a durability policy may be modified according to retention time for a data suffering a loss of redundancy.

Abstract: A network-based data store may implement retention-based management techniques for data stored at the network-based data store. When data is received for storage at the network-based data store, a retention time for the data may be determined. Storage locations at persistent storage devices of the network-based data store may be selected according to the retention time. The data may then be placed at the storage locations. When a request to delete data is received, retention times of co-located data may be evaluated to determine whether the deletion may be delayed. Delayed deletions may allow the data to be subsequently deleted with at least some of the co-located data. Repair operations to maintain the data according to a durability policy may be modified according to retention time for a data suffering a loss of redundancy.

Abstract: Erasure encoded fragments are originally generated by applying an erasure encoding scheme to a data file. An erasure encoded fragment is subsequently generated directly from previously generated erasure encoded fragments or by reconstructing the original data file and then erasure encoding the reconstructed data file. The integrity or fidelity of such a subsequently generated erasure encoded fragment is verified by newly generating an error detection code, such as but not limited to a checksum, for the subsequently generated erasure encoded fragment, and comparing that subsequently error detection code against an error detection code previously generated for a previous or original version of the erasure encoded fragment. Each error detection code is preferably stored in association with its corresponding erasure encoded fragment and with one or more other erasure encoded fragments. Thus, each error detection code is saved in at least two locations.

Abstract: Erasure encoded fragments may be originally generated by applying an erasure encoding scheme to a data file. An erasure encoded fragment may be subsequently generated directly from previously generated erasure encoded fragments or by reconstructing the original data file and then erasure encoding the reconstructed data file. The integrity or fidelity of such a subsequently generated erasure encoded fragment may be verified by newly generating an error detection code, such as but not limited to a checksum, for the subsequently generated erasure encoded fragment, and comparing that subsequently generated error detection code against an error detection code previously generated for a previous or original version of the erasure encoded fragment. Each error detection code is preferably stored in association with its corresponding erasure encoded fragment and with one or more other erasure encoded fragments. Thus, each error detection code is saved in at least two locations.