Abstract: Techniques for encoding data in grid encoded data storage systems are described herein. Data is stored in a grid of shards using grid encoding techniques that store the data in a combination of data shards and derived shards. Because the data is redundantly encoded in the grid of shards, a plurality of distinct copies of the data are reproducible from the grid of shards where each distinct copy is reproducible from a non-overlapping set of shards of the grid of shards.

Abstract: A data center may include a tape library rack module along with rack computer systems. The rack computer systems may be configured to provide computing capacity within a data center environment. In some embodiments, the tape library rack module may include an enclosure encompassing an interior of the tape library rack module, a rack within the interior, and a tape library unit mounted on the rack. The tape library rack unit may include tape cartridges configured to store data within a tape environment that is different than the data center environment. The tape library rack unit may be within a portion of the interior that is enclosed such that it is environmentally isolated from the data center environment. In some examples, the tape library rack module may include a cooling unit and/or a humidifier unit, which may provide the tape environment to the environmentally isolated portion of the interior of the tape library rack module.

Abstract: A logical storage layer for shared storage systems interposes between address ranges of the shards and the storage devices on which the shards are stored. The shards may be logically addressed using a plurality of addressable zones, to which the storage devices are independently mapped. Data requests related to a given piece of data associated with a shard may involve multiple storage devices, and vice versa.

Abstract: Techniques described and suggested herein include various methods and systems for verifying integrity of redundancy coded data, such as erasure coded data shards. In some embodiments, a quantity of redundancy coded data elements, hereafter referred to as data shards (e.g., erasure coded data shards), sufficient to reconstruct the original data element from which the redundancy coded data elements are derived, is used to generate reconstructed data shards to be used for checking the validity of analogous data shards stored for the original data element.

Abstract: A data center may include a tape library rack module along with rack computer systems. The rack computer systems may be configured to provide computing capacity within a data center environment. In some embodiments, the tape library rack module may include an enclosure encompassing an interior of the tape library rack module, a rack within the interior, and a tape library unit mounted on the rack. The tape library rack unit may include tape cartridges configured to store data within a tape environment that is different than the data center environment. The tape library rack unit may be within a portion of the interior that is enclosed such that it is environmentally isolated from the data center environment. In some examples, the tape library rack module may include a cooling unit and/or a humidifier unit, which may provide the tape environment to the environmentally isolated portion of the interior of the tape library rack module.

Abstract: One or more grids of redundancy coded shards, such as those stored or otherwise represented on grid encoded storage systems, are combinable or extensible. For example, a generator matrix of a redundancy code may be configured so as to have a sufficient number of fields to generate a grid. The generator matrix may initially be used to generate smaller grids, which can be combined into the target grid without re-encoding most or all of the data represented thereon. In some cases, vertically derived shards of the input grids may be combined using, e.g., matrix addition, which may then be directly allocated to the target grid, while data shards and horizontally derived shards may be allocated to the target grid with no further transformation.

Abstract: Techniques for validating grid encoded data storage systems are described herein. Data stored is stored in a grid of shards using grid encoding techniques that store the data in a combination of data shards and derived shards. Each of the shards has at least a first index corresponding to one dimension of the grid, a second index corresponding to a second dimension of the grid, and a set of error-detection code values. Updates that alter the grid of shards cause updates to the error-detection code values and the update can be validated based on the updated error-detection code values.

Abstract: A computer system, such as a data storage system, implements techniques for segregating and controlling access to data stored in multiple regions. In some embodiments, redundancy coded shards generated from the data and stored in durable storage of a data storage system is allocated across multiple regions, but in a fashion that prevents actors with access to regions outside that of a “home” region from recovering a sufficient number of unique shards to regenerate the data represented thereby. In some embodiments, encryption is used to segregate the data by encrypting the generated shards, then storing the cryptographic information on or otherwise controlling access on hosts or other devices of only the home region.

Abstract: Techniques described and suggested herein include various methods and systems for verifying integrity of redundancy coded data, such as erasure coded data shards. In some embodiments, a quantity of redundancy coded data elements, hereafter referred to as data shards (e.g., erasure coded data shards), sufficient to reconstruct the original data element from which the redundancy coded data elements are derived, is used to generate reconstructed data shards to be used for checking the validity of analogous data shards stored for the original data element.

Abstract: Techniques for encoding data storage systems using grid encoded data storage systems are described herein. Data to be stored in a data storage system is obtained and the data is stored in a grid of shards using grid encoding techniques that store the data in a combination of data shards and derived shards. Each of the shards has at least a first index corresponding to one dimension of the grid and a second index corresponding to a second dimension of the grid. Loss of a plurality of data shards can be repaired because each shard is reproducible from one or more shards with a first index that is associated with the first index of the shard and is also reproducible from one or more shards with a second index that is associated with the second index of the shard.

Abstract: Techniques for storing incremental backups in long-term storage are described herein. A backup data set is generated from a set of data to backup and a deletion time for that backup data set is determined. A scheduled time period to perform the backup is selected based at least in part on a requested time period to perform the backup. The requested time period is altered if the performing the backup at the requested time period would produce a period of high resource contention at the destination for the backup. The backup is then stored at the destination at the scheduled time.

Abstract: Techniques described and suggested herein include systems and methods for precomputing regeneration information for data archives (“archives”) that have been processed and stored using redundancy coding techniques. For example, regeneration information, such as redundancy code-related matrices (such as inverted matrices based on, e.g., a generator matrix for the selected redundancy code) corresponding to subsets of the shards, is computed for each subset and, in some embodiments, stored for use in the event that one or more shards becomes unavailable, e.g., so as to more efficiently and/or quickly regenerate a replacement shard.

Abstract: One or more grids of redundancy coded shards, such as those stored or otherwise represented on grid encoded storage systems, are configured to be extensible. For example, a grid of shards may include data shards, derived shards (derived from the data shards), and null shard, indexed by, e.g., row and column. A grid of shards so configured may include data shards and derived shards in one set of columns of the grid, and the null shards in another set of columns of the grid. As additional data is added to the grid, the grid may be extended by converting some of the null shards into data or derived shards, on a row-by-row basis, and regenerating or re-deriving additional shards as necessary.

Abstract: Techniques for extending a grid encoded data storage system to additional datacenters are described herein. A grid of shards with a first index and a second index is created and a set of null shards is added to the grid of shards. When a data object is received for storage in the grid of shards, a set of shards with the same first index is selected for the storage location with at least one null shard and one or more other shards. The null shard is enabled for data storage by allocating a storage device for the null shard. The grid is then updated by storing at least a portion of the data object in the set of shards, updating derived shards in the set of shards, and updating derived shards with the same second index as the updated shards.

Abstract: A switching device is implemented in a network-attachable data transfer device to provide data storage access to other such devices. In some embodiments, network-attachable data transfer devices are arranged in a clustered configuration to provide various computational and storage services. When one or more devices of the cluster fails, various implementations associated with the switching device, via an external data interface, provide operational mitigation, optimized data recovery, and efficient reinstatement of normal operation of the cluster.

Abstract: A switching device is implemented in a network-attachable data transfer device to provide data storage access to other such devices. In some embodiments, network-attachable data transfer devices are arranged in a clustered configuration to provide various computational and storage services. When one or more devices of the cluster fails, various implementations associated with the switching device, via an external data interface, provide operational mitigation, optimized data recovery, and efficient reinstatement of normal operation of the cluster.

Abstract: A switching device is implemented in a network-attachable data transfer device to provide data storage access to other such devices. In some embodiments, network-attachable data transfer devices are arranged in a clustered configuration to provide various computational and storage services. When one or more devices of the cluster fails, various implementations associated with the switching device, via an external data interface, provide operational mitigation, optimized data recovery, and efficient reinstatement of normal operation of the cluster.

Abstract: Techniques for incrementally updating grid encoding data storage systems are described herein. A grid of shards with a plurality of virtual shards is created where each virtual shard is a representation of a shard in the grid of shards that is not backed by a data storage device and where each shard of the grid of shards has an index value. Data is then stored in the grid of shards by updating a shard to store the data and by also updating a second shard based on a set of shards with the same index value as the shard updated to store the data.

Abstract: Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored. The volumes may be apportioned into failure-decorrelated subsets, and archives stored thereto may be apportioned to such subsets.

Abstract: Techniques described and suggested herein include various methods and systems for verifying integrity of redundancy coded data, such as erasure coded data shards. In some embodiments, a quantity of redundancy coded data elements, hereafter referred to as data shards (e.g., erasure coded data shards), sufficient to reconstruct the original data element from which the redundancy coded data elements are derived, is used to generate reconstructed data shards to be used for checking the validity of analogous data shards stored for the original data element.