Abstract: Techniques described and suggested herein include various methods and systems for verifying integrity of redundancy coded data, such as erasure coded data shards. In some embodiments, a quantity of redundancy coded data elements, hereafter referred to as data shards (e.g., erasure coded data shards), sufficient to reconstruct the original data element from which the redundancy coded data elements are derived, is used to generate reconstructed data shards to be used for checking the validity of analogous data shards stored for the original data element.

Abstract: Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored.

Abstract: Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored.

Abstract: A data center may include a tape library rack module along with rack computer systems. The rack computer systems may be configured to provide computing capacity within a data center environment. In some embodiments, the tape library rack module may include an enclosure encompassing an interior of the tape library rack module, a rack within the interior, and a tape library unit mounted on the rack. The tape library rack unit may include tape cartridges configured to store data within a tape environment that is different than the data center environment. The tape library rack unit may be within a portion of the interior that is enclosed such that it is environmentally isolated from the data center environment. In some examples, the tape library rack module may include a cooling unit and/or a humidifier unit, which may provide the tape environment to the environmentally isolated portion of the interior of the tape library rack module.

Abstract: Techniques and systems for storing and retrieving data storage devices of a data storage system are disclosed. In some embodiments, inventory holders are used to store data storage devices used by a data storage system. When data is to be transacted with the data storage devices, mobile drive units locate appropriate inventory holders and transport them to a device reading station, where an appropriate device retrieval unit transacts the data. In some embodiments, each inventory holder includes a plurality of trays that are independently accessible to the device retrieval unit. After the data has been transacted, the data storage devices are returned to the appropriate inventory holders, and the inventory holders are placed by the mobile drive units in locations where they may be accessed in response to further data transactions.

Abstract: Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored.

Abstract: A computer system, such as a data storage system, implements techniques for segregating and controlling access to data stored in multiple regions. In some embodiments, redundancy coded shards generated from the data and stored in durable storage of a data storage system is allocated across multiple regions, but in a fashion that prevents actors with access to regions outside that of a “home” region from recovering a sufficient number of unique shards to regenerate the data represented thereby. In some embodiments, encryption is used to segregate the data by encrypting the generated shards, then storing the cryptographic information on or otherwise controlling access on hosts or other devices of only the home region.

Abstract: Techniques for exploiting variable media sizes to create new redundancy encoded data storage systems are described herein. A set of storage devices is selected based at least in part on each storage device having an available capacity and, using the set of storage devices, a set of shards for a redundancy encoded data storage system is generated such that each shard of the set of shards has a storage capacity corresponding to the minimum available capacity of the set of storage devices.

Abstract: An enclosure-based tape drive server, comprises an enclosure, a controller, a memory linked to the controller, a plurality of at least eight tape drives, a host bus adapter and a network controller. The controller is positioned within the enclosure. The at least eight tape drives are positioned within the enclosure, and each of the tape drives is accessible from a front side of the enclosure. The host bus adapter couples the plurality of tape drives to the controller. The network controller is connectible to a network to couple the server to the network.

Abstract: Techniques and systems for storing and retrieving data storage devices of a data storage system are disclosed. In some embodiments, inventory holders are used to store data storage devices used by a data storage system. When data is to be transacted with the data storage devices, mobile drive units locate appropriate inventory holders and transport them to a device reading station, where an appropriate device retrieval unit transacts the data. The inventory holders are configured such that any of the mobile drive units may interchangeably interact with any of the inventory holders and/or the data device reading stations. After the data has been transacted, the data storage devices are returned to the appropriate inventory holders, and the inventory holders are placed by the mobile drive units in locations where they may be accessed in response to further data transactions.

Abstract: A data center may include a tape library rack module along with rack computer systems. The rack computer systems may be configured to provide computing capacity within a data center environment. In some embodiments, the tape library rack module may include an enclosure encompassing an interior of the tape library rack module, a rack within the interior, and a tape library unit mounted on the rack. The tape library rack unit may include tape cartridges configured to store data within a tape environment that is different than the data center environment. The tape library rack unit may be within a portion of the interior that is enclosed such that it is environmentally isolated from the data center environment. In some examples, the tape library rack module may include a cooling unit and/or a humidifier unit, which may provide the tape environment to the environmentally isolated portion of the interior of the tape library rack module.

Abstract: A data storage service distributes a plurality of data fragments corresponding to a data object among one or more data storage host groupings in a manner that avoids a possibility of correlated loss of multiple data fragments by consolidation of data of a data storage host grouping onto a single data storage host. The data storage service selects a data storage host grouping and determines an amount of used capacity for the selected data storage host grouping. If the selected grouping satisfies an emptiness threshold, the data storage service selects a data storage host from the grouping and consolidates one or more data sets of the grouping onto the selected data storage host. Subsequently, the data storage service updates metadata for each data storage host of the selected data storage host grouping to specify a location of data stored therein.

Abstract: Techniques described and suggested herein include systems and methods for optimizing random access characteristics for data archives stored on data storage systems using redundancy coding techniques. For example, redundancy coded shards, which may include identity shards that contain unencoded original data of archives, may be configured such that a variable number of the shards can be leveraged to meet random access requirements for retrieval requests associated with the archives stored and/or encoded therein. Implementing systems may monitor random access rates, capabilities, and burdens, so as to adaptively account for changes to some or all of the monitored parameters.

Abstract: Methods and apparatus for data durability using un-encoded copies together with combination encodings are disclosed. Additional storage objects to be included in a durability group with a particular storage object are identified. An encoding of a combination of the particular object and another object of the group is generated. The encoding, as well as an un-encoded copy of the particular object, may be stored at respective nodes of a service. A request to perform a computation operation may be directed to a node with the un-encoded copy. In the event that the un-encoded copy becomes unavailable, a new un-encoded copy may be generated using the encoding.

Abstract: Embodiments of the present disclosure are directed to, among other things, managing space allocation of one or more data storage devices. In some examples, a storage service may identify at least one computer-readable storage device. The computer-readable storage device may, in some examples, have availability level that corresponds to a size of data to be stored. The storage service may also identify a target storage device based at least in part on a capacity of a subset of the storage devices. In some cases, this may be performed for one or more of the devices of the subset. Further, the storage service may store the data in the identified target storage device.

Abstract: Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored. The volumes may be apportioned into failure-decorrelated subsets, and archives stored thereto may be apportioned to such subsets.

Abstract: Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored. The volumes may be apportioned into failure-decorrelated subsets, and archives stored thereto may be apportioned to such subsets.

Abstract: Techniques described and suggested herein include systems and methods for precomputing regeneration information for data archives (“archives”) that have been processed and stored using redundancy coding techniques. For example, regeneration information, such as redundancy code-related matrices (such as inverted matrices based on, e.g., a generator matrix for the selected redundancy code) corresponding to subsets of the shards, is computed for each subset and, in some embodiments, stored for use in the event that one or more shards becomes unavailable, e.g., so as to more efficiently and/or quickly regenerate a replacement shard.

Abstract: Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored. The volumes may be apportioned into failure-decorrelated subsets, and archives stored thereto may be apportioned to such subsets.

Abstract: Erasure encoded fragments are originally generated by applying an erasure encoding scheme to a data file. An erasure encoded fragment is subsequently generated directly from previously generated erasure encoded fragments or by reconstructing the original data file and then erasure encoding the reconstructed data file. The integrity or fidelity of such a subsequently generated erasure encoded fragment is verified by newly generating an error detection code, such as but not limited to a checksum, for the subsequently generated erasure encoded fragment, and comparing that subsequently error detection code against an error detection code previously generated for a previous or original version of the erasure encoded fragment. Each error detection code is preferably stored in association with its corresponding erasure encoded fragment and with one or more other erasure encoded fragments. Thus, each error detection code is saved in at least two locations.