Abstract: Aspects of a data environment, such as the creation, provisioning, and management of data stores and instances, are managed using a separate control environment. A user can call into an externally-facing interface of the control environment, the call being analyzed to determine actions to be performed in the data environment. A monitoring component of the control plane also can periodically communicate with the data environment to determine any necessary actions to be performed, such as to recover from faults or events in the data environment. A workflow can be instantiated that includes tasks necessary to perform the action. For each task, state information can be passed to a component in the data environment operable to perform the task, until all tasks for an action are completed. Data in the data environment can be accessed directly using an externally-facing interface of the data environment, without accessing the control plane.

Abstract: Aspects of a data environment, such as the creation, provisioning, and management of data stores and instances, are managed using a separate control environment. A user can call into an externally-facing interface of the control environment, the call being analyzed to determine actions to be performed in the data environment. A monitoring component of the control plane also can periodically communicate with the data environment to determine any necessary actions to be performed, such as to recover from faults or events in the data environment. A workflow can be instantiated that includes tasks necessary to perform the action. For each task, state information can be passed to a component in the data environment operable to perform the task, until all tasks for an action are completed. Data in the data environment can be accessed directly using an externally-facing interface of the data environment, without accessing the control plane.

Abstract: A switching device is implemented in a network-attachable data transfer device to provide data storage access to other such devices. In some embodiments, network-attachable data transfer devices are arranged in a clustered configuration to provide various computational and storage services. When one or more devices of the cluster fails, various implementations associated with the switching device, via an external data interface, provide operational mitigation, optimized data recovery, and efficient reinstatement of normal operation of the cluster.

Abstract: Aspects of a data environment, such as the creation, provisioning, and management of data stores and instances, are managed using a separate control environment. A user can call into an externally-facing interface of the control environment, the call being analyzed to determine actions to be performed in the data environment. A monitoring component of the control plane also can periodically communicate with the data environment to determine any necessary actions to be performed, such as to recover from faults or events in the data environment. A workflow can be instantiated that includes tasks necessary to perform the action. For each task, state information can be passed to a component in the data environment operable to perform the task, until all tasks for an action are completed. Data in the data environment can be accessed directly using an externally-facing interface of the data environment, without accessing the control plane.

Abstract: A processing device receives a request to identify whether a file is a complete file. The processing device determines a first size of the file. The processing device determines whether the first size satisfies a size criterion. The processing device determines that the file is a complete file responsive to the determination that the first size satisfies the size criterion.

Abstract: Techniques and systems for storing and retrieving data storage devices of a data storage system are disclosed. In some embodiments, inventory holders are used to store data storage devices used by a data storage system. When data is to be transacted with the data storage devices, mobile drive units locate appropriate inventory holders and transport them to a device reading station, where an appropriate device retrieval unit transacts the data. After the data has been transacted, the data storage devices are returned to the appropriate inventory holders, and the inventory holders are placed by the mobile drive units in locations where they may be accessed in response to further data transactions.

Abstract: A data center may include a tape library rack module along with rack computer systems. The rack computer systems may be configured to provide computing capacity within a data center environment. In some embodiments, the tape library rack module may include an enclosure encompassing an interior of the tape library rack module, a rack within the interior, and a tape library unit mounted on the rack. The tape library rack unit may include tape cartridges configured to store data within a tape environment that is different than the data center environment. The tape library rack unit may be within a portion of the interior that is enclosed such that it is environmentally isolated from the data center environment. In some examples, the tape library rack module may include a cooling unit and/or a humidifier unit, which may provide the tape environment to the environmentally isolated portion of the interior of the tape library rack module.

Abstract: A data storage service stores a dataset on a set of storage nodes in accordance with a first encoding. A set of shards constituting quorum, and one or more additional shards, are stored on the storage nodes. The data storage system determines to store the dataset according to a second encoding, in which the second encoding has fewer total shards. The data storage system reconfigures the storage of the dataset in accordance with the second encoding, such that the reconfigured storage comprises subsets of shards from the first encoding that were not re-encoded in forming the second encoding.

Abstract: Aspects of a data environment, such as the creation, provisioning, and management of data stores and instances, are managed using a separate control environment. A user can call into an externally-facing interface of the control environment, the call being analyzed to determine actions to be performed in the data environment. A monitoring component of the control plane also can periodically communicate with the data environment to determine any necessary actions to be performed, such as to recover from faults or events in the data environment. A workflow can be instantiated that includes tasks necessary to perform the action. For each task, state information can be passed to a component in the data environment operable to perform the task, until all tasks for an action are completed. Data in the data environment can be accessed directly using an externally-facing interface of the data environment, without accessing the control plane.

Abstract: Cohorts may be created on storage nodes in an object-redundant storage system that uses replication and/or a redundant encoding technique. In a cohort with N nodes, M data elements (replicas or shards) of an object are stored to M of the nodes that are randomly (or otherwise) selected from the N nodes. Metadata for locating other data elements for an object in the cohort may be stored with one or more of the data elements in the cohort. To retrieve an object from a cohort, a subset of the nodes may be queried; the subset may be randomly or otherwise selected. If enough data elements are retrieved for the object from the queried nodes, the object is provided to the requester. Otherwise, additional data elements may be retrieved according to the metadata returned with a data element or by querying additional nodes.

Abstract: Techniques and systems for storing and retrieving data storage devices of a data storage system are disclosed. In some embodiments, inventory holders are used to store data storage devices used by a data storage system. When data is to be transacted with the data storage devices, mobile drive units locate appropriate inventory holders and transport them to a device reading station, where an appropriate device retrieval unit transacts the data. In some embodiments, each inventory holder includes a heterogenous mix of data storage device types, the layout of which may be calculated according to the specific mix allocated to a given inventory holder. After the data has been transacted, the data storage devices are returned to the appropriate inventory holders, and the inventory holders are placed by the mobile drive units in locations where they may be accessed in response to further data transactions.

Abstract: A switching device is implemented in a network-attachable data transfer device to provide data storage access to other such devices. In some embodiments, network-attachable data transfer devices are arranged in a clustered configuration to provide various computational and storage services. When one or more devices of the cluster fails, various implementations associated with the switching device, via an external data interface, provide operational mitigation, optimized data recovery, and efficient reinstatement of normal operation of the cluster.

Abstract: A system and method for dynamically implementing a resource budget based at least in part on receiving information that prompts a determination of whether to adjust a maximum amount of resources available for utilization at least in part by data storage operations. As a result of the determination, the system and method produce, based at least in part on the information, a resource budget that reflects an adjustment to the maximum amount of resources available for utilization at least in part by the data storage operations, and implements the resource budget such that performance of the data storage operations is adjusted in accordance with the adjustment to the maximum amount of resources available for utilization.

Abstract: A multilayered index is implemented for data stored in a data storage system. The multilayered index may include a granular index that specifies the location of specific files or abstractions within the data storage system, while a skip table specifies segments of the abstractions, and may point to both the granular index and the data storage devices of the data storage system. In redundancy coded systems, the use of multilayered indices may enable asynchronous retrieval of data during normal or adverse operation (such as during periods of varying data availability).

Abstract: A data storage service stores a dataset on a set of storage nodes in accordance with a first encoding. A set of shards constituting quorum, and one or more additional shards, are stored on the storage nodes. The data storage system determines to store the dataset according to a second encoding, in which the second encoding has a greater number of shards. The data storage system reconfigures the storage of the dataset in accordance with the second encoding, such that the reconfigured storage forms additional shards for the second encoding by combining portions of shards of the first encoding.

Abstract: A switching device is implemented in a network-attachable data transfer device to provide data storage access to other such devices. In some embodiments, network-attachable data transfer devices are arranged in a clustered configuration to provide various computational and storage services. When one or more devices of the cluster fails, various implementations associated with the switching device, via an external data interface, provide operational mitigation, optimized data recovery, and efficient reinstatement of normal operation of the cluster.

Abstract: A magnetic tape drive includes a head assembly with a write head and a read head. The read head is positioned ahead of the write head, so that as a magnetic tape is moved across the head, the read head is able to read data that is about to be overwritten by the write head. When a client computer system commands the tape drive to write data to the magnetic tape, the tape drive writes the data to the tape but preserves any data that is overwritten by reading the data before it is overwritten. The preserved data is returned to the client computer system. In various examples, the returned data can be written back to the tape, discarded, or stored elsewhere under the control of the client computer system.

Abstract: A tape drive allows for an increased track density by partially overlapping adjacent tracks as each successive track is written. Each successive track overwrites a portion of the width of the previous track, thereby reducing the width of the previous track. This process can be applied repeatedly thereby producing an arrangement of shingled data tracks across the width of the tape. In an embodiment, single-pass verification of data written in this manner is accomplished using a head assembly with two read heads per track. A first read head positioned behind the write head verifies that the data written is correct, and a second read head positioned over the previous track verifies that data on the previous track remains intact after being partially overwritten.

Abstract: Techniques for encoding data in grid encoded data storage systems are described herein. Data is stored in a grid of shards using grid encoding techniques that store the data in a combination of data shards and derived shards. Because the data is redundantly encoded in the grid of shards, a plurality of distinct copies of the data are reproducible from the grid of shards where each distinct copy is reproducible from a non-overlapping set of shards of the grid of shards.

Abstract: Techniques and systems for storing and retrieving data storage devices of a data storage system are disclosed. In some embodiments, inventory holders are used to store data storage devices used by a data storage system. When data is to be transacted with the data storage devices, mobile drive units locate appropriate inventory holders and transport them to a device reading station, where an appropriate device retrieval unit transacts the data. After the data has been transacted, the data storage devices are returned to the appropriate inventory holders, and the inventory holders are placed by the mobile drive units in locations where they may be accessed in response to further data transactions.