Abstract: A cost-effective, durable and scalable archival data storage system is provided herein that allow customers to store, retrieve and delete archival data objects, among other operations. For data storage, in an embodiment, the system stores data in a transient data store and provides a data object identifier may be used by subsequent requests. For data retrieval, in an embodiment, the system creates a job corresponding to the data retrieval and provides a job identifier associated with the created job. Once the job is executed, data retrieved is provided in a transient data store to enable customer download. In various embodiments, jobs associated with storage, retrieval and deletion are scheduled and executed using various optimization techniques such as load balancing, batch processed and partitioning. Data is redundantly encoded and stored in self-describing storage entities increasing reliability while reducing storage costs. Data integrity is ensured by integrity checks along data paths.

Abstract: Methods and apparatus for supporting cached volumes at storage gateways are disclosed. A storage gateway appliance is configured to cache at least a portion of a storage object of a remote storage service at local storage devices. In response to a client's write request, directed to at least a portion of a data chunk of the storage object, the appliance stores a data modification indicated in the write request at a storage device, and asynchronously uploads the modification to the storage service. In response to a client's read request, directed to a different portion of the data chunk, the appliance downloads the requested data from the storage service to the storage device, and provides the requested data to the client.

Abstract: Techniques for performing data-related operations using data storage devices are described herein. Data storage devices are configured and/or enabled to perform data operations against one or more logical data addresses thereon. The data storage device receives a request including at least executable instructions defining the data operations to be performed and a range of logical data addresses upon which to execute the data operations. Upon request, either the same request as the one defining the data operations or in a separate request, the defined data operations are executed against the specified logical data addresses.

Abstract: A system for storing data includes a rack, one or more data storage drive assemblies coupled to the rack, and a data control module coupled to the rack. The data storage drive assemblies include one or more drive mechanical modules configured to store data and one or more drive control modules coupled to the drive mechanical modules. The drive control modules control mechanical operations in the drive mechanical modules. The drive mechanical modules and the associated drive control modules are separable from one another without removing the other module from the at least one data storage drive assembly.

Abstract: Techniques for performing data-related operations using data storage devices are described herein. Data storage devices are configured and/or enabled to perform data operations against one or more logical data addresses thereon. The data storage device receives a request including at least executable instructions defining the data operations to be performed and a range of logical data addresses upon which to execute the data operations. Upon request, either the same request as the one defining the data operations or in a separate request, the defined data operations are executed against the specified logical data addresses.

Abstract: Methods and apparatus for supporting cached volumes at storage gateways are disclosed. A storage gateway appliance is configured to cache at least a portion of a storage object of a remote storage service at local storage devices. In response to a client's write request, directed to at least a portion of a data chunk of the storage object, the appliance stores a data modification indicated in the write request at a storage device, and asynchronously uploads the modification to the storage service. In response to a client's read request, directed to a different portion of the data chunk, the appliance downloads the requested data from the storage service to the storage device, and provides the requested data to the client.

Abstract: Methods and systems are provided herein to allow efficient management of data flowing in and out of an archival data storage system. In an embodiment, storage entities keep very little state information in memory to provide higher throughput. Further, storage entities may send data in large chunks to facilitate high throughput. Techniques such as batching and coalescing may be used by various storage entities to provide efficiency.

Abstract: Embodiments of the present disclosure are directed to, among other things, validating the integrity of received and/or stored data payloads. In some examples, a storage service may perform a first partitioning of a data object into first partitions based at least in part on a first operation. The storage service may also verify the data object, by utilizing a verification algorithm, to generate a first verification value. In some cases, the storage service may additionally perform a second partitioning of the data object into second partitions based at least in part on a second operation. The second partitions may be different from the first partitions. Additionally, the archival data storage service may verify the data object using the verification algorithm to generate a second verification value. Further, the storage service may determine whether the second verification value equals the first verification value.

Abstract: Various embodiments (“systems”) are described for transferring data from a primary storage (e.g., magnetic disk drives, solid state drives, etc.) to an optical cold storage rack. The optical cold storage rack may include many physical optical storage disks, but a much smaller number of burners and readers (e.g., optical disk drives). When data is to be transferred to the optical cold storage rack, the system may generate a plan for performing the transfer. “Migration worker” components may then implement the plan and may be exclusively dedicated to implementing such plans. In various embodiments, the plan may specify how large data file “aggregates” (collections of portions of one or more data files) are to be distributed across optical disks (“disks”) to improve throughput during subsequent reading operations from the optical cold storage rack. The plan may also anticipate the relation between the limited number of burners/readers and the overall optical cold storage rack disk capacity.

Abstract: A system for storing data includes a rack, one or more data storage drive assemblies coupled to the rack, and a data control module coupled to the rack. The data storage drive assemblies include one or more drive mechanical modules configured to store data and one or more drive control modules coupled to the drive mechanical modules. The drive control modules control mechanical operations in the drive mechanical modules. The drive mechanical modules and the associated drive control modules are separable from one another without removing the other module from the at least one data storage drive assembly.

Abstract: Various embodiments (“systems”) are described for transferring data from a primary storage (e.g., magnetic disk drives, solid state drives, etc.) to an optical cold storage rack. The optical cold storage rack may include many physical optical storage disks, but a much smaller number of burners and readers (e.g., optical disk drives). When data is to be transferred to the optical cold storage rack, the system may generate a plan for performing the transfer. “Migration worker” components may then implement the plan and may be exclusively dedicated to implementing such plans. In various embodiments, the plan may specify how large data file “aggregates” (collections of portions of one or more data files) are to be distributed across optical disks (“disks”) to improve throughput during subsequent reading operations from the optical cold storage rack. The plan may also anticipate the relation between the limited number of burners/readers and the overall optical cold storage rack disk capacity.

Abstract: Embodiments of the present disclosure are directed to, among other things, validating the integrity of received and/or stored data payloads. In some examples, a storage service may perform a first partitioning of a data object into first partitions based at least in part on a first operation. The storage service may also verify the data object, by utilizing a verification algorithm, to generate a first verification value. In some cases, the storage service may additionally perform a second partitioning of the data object into second partitions based at least in part on a second operation. The second partitions may be different from the first partitions. Additionally, the archival data storage service may verify the data object using the verification algorithm to generate a second verification value. Further, the storage service may determine whether the second verification value equals the first verification value.

Abstract: Restarting transmission and application of changed data. A method includes receiving from the second location a first checkpoint. The first checkpoint points to a last record applied to replication data at the second location to replicate data from the first location. The method further includes receiving from the second location a second checkpoint. The second checkpoint is received by the second location from the first location. The second checkpoint identifies a last record in the set of transmitted records. The method further includes transmitting the received first checkpoint back to the second location with a command to restart applying records from the last record applied pointed to in the first checkpoint. The method further includes transmitting the received second checkpoint to the first location with a command for the first location to restart transmission of records from the last record in the set of records.

Abstract: Methods and apparatus for supporting cached volumes at storage gateways are disclosed. A storage gateway appliance is configured to cache at least a portion of a storage object of a remote storage service at local storage devices. In response to a client's write request, directed to at least a portion of a data chunk of the storage object, the appliance stores a data modification indicated in the write request at a storage device, and asynchronously uploads the modification to the storage service. In response to a client's read request, directed to a different portion of the data chunk, the appliance downloads the requested data from the storage service to the storage device, and provides the requested data to the client.

Abstract: Embodiments of the present disclosure are directed to, among other things, managing power of one or more data storage devices. In some examples, a storage service may obtain a schedule associated with enabling different storage devices at different times. The storage service may also identify a request of a batch of requests for accessing the storage devices. In some cases, the storage service may also determine which storage device to activate based at least in part on the schedule and/or the request. Further, the storage service may manage power of a storage device based at least in part on the determination of which storage device to activate.

Abstract: Various embodiments (“systems”) are described for transferring data from a primary storage (e.g., magnetic disk drives, solid state drives, etc.) to an optical cold storage rack. The optical cold storage rack may include many physical optical storage disks, but a much smaller number of burners and readers (e.g., optical disk drives). When data is to be transferred to the optical cold storage rack, the system may generate a plan for performing the transfer. “Migration worker” components may then implement the plan and may be exclusively dedicated to implementing such plans. In various embodiments, the plan may specify how large data file “aggregates” (collections of portions of one or more data files) are to be distributed across optical disks (“disks”) to improve throughput during subsequent reading operations from the optical cold storage rack. The plan may also anticipate the relation between the limited number of burners/readers and the overall optical cold storage rack disk capacity.

Abstract: An application programming interface for a data storage service provides a convenient mechanism for clients of the data storage service to access its various capabilities. An API call may be made to initiate a job and in response a job identifier may be provided. A separate API call specifying the job identifier may be made and a response providing information related to the job may result. Various API calls may be used to store data, retrieve data, obtain an inventory of stored data, and to obtain other information relating to stored data.

Abstract: A request to retrieve a persistently stored data object is received, the request including a data object identifier that encodes at least storage location information and validation information related to the data object. The data object is retrieved using at least the storage location information to form a retrieved data object, and validation is performed using at least the validation information.

Abstract: Methods and apparatus for intelligent cache eviction at storage gateways are disclosed. A system comprises computing devices configured to determine whether the number of free chunks of storage at a storage appliance for caching portions of a storage object is below a threshold value. If the number is below the threshold, the computing devices identify an eviction set of chunks to be freed, and generate a respective new instance identifier for each chunk of the eviction set. The identifier of a given chunk may be used to determine a validity of a block of the chunk. The devices store, within metadata storage of the appliance, the new instance identifiers of the eviction set, and indicate that the chunks of the eviction set are available for caching data of the storage object.

Abstract: Methods and apparatus for efficient recovery of cached volumes at storage gateways are disclosed. To recover, after an unplanned shutdown, a storage gateway appliance configured to cache chunks of a storage object, chunk metadata corresponding to a particular chunk is read into an in-memory metadata region from a first metadata location. Based on analysis of the chunk metadata, a validation requirement indication for the particular chunk is stored, and the chunk is designated as being accessible for client I/O requests. In response to receiving a subsequent I/O request targeted to the particular chunk, the chunk metadata is validated using a different metadata location prior to performing the requested I/O operation.