Abstract: According to examples, a system for rate-based load balancing may include a processor and a memory storing instructions. The processor may, through execution of the instructions, cause the system to receive a request for processing. The system may further identify a target server to transmit the request using a rate-based load balancing technique. In some examples, the rate-based load balancing technique may include: selecting a server, from a plurality of servers, as a potential target; receiving a readiness indicator for the selected server; and designating the selected server as the target server based on the readiness indicator. The system may transmit the request to the target server for processing.

Abstract: According to examples, a system for rate-based load balancing may include a processor and a memory storing instructions. The processor may, through execution of the instructions, cause the system to receive a request for processing. The system may further identify a target server to transmit the request using a rate-based load balancing technique. In some examples, the rate-based load balancing technique may include: selecting a server, from a plurality of servers, as a potential target; receiving a readiness indicator for the selected server; and designating the selected server as the target server based on the readiness indicator. The system may transmit the request to the target server for processing.

Abstract: Methods and apparatus for supporting cached volumes at storage gateways are disclosed. A storage gateway appliance is configured to cache at least a portion of a storage object of a remote storage service at local storage devices. In response to a client's write request, directed to at least a portion of a data chunk of the storage object, the appliance stores a data modification indicated in the write request at a storage device, and asynchronously uploads the modification to the storage service. In response to a client's read request, directed to a different portion of the data chunk, the appliance downloads the requested data from the storage service to the storage device, and provides the requested data to the client.

Abstract: An application programming interface for a data storage service provides a convenient mechanism for clients of the data storage service to access its various capabilities. An API call may be made to initiate a job and in response a job identifier may be provided. A separate API call specifying the job identifier may be made and a response providing information related to the job may result. Various API calls may be used to store data, retrieve data, obtain an inventory of stored data, and to obtain other information relating to stored data.

Abstract: An application programming interface for a data storage service provides a convenient mechanism for clients of the data storage service to access its various capabilities. An API call may be made to initiate a job and in response a job identifier may be provided. A separate API call specifying the job identifier may be made and a response providing information related to the job may result. Various API calls may be used to store data, retrieve data, obtain an inventory of stored data, and to obtain other information relating to stored data.

Abstract: Methods and apparatus for supporting cached volumes at storage gateways are disclosed. A storage gateway appliance is configured to cache at least a portion of a storage object of a remote storage service at local storage devices. In response to a client's write request, directed to at least a portion of a data chunk of the storage object, the appliance stores a data modification indicated in the write request at a storage device, and asynchronously uploads the modification to the storage service. In response to a client's read request, directed to a different portion of the data chunk, the appliance downloads the requested data from the storage service to the storage device, and provides the requested data to the client.

Abstract: Methods and apparatus for supporting cached volumes at storage gateways are disclosed. A storage gateway appliance is configured to cache at least a portion of a storage object of a remote storage service at local storage devices. In response to a client's write request, directed to at least a portion of a data chunk of the storage object, the appliance stores a data modification indicated in the write request at a storage device, and asynchronously uploads the modification to the storage service. In response to a client's read request, directed to a different portion of the data chunk, the appliance downloads the requested data from the storage service to the storage device, and provides the requested data to the client.

Abstract: Embodiments of the present disclosure are directed to, among other things, validating the integrity of received and/or stored data payloads. In some examples, a storage service may perform a first partitioning of a data object into first partitions based at least in part on a first operation. The storage service may also verify the data object, by utilizing a verification algorithm, to generate a first verification value. In some cases, the storage service may additionally perform a second partitioning of the data object into second partitions based at least in part on a second operation. The second partitions may be different from the first partitions. Additionally, the archival data storage service may verify the data object using the verification algorithm to generate a second verification value. Further, the storage service may determine whether the second verification value equals the first verification value.

Abstract: Methods and apparatus for supporting cached volumes at storage gateways are disclosed. A storage gateway appliance is configured to cache at least a portion of a storage object of a remote storage service at local storage devices. In response to a client's write request, directed to at least a portion of a data chunk of the storage object, the appliance stores a data modification indicated in the write request at a storage device, and asynchronously uploads the modification to the storage service. In response to a client's read request, directed to a different portion of the data chunk, the appliance downloads the requested data from the storage service to the storage device, and provides the requested data to the client.

Abstract: Techniques for implementing a data management scheme for optimizing data storage are described herein. A deletion quantity or other metric relating to deletions of data within a first storage zone are tracked. Upon detection that the tracked deletion metric meets certain criteria, the data within the first storage zone are moved to a second storage zone. A verification of the data to be moved is also performed, and if such verification indicates that at least a portion of the data is corrupted, routines repairing and/or restoring at least the corrupted portion are initiated.

Abstract: Embodiments of the present disclosure are directed to, among other things, validating the integrity of received and/or stored data payloads. In some examples, a storage service may perform a first partitioning of a data object into first partitions based at least in part on a first operation. The storage service may also verify the data object, by utilizing a verification algorithm, to generate a first verification value. In some cases, the storage service may additionally perform a second partitioning of the data object into second partitions based at least in part on a second operation. The second partitions may be different from the first partitions. Additionally, the archival data storage service may verify the data object using the verification algorithm to generate a second verification value. Further, the storage service may determine whether the second verification value equals the first verification value.

Abstract: Methods and apparatus for supporting cached volumes at storage gateways are disclosed. A storage gateway appliance is configured to cache at least a portion of a storage object of a remote storage service at local storage devices. In response to a client's write request, directed to at least a portion of a data chunk of the storage object, the appliance stores a data modification indicated in the write request at a storage device, and asynchronously uploads the modification to the storage service. In response to a client's read request, directed to a different portion of the data chunk, the appliance downloads the requested data from the storage service to the storage device, and provides the requested data to the client.

Abstract: A system for storing data includes a rack, one or more data storage drive assemblies coupled to the rack, and a data control module coupled to the rack. The data storage drive assemblies include one or more drive mechanical modules configured to store data and one or more drive control modules coupled to the drive mechanical modules. The drive control modules control mechanical operations in the drive mechanical modules. The drive mechanical modules and the associated drive control modules are separable from one another without removing the other module from the at least one data storage drive assembly.

Abstract: Methods and systems are described herein to provide efficient data retrieval in a data storage system. Specifically, in cases where users of a data storage system are not overly sensitive to data retrieval time, such as the case for backup and archival data storage systems, random read requests may be fulfilled as part of sequential reads to reduce I/O operations. A data storage system may be divided into data storage zones. Sequential reads may be performed for data stored in those data storage zones with pending data retrieval requests. Data retrieval requests may be fulfilled based at least in part on the sequentially-read data.

Abstract: Embodiments of the present disclosure are directed to, among other things, managing space allocation of one or more data storage devices. In some examples, a storage service may identify at least one computer-readable storage device. The computer-readable storage device may, in some examples, have availability level that corresponds to a size of data to be stored. The storage service may also identify a target storage device based at least in part on a capacity of a subset of the storage devices. In some cases, this may be performed for one or more of the devices of the subset. Further, the storage service may store the data in the identified target storage device.

Abstract: Techniques for implementing a log-based storage scheme upon data storage devices are described herein. A data storage device is initialized by writing an identifying record. For each portion of data to be written to the drive, a first record including information regarding the anticipated nature of the portion of data is written prior to the data. The data is then written as a second record that includes at least the raw data as well as integrity verification information. A third record is stored following the second record, and includes an accounting and/or index of the data successfully written in the second record. On sequentially written devices, the information in the first stored record may be used to locate the third record, which in turn may be used to record data in the second record as well as the location of a first record of another portion of data.

Abstract: A cost-effective, durable and scalable archival data storage system is provided herein that allow customers to store, retrieve and delete archival data objects, among other operations. For data storage, in an embodiment, the system stores data in a transient data store and provides a data object identifier may be used by subsequent requests. For data retrieval, in an embodiment, the system creates a job corresponding to the data retrieval and provides a job identifier associated with the created job. Once the job is executed, data retrieved is provided in a transient data store to enable customer download. In various embodiments, jobs associated with storage, retrieval and deletion are scheduled and executed using various optimization techniques such as load balancing, batch processed and partitioning. Data is redundantly encoded and stored in self-describing storage entities increasing reliability while reducing storage costs. Data integrity is ensured by integrity checks along data paths.

Abstract: Embodiments of the present disclosure are directed to, among other things, managing inventory indexing of one or more data storage devices. In some examples, a storage service may store an index associated with archived data. Additionally, the storage service may receive information associated with an operation performed on the archived data. The storage service may also partition the received information into subsets corresponding to an identifier. In some cases, the identifier may be received with or otherwise be part of the received information. The storage service may also retrieve at least a portion of the index that corresponds to the subset. Further, the storage service may update the retrieved portion of the index with at least part of the received information. The updating may be based at least in part on the subsets.

Abstract: Various embodiments (“systems”) are described for transferring data from a primary storage (e.g., magnetic disk drives, solid state drives, etc.) to an optical cold storage rack. The optical cold storage rack may include many physical optical storage disks, but a much smaller number of burners and readers (e.g., optical disk drives). When data is to be transferred to the optical cold storage rack, the system may generate a plan for performing the transfer. “Migration worker” components may then implement the plan and may be exclusively dedicated to implementing such plans. In various embodiments, the plan may specify how large data file “aggregates” (collections of portions of one or more data files) are to be distributed across optical disks (“disks”) to improve throughput during subsequent reading operations from the optical cold storage rack. The plan may also anticipate the relation between the limited number of burners/readers and the overall optical cold storage rack disk capacity.

Abstract: Techniques for implementing a log-based storage scheme upon data storage devices are described herein. A data storage device is initialized by writing an identifying record. For each portion of data to be written to the drive, a first record including information regarding the anticipated nature of the portion of data is written prior to the data. The data is then written as a second record that includes at least the raw data as well as integrity verification information. A third record is stored following the second record, and includes an accounting and/or index of the data successfully written in the second record. On sequentially written devices, the information in the first stored record may be used to locate the third record, which in turn may be used to record data in the second record as well as the location of a first record of another portion of data.