Abstract: An application programming interface for a data storage service provides a convenient mechanism for clients of the data storage service to access its various capabilities. An API call may be made to initiate a job and in response a job identifier may be provided. A separate API call specifying the job identifier may be made and a response providing information related to the job may result. Various API calls may be used to store data, retrieve data, obtain an inventory of stored data, and to obtain other information relating to stored data.

Abstract: An application programming interface for a data storage service provides a convenient mechanism for clients of the data storage service to access its various capabilities. An API call may be made to initiate a job and in response a job identifier may be provided. A separate API call specifying the job identifier may be made and a response providing information related to the job may result. Various API calls may be used to store data, retrieve data, obtain an inventory of stored data, and to obtain other information relating to stored data.

Abstract: Embodiments of the present disclosure are directed to, among other things, validating the integrity of received and/or stored data payloads. In some examples, a storage service may perform a first partitioning of a data object into first partitions based at least in part on a first operation. The storage service may also verify the data object, by utilizing a verification algorithm, to generate a first verification value. In some cases, the storage service may additionally perform a second partitioning of the data object into second partitions based at least in part on a second operation. The second partitions may be different from the first partitions. Additionally, the archival data storage service may verify the data object using the verification algorithm to generate a second verification value. Further, the storage service may determine whether the second verification value equals the first verification value.

Abstract: Techniques for implementing a data management scheme for optimizing data storage are described herein. A deletion quantity or other metric relating to deletions of data within a first storage zone are tracked. Upon detection that the tracked deletion metric meets certain criteria, the data within the first storage zone are moved to a second storage zone. A verification of the data to be moved is also performed, and if such verification indicates that at least a portion of the data is corrupted, routines repairing and/or restoring at least the corrupted portion are initiated.

Abstract: Embodiments of the present disclosure are directed to, among other things, validating the integrity of received and/or stored data payloads. In some examples, a storage service may perform a first partitioning of a data object into first partitions based at least in part on a first operation. The storage service may also verify the data object, by utilizing a verification algorithm, to generate a first verification value. In some cases, the storage service may additionally perform a second partitioning of the data object into second partitions based at least in part on a second operation. The second partitions may be different from the first partitions. Additionally, the archival data storage service may verify the data object using the verification algorithm to generate a second verification value. Further, the storage service may determine whether the second verification value equals the first verification value.

Abstract: In a system in which documents are generated dynamically in response to user requests, historical data is collected regarding data retrieval subtasks, such as service requests, that are performed to generate such documents. This data is used to predict the specific subtasks that will be performed to respond to specific document requests, such that these subtasks may be initiated preemptively at or near the outset of the associated document generation task. In one embodiment, the historical data is included within, or is used to generate, a mapping table that maps document generation tasks (which may correspond to specific URLs) to the data retrieval subtasks that are frequently performed within such tasks.

Abstract: Methods and systems are described herein to provide efficient data retrieval in a data storage system. Specifically, in cases where users of a data storage system are not overly sensitive to data retrieval time, such as the case for backup and archival data storage systems, random read requests may be fulfilled as part of sequential reads to reduce I/O operations. A data storage system may be divided into data storage zones. Sequential reads may be performed for data stored in those data storage zones with pending data retrieval requests. Data retrieval requests may be fulfilled based at least in part on the sequentially-read data.

Abstract: A cost-effective, durable and scalable archival data storage system is provided herein that allow customers to store, retrieve and delete archival data objects, among other operations. For data storage, in an embodiment, the system stores data in a transient data store and provides a data object identifier may be used by subsequent requests. For data retrieval, in an embodiment, the system creates a job corresponding to the data retrieval and provides a job identifier associated with the created job. Once the job is executed, data retrieved is provided in a transient data store to enable customer download. In various embodiments, jobs associated with storage, retrieval and deletion are scheduled and executed using various optimization techniques such as load balancing, batch processed and partitioning. Data is redundantly encoded and stored in self-describing storage entities increasing reliability while reducing storage costs. Data integrity is ensured by integrity checks along data paths.

Abstract: A cost-effective, durable and scalable archival data storage system is provided herein that allow customers to store, retrieve and delete archival data objects, among other operations. For data storage, in an embodiment, the system stores data in a transient data store and provides a data object identifier may be used by subsequent requests. For data retrieval, in an embodiment, the system creates a job corresponding to the data retrieval and provides a job identifier associated with the created job. Once the job is executed, data retrieved is provided in a transient data store to enable customer download. In various embodiments, jobs associated with storage, retrieval and deletion are scheduled and executed using various optimization techniques such as load balancing, batch processed and partitioning. Data is redundantly encoded and stored in self-describing storage entities increasing reliability while reducing storage costs. Data integrity is ensured by integrity checks along data paths.

Abstract: Methods and systems are provided herein to allow efficient management of data flowing in and out of an archival data storage system. In an embodiment, storage entities keep very little state information in memory to provide higher throughput. Further, storage entities may send data in large chunks to facilitate high throughput. Techniques such as batching and coalescing may be used by various storage entities to provide efficiency.

Abstract: Embodiments of the present disclosure are directed to, among other things, validating the integrity of received and/or stored data payloads. In some examples, a storage service may perform a first partitioning of a data object into first partitions based at least in part on a first operation. The storage service may also verify the data object, by utilizing a verification algorithm, to generate a first verification value. In some cases, the storage service may additionally perform a second partitioning of the data object into second partitions based at least in part on a second operation. The second partitions may be different from the first partitions. Additionally, the archival data storage service may verify the data object using the verification algorithm to generate a second verification value. Further, the storage service may determine whether the second verification value equals the first verification value.

Abstract: In this disclosure, a resource scheduler is described that allows virtual machine instances to earn resource credits during the low activity levels. Virtual machine instances that spend a predominant amount of time operating at low activity levels are able to quickly gain resource credits. Once these virtual machine instances acquire enough resource credits to surpass a threshold level, the resource scheduler can assign a high priority level to the virtual machine instances that provide them with priority access to CPU resources. The next time that the virtual machine instances enter a high activity level, they have a high priority level that allows them to preempt other, lower priority virtual machine instances. Thus, these virtual machine instances are able to process operations and/or respond to user requests with low latency.

Abstract: Embodiments of the present disclosure are directed to, among other things, validating the integrity of received and/or stored data payloads. In some examples, a storage service may perform a first partitioning of a data object into first partitions based at least in part on a first operation. The storage service may also verify the data object, by utilizing a verification algorithm, to generate a first verification value. In some cases, the storage service may additionally perform a second partitioning of the data object into second partitions based at least in part on a second operation. The second partitions may be different from the first partitions. Additionally, the archival data storage service may verify the data object using the verification algorithm to generate a second verification value. Further, the storage service may determine whether the second verification value equals the first verification value.

Abstract: Embodiments of the present disclosure are directed to, among other things, managing power of one or more data storage devices. In some examples, a storage service may obtain a schedule associated with enabling different storage devices at different times. The storage service may also identify a request of a batch of requests for accessing the storage devices. In some cases, the storage service may also determine which storage device to activate based at least in part on the schedule and/or the request. Further, the storage service may manage power of a storage device based at least in part on the determination of which storage device to activate.

Abstract: An application programming interface for a data storage service provides a convenient mechanism for clients of the data storage service to access its various capabilities. An API call may be made to initiate a job and in response a job identifier may be provided. A separate API call specifying the job identifier may be made and a response providing information related to the job may result. Various API calls may be used to store data, retrieve data, obtain an inventory of stored data, and to obtain other information relating to stored data.

Abstract: A request to retrieve a persistently stored data object is received, the request including a data object identifier that encodes at least storage location information and validation information related to the data object. The data object is retrieved using at least the storage location information to form a retrieved data object, and validation is performed using at least the validation information.

Abstract: Techniques for optimizing data storage are disclosed herein. In particular, methods and systems for implementing redundancy encoding schemes with data storage systems are described. The redundancy encoding schemes may be scheduled according to system and data characteristics. The schemes may span multiple tiers or layers of a storage system. The schemes may be generated, for example, in accordance with a transaction rate requirement, a data durability requirement or in the context of the age of the stored data. The schemes may be designed to rectify entropy-related effects upon data storage. The schemes may include one or more erasure codes or erasure coding schemes. Additionally, methods and systems for improving and/or accounting for failure correlation of various components of the storage system, including that of storage devices such as hard disk drives, are described.

Abstract: An application programming interface for a data storage service provides a convenient mechanism for clients of the data storage service to access its various capabilities. An API call may be made to initiate a job and in response a job identifier may be provided. A separate API call specifying the job identifier may be made and a response providing information related to the job may result. Various API calls may be used to store data, retrieve data, obtain an inventory of stored data, and to obtain other information relating to stored data.

Abstract: Methods and systems are provided herein that facilitate cost-effective and reliable data identification in an archival data storage system. In an embodiment, a data object stored in an archival data storage system is identified by a data object identifier which encodes storage location information that may be used to locate a data object stored in an archival data storage system that reduces the cost to store a global index. The data object identifier may also encode policy information such as access control information usable for validating subsequent access to the data object, payload validation information such as size and digest usable for verifying the integrity of the payload data, metadata validation information such as error-detection codes usable for validating integrity of the data object identifier and other information.

Abstract: Systems and methods are provided herein for storing data to enable inexpensive and/or guaranteed deletion of data. In various embodiments, a customer specifies a data deletion indication associated with a data object to be stored, specifying when and/or how to delete the data object. Such a data deletion indication may be based, for example, on a regulatory compliance requirement. Based at least in part on the data deletion indication, the storage system may select, from a plurality of storage devices, a storage device to store the data object. Data objects with similar data deletion indications may be stored in the same storage device. In some embodiments, a data object stored in a storage device using the methods described herein may be deleted as part of the deletion of all or a portion of the storage device near a time specified by the data deletion indication of the data object.