Abstract: Techniques described and suggested herein include systems and methods for precomputing regeneration information for data archives (“archives”) that have been processed and stored using redundancy coding techniques. For example, regeneration information, such as redundancy code-related matrices (such as inverted matrices based on, e.g., a generator matrix for the selected redundancy code) corresponding to subsets of the shards, is computed for each subset and, in some embodiments, stored for use in the event that one or more shards becomes unavailable, e.g., so as to more efficiently and/or quickly regenerate a replacement shard.

Abstract: Erasure encoded fragments may be originally generated by applying an erasure encoding scheme to a data file. An erasure encoded fragment may be subsequently generated directly from previously generated erasure encoded fragments or by reconstructing the original data file and then erasure encoding the reconstructed data file. The integrity or fidelity of such a subsequently generated erasure encoded fragment may be verified by newly generating an error detection code, such as but not limited to a checksum, for the subsequently generated erasure encoded fragment, and comparing that subsequently generated error detection code against an error detection code previously generated for a previous or original version of the erasure encoded fragment. Each error detection code is preferably stored in association with its corresponding erasure encoded fragment and with one or more other erasure encoded fragments. Thus, each error detection code is saved in at least two locations.

Abstract: Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored. The volumes may be apportioned into failure-decorrelated subsets, and archives stored thereto may be apportioned to such subsets.

Abstract: Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored.

Abstract: Reliable and efficient storage and reconstruction of secure data files is provided. Encrypted fragments are generated by exclusive-OR (XOR) based erasure-encoding and XOR encryption of data files. At least some of the encrypted fragments, and preferably at least two copies of such encrypted fragments, are stored at two or more locations, such as but not limited to two or more servers in two or more regional storage systems. Fragments are retrieved from one or more of the multiple locations and the original data file is reconstructed, even if different encryption techniques have been used. If not enough valid fragments from that original data file can be identified then hash values, checksums, seeds, and other techniques may be used to distinguish files and to identify related or identical files which may be used to reconstruct the data file.

Abstract: If none of the fragments of an erasure-coded data file have become corrupted then the original data file can be readily reconstructed. If one or more fragments of an erasure-coded data file have become corrupted it may still be possible to find a combination of fragments that reconstruct the original data file, but the number of possible combinations may be impracticably large. If an attempt with a first set of fragments fails, an efficient approach is to use an independent set of fragments for the second attempt. Then, for further attempts, the results of a current attempt are compared with previous results. If a match is found then the original data file has been reconstructed. An original data file may also be reconstructed by separately recovering each data block of the data file from corresponding fragment blocks and assembling the data file from the recovered data blocks.

Abstract: If none of the fragments of an erasure-coded data file have become corrupted then the original data file can be readily reconstructed. If one or more fragments of an erasure-coded data file have become corrupted it may still be possible to find a combination of fragments that reconstruct the original data file, but the number of possible combinations may be impracticably large. If an attempt with a first set of fragments fails, an efficient approach is to use an independent set of fragments for the second attempt. Then, for further attempts, the results of a current attempt are compared with previous results. If a match is found then the original data file has been reconstructed. An original data file may also be reconstructed by separately recovering each data block of the data file from corresponding fragment blocks and assembling the data file from the recovered data blocks.

Abstract: Cohorts may be created on storage nodes in an object-redundant storage system that uses replication and/or a redundant encoding technique. In a cohort with N nodes, M data elements (replicas or shards) of an object are stored to M of the nodes that are randomly (or otherwise) selected from the N nodes. Metadata for locating other data elements for an object in the cohort may be stored with one or more of the data elements in the cohort. To retrieve an object from a cohort, a subset of the nodes may be queried; the subset may be randomly or otherwise selected. If enough data elements are retrieved for the object from the queried nodes, the object is provided to the requester. Otherwise, additional data elements may be retrieved according to the metadata returned with a data element or by querying additional nodes.

Abstract: Aspects of a data environment, such as the creation, provisioning, and management of data stores and instances, are managed using a separate control environment. A user can call into an externally-facing interface of the control environment, the call being analyzed to determine actions to be performed in the data environment. A monitoring component of the control plane also can periodically communicate with the data environment to determine any necessary actions to be performed, such as to recover from faults or events in the data environment. A workflow can be instantiated that includes tasks necessary to perform the action. For each task, state information can be passed to a component in the data environment operable to perform the task, until all tasks for an action are completed. Data in the data environment can be accessed directly using an externally-facing interface of the data environment, without accessing the control plane.

Abstract: Aspects of a data environment, such as the creation, provisioning, and management of data stores and instances, are managed using a separate control environment. A user can call into an externally-facing interface of the control environment, the call being analyzed to determine actions to be performed in the data environment. A monitoring component of the control plane also can periodically communicate with the data environment to determine any necessary actions to be performed, such as to recover from faults or events in the data environment. A workflow can be instantiated that includes tasks necessary to perform the action. For each task, state information can be passed to a component in the data environment operable to perform the task, until all tasks for an action are completed. Data in the data environment can be accessed directly using an externally-facing interface of the data environment, without accessing the control plane.

Abstract: Aspects of a data environment, such as the creation, provisioning, and management of data stores and instances, are managed using a separate control environment. A user can call into an interface of the control environment, the call being analyzed to determine actions to be performed in the data environment. A monitoring component of the control plane also can periodically communicate with the data environment to determine any necessary actions to be performed, such as to recover from faults or events in the data environment. A workflow can be instantiated that includes tasks necessary to perform the action. For each task, state information can be passed to a component in the data environment operable to perform the task, until all tasks for an action are completed. Data in the data environment can be accessed directly using a separate interface of the data environment, without accessing the control plane.

Abstract: Aspects of a data environment, such as the creation, provisioning, and management of data stores and instances, are managed using a separate control environment. A user can call into an interface of the control environment, the call being analyzed to determine actions to be performed in the data environment. A monitoring component of the control plane also can periodically communicate with the data environment to determine any necessary actions to be performed, such as to recover from faults or events in the data environment. A workflow can be instantiated that includes tasks necessary to perform the action. For each task, state information can be passed to a component in the data environment operable to perform the task, until all tasks for an action are completed. Data in the data environment can be accessed directly using a separate interface of the data environment, without accessing the control plane.

Abstract: Aspects of a data environment, such as the creation, provisioning, and management of data stores and instances, are managed using a separate control environment. A user can call into an externally-facing interface of the control environment, the call being analyzed to determine actions to be performed in the data environment. A monitoring component of the control plane also can periodically communicate with the data environment to determine any necessary actions to be performed, such as to recover from faults or events in the data environment. A workflow can be instantiated that includes tasks necessary to perform the action. For each task, state information can be passed to a component in the data environment operable to perform the task, until all tasks for an action are completed. Data in the data environment can be accessed directly using an externally-facing interface of the data environment, without accessing the control plane.

Abstract: A configurable computing architecture (10) has its functionality controlled by a combination of static and dynamic control, wherein the configuration is referred to as static control and instructions are referred to as dynamic control. A reconfigurable data path (12) has a plurality of elements including functional units (32, 36), registers (30), and memories (34) whose interconnection and functionality is determined by a combination of static and dynamic control. These elements are connected together, using the static configuration, into a pipelined data path that performs a computation of interest. The dynamic control signals (21) are suitably used to change the operation of a functional unit and the routing of signals between functional units. The static control signals (23) are provided each by a static memory cell (62) that is written by a host (13). The controller (14) generates control instructions (16) that are interpreted by a control path (18) that computes the dynamic control signals.