Abstract: Example distributed storage systems, replication managers, and methods provide replication barriers for dependent data transfers between data stores. An object data store may include a barrier object and be configured to identify dependencies between a dependency set of data objects and the barrier object. When replicating data objects to another data store, the dependency set of data objects may be transferred first, delaying the transfer of the barrier object while the dependency set is being transferred.

Abstract: Example distributed storage systems, controller nodes, and methods provide selective allocation of redundant data blocks to background operations. Background operations may be identified targeting a data unit stored in redundant data blocks in a storage pool with a plurality of storage elements. A subset of data units may be selected for the background operation and the system components including those data units may be isolated. Data requests to the isolated system components may be selectively prevented while the background operation executes on the subset of data units in the isolated system components.

Abstract: Example storage systems, storage nodes, and methods provide storage node processing of predefined data functions, such as map-reduce functions. Storage nodes storing a plurality of symbols for a data unit are configured to select a predefined function using a data type of the data unit. Each storage node identifies subunits of the data unit from the symbols and processes the subunits using the predefined function to generate function results. A final result is returned based on the function results from each storage node.

Abstract: Example storage systems, storage nodes, and methods provide storage node processing of predefined data functions, such as map-reduce functions. Storage nodes storing a plurality of symbols for a data unit are configured to select a predefined function using a data type of the data unit. Each storage node identifies subunits of the data unit from the symbols and processes the subunits using the predefined function to generate function results. A final result is returned based on the function results from each storage node.

Abstract: Example storage systems, storage nodes, and methods provide storage node processing of data functions, such as map-reduce functions, using overlapping symbols. Storage nodes are configured to partition data units into symbols that include an overlap data portion of an adjacent symbol and erasure encode the symbols. The storage nodes may then decode erasure encoded symbols, identify subunits of a data unit from the decoded symbols, and process the subunits using map-functions to generate intermediate contexts. A reduce-function may be used to determine a function result using the intermediate contexts.

Abstract: Example storage systems, storage nodes, and methods provide storage node processing of data functions, such as serial functions. Storage nodes are configured to partition decode erasure encoded symbols, identify subunits of a data unit from the decoded symbols, process the subunits using a serial function to generate intermediate contexts, and send the intermediate context to a next storage node for continued processing using the serial function.

Abstract: Example storage systems, storage nodes, and methods provide parallel storage node processing of data functions, such as map-reduce functions. Storage nodes are configured to decode erasure encoded symbols, identify subunits of a data unit from the decoded symbols, and process the subunits in parallel using map-functions to generate intermediate contexts. A reduce-function may be used to determine a function result using the intermediate contexts.

Abstract: Example storage systems and methods provide data storage management using a key data store with progress values. A key data store includes a set of key data entries that each include a key value associated with a storage operation and a timestamp corresponding to a creation time of the key data entry. Storage management processes are executed on the set of key data entries and progress values for the storage management processes are tracked using the timestamps of the key data entries to manage the relative progress of the storage management processes.

Abstract: Systems and methods for metadata compaction in a distributed storage system with a file system interface are described. A file system interface and an object storage system interface use a metadata index for mapping object identifiers from the object storage system to location identifiers for the file system. When the metadata index includes a number of entries for continuous data blocks with overlapping intervals, a defragmentation operation may generate a defragmented entry for a defragmentation interval overlapping the overlapping data blocks.

Abstract: Example storage systems and methods provide data storage management using parallel processing of filtered transaction logs. Transaction logs are comprised of log entries corresponding to storage operations for at least one storage node. Sets of log entries are sequentially retrieved from the transaction log and filtered through multiple transaction log filters to generate multiple subsets of the log entries. Different metadata operations are executed in parallel using the different filtered subsets of log entries.

Abstract: Novel key ticketing technology includes an example method in which a first request associated with a first object storage operation is received. The first request includes a first timestamp associated with the first object storage operation and a first object identifier identifying a first object associated with the first object storage operation. The method calculates a first inverse timestamp based on the first timestamp, and generates a first object key corresponding to the first object storage operation. The first object key includes at least the first object identifier and the first inverse timestamp. The method further inserts a first entry including the first object key into a key data store at a position relative to other object key entries based on the first object identifier and the first inverse timestamp included in the first object key.

Abstract: Example storage systems and methods provide data storage management using generation markers in a key data store. A key data store includes a set of key data entries that each include a key value and a property value associated with a storage operation. An active generation of the key data entries include an active generation marker and a base generation does not. A base storage parameter is calculated from a data scan and a current storage parameter is calculated from the base storage parameter and the property values of the active generation of key data entries. The calculated storage parameter may be reported to manage storage configuration and operations in the storage system.

Abstract: Example storage systems and methods provide data storage management using change values in a key data store. A key data store includes a set of key data entries that each include a key value associated with a storage operation, a precondition value, and a postcondition value. A storage parameter is calculated using the precondition value and the postcondition value for each key data entry the set of key data entries. The calculated storage parameter may be reported to manage storage configuration and operations in the storage system.

Abstract: Example distributed storage systems, file system interfaces, and methods provide duplicate request checking. A file interface manages a plurality of file data requests from client systems using a file system protocol. A metadata store acts as a cross-reference between file data references and storage data references. A request data store in the metadata store includes request entries with transaction identifiers for corresponding file data requests. A duplicate checker searches the request data store and excludes incoming file data requests when it finds a matching request entry.

Abstract: Novel key ticketing technology includes an example method in which a first request associated with a first object storage operation is received. The first request includes a first timestamp associated with the first object storage operation and a first object identifier identifying a first object associated with the first object storage operation. The method calculates a first inverse timestamp based on the first timestamp, and generates a first object key corresponding to the first object storage operation. The first object key includes at least the first object identifier and the first inverse timestamp. The method further inserts a first entry including the first object key into a key data store at a position relative to other object key entries based on the first object identifier and the first inverse timestamp included in the first object key.

Abstract: A novel distributed data storage system is disclosed. In an example method, a first plurality of key entries is stored in a first key data store at a first location and a second plurality of key entries is stored in a second key data at a second location. A key entry in comprises a corresponding key having an object identifier, an inverse timestamp, and a source identifier. The method further replicates a set of the first key entries to the second key data store. The method further inserts each first key entry from the set of the first key entries into the second key data store based on the object identifier, the inverse timestamp, and the source identifier of the first key included in that first key entry, the first key entries and the second key entries being interwoven to form a plurality of interwoven ordered key entries.

Abstract: In an example embodiment, a method comprises determining that a multipart upload request to upload a data object in separate object parts has been received by an object storage service; generating temporary keys for the separate object parts of the data object; storing the temporary keys in a temporary key data store; generating, based on the temporary keys, a multipart key entry for the data object, the multipart key entry comprising a multipart key that contains an object identifier identifying the data object and an inverse timestamp; and inserting the multipart key entry in a persistent key data store storing an ordered set of key entries in a position determined by the object identifier and the inverse timestamp.

Abstract: In an example embodiment, a method comprises determining an ordered set of key entries; determining a first key entry for a first object in the ordered set of key entries; determining an object storage operation represented by a key of the first key entry; determining the object storage operation represented by the key of the first key entry to comprise a delete operation; and responsive to determining the object storage operation represented by the key of the first key entry to comprise the delete operation, skipping over subsequent key entries associated with the first object in the ordered set of key entries.