Abstract: Example storage systems and methods provide data storage management using a key data store with progress values. A key data store includes a set of key data entries that each include a key value associated with a storage operation and a timestamp corresponding to a creation time of the key data entry. Storage management processes are executed on the set of key data entries and progress values for the storage management processes are tracked using the timestamps of the key data entries to manage the relative progress of the storage management processes.

Abstract: A system, method and apparatus for storing metadata in a metadata store in a robust and efficient manner including receiving a request from a client to perform a data transaction, updating a key-value pair in a metadata store based on the request, entering the data transaction in a transaction log, updating a read cache with the key-value pair, and replicating the last transaction log entry in at least one other storage node in the metadata store.

Abstract: A system, method and apparatus for storing metadata in a metadata store in a robust and efficient manner including receiving a request from a client to perform a data transaction, updating a key-value pair in a metadata store based on the request, entering the data transaction in a transaction log, replicating the last transaction log entry in at least one other storage node in the metadata store and, in response to a threshold event, migrating a first portion of the transaction log from the a data storage medium to a second data storage medium of each of the plurality of data storage nodes.

Abstract: Example storage systems and methods provide data storage management using change values in a key data store. A key data store includes a set of key data entries that each include a key value associated with a storage operation, a precondition value, and a postcondition value. A storage parameter is calculated using the precondition value and the postcondition value for each key data entry the set of key data entries. The calculated storage parameter may be reported to manage storage configuration and operations in the storage system.

Abstract: A novel distributed data storage system is disclosed. In an example method, a first plurality of key entries is stored in a first key data store at a first location and a second plurality of key entries is stored in a second key data at a second location. A key entry in comprises a corresponding key having an object identifier, an inverse timestamp, and a source identifier. The method further replicates a set of the first key entries to the second key data store. The method further inserts each first key entry from the set of the first key entries into the second key data store based on the object identifier, the inverse timestamp, and the source identifier of the first key included in that first key entry, the first key entries and the second key entries being interwoven to form a plurality of interwoven ordered key entries.

Abstract: In an example embodiment, a method comprises determining that a multipart upload request to upload a data object in separate object parts has been received by an object storage service; generating temporary keys for the separate object parts of the data object; storing the temporary keys in a temporary key data store; generating, based on the temporary keys, a multipart key entry for the data object, the multipart key entry comprising a multipart key that contains an object identifier identifying the data object and an inverse timestamp; and inserting the multipart key entry in a persistent key data store storing an ordered set of key entries in a position determined by the object identifier and the inverse timestamp.

Abstract: Example storage systems and methods provide data storage management using generation markers in a key data store. A key data store includes a set of key data entries that each include a key value and a property value associated with a storage operation. An active generation of the key data entries include an active generation marker and a base generation does not. A base storage parameter is calculated from a data scan and a current storage parameter is calculated from the base storage parameter and the property values of the active generation of key data entries. The calculated storage parameter may be reported to manage storage configuration and operations in the storage system.

Abstract: In an example embodiment, a method comprises determining an ordered set of key entries; determining a first key entry for a first object in the ordered set of key entries; determining an object storage operation represented by a key of the first key entry; determining the object storage operation represented by the key of the first key entry to comprise a delete operation; and responsive to determining the object storage operation represented by the key of the first key entry to comprise the delete operation, skipping over subsequent key entries associated with the first object in the ordered set of key entries.

Abstract: Example storage systems and methods provide data storage management using a key data store with progress values. A key data store includes a set of key data entries that each include a key value associated with a storage operation and a timestamp corresponding to a creation time of the key data entry. Storage management processes are executed on the set of key data entries and progress values for the storage management processes are tracked using the timestamps of the key data entries to manage the relative progress of the storage management processes.

Abstract: Novel key ticketing technology includes an example method in which a first request associated with a first object storage operation is received. The first request includes a first timestamp associated with the first object storage operation and a first object identifier identifying a first object associated with the first object storage operation. The method calculates a first inverse timestamp based on the first timestamp, and generates a first object key corresponding to the first object storage operation. The first object key includes at least the first object identifier and the first inverse timestamp. The method further inserts a first entry including the first object key into a key data store at a position relative to other object key entries based on the first object identifier and the first inverse timestamp included in the first object key.

Abstract: An object is divided into SD1 first-level pieces. Each first-level piece is stored in a first-level container on a first-level storage entity. A redundant encoding of the first-level containers is stored in RL1 additional first-level containers on RL1 additional first-level storage entities. On each of the first-level storage entities, the locally-stored first-level container is divided into SD2 local second-level pieces. Each second-level piece is stored in a second-level container on a second-level storage entity of the specific first-level storage entity. Each first-level storage entities contains SD2 plus RL2 second-level storage entities. A redundant encoding of the second-level containers is stored in RL2 additional second-level containers on RL2 additional second-level storage entities.

Abstract: Example storage systems and methods provide data storage management using generation markers in a key data store. A key data store includes a set of key data entries that each include a key value and a property value associated with a storage operation. An active generation of the key data entries include an active generation marker and a base generation does not. A base storage parameter is calculated from a data scan and a current storage parameter is calculated from the base storage parameter and the property values of the active generation of key data entries. The calculated storage parameter may be reported to manage storage configuration and operations in the storage system.

Abstract: Example storage systems and methods provide data storage management using change values in a key data store. A key data store includes a set of key data entries that each include a key value associated with a storage operation, a precondition value, and a postcondition value. A storage parameter is calculated using the precondition value and the postcondition value for each key data entry the set of key data entries. The calculated storage parameter may be reported to manage storage configuration and operations in the storage system.

Abstract: The present disclosure relates to systems and methods for dynamic sharding of a database comprising data identifiable by keys comprised within a global ordered range. When handling a request for data of at least one key: providing the request to a predetermined shard store; the predetermined shard store verifying, by means of its local subrange collection, whether the at least one key is present in a local subrange of a shard stored on the predetermined shard store; and when at least one key is not present in a local subrange of a shard stored on the predetermined shard store, returning a message comprising the local subrange collection of the predetermined shard store.

Abstract: An ordered data object identifier denoted by a value is assigned to each data object grouped to a container object, wherein data object identifiers with successively incremented values are assigned to successive data objects. An ordered container identifier is assigned to each container object with the greatest value of the identifiers of the data objects grouped thereto. A metadata structure with an entry for each container objects is stored. Each entry comprises the ordered container identifier and a reference to the corresponding data. The metadata structure is ordered according to the values of the container identifiers. A request to read a data object contains a corresponding data object identifier. It is determined to which container object the data object is grouped, by locating the first container identifier in the metadata structure with a value greater than or equal to that the requested data object.

Abstract: A set of proxy servers respectively located on a set of servers communicatively coupled via a first network switch and a second network switch are configured to listen on a predefined proxy IP address, and forward communication requests designated for the predefined proxy IP address to a connection pool of IP addresses for a first server. A proxy server hosted on the first server is configured to listen on IP addresses of the first server, and forward communication requests received on the first IP address and the second IP address to the predefined proxy IP address. The IP addresses of the first server including a first IP address associated with the first server's interface with the first network switch and a second IP address associated with the first server's interface with the second network switch.

Abstract: An object is divided into SD1 first-level pieces. Each first-level piece is stored in a first-level container on a first-level storage entity. A redundant encoding of the first-level containers is stored in RL1 additional first-level containers on RL1 additional first-level storage entities. On each of the first-level storage entities, the locally-stored first-level container is divided into SD2 local second-level pieces. Each second-level piece is stored in a second-level container on a second-level storage entity of the specific first-level storage entity. Each first-level storage entities contains SD2 plus RL2 second-level storage entities. A redundant encoding of the second-level containers is stored in RL2 additional second-level containers on RL2 additional second-level storage entities.

Abstract: A single-geo system includes a network configuration generator and a first computing system rack, or a multi-geo system includes a network configuration generator, a first computing system rack, and a second computing system rack are described. The system receives network information for a first plurality of nodes and a second plurality of nodes, and generates a system-wide network configuration file including network configuration information for the first plurality of nodes and the second plurality of nodes. Each plurality of nodes includes a controller node to receive the system-wide network configuration file, identify network configuration information for this plurality of nodes in the system-wide network configuration file as being part of the computing system rack, and update network configuration for this plurality of nodes based on the identified network configuration information for the computing system rack.

Abstract: A computer-implemented method for redistributing data between memory clusters in a key value store. The data is redistributed according to a load balancing algorithm. Memory clusters are sorted into lists according to the number of intervals each cluster contains. Migration jobs are created by the load balancing algorithm to push data fragments from a larger cluster to a smaller cluster. Data fragments, or intervals, are selected for migration according to corresponding data fragments on a target cluster or the number of occupying data fragments on a target cluster. The redistribution of data helps avoid fragmentation of key ranges of intervals, decrease the overall number of non-adjacent key ranges, and to conform to the routing table requirements.

Abstract: An ordered data object identifier denoted by a value is assigned to each data object grouped to a container object, wherein data object identifiers with successively incremented values are assigned to successive data objects. An ordered container identifier is assigned to each container object with the greatest value of the identifiers of the data objects grouped thereto. A metadata structure with an entry for each container objects is stored. Each entry comprises the ordered container identifier and a reference to the corresponding data. The metadata structure is ordered according to the values of the container identifiers. A request to read a data object contains a corresponding data object identifier. It is determined to which container object the data object is grouped, by locating the first container identifier in the metadata structure with a value greater than or equal to that the requested data object.