Abstract: Systems and methods for determining garbage collection (GC) scope in a distribute storage system using chunk-based storage. The systems and methods are compatible with multi-version concurrency control (MVCC) semantics.

Abstract: Method for tree management of trees under multi-version concurrency control is described herein. Method starts by detecting change in a size of a cluster. The number of nodes in the cluster is counted to obtain the changed size of the cluster. The number of trees needed for the number of nodes in the cluster is determined. The number of trees may be based on the number of nodes in the cluster and predetermined system balancing coefficient. When the number of trees needed is greater than existing number of trees, existing number of trees is doubled as a single-step operation which includes modifying and scaling hash function used to derive each tree to generate new hash function and using new hash function to partition each tree instantly into two new trees. Scaling happens on demand without service disruption. Hash function scales automatically when number of trees increases. Other embodiments are described.

Abstract: One embodiment is related to a method for creating a redundancy data chunk for data protection with a chain topology, comprising: transmitting a data chunk of a first frontend zone of a data storage system to a second frontend zone of the data storage system; creating a redundancy data chunk at the second frontend zone of the data storage system based on the data chunk of the first frontend zone and a data chunk of the second frontend zone; passing the redundancy data chunk onto one or more subsequent frontend zones of the data storage system from the second frontend zone, wherein at each subsequent frontend zone the redundancy data chunk is updated based on the received redundancy data chunk and a data chunk of the respective subsequent frontend zone, and wherein the redundancy data chunk is passed through each subsequent frontend zone exactly once; and at a last subsequent frontend zone of the data storage system, forwarding the updated redundancy data chunk to a backend zone of the data storage system for

Abstract: Data protection in a distributed storage system is provided using a combination of a data protection operation and an exclusive or XOR operation. Chunks of data subject to replication are encoded with a protection operation and an XOR operation commutative with the protection operation to generate a combined protected data chunk. The protected data chunks from which the combined protected data chunk was generated can be safely deleted to reduce data protection overhead. Portions of the protected data that later become unavailable due to failure in the distributed storage system can be recovered from other portions that are available and the combined protect data chunk using the XOR operation. The protection operation includes a matrix-based erasure coding operation commutative with the XOR operation.

Abstract: A transformation service is provided for a plurality of transformation jobs associated with one or more storage containers. A single transformation user runs enumeration tasks associated with data stored in the one or more storage containers. When two or more of the plurality of transformation jobs are active during a time interval, the single transformation user runs an enumeration task for all of the active transformation jobs during said time interval.

Abstract: A computer program product, system, and method for determining containers, within a secondary storage, to transform; determining users having access to the containers to transform; synchronizing storage configuration between the secondary storage and a primary storage; enumerating objects within the containers to transform; and adding object identifiers and object metadata for the enumerated objects to an object index within the primary storage.

Abstract: A computer program product, system, and method for determining a list of objects, within source storage, to migrate; generating a chunk layout for the objects to migrate; and for each unencoded chunk within the chunk layout: retrieving objects from source storage specified by the unencoded chunk within the chunk layout; generating data and coded fragments for the unencoded chunk using the retrieved objects; and storing the data and coded fragments to primary storage.

Abstract: A computer-executable method, computer program product, and system of managing I/O requests in a distributed data storage system, wherein the distributed data storage system includes a first node and one or more data storage array, the computer-executable method, computer program product, and system comprising receiving an I/O request at the first node enabled to utilize a storage engine to process the I/O request, wherein the storage engine is comprised of two or more layers, analyzing the I/O request using a first layer of the two or more layers to determine whether the I/O request relates to a portion of metadata managed by the first layer of the two or more layers, and processing the I/O request based on the determination.

Abstract: A method comprising: processing an update to a search tree and updating statistics, the search tree storing information about one or more objects indexed by corresponding object keys; determining to rebuild a first Bloom filter based on the statistics, the first Bloom filter associated with the search tree; generating a second Bloom filter associated with the search tree; populating the second Bloom filter as part of a tracing garbage collection process; and replacing the first Bloom filter with the second Bloom filter.

Abstract: A method, computer program product, and computer system for receiving a change to a configuration parameter associated with a first cluster node in a cluster configuration. The configuration parameter may be stored in a shared cluster memory. The configuration parameter may be distributed from the shared cluster memory to the cluster configuration.

Abstract: A system, computer program product, and computer-executable method for use with a distributed storage system comprising a plurality of storage nodes each having attached storage devices, the system, computer program product, and computer-executable method including receiving a request, at a first storage node of the plurality of storage nodes, to store a large portion of data, using at least one of a first type of data chunk and a plurality of a second type of data chunks to store the large portion of data, processing each of the plurality of the second type of data chunks, processing each of the at least one of the first type of data chunk, and returning an acknowledgement to the request.

Abstract: A method, computer program product, and computer system for identifying, by a computing device, a plurality of used chunks, wherein the plurality of used chunks may include a plurality of free blocks and at least one non-free block. A capacity of the plurality of free blocks that is available to reclaim may be verified. The capacity of the plurality of free blocks may be reclaimed. A composite chunk from the plurality of free blocks may be created.

Abstract: A technique to accelerate Galois Field (GF) arithmetic. The technique, which does not rely on any specific processor instruction set, can be used to accelerate erasure coding within storage systems.

Abstract: Systems and methods for efficiently protecting data within a distributed storage system using erasure coding. Unnecessary network traffic can be eliminated by scheduling executing erasure coding tasks on storage nodes that have local copies of data. Encoding may be performed in parallel by multiple nodes to reduce elapsed encoding time.

Abstract: Systems and methods for determining garbage collection (GC) scope in a distribute storage system using chunk-based storage. The systems and methods are compatible with multi-version concurrency control (MVCC) semantics.

Abstract: Structures and processes for garbage collection of search trees under Multi-Version Concurrency Control (MVCC). Such search trees may be used to store data within a distributed storage system. A process detects live search tree elements using tracing and then identify storage chunks having no live elements as garbage to be reclaimed. The process can be paused and resumed to reduce impact on other system processing. To reduce disk fragmentation, a garbage collector may copy pages between chunks prior to reclaiming chunk capacity. Also described is a resource efficient scheduler for a garbage collection.

Abstract: Systems and processes to prevent storage capacity exhaustion within distributed storage systems using chunk-based storage. The systems and processes control capacity by classifying various types of storage chunks into groups and impose used capacity thresholds on chunk groups.

Abstract: Structures and processes for garbage collection of search trees under Multi-Version Concurrency Control (MVCC). Such search trees may be used to store data within a distributed storage system. A process detects live search tree elements using tracing and then identify storage chunks having no live elements as garbage to be reclaimed. The process can be paused and resumed to reduce impact on other system processing.

Abstract: A data object content verification technique provides perfect reliability and low storage overhead. Object data is generated in a reproducible manner based upon object locally stored object metadata. The object data is stored to an object storage system. The stored object data is subsequently verified by retrieving the object metadata, regenerating the original object data, and comparing the stored and original object data.