Abstract: The present disclosure provides an advantageous systems and related methods for scalable object storage. In accordance with an embodiment of the invention, a stream session is opened to initiate a retention epoch during which a series of edits to an object by a source may be performed without seeing edits from other stream sessions. Backreferences identifying the stream session (i.e. stream session backreferences) are applied to retained chunks. While a payload chunk has one or more such associated stream session backreferences, the chunk is retained and not expunged. Advantageously, transitory chunks that are put during the retention epoch may be unput promptly so as to release valuable cache resources. In comparison, the prior approach only eventually releases the data storage resources for expunged version manifests. Other embodiments, aspects and features are also disclosed.

Abstract: The present disclosure provides an advantageous systems and related methods for scalable object storage. In accordance with an embodiment of the invention, a stream session is opened to initiate a retention epoch during which a series of edits to an object by a source may be performed without seeing edits from other stream sessions. Backreferences identifying the stream session (i.e. stream session backreferences) are applied to retained chunks. While a payload chunk has one or more such associated stream session backreferences, the chunk is retained and not expunged. Advantageously, transitory chunks that are put during the retention epoch may be unput promptly so as to release valuable cache resources. In comparison, the prior approach only eventually releases the data storage resources for expunged version manifests. Other embodiments, aspects and features are also disclosed.

Abstract: The present disclosure provides methods and systems for multicast collaborative erasure encoding and methods and systems for distributed parity protection. One embodiment relates to a method of multicast collaborative erasure encoding of a chunk stored in a distributed object storage cluster. A roll-call request is multicast to every storage server in a negotiating group for the chunk. Roll-call inventory responses are generated and multicast by every storage server in the negotiating group. The roll-call inventory responses are collected by every storage server in the negotiating group from other storage servers in the negotiating group to form a set of roll-call inventory responses. A logical evaluation of the set of roll-call inventory responses may then be performed by every storage server in the negotiating group. Other embodiments, aspects and features are also disclosed.

Abstract: The present invention introduces a specific form of parity protection chunk (PPC) that allows for distributed creation of coordinated PPCs that can reliably provide protection against the concurrent loss of two or more storage servers of devices. Coordinated PPCs can protect against the concurrent loss of multiple lost chunks by ensuring that the PPCSs protecting any specific chunk have at most a single overlapping failure domain. This is done without requiring full centralized control over the assignment of protected chunks to specific PPCs. The PPCs are created as part of the put transaction for a chunk. The chunk itself is stored as a whole replica, and the PPCs provide parity protection for the chunk.

Abstract: The present disclosure provides techniques for efficiently updating and searching sharded key-value record stores in an object storage cluster. The disclosed techniques use shard groups, instead of using negotiating groups and rendezvous groups as in a previously-disclosed multicast replication technique. The use of shard groups results in fewer messages being required to complete an update or a search than would have been required using the previously-disclosed technique. The use of shard groups is particularly beneficial when applied to system maintained objects, such as a namespace manifest.

Abstract: The present disclosure provides systems and methods for Key-Value-Tuple-encoded (KVT-encoded) object storage. In accordance with an embodiment of the invention, chunks of objects, storing payload data or metadata, are persistently stored by servers using KVT-encoded storage. Furthermore, the KVT encoding may be applied to advantageously re-structure content within a distributed object storage cluster, especially for object storage systems that allow payload references to be cached extensively. For such systems, it is of considerable value to honor (use as valid) existing chunk references after the underlying content has been re-structured. Further, the KVT encoding taught herein is completely compatible with a fully-distributed object storage cluster. One embodiment relates to a storage server that includes a persistent storage module that provides a key-value application programming interface that encodes multiple key-value-tuple (KVT) entries for one chunk.

Abstract: The present disclosure provides an innovative technique for parallel transparent restructuring of immutable content in a distributed object storage system. The content restructuring technique disclosed herein may be implemented with parallel operations by multiple storage servers in a live system. The restructuring is transparent in that the original CHIT may still be used to retrieve and validate the original content, such that a client requesting the original content need not be aware that the original content has been restructured. Furthermore, the restructuring is performed while maintaining support for pre-existing immutable metadata. Other embodiments, aspects and features are also disclosed. The content remains immutable from the perspective of the end user, but still may be advantageously restructured to optimize storage efficiency.

Abstract: The present disclosure provides a method performed by an object storage cluster with distributed metadata. The distributed metadata is defined and stored in a form so as to be guaranteed to be commutative. For eventual edits to the distributed metadata, the system accumulates the edits for subsequent batch processing at relevant storage servers. For transactional edits to the distributed metadata, the system has the relevant storage servers perform a targeted search for older eventual edits to the distributed metadata for the same target object in the accumulation of eventual edits at the relevant storage servers. Before performing the transactional edit, any older eventual edits found by the targeted search are performed by the relevant storage servers.

Abstract: The present invention relates to a method and system for providing parity protection in an object storage system. The present invention allows for tracking the storage requirements for chunks in a distributed storage cluster when transitioning from replica-based protection to parity or erasure coding-based protection and when transitioning from parity or erasure coding-based protection to replica-based protection.

Abstract: One difficult problem is that a straightforward renaming of a directory containing a very large number of subdirectories and files requires locking and then updating, in parallel, the names of vast numbers of directory records. Another problem is that renaming a directory with a very large number of subdirectories and files may cause a mass migration of the metadata for the renamed objects due to their object names being changed. The presently-disclosed solution involves at least a) introducing a “folder” object and b) extending the distributed searchable set of records in the namespace manifest with a “folder” index record. In an exemplary implementation, each instance of a folder object created is described by an instance of a folder index record that is recorded in a namespace manifest. Different embodiments of the solution may be particularly suited to different use cases.

Abstract: The present disclosure provides methods and systems for multicast collaborative erasure encoding and methods and systems for distributed parity protection. One embodiment relates to a method of multicast collaborative erasure encoding of a chunk stored in a distributed object storage cluster. A roll-call request is multicast to every storage server in a negotiating group for the chunk. Roll-call inventory responses are generated and multicast by every storage server in the negotiating group. The roll-call inventory responses are collected by every storage server in the negotiating group from other storage servers in the negotiating group to form a set of roll-call inventory responses. A logical evaluation of the set of roll-call inventory responses may then be performed by every storage server in the negotiating group. Other embodiments, aspects and features are also disclosed.

Abstract: The present disclosure provides a system and method to perform access control authentication using a cryptographic hash of the encoding of access control rules. The compact cryptographic hash identifier of the access control rules is suitable for inclusion in a name indexing entry, whereas inclusion of the full encoding would result in a large name indexing entry, resulting in disadvantageously large storage requirements and bandwidth usage. In accordance with an embodiment of the invention, a common manifest is used to store and access versions of a named object stored in a distributed data storage system. The common manifest for the named object may encode a set of key-value metadata pairs which have been inherited from a parent object, such as an enclosing folder, for example. Other embodiments, aspects and features are also disclosed.

Abstract: The present disclosure provides methods and systems for multicast collaborative erasure encoding and methods and systems for distributed parity protection. One embodiment relates to a method of multicast collaborative erasure encoding of a chunk stored in a distributed object storage cluster. A roll-call request is multicast to every storage server in a negotiating group for the chunk. Roll-call inventory responses are generated and multicast by every storage server in the negotiating group. The roll-call inventory responses are collected by every storage server in the negotiating group from other storage servers in the negotiating group to form a set of roll-call inventory responses. A logical evaluation of the set of roll-call inventory responses may then be performed by every storage server in the negotiating group. Other embodiments, aspects and features are also disclosed.

Abstract: The present invention introduces a specific form of parity protection chunk (PPC) that allows for distributed creation of coordinated PPCs that can reliably provide protection against the concurrent loss of two or more storage servers of devices. Coordinated PPCs can protect against the concurrent loss of multiple lost chunks by ensuring that the PPCSs protecting any specific chunk have at most a single overlapping failure domain. This is done without requiring full centralized control over the assignment of protected chunks to specific PPCs. The PPCs are created as part of the put transaction for a chunk. The chunk itself is stored as a whole replica, and the PPCs provide parity protection for the chunk.

Abstract: The present invention relates to a method and system for providing parity protection in an object storage system. The present invention allows for tracking the storage requirements for chunks in a distributed storage cluster when transitioning from replica-based protection to parity or erasure coding-based protection and when transitioning from parity or erasure coding-based protection to replica-based protection.

Abstract: The present disclosure also provides systems and methods for sharding objects stored in a distributed storage system. In accordance with one embodiment disclosed herein, a key sharding technique is used. Key sharding is an advantageously efficient technique when dealing with an object containing a collection of key-value records. In accordance with an embodiment of the invention, referenced chunks identified by the key shards may each store a subset of the collection of the key-value records, and the key-value records in the subset have key hashes that have a range of matching bits in common. One embodiment disclosed herein provides a method of performing a delta edit of a named object stored in a distributed storage system in which a payload of the named object is stored in key shards. Other embodiments, aspects and features are also disclosed.

Abstract: The present disclosure provides systems and methods for sharding objects stored in a distributed storage system. Such sharding may be advantageously utilized for an object that stores a collection of key-value records and for otherwise encoded objects. The object sharding techniques disclosed herein advantageously enable a read-modify-write process at the storage server level. One embodiment disclosed herein provides a method of creating a new chunk by modifying a payload of an existing chunk at the storage server level in a distributed object storage system. The method includes validating the new chunk at the gateway server. Other embodiments, aspects and features are also disclosed.

Abstract: The present invention relates to object storage systems that support hierarchical directories within a namespace manifest stored as an object. The namespace manifest is stored as namespace manifest shards that are determined based on a partial key derived from the full name of the referenced version manifest in each directory entry. Each storage server maintains a local transaction log that keeps track of changes to the namespace manifest in response to put transactions. The namespace manifest can be updated after each put transaction is acknowledged, which minimizes the number of write operations required for a put transaction. Updates to the namespace manifest are batched, distributed, and processed concurrently and in parallel by the storage servers that store the corresponding shards. This reduces the latency that would otherwise exist in the object storage cluster as a result of a put transaction. Updates to namespace manifest shards optionally occur in a batch process using MapReduce techniques.

Abstract: The present invention defines a new type of negotiating proxy server which optimizes a multi-node collaboration of distributed servers, where the goal of the negotiation is to synchronize rendezvous interactions between the initiator and subsets of servers in multicast groups. For example, in distributed storage clusters servers commit to receiving and then storing a chunk of data in parallel at a specified time, while in compute clusters servers perform one portion of a calculation in parallel. Negotiating proxy server optimizes these interactions as disclosed herein.

Abstract: The present invention relates to a distributed object storage system that supports snapshots and clones without requiring any form of distributed locking—or any form of centralized processing. A clone tree can be modified in isolation and the modifications then either discarded or merged into the main tree of the distributed object storage system.