Abstract: Methods and apparatus to dynamically assign and relocate object fragments in distributed storage systems are disclosed.

Abstract: Methods and apparatus to dynamically assign and relocate object fragments in distributed storage systems are disclosed.

Abstract: Methods and apparatus to dynamically assign and relocate object fragments in distributed storage systems are disclosed.

Abstract: A technique is applied to eventually converge on a single data storage strategy for any set of object data which had an inconsistent data storage strategy applied during storage while there was a network partition. This state could occur in instances of a highly available distributed object storage system which can store objects according to multiple data storage strategies. Upon the healing of a network partition, the technique discovers if multiple data storage strategies were applied to the object data stored during a network partition, deterministically identifies which data storage strategy represents the correct strategy, for example based on the log of state transitions requested by the client according to the API contract, and ensures that this strategy is consistently applied to all object data in the collection.

Abstract: Methods and apparatus to dynamically assign and relocate object fragments in distributed storage systems are disclosed.

Abstract: Methods and apparatus to dynamically assign and relocate object fragments in distributed storage systems are disclosed. In some examples, the methods and apparatus encode an object with error correction coding to separate the object into fragments, create a first index indicative of storage nodes where the fragments of the object are to be stored, encode a second index into identifiers of the fragments of the object, the second index based on the first index, and store the fragments of the object and the corresponding second index encoded identifiers in the storage nodes based on the first index.

Abstract: An improved scalable object storage system allows multiple clusters to work together. In one embodiment, a trust and federation relationship is established between a first cluster and a second cluster. This is done by designating a first cluster as a trust root. The trust root receives contact from another cluster, and the two clusters exchange cryptographic credentials. The two clusters mutually authenticate each other based upon the credentials, and optionally relative to a third information service, and establish a service connection. Services from the remote cluster are registered as being available to the cluster designated as the trust root. Multi-cluster gateways can also be designated as the trust root, and joined clusters can be mutually untrusting. Two one-way trust and federation relationships can be set up to form a trusted bidirectional channel.

Abstract: A technique is applied to eventually converge on a single data storage strategy for any set of object data which had an inconsistent data storage strategy applied during storage while there was a network partition. This state could occur in instances of a highly available distributed object storage system which can store objects according to multiple data storage strategies. Upon the healing of a network partition, the technique discovers if multiple data storage strategies were applied to the object data stored during a network partition, deterministically identifies which data storage strategy represents the correct strategy, for example based on the log of state transitions requested by the client according to the API contract, and ensures that this strategy is consistently applied to all object data in the collection.

Abstract: A technique is applied to eventually converge on a single data storage strategy for any set of object data which had an inconsistent data storage strategy applied during storage while there was a network partition. This state could occur in instances of a highly available distributed object storage system which can store objects according to multiple data storage strategies. Upon the healing of a network partition, the technique discovers if multiple data storage strategies were applied to the object data stored during a network partition, deterministically identifies which data storage strategy represents the correct strategy, for example based on the log of state transitions requested by the client according to the API contract, and ensures that this strategy is consistently applied to all object data in the collection.

Abstract: An improved scalable object storage system allows multiple clusters to work together. In one embodiment, a trust and federation relationship is established between a first cluster and a second cluster. This is done by designating a first cluster as a trust root. The trust root receives contact from another cluster. and the two clusters exchange cryptographic credentials. The two clusters mutually authenticate each other based upon the credentials, and optionally relative to a third information service, and establish a service connection. Services from the remote cluster are registered as being available to the cluster designated as the trust root. Multi-cluster gateways can also be designated as the trust root, and joined clusters can be mutually untrusting. Two one-way trust and federation relationships can be set up to form a trusted bidirectional channel.

Abstract: Methods and apparatus to dynamically assign and relocate object fragments in distributed storage systems are disclosed. In some examples, the methods and apparatus encode an object with error correction coding to separate the object into fragments, create a first index indicative of storage nodes where the fragments of the object are to be stored, encode a second index into identifiers of the fragments of the object, the second index based on the first index, and store the fragments of the object and the corresponding second index encoded identifiers in the storage nodes based on the first index.

Abstract: Techniques are described for achieving durability of a data object stored in a network storage system. In some embodiments, erasure coding is applied to break a data object into fragments wherein the original data object can be recovered with fewer than all of the fragments. These fragments are stored on multiple storage nodes in a distributed storage cluster of a network storage system. So that individual storage nodes have knowledge of the state of the stored data object, a proxy server acing as a central agent can wait for acknowledgments indicating that the fragments have been successfully stored at the storage nodes. If the proxy server receives successful write responses from a sufficient number of the storage nodes, the proxy server can report that the data object is durably stored by placing markers on the storage nodes.

Abstract: An improved scalable object storage system includes methods and systems allowing multiple clusters to work together. In one embodiment, there is a multi-cluster synchronization system between two or more clusters. The multi-cluster synchronization system uses variable compression to optimize the transfer of information between the clusters. Compression is used not only to minimize the total number of bytes sent between the two clusters, but to dynamically vary the size of the objects sent across the wire to optimize for higher throughput after considering packet loss, TCP windows, and block sizes. This includes both the packaging of multiple small files together into one larger compressed file, saving on TCP and header overhead, but also the chunking of large files into multiple smaller files that are less likely to have difficulties due to intermittent network congestion or errors. Depending on the state of the network and disks, the best size can vary.

Abstract: An improved scalable object storage system allows multiple clusters to work together. In one embodiment, a trust and federation relationship is established between a first cluster and a second cluster. This is done by designating a first cluster as a trust root. The trust root receives contact from another cluster, and the two clusters exchange cryptographic credentials. The two clusters mutually authenticate each other based upon the credentials, and optionally relative to a third information service, and establish a service connection. Services from the remote cluster are registered as being available to the cluster designated as the trust root. Multi-cluster gateways can also be designated as the trust root, and joined clusters can be mutually untrusting. Two one-way trust and federation relationships can be set up to form a trusted bidirectional channel.

Abstract: In one embodiment, there is a multi-cluster synchronization system between two or more clusters. The multi-cluster synchronization system uses variable compression to optimize the transfer of information between the clusters. Compression is used not only to minimize the total number of bytes sent between the two clusters, but to dynamically vary the size of the objects sent across the wire to optimize for higher throughput after considering packet loss, TCP windows, and block sizes. This includes both the packaging of multiple small files together into one larger compressed file, saving on TCP and header overhead, but also the chunking of large files into multiple smaller files that are less likely to have difficulties due to intermittent network congestion or errors. A further embodiment uses forward error correction to maximize the chances that the remote end will be able to correctly reconstitute the transmission.

Abstract: A technique is applied to eventually converge on a single data storage strategy for any set of object data which had an inconsistent data storage strategy applied during storage while there was a network partition. This state could occur in instances of a highly available distributed object storage system which can store objects according to multiple data storage strategies. Upon the healing of a network partition, the technique discovers if multiple data storage strategies were applied to the object data stored during a network partition, deterministically identifies which data storage strategy represents the correct strategy, for example based on the log of state transitions requested by the client according to the API contract, and ensures that this strategy is consistently applied to all object data in the collection.

Abstract: Methods and systems to configure an object-based storage cluster with multiple selectable data handling policies, including to map an object to a storage device/node of the cluster based on the policy associated with the object. In an embodiment, each policy is associated with a respective one of multiple rings, partitions of which are mapped to storage devices of the same cluster, objects are associated with buckets/containers, and each bucket/container is associated within a user-selectable one of the policies, such as with a metadata-based policy index, and an object is mapped to a storage device/node of the cluster based on the ring associated with the policy index of the object's container.

Abstract: An improved scalable object storage system allows multiple clusters to work together. Users working with a first cluster, or with a multi-cluster gateway, can ask for services and have the request or data transparently proxied to a second cluster. This gives transparent cross-cluster replication, as well as multi-cluster compute or storage farms based upon spot availability or various provisioning policies. Vendors providing a cloud storage “frontend” can provide multiple backends simultaneously. In one embodiment, a multi-cluster gateway can have a two, three, or higher-level ring that transparently matches an incoming request with the correct cluster. In the ring, a request is first mapped to an abstract “partition” based on a consistent hash function, and then one or more constrained mappings map the partition number to an actual resource. In another embodiment, the multi-cluster gateway is a dumb gateway, and the rings are located only at the cluster level.

Abstract: An improved scalable object storage system includes methods and systems allowing multiple clusters to work together. In one embodiment, there is a multi-cluster synchronization system between two or more clusters. Each cluster has a cluster-internal network, with object storage services and container services. The container services track and replicate metadata associated with the object storage service. An intercluster network connects the two clusters and performs a one-way synchronization of the objects and metadata associated with a particular container. This can be done either through the direct association of the container and object storage services, such as through a trust and federation relationship, or it can be opaque, so that the cross-cluster replication treats the remote repository as a black box and uses the external API to call and manipulate the files.

Abstract: An improved scalable object storage system includes methods and systems allowing multiple clusters to work together. Users working with a first cluster, or with a multi-cluster gateway, can ask for services and have the request or data transparently proxied to a second cluster. This gives transparent cross-cluster replication, as well as multi-cluster compute or storage farms based upon spot availability or various provisioning policies. Vendors providing a cloud storage “frontend” can provide multiple backends simultaneously. In one embodiment, a multi-cluster gateway can have a two, three, or higher-level ring that transparently matches an incoming request with the correct cluster. In the ring, a request is first mapped to an abstract “partition” based on a consistent hash function, and then one or more constrained mappings map the partition number to an actual resource. In another embodiment, the multi-cluster gateway is a dumb gateway, and the rings are located only at the cluster level.