Abstract: A method may include selecting a cohort of entities for migration from a source storage repository to a target storage repository, obtaining a mapping between a source storage schema of the source storage repository to a target storage schema of the target storage repository, and migrating data for the entities in the cohort. Migrating the data of an entity may include copying, without locking the data in the source storage repository and in the target storage repository, the data from the source storage repository to the target storage repository, verifying, while the data is locked, that the data in the source storage repository is the same as the data in the target storage repository, changing, while the data in the source storage repository and the target storage repository is locked, an entity pointer for the entity to the target storage repository based on the verifying, and unlocking the data.

Abstract: In one embodiment, a device performs a plurality of iterations of write operations on portions of a file tree of a file system to a first storage cluster, each write operation flagging the written portion of the file tree with a generation identifier indicative of the associated iteration of the write operations. The device identifies a set of the written portions of the file tree as unsynchronized between the first storage cluster and a second storage cluster by performing a search for changes made to the written portions of the file tree between a generation identifier associated with the first storage cluster and a generation identifier associated with the second storage cluster. The device synchronizes the first and second storage clusters by sending, via a network, the unsynchronized portions of the file tree from the first storage cluster to the second storage cluster.

Abstract: In one embodiment, a device performs a plurality of iterations of write operations on portions of a file tree of a file system to a first storage cluster, each write operation flagging the written portion of the file tree with a generation identifier indicative of the associated iteration of the write operations. The device identifies a set of the written portions of the file tree as unsynchronized between the first storage cluster and a second storage cluster by performing a search for changes made to the written portions of the file tree between a generation identifier associated with the first storage cluster and a generation identifier associated with the second storage cluster. The device synchronizes the first and second storage clusters by sending, via a network, the unsynchronized portions of the file tree from the first storage cluster to the second storage cluster.

Abstract: Creating and using highly available transaction logs in a distributed storage system is described, where the storage controller functions of the distributed storage system are separated from that of distributed storage system storage media. In an exemplary embodiment, a storage controller server receives a transaction entry at a first storage pool of the distributed storage system. The storage controller server further looks up a transaction log to store the transaction entry, where the transaction log is associated with a second storage pool. The storage controller server routes the transaction entry to the second storage pool, wherein the second storage pool stores the transaction entry.

Abstract: A distributed storage system called StorFS that performs distributed data deduplication is described. In an exemplary embodiment, a storage controller server determines if there is duplicative data in a distributed storage system. In this embodiment, the storage controller server determines if an entity is duplicated in the distributed storage system in line with an incoming input/output operation. The storage controller server determines if the entity is duplicated in the distributed storage system by receiving the entity and looking up the entity in a data deduplication table. If the entity exists in the data deduplication table, the storage controller server updates the metadata for the entity to point to the duplicate entity.

Abstract: A distributed storage system that dispatches an input/output request is described. In an exemplary embodiment, a storage controller client receives the input/output request, wherein the distributed storage system includes the storage controller client, a plurality of storage controller servers, a plurality of virtual nodes distributed among a plurality of physical nodes, and each of the plurality of physical nodes is hosted on one of the plurality of storage controller servers. The storage controller client further computes a target virtual node for the input/output request, where the target virtual node is one of the plurality of virtual nodes. Using the computed target virtual node, the storage controller client determines a target physical node that corresponds to the target virtual node, where the target physical node is one of the plurality of physical nodes.

Abstract: A distributed snapshot in a distributed storage system is described, where the storage controller functions of the distributed storage system are separated from that of distributed storage system storage media. In an exemplary embodiment, a storage controller server receives a request to create the snapshot in the distributed storage system, where the distributed storage system includes a plurality of virtual nodes and a plurality of physical nodes, and the source object includes a plurality of stripes. The storage controller server further determines a set of virtual nodes from the plurality of virtual nodes, where each of the set of virtual nodes owns one of the plurality of stripes of the source object. For each of the set of virtual nodes, the storage controller server sends a clone request to that virtual node, where the request is to create a snapshot for the stripe hosted by that virtual node.

Abstract: A distributed multi-layer cache in a distributed storage system is described, where the storage controller functions of the distributed storage system are separated from that of distributed storage system storage media. In an exemplary embodiment, a storage controller server determines if an object is in a cache that stores a plurality of objects. In addition, the distributed storage system includes the cache and a distributed object layer for persistently storing the plurality of objects. The cache further includes a de-duplicated cache layer. The storage controller server accesses the object from the cache if the object is in the cache and accesses the object from the distributed object layer if the object is not in the cache.

Abstract: A distributed storage system that dispatches an input/output request is described. In an exemplary embodiment, a storage controller client receives the input/output request, wherein the distributed storage system includes the storage controller client, a plurality of storage controller servers, a plurality of virtual nodes distributed among a plurality of physical nodes, and each of the plurality of physical nodes is hosted on one of the plurality of storage controller servers. The storage controller client further computes a target virtual node for the input/output request, where the target virtual node is one of the plurality of virtual nodes. Using the computed target virtual node, the storage controller client determines a target physical node that corresponds to the target virtual node, where the target physical node is one of the plurality of physical nodes.

Abstract: A distributed garbage collection in a distributed storage system is described, where the storage controller functions of the distributed storage system are separated from that of distributed storage system storage media. In an exemplary embodiment, a storage controller server generates a live object map of live objects stored on the distributed storage system in a plurality of block segments distributed across a plurality of storage controller servers. The storage controller server further scans the plurality of block segments to generate segment summary statistics, where the segment summary statistics indicates the number of live objects stored in the plurality of block segments. In addition, the storage controller server compacts each of the plurality of block segments that have a low utilization based on the segment summary statistics. Furthermore, the live object map is a probabilistic data structure storing a list of valid objects.

Abstract: A distributed storage system that performs automated load balancing is described. In an exemplary embodiment, a storage controller server determines if there is duplicative data in a distributed storage system. In this embodiment, the storage controller server detects a load balancing event in the distributed storage system, where the distributed storage system includes a plurality of virtual nodes distributed across a plurality of physical nodes. In response to detecting the load balancing event, the storage controller server determines that a current virtual node is to move from a source physical node to a destination physical node. In addition, the current virtual node is one of the plurality of virtual nodes and the source and destination physical nodes are in the plurality of physical nodes. The storage controller server further moves the current virtual node from the source physical node to the destination physical node.