Abstract: Systems, methods, and computer-readable media are provided for utilizing distributed erasure encoding in a redundant array of independent disks (RAID) system. An example method can include generating a plurality of virtual redundant array of independent disk (vRAID) stripes, each of the plurality of vRAID stripes including a segment having a plurality of data, each of the plurality of data including metadata, the metadata including a checksum of a corresponding data of the plurality of data, distributing the segment of each of the plurality of vRAID stripes over a plurality of virtual nodes, mapping at least one of logical files, volumes, or objects to the plurality of data chunks and the at least one parity chunk of the plurality of vRAID stripes to avoid write-hole issues, and verifying data integrity of the corresponding data of the plurality of data using the checksum of the corresponding data.

Abstract: Systems, methods, and computer-readable media are provided for utilizing distributed erasure encoding in a redundant array of independent disks (RAID) system. An example method can include generating a plurality of virtual redundant array of independent disk (vRAID) stripes, each of the plurality of vRAID stripes including a segment having a plurality of data, each of the plurality of data including metadata, the metadata including a checksum of a corresponding data of the plurality of data, distributing the segment of each of the plurality of vRAID stripes over a plurality of virtual nodes, mapping at least one of logical files, volumes, or objects to the plurality of data chunks and the at least one parity chunk of the plurality of vRAID stripes to avoid write-hole issues, and verifying data integrity of the corresponding data of the plurality of data using the checksum of the corresponding data.

Abstract: In one embodiment, a device performs a plurality of iterations of write operations on portions of a file tree of a file system to a first storage cluster, each write operation flagging the written portion of the file tree with a generation identifier indicative of the associated iteration of the write operations. The device identifies a set of the written portions of the file tree as unsynchronized between the first storage cluster and a second storage cluster by performing a search for changes made to the written portions of the file tree between a generation identifier associated with the first storage cluster and a generation identifier associated with the second storage cluster. The device synchronizes the first and second storage clusters by sending, via a network, the unsynchronized portions of the file tree from the first storage cluster to the second storage cluster.

Abstract: Techniques for filesystem management for cloud object storage are described. In one embodiment, a method includes writing, by a filesystem layer, a plurality of entries to a log structured file tree, including filesystem metadata and filesystem data. The method includes performing a flush operation of the entries from the filesystem layer to one or more objects in a distributed cloud object storage layer. The method includes storing the filesystem metadata and the filesystem data to the one or more objects in the distributed cloud object storage layer. The method further includes storing flush metadata generated during each flush operation, including a flush sequence number associated with each flush operation. Each object of the one or more objects in the distributed cloud object storage layer is identified by a key that identifies the flush sequence number, an object identifier, and a rebirth identifier.

Abstract: Systems and methods provide for optimizing data deduplication. A physical storage node can determine a cost of performing each exclusive action of a set of exclusive actions associated with data deduplication for a predetermined number of epochs to populate an exclusive action cost matrix. The node can select an optimal exclusive action from among the set of exclusive actions based on the cost of performing each exclusive action. In response to a random number being below a threshold value, the node can determine a first cost of performing a randomly selected exclusive action from the set of exclusive actions and update the exclusive action cost matrix with the first cost. In response to the random number exceeding the threshold value, the node can determine a second cost of performing the optimal exclusive action and update the exclusive action cost matrix with the second cost.

Abstract: Techniques for filesystem durable write operations to cloud object storage are described. In one embodiment, a method includes receiving at least one of a read operation or a write operation from a filesystem layer. The read operation identifies content stored in a distributed content object storage layer that is to be retrieved and the write operation identifies content that is to be stored to the distributed content object storage layer. The method includes converting the read operation into a get operation or the write operation into a put operation. The method further includes storing an index that maps the read or write operation to the get or put operation and includes an association between the content identified by the filesystem layer to objects in the distributed content object storage layer. The method includes sending the get or put operation to the distributed content object storage layer for the identified content.

Abstract: Techniques for implementing distributed quality of service controls for data transmissions between clusters in a clustered distributed storage system are described. In one embodiment, a method of distributed quality of service control includes establishing a control channel between a first plurality of nodes associated with a first cluster. The method also includes setting a quality of service (QoS) policy for the first plurality of nodes that establishes a QoS parameter for data transmitted over a shared network from the first cluster to a second cluster. The method further includes communicating messages between the first plurality of nodes over the control channel to signal an allocated portion of the QoS parameter for each node. The method also includes transmitting data by the first plurality of nodes of the first cluster to a second plurality of nodes of the second cluster over the shared network according to the allocated portion.

Abstract: In one embodiment, a device performs a plurality of iterations of write operations on portions of a file tree of a file system to a first storage cluster, each write operation flagging the written portion of the file tree with a generation identifier indicative of the associated iteration of the write operations. The device identifies a set of the written portions of the file tree as unsynchronized between the first storage cluster and a second storage cluster by performing a search for changes made to the written portions of the file tree between a generation identifier associated with the first storage cluster and a generation identifier associated with the second storage cluster. The device synchronizes the first and second storage clusters by sending, via a network, the unsynchronized portions of the file tree from the first storage cluster to the second storage cluster.

Abstract: Systems and methods provide for optimizing data deduplication. A physical storage node can determine a cost of performing each exclusive action of a set of exclusive actions associated with data deduplication for a predetermined number of epochs to populate an exclusive action cost matrix. The node can select an optimal exclusive action from among the set of exclusive actions based on the cost of performing each exclusive action. In response to a random number being below a threshold value, the node can determine a first cost of performing a randomly selected exclusive action from the set of exclusive actions and update the exclusive action cost matrix with the first cost. In response to the random number exceeding the threshold value, the node can determine a second cost of performing the optimal exclusive action and update the exclusive action cost matrix with the second cost.

Abstract: Aspects of the subject technology relate to a system configured to receive a request for data associated with a key, identify the key in a first pre-filter in a set of cascading pre-filters for the cache memory, wherein the set of cascading pre-filters is arranged in an order, and store, in response to the key being in the first pre-filter, the key in a second pre-filter in the set of cascading pre-filters, wherein the second pre-filter is next in the order after the first pre-filter.

Abstract: Techniques for filesystem management for cloud object storage are described. In one embodiment, a method includes writing, by a filesystem layer, a plurality of entries to a log structured file tree, including filesystem metadata and filesystem data. The method includes performing a flush operation of the entries from the filesystem layer to one or more objects in a distributed cloud object storage layer. The method includes storing the filesystem metadata and the filesystem data to the one or more objects in the distributed cloud object storage layer. The method further includes storing flush metadata generated during each flush operation, including a flush sequence number associated with each flush operation. Each object of the one or more objects in the distributed cloud object storage layer is identified by a key that identifies the flush sequence number, an object identifier, and a rebirth identifier.

Abstract: Techniques for filesystem durable write operations to cloud object storage are described. In one embodiment, a method includes receiving at least one of a read operation or a write operation from a filesystem layer. The read operation identifies content stored in a distributed content object storage layer that is to be retrieved and the write operation identifies content that is to be stored to the distributed content object storage layer. The method includes converting the read operation into a get operation or the write operation into a put operation. The method further includes storing an index that maps the read or write operation to the get or put operation and includes an association between the content identified by the filesystem layer to objects in the distributed content object storage layer. The method includes sending the get or put operation to the distributed content object storage layer for the identified content.

Abstract: Aspects of the subject technology relate to a system configured to receive a request for data associated with a key, identify the key in a first pre-filter in a set of cascading pre-filters for the cache memory, wherein the set of cascading pre-filters is arranged in an order, and store, in response to the key being in the first pre-filter, the key in a second pre-filter in the set of cascading pre-filters, wherein the second pre-filter is next in the order after the first pre-filter.

Abstract: I/O bandwidth reduction using storage-level common page information is implemented by a storage server, in response to receiving a request from a client for a page stored at a first virtual address, determining that the first virtual address maps to a page that is a duplicate of a page stored at a second virtual address or that the first and second virtual addresses map to a deduplicated page within a storage system, and transmitting metadata to the client mapping the first virtual address to a second virtual address that also maps to the deduplicated page. For one embodiment, the metadata is transmitted in anticipation of a request for the redundant/deduplicated page via the second virtual address. For an alternate embodiment, the metadata is sent in response to a determination that a page that maps to the second virtual address was previously sent to the client.

Abstract: Techniques for implementing distributed quality of service controls for data transmissions between clusters in a clustered distributed storage system are described. In one embodiment, a method of distributed quality of service control includes establishing a control channel between a first plurality of nodes associated with a first cluster. The method also includes setting a quality of service (QoS) policy for the first plurality of nodes that establishes a QoS parameter for data transmitted over a shared network from the first cluster to a second cluster. The method further includes communicating messages between the first plurality of nodes over the control channel to signal an allocated portion of the QoS parameter for each node. The method also includes transmitting data by the first plurality of nodes of the first cluster to a second plurality of nodes of the second cluster over the shared network according to the allocated portion.

Abstract: Creating and using highly available transaction logs in a distributed storage system is described, where the storage controller functions of the distributed storage system are separated from that of distributed storage system storage media. In an exemplary embodiment, a storage controller server receives a transaction entry at a first storage pool of the distributed storage system. The storage controller server further looks up a transaction log to store the transaction entry, where the transaction log is associated with a second storage pool. The storage controller server routes the transaction entry to the second storage pool, wherein the second storage pool stores the transaction entry.

Abstract: I/O bandwidth reduction using storage-level common page information is implemented by a storage server, in response to receiving a request from a client for a page stored at a first virtual address, determining that the first virtual address maps to a page that is a duplicate of a page stored at a second virtual address or that the first and second virtual addresses map to a deduplicated page within a storage system, and transmitting metadata to the client mapping the first virtual address to a second virtual address that also maps to the deduplicated page. For one embodiment, the metadata is transmitted in anticipation of a request for the redundant/deduplicated page via the second virtual address. For an alternate embodiment, the metadata is sent in response to a determination that a page that maps to the second virtual address was previously sent to the client.

Abstract: A distributed storage system called StorFS that performs distributed data deduplication is described. In an exemplary embodiment, a storage controller server determines if there is duplicative data in a distributed storage system. In this embodiment, the storage controller server determines if an entity is duplicated in the distributed storage system in line with an incoming input/output operation. The storage controller server determines if the entity is duplicated in the distributed storage system by receiving the entity and looking up the entity in a data deduplication table. If the entity exists in the data deduplication table, the storage controller server updates the metadata for the entity to point to the duplicate entity.

Abstract: I/O bandwidth reduction using storage-level common page information is implemented by a storage server, in response to receiving a request from a client for a page stored at a first virtual address, determining that the first virtual address maps to a page that is a duplicate of a page stored at a second virtual address or that the first and second virtual addresses map to a deduplicated page within a storage system, and transmitting metadata to the client mapping the first virtual address to a second virtual address that also maps to the deduplicated page. For one embodiment, the metadata is transmitted in anticipation of a request for the redundant/deduplicated page via the second virtual address. For an alternate embodiment, the metadata is sent in response to a determination that a page that maps to the second virtual address was previously sent to the client.

Abstract: One or more techniques and/or systems are provided for hosting a virtual machine from a snapshot. In particular, a snapshot of a virtual machine hosted on a primary computing device may be created. The virtual machine may be hosted on a secondary computing device using the snapshot, for example, when a failure of the virtual machine on the primary computing device occurs. If a virtual machine type (format) of the snapshot is not supported by the secondary computing device, then the virtual machine within the snapshot may be converted to a virtual machine type supported by the secondary computing device. In this way, the virtual machine may be operable and/or accessible on the secondary computing device despite the failure. Hosting the virtual machine on the secondary computing device provides, among other things, fault tolerance for the virtual machine and/or applications comprised therein.