Abstract: A technique provides efficient data protection, such as erasure coding, for data blocks of volumes served by storage nodes of a cluster. Data blocks associated with write requests of unpredictable client workload patterns may be compressed. A set of the compressed data blocks may be selected to form a write group and an erasure code may be applied to the group to algorithmically generate one or more encoded blocks in addition to the data blocks. Due to the unpredictability of the data workload patterns, the compressed data blocks may have varying sizes. A pool of the various-sized compressed data blocks may be established and maintained from which the data blocks of the write group are selected. Establishment and maintenance of the pool enables selection of compressed data blocks that are substantially close to the same size and, thus, that require minimal padding.

Abstract: A technique is configured to provide various data protection schemes, such as replication and erasure coding, for data blocks of volumes served by storage nodes of a cluster configured to perform deduplication of the data blocks. Additionally, the technique is configured to ensure that each deduplicated data block complies with data redundancy guarantees of the data protection schemes, while improving storage space of the storage nodes. In order to satisfy the data integrity guarantees while improving available storage space, the storage nodes perform periodic garbage collection for data blocks to optimize storage in accordance with currently applicable data protection schemes.

Abstract: A bin syncing technique ensures continuous data protection, such as replication and erasure coding, for content driven distribution of data served by storage nodes of a cluster in the event of failure to one or more block services configured to process the data. The cluster maintains information about the block services assigned to host a bin with a copy of the data in a bin assignment table associated with a state. The copies of the data are named, e.g., replica 0 (R0), replica 1 (R1) or replica 2 (R2). In response to failure of one or more block services assigned to host a bin with a replica of the data, an alternate or replacement block service may access the assignments maintained in the bin assignment table, which specify names of the replicas associated with the state.

Abstract: A technique is configured to provide various data protection schemes, such as replication and erasure coding, for data blocks of volumes served by storage nodes of a cluster configured to perform deduplication of the data blocks. Additionally, the technique is configured to ensure that each deduplicated data block complies with data redundancy guarantees of the data protection schemes, while improving storage space of the storage nodes. In order to satisfy the data integrity guarantees while improving available storage space, the storage nodes perform periodic garbage collection for data blocks to optimize storage in accordance with currently applicable data protection schemes.

Abstract: A technique manages bandwidth allocated among input/output operations, such as reads and writes, to storage devices coupled to storage nodes of a cluster. The technique balances the writes in a manner that reduces latency of reads, while allowing the writes to complete in a desired amount of time. The writes include write types, such as client writes, data migration writes, block transfer writes, and recycling writes, which are defined by differing characteristics and relative priorities. To ensure timely completion of the write types, the technique provides periodic time intervals over which the writes may be balanced and allocated sufficient bandwidth to access the storage devices. The time intervals may include shuffle intervals within a larger distribution interval. In addition, the technique throttles certain write types at the storage device level to maintain consistent read performance. Throttling is based on a credit system that allocates bandwidth as “credits” based on write type.

Abstract: A technique manages bandwidth allocated among input/output operations, such as reads and writes, to storage devices coupled to storage nodes of a cluster. The technique balances the writes in a manner that reduces latency of reads, while allowing the writes to complete in a desired amount of time. The writes include write types, such as client writes, data migration writes, block transfer writes, and recycling writes, which are defined by differing characteristics and relative priorities. To ensure timely completion of the write types, the technique provides periodic time intervals over which the writes may be balanced and allocated sufficient bandwidth to access the storage devices. The time intervals may include shuffle intervals within a larger distribution interval. In addition, the technique throttles certain write types at the storage device level to maintain consistent read performance. Throttling is based on a credit system that allocates bandwidth as “credits” based on write type.

Abstract: A technique provides efficient data protection, such as erasure coding, for data blocks of volumes served by storage nodes of a cluster. Data blocks associated with write requests of unpredictable client workload patterns may be compressed. A set of the compressed data blocks may be selected to form a write group and an erasure code may be applied to the group to algorithmically generate one or more encoded blocks in addition to the data blocks. Due to the unpredictability of the data workload patterns, the compressed data blocks may have varying sizes. A pool of the various-sized compressed data blocks may be established and maintained from which the data blocks of the write group are selected. Establishment and maintenance of the pool enables selection of compressed data blocks that are substantially close to the same size and, thus, that require minimal padding.

Abstract: An Updating No Sync (UNS) technique ensures continuous data protection for content driven distribution of data served by storage nodes of a fault tolerant cluster in the event of a degraded cluster state. A storage service implemented in each node includes one or more slice services (SSs) configured to process and store metadata describing the data served by the storage nodes and one or more block services (BSs) configured to process and store the data on storage devices of the node. A bin assignment service may coopt one or more healthy BSs to temporarily store updates of data and metadata received at the SS while the cluster is in the degraded state as an overflow data path (hence the term “Updating No Sync,” which denotes updating without synchronizing, i.e., not distributing the data within the cluster, but only accumulating an overflow of SS information). Once the cluster is no longer degraded, the accumulated overflow SS information at the BSs may be synchronized back to restored BSs, i.e.

Abstract: A technique is configured to maintain multiple copies of data served by storage nodes of a cluster during upgrade of a storage node to ensure continuous protection of the data served by the nodes. The data is logically organized as one or more volumes on storage devices of the cluster and includes metadata that describe the data of each volume. A data protection system may be configured to maintain at least two copies of the data in the cluster during upgrade to a storage node that is assigned to host one of the copies of the data but that is taken offline during the upgrade. As a result, an original slice service of the node may be rendered unavailable during the upgrade. In response, the technique redirects replicated data targeted to the original slice service to a standby pool of slice services in accordance with a degraded redundant metadata service of the cluster.

Abstract: A technique provides efficient data protection, such as erasure coding, for data blocks of volumes served by storage nodes of a cluster. Data blocks associated with write requests of unpredictable client workload patterns may be compressed. A set of the compressed data blocks may be selected to form a write group and an erasure code may be applied to the group to algorithmically generate one or more encoded blocks in addition to the data blocks. Due to the unpredictability of the data workload patterns, the compressed data blocks may have varying sizes. A pool of the various-sized compressed data blocks may be established and maintained from which the data blocks of the write group are selected. Establishment and maintenance of the pool enables selection of compressed data blocks that are substantially close to the same size and, thus, that require minimal padding.

Abstract: A technique provides efficient data protection, such as erasure coding, for data blocks of volumes served by storage nodes of a cluster. Data blocks associated with write requests of unpredictable client workload patterns may be compressed. A set of the compressed data blocks may be selected to form a write group and an erasure code may be applied to the group to algorithmically generate one or more encoded blocks in addition to the data blocks. Due to the unpredictability of the data workload patterns, the compressed data blocks may have varying sizes. A pool of the various-sized compressed data blocks may be established and maintained from which the data blocks of the write group are selected. Establishment and maintenance of the pool enables selection of compressed data blocks that are substantially close to the same size and, thus, that require minimal padding.

Abstract: A technique transitions data blocks of volumes served by storage nodes of a storage cluster from an old data protection scheme (DPS) to a new DPS in a non-disruptive manner. Slice services of the storage nodes forward the data blocks associated with write requests to the block services for storage on storage devices of the nodes. Mapping of volume logical block addresses to block identifiers are contained in slice files, wherein there is a single slice file for each volume. To transition a volume between the old and new DPSs, the slice service tags the data blocks with the new DPS when forwarding new write requests to the block services. In accordance with a background transitioning process, the slice service also retrieves every data block referenced by the to slice file and then resends the data to the block service with the new DPS.

Abstract: An Updating No Sync (UNS) technique ensures continuous data protection for content driven distribution of data served by storage nodes of a fault tolerant cluster in the event of a degraded cluster state. A storage service implemented in each node includes one or more slice services (SSs) configured to process and store metadata describing the data served by the storage nodes and one or more block services (BSs) configured to process and store the data on storage devices of the node. A bin assignment service may coopt one or more healthy BSs to temporarily store updates of data and metadata received at the SS while the cluster is in the degraded state as an overflow data path (hence the term “Updating No Sync,” which denotes updating without synchronizing, i.e., not distributing the data within the cluster, but only accumulating an overflow of SS information). Once the cluster is no longer degraded, the accumulated overflow SS information at the BSs may be synchronized back to restored BSs, i.e.

Abstract: A bin syncing technique ensures continuous data protection, such as replication and erasure coding, for content driven distribution of data served by storage nodes of a cluster in the event of failure to one or more block services configured to process the data. The cluster maintains information about the block services assigned to host a bin with a copy of the data in a bin assignment table associated with a state. The copies of the data are named, e.g., replica 0 (R0), replica 1 (R1) or replica 2 (R2). In response to failure of one or more block services assigned to host a bin with a replica of the data, an alternate or replacement block service may access the assignments maintained in the bin assignment table, which specify names of the replicas associated with the state.

Abstract: A technique is configured to maintain multiple copies of data served by storage nodes of a cluster during upgrade of a storage node to ensure continuous protection of the data served by the nodes. The data is logically organized as one or more volumes on storage devices of the cluster and includes metadata that describe the data of each volume. A data protection system may be configured to maintain at least two copies of the data in the cluster during upgrade to a storage node that is assigned to host one of the copies of the data but that is taken offline during the upgrade. As a result, an original slice service of the node may be rendered unavailable during the upgrade. In response, the technique redirects replicated data targeted to the original slice service to a standby pool of slice services in accordance with a degraded redundant metadata service of the cluster.

Abstract: A technique is configured to provide data protection, such as replication and erasure coding, of content driven distribution of data blocks served by storage nodes of a cluster. When providing data protection in the form of replication (redundancy), a slice service of the storage node generates one or more copies or replicas of a data block for storage on the cluster. Each replicated data block is illustratively organized within a bin that is maintained by block services of the nodes for storage on storage devices. When providing data protection in the form of erasure coding, the block services may select data blocks to be erasure coded. A set of data blocks for erasure coding may then be grouped together to form a write group. According to the technique, EC group membership is guided by varying bin groups so the data is resilient against failure. Slice services of the storage nodes assign data blocks of different bins and replicas to a write group.

Abstract: A technique is configured to provide various data protection schemes, such as replication and erasure coding, for data blocks of volumes served by storage nodes of a cluster configured to perform deduplication of the data blocks. Additionally, the technique is configured to ensure that each deduplicated data block complies with data redundancy guarantees of the data protection schemes, while improving storage space of the storage nodes. In order to satisfy the data integrity guarantees while improving available storage space, the storage nodes perform periodic garbage collection for data blocks to optimize storage in accordance with currently applicable data protection schemes.