Abstract: A storage control system receives a first write request and a second write request following the first write request. The first and second write requests comprise respective first and second data items for storage to a primary storage. First and second cache write operations are performed in parallel to write the first and second data items a persistent write cache. The first cache write operation comprises a split write operation which comprises writing a parsing header for the first data item to the write cache, and writing a payload of the first data item to the write cache. The second cache write operation comprises storing the second data item and associated metadata in the write cache, and waiting for an acknowledgment that the parsing header for the first data item has been successfully stored in the write cache before returning an acknowledgment indicating successful storage of the second data item.

Abstract: Techniques are disclosed for utilizing checkpoints to achieve resiliency of metadata in a storage system. A storage control system writes metadata to a persistent write cache. The storage control system performs a checkpoint generation process to generate a new metadata checkpoint which includes at least a portion of the metadata in the persistent write cache. The checkpoint generation process comprises placing a lock on processing to enable metadata in the persistent write cache to reach a consistent state, creating a metadata checkpoint structure in memory, removing the lock on processing to allow metadata updates in the persistent write cache, destaging at least a portion of the metadata from the persistent write cache to the metadata checkpoint structure, and persistently storing the metadata checkpoint structure.

Abstract: A processing device obtains a write operation which comprises first data and second data to be stored in first and second strips of a given stripe. The processing device stores the first data in the first strip and determines that the second strip is unavailable. The processing device determines a parity based on the first data and the second data and stores the parity in a parity strip. The processing device updates metadata to indicate that the second data was not stored in the second strip. In some embodiments, the updated metadata is non-persistent and the processing device may be further configured to rebuild the given stripe, update persistent metadata corresponding to a sector of stripes including the given stripe and clear the non-persistent metadata based at least in part on a completion of the rebuild.

Abstract: Techniques are disclosed for managing metadata of a storage system. A storage control system receives data to be written to primary storage, and writes the received data together with metadata to a write cache. The storage control system destages the metadata from the write cache to a primary metadata structure which is configured to persistently store and index the metadata. The primary metadata structure comprises (i) a first data structure that is configured to accumulate the metadata destaged from the write cache and organize the accumulated metadata in blocks of metadata sorted by index keys, and (ii) a second data structure that is configured to receive the accumulated metadata from the first data structure, and organize the received metadata using an index structure that enables random-access to the metadata using the index keys.

Abstract: A storage control system manages a utilization of data blocks of a storage volume which is partitioned into data blocks having a unique block identifier (ID) and a same block size. The storage control system receives data items and assigns a respective unique data ID to each data item, which include consecutive data IDs. The data items are written to a free data block as a whole, and a record for the written data block is inserted into a node of a first tree structure. The record includes the unique block ID of the written data block, a first data ID of the data items, and a bitmap which maps the consecutive data IDs of the data items in the written data block, starting from the first data ID, to a respective bit whose value indicates whether the data item associated with the data ID is valid or invalid.

Abstract: A data deduplication process is performed in a storage system which includes storage nodes, and storage control nodes which can access data directly from each storage node. A first storage control node sends a message to a second storage control node to initiate a deduplication process with respect to a given data block and an original data block owned by the second storage control node. The second storage control node increments a reference counter associated with the original data block, and sends a message to the first storage control which includes metadata. The first storage control node uses the metadata to read the original data block from a given storage node, performs a data compare process to determine whether the given data block matches the original data block, and creates a reference to the original data block, if the given data block matches the original data block.

Abstract: Methods that can provide distributed data deduplication reference counting are disclosed herein. One method includes generating, by a processor, a secondary reference in response to a determination that a data chunk is a copy of the primary reference in which the primary reference includes a pointer to an existing data chunk stored on a storage device. The method further includes incrementing a secondary reference counter on the primary reference in response to generating the secondary reference. Apparatus and computer program products that can include, perform, and/or implement the methods are also disclosed herein.

Abstract: An apparatus comprises a processing device configured to receive a request to create a given storage volume in a storage system, the storage system providing a plurality of storage features. The processing device is also configured to select, for the given storage volume, one of a set of one or more volume tiers, each of the volume tiers specifying whether respective ones of the plurality of storage features provided by the storage system are enabled or disabled for storage volumes associated with that volume tier. The processing device is further configured to create the given storage volume in the storage system, and to associate the selected volume tier with the given storage volume, wherein associating the selected volume tier with the given storage volume comprises enabling or disabling respective ones of the plurality of storage features provided by the storage system as specified by the selected volume tier.

Abstract: A processing device obtains a write operation which comprises first data and second data to be stored in first and second strips of a given stripe. The processing device stores the first data in the first strip and determines that the second strip is unavailable. The processing device determines a parity based on the first data and the second data and stores the parity in a parity strip. The processing device updates metadata to indicate that the second data was not stored in the second strip. In some embodiments, the updated metadata is non-persistent and the processing device may be further configured to rebuild the given stripe, update persistent metadata corresponding to a sector of stripes including the given stripe and clear the non-persistent metadata based at least in part on a completion of the rebuild.

Abstract: A storage control system is configured to obtain measurement information comprising an indication of an amount of utilization of a hardware resource of a first server node by a plurality of processing groups and to determine that the amount of utilization of the hardware resource is above a threshold amount of utilization. The at least one processor is further configured to select a given processing group for redistribution based at least in part on the determination that the amount of utilization of the hardware resource is above the threshold amount and on an amount of utilization of the hardware resource by the given processing group. The at least one processor is further configured to determine that a second server node comprises enough available capacity of the hardware resource and to redistribute the given processing group to the second server node based at least in part on the determination.

Abstract: Methods that can crypto-erase deduplicated data are disclosed herein. One method includes encrypting, by a processor, each data chunk on a storage device with a unique first encryption key that is different from each other first encryption key and encrypting each storage object on the storage device with a unique second encryption key that is different from each first encryption key and each other second encryption key in which each second encryption key encrypts a first encryption key for a data chunk. Apparatus, systems, and computer program products that can include, perform, and/or implement the methods are also disclosed herein.

Abstract: A mechanism is provided for dispersed location-based data storage. A request is received to write a data file to a referrer memory region in a set of memory regions. For each data chunk of the data file, responsive to a comparison of a hash value for the data chunk to other hash values for other stored data chunks referenced in the referrer memory region indicating that the data chunk fails to exist in the referrer memory region, responsive to the data chunk existing in another memory region in the set of memory regions, responsive to the memory region failing to be one of the predetermined number N of owner memory regions associated with the referrer memory region, and responsive to the predetermined number N of owner memory regions failing to have been met, a reference to the data chunk is stored in the referrer memory region.

Abstract: An apparatus comprises a processing device configured to receive a request to create a given storage volume in a storage system, the storage system providing a plurality of storage features. The processing device is also configured to select, for the given storage volume, one of a set of one or more volume tiers, each of the volume tiers specifying whether respective ones of the plurality of storage features provided by the storage system are enabled or disabled for storage volumes associated with that volume tier. The processing device is further configured to create the given storage volume in the storage system, and to associate the selected volume tier with the given storage volume, wherein associating the selected volume tier with the given storage volume comprises enabling or disabling respective ones of the plurality of storage features provided by the storage system as specified by the selected volume tier.

Abstract: Techniques are provided for processing user input/output (I/O) write requests in a fault-tolerant data storage system (e.g., a RAID storage system) by selecting between performing a degraded write operation or a write operation to spare capacity, when the fault-tolerant data storage system is operating in a degraded mode. A method includes receiving a user I/O write request comprising data to be written to a RAID array operating in a degraded mode, and determining whether spare capacity has been allocated for rebuilding missing data of an inaccessible storage device of the RAID array and whether a missing data block, which is associated with I/O write request, has been rebuilt to the spare capacity. A degraded write operation is performed without using the spare capacity, when the missing data block, which is associated with the data of the I/O write request, has not been rebuilt to the allocated spare capacity.

Abstract: A mechanism is provided for dispersed location-based data storage. A request is received to write a data file to a referrer memory region in a set of memory regions. For each data chunk of the data file, responsive to a comparison of a hash value for the data chunk to other hash values for other stored data chunks referenced in the referrer memory region indicating that the data chunk fails to exist in the referrer memory region, responsive to the data chunk existing in another memory region in the set of memory regions, responsive to the memory region failing to be one of the predetermined number N of owner memory regions associated with the referrer memory region, and responsive to the predetermined number N of owner memory regions failing to have been met, a reference to the data chunk is stored in the referrer memory region.

Abstract: A storage system comprises mixed-capacity storage devices, including at least one or more storage devices having relatively low capacities and one or more storage devices having relatively high capacities. The storage system is configured to establish a redundant array of independent disks (RAID) arrangement comprising a plurality of stripes, with each of the plurality of stripes comprising a plurality of data blocks and one or more corresponding parity blocks, the data blocks and parity blocks being distributed across multiple ones of the storage devices. In conjunction with establishment of the RAID arrangement, the storage system is further configured to concentrate storage of the parity blocks in the one or more storage devices having the relatively high capacities. The storage devices in some embodiments comprise at least a first set of storage devices each having a first capacity and a second set of storage devices each having a second capacity higher than the first capacity.

Abstract: A storage system comprises a plurality of storage devices, and is configured to establish a redundant array of independent disks (RAID) arrangement comprising a plurality of stripes, with each of the plurality of stripes comprising a plurality of blocks, the blocks being distributed across multiple ones of the storage devices. In conjunction with establishment of the RAID arrangement, the storage system is further configured, for each of the plurality of stripes, to designate a particular one of the storage devices as a spare storage device for that stripe, and for each of the storage devices, to determine numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices. A particular number of spare blocks is reserved for each of the storage devices based at least in part on its determined numbers of the stripes.

Abstract: Techniques are provided for processing user input/output (I/O) write requests in a fault-tolerant data storage system (e.g., a RAID storage system) by selecting between performing a degraded write operation or a write operation to spare capacity, when the fault-tolerant data storage system is operating in a degraded mode. A method includes receiving a user I/O write request comprising data to be written to a RAID array operating in a degraded mode, and determining whether spare capacity has been allocated for rebuilding missing data of an inaccessible storage device of the RAID array and whether a missing data block, which is associated with I/O write request, has been rebuilt to the spare capacity. A degraded write operation is performed without using the spare capacity, when the missing data block, which is associated with the data of the I/O write request, has not been rebuilt to the allocated spare capacity.

Abstract: A storage system comprises a plurality of storage devices, and is configured to establish a redundant array of independent disks (RAID) arrangement comprising a plurality of stripes, with each of the plurality of stripes comprising a plurality of blocks, the blocks being distributed across multiple ones of the storage devices. In conjunction with establishment of the RAID arrangement, the storage system is further configured, for each of the plurality of stripes, to designate multiple ones of the storage devices as respective spare devices for that stripe, and for each of the storage devices, to determine numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices in each of multiple spare levels and for each of multiple failure combinations. A particular number of spare blocks is reserved for each of the storage devices using the determined numbers.

Abstract: A storage system comprises a plurality of storage devices, and is configured to establish a redundant array of independent disks (RAID) arrangement comprising a plurality of stripes, with each of the plurality of stripes comprising a plurality of blocks, the blocks being distributed across multiple ones of the storage devices. In conjunction with establishment of the RAID arrangement, the storage system is further configured, for each of the plurality of stripes, to designate a particular one of the storage devices as a spare storage device for that stripe, and for each of the storage devices, to determine numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices. A particular number of spare blocks is reserved for each of the storage devices based at least in part on its determined numbers of the stripes.