Abstract: Example embodiments of the present invention relate and a method and an apparatus for double hashing. The method including receiving a hash signature, including a short hash handle, for a data block. The method then includes determining a bucket with which the hash signature should be associated and associating the hash signature with the bucket.

Abstract: A method comprises, in an information processing system implementing data deduplication and compression, wherein the information processing system comprises a set of data storage devices, receiving by at least one of the data storage devices comprising a processing device an instruction from the information processing system to perform at least a portion of a compression operation. The method also comprises performing the portion of the compression operation in response to the instruction, and sending a result of the performed portion of the compression operation to the information processing system.

Abstract: An aspect of dual layer deduplication for a file system running over a deduplication block storage system includes accessing a file by a file system driver of the file system. The file is stored as one of a plurality of files in the file system. A further aspect includes breaking the file down into multiple blocks and generating chunks from the blocks. For at least one of the chunks, a trail of zeros is added until a size of a respective one of the chunks is a multiple of a block size of the deduplication block storage system.

Abstract: In an information processing system including a set of data storage devices for storing data blocks arranged in respective columns on each data storage device and rows across the set of data storage devices to form at least one data stripe, and a set of parity storage devices for storing parity blocks computed via one or more parity operations based on the data blocks of the at least one data stripe, at least one of the data storage devices includes a processing device configured to: receive from the information processing system an instruction to perform at least a portion of a parity operation; perform the portion of the parity operation; and send a result of the performed portion of the parity operation to the information processing system, wherein the result is useable by the information processing system for performing another portion of the parity operation.

Abstract: An apparatus comprises at least one processing device comprising a processor coupled to a memory. The processing device is configured to select a stripe column size for stripes of a data storage system, to determine a first compress block size for a first one of the stripes based on compressibility of data to be stored, to select a first prime number for computing parity blocks for the first stripe and a first number of sub-stripes for splitting stripe columns of the first stripe, to generate metadata specifying the first compress block size, the first prime number and the first number of sub-stripes for the first stripe, and to store data compressed using the first compress block size in the first stripe. The first prime number and first number of sub-stripes for the first stripe is different than a second prime number and second number of sub-stripes for a second stripe.

Abstract: An apparatus in one embodiment comprises at least one processing device comprising a processor coupled to a memory. The processing device is configured to receive a request to write a given data object to a write buffer of at least one storage system, to allocate an entry in the write buffer, to generate a header for the given data object comprising a first portion indicating a validation status, to write the header to the allocated entry of the write buffer with the first portion of the header comprising a value indicating invalid status, to write the given data object to the allocated entry of the write buffer, and to update the header of the allocated entry by modifying the value of the first portion of the header to indicate valid status responsive to a successful write of the given data object to the allocated entry of the write buffer.

Abstract: An apparatus comprises at least one processing device comprising a processor coupled to a memory. The processing device is configured to select a stripe column size for stripes of a data storage system, to determine a first compress block size for a first one of the stripes based on compressibility of data to be stored, to select a first prime number for computing parity blocks for the first stripe and a first number of sub-stripes for splitting stripe columns of the first stripe, to generate metadata specifying the first compress block size, the first prime number and the first number of sub-stripes for the first stripe, and to store data compressed using the first compress block size in the first stripe. The first prime number and first number of sub-stripes for the first stripe is different than a second prime number and second number of sub-stripes for a second stripe.

Abstract: A clustered storage system in one embodiment comprises a plurality of nodes, with each of at least a subset of the nodes comprising a set of processing modules configured to communicate over one or more networks with corresponding sets of processing modules on other ones of the nodes. In conjunction with a failure of a first instance of a process running on a given one of the nodes and a subsequent restart of a second instance of the process, at least one of the processing modules is to identify at least one transfer buffer command of the first instance of the process, to identify a plurality of logically ordered commands of the first instance of the process, and to provide distinct treatment of the transfer buffer command relative to treatment of the logically ordered commands in a manner that ensures that the restart of the second instance of the process is not delayed to await completion of the transfer buffer command or the logically ordered commands.

Abstract: An aspect includes reading a first page and a corresponding second page from a storage device. The first page specifies a metadata page stored in persistent storage and having logical addresses of metadata, and the second page associates logical block addresses (LBAs) with corresponding physical locations for the metadata. An aspect also includes reading data for a RAID stripe according to an associated physical offset in the second page, accessing a stripe counter from the second page, and comparing the stripe counter from the second page to a stripe counter held in memory. Upon determining the stripe counter from the second page is not the same, a third page is loaded, and a physical location of the data is read from the third page that provides a hash value of the data and corresponding physical location. The physical location of the data is accessed, and the second page is updated.

Abstract: A storage system in one embodiment comprises a plurality of storage devices and a storage controller. The storage controller is configured to receive a plurality of logical addresses. Each logical address has one of a content-based mapping type and an address-based mapping type. Responsive to a first logical address of the plurality of logical addresses having the content-based mapping type, the storage controller is configured to utilize a content-based mapping generated based on content of a data page associated with the first logical address to identify a corresponding physical address. Responsive to a second logical address of the plurality of logical addresses having the address-based mapping type, the storage controller is configured to utilize an address-based mapping generated based on the second logical address to identify a corresponding physical address.

Abstract: An apparatus in one embodiment comprises at least one processing device comprising a processor coupled to a memory. The processing device is configured to detect an error relating to storage of a given data page in a first storage system, and to send a message to at least one additional storage system, the message containing a hash digest of the given data page. The processing device is further configured to receive from the additional storage system responsive to the message a recovery data page identified by the additional storage system as having the same hash digest as the given data page. The recovery data page received from the additional storage system is verified by computing a hash digest of the recovery data page and comparing the computed hash digest to the hash digest of the given data page. Separate instances of the message containing the hash digest of the given data page may be sent to each of a plurality of additional storage systems.

Abstract: An aspect of memory management is provided. An aspect includes evaluating performance parameters of caches of a control module. The caches of the control module have two types of entries: address, hash, and physical location values, and address-to-short-hash (A2SH) values. An aspect further includes evaluating performance parameters of caches of a data module of the multi-layer cache system. The caches of the data module cache include three types of entries: a short-hash-to-physical address, a full-hash-and-short-hash-to-physical address, and a filter mechanism. An aspect further includes predicting an effect that a modification to a size of one of the caches o is on performance of operations at the multi-level cache based on results of the calculating the performance parameters of the caches. Upon estimating an increase in performance, an aspect includes increasing allocation to the cache is determined to have increased performance responsive to the estimating, and decreasing allocation from another cache.

Abstract: A method comprises, in an information processing system implementing data deduplication and compression, wherein the information processing system comprises a set of data storage devices, receiving by at least one of the data storage devices comprising a processing device an instruction from the information processing system to perform at least a portion of a compression operation. The method also comprises performing the portion of the compression operation in response to the instruction, and sending a result of the performed portion of the compression operation to the information processing system.

Abstract: A storage system comprising a plurality of storage devices and an associated storage controller. The plurality of storage devices are configured to store data blocks distributed across the plurality of storage devices in a plurality of data stripes. The plurality of data stripes comprise a first set of data stripes and a second set of data stripes. The storage controller is configured to receive data associated with at least one input-output request and to store the received data sequentially in at least one data stripe of the first set of data stripes. The controller is further configured to determine whether or not an amount of data stored in the first set of data stripes is greater than a threshold amount of data and in response to determining that the amount of data stored in the first set of data stripes is greater than the threshold amount of data, to destage the at least one data stripe of the first set of data stripes to the second set of data stripes.

Abstract: An apparatus in one embodiment comprises at least one processing device comprising a processor coupled to a memory. The at least one processing device is configured to obtain a file and to determine a type of the file. A structure of the file may be determined based at least in part on the determined type of the file and at least one location in the file may be identified based at least in part on the determined structure. The file may be divided at the identified at least one location into a plurality of chunks and the plurality of chunks may be provided to a block deduplication module of a storage system where the block deduplication module is configured to perform a deduplication process based at least in part on the plurality of chunks.

Abstract: In an information processing system including a set of data storage devices for storing data blocks arranged in respective columns on each data storage device and rows across the set of data storage devices to form at least one data stripe, and a set of parity storage devices for storing parity blocks computed via one or more parity operations based on the data blocks of the at least one data stripe, at least one of the data storage devices includes a processing device configured to: receive from the information processing system an instruction to perform at least a portion of a parity operation; perform the portion of the parity operation; and send a result of the performed portion of the parity operation to the information processing system, wherein the result is useable by the information processing system for performing another portion of the parity operation.

Abstract: An apparatus in one embodiment comprises at least one processing device comprising a processor coupled to a memory. The processing device is configured, in conjunction with an online cluster expansion process for a clustered storage system having an address space distributed across a set of storage controllers each associated with one or more storage devices, to modify the distribution of the address space across the set of storage controllers responsive to adding at least one of compute and storage capacity to the set of storage controllers of the clustered storage system. The processing device is also configured to identify a set of pending input-output requests directed to deletion of data in the clustered storage system, and to execute the set of pending input-output requests prior to relocating data in the clustered storage system based at least in part on the modified distribution of the address space.

Abstract: A method is disclosed for dynamic multitasking in a storage system, the storage system including a first storage server configured to execute a first I/O service process and one or more second storage servers, the method comprising: detecting a first event for triggering a context switch; transmitting to each of the second storage servers an instruction to stop transmitting internal I/O requests to the first I/O service process, the instruction including an identifier corresponding to the first I/O service process, the identifier being arranged to distinguish the first I/O service process from other first I/O service processes that are executed by the first storage server concurrently with the first I/O service process; deactivating the first I/O service process by pausing a frontend of the first I/O service process, and pausing one or more I/O providers of the first I/O service process; and executing a first context switch between the first I/O service process and a second process.

Abstract: A storage system in one embodiment comprises a plurality of storage devices and a storage controller. The storage controller is configured to receive a plurality of logical addresses. Each logical address has one of a content-based mapping type and an address-based mapping type. Responsive to a first logical address of the plurality of logical addresses having the content-based mapping type, the storage controller is configured to utilize a content-based mapping generated based on content of a data page associated with the first logical address to identify a corresponding physical address. Responsive to a second logical address of the plurality of logical addresses having the address-based mapping type, the storage controller is configured to utilize an address-based mapping generated based on the second logical address to identify a corresponding physical address.

Abstract: A storage system comprises a plurality of storage nodes each comprising one or more storage devices. At least a given one of the storage nodes is configured to read data blocks from its one or more storage devices, and for a given one of the data blocks, to determine based at least in part on a content-based signature of that data block whether or not the given data block is appropriate for use in a prefilling operation of the given storage node. Responsive to the given data block being appropriate for use in the prefilling operation of the given storage node, the given storage node uses the data block in the prefilling operation of the given storage node. Responsive to the given data block not being appropriate for use in the prefilling operation of the given storage node, the given storage node sends the data block to another one of the storage nodes for use in a prefilling operation of that other storage node.