Abstract: Aspects of the present disclosure relate to reducing the latency of data deduplication. In embodiments, an input/output (IO) workload received by a storage array is monitored. Further, at least one IO write operation in the IO workload is identified. A space-efficient probabilistic data structure is used to determine if a director board is associated with the IO write. Additionally, the IO write operation is processed based on the determination.

Abstract: Aspects of the present disclosure relate to reducing the latency of data deduplication. In embodiments, an input/output (IO) workload received by a storage array is monitored. Further, at least one IO write operation in the IO workload is identified. A space-efficient probabilistic data structure is used to determine if a director board is associated with the IO write. Additionally, the IO write operation is processed based on the determination.

Abstract: A deduplication engine maintains a deduplication hash table containing a plurality of fingerprint entries identifying a track of current data and a hash value computed from the data currently stored at the identified track. A hash table manager implements a dynamic deduplication hash table management process, that includes a fingerprint entry purge process and a fingerprint entry re-entry process. The purge process is implemented on the deduplication hash table to remove selected fingerprint entries of tracks of data currently stored on the managed drives from the deduplication hash table based on a first set of Thin Device (TDev) selection rules and a second set of track selection purge rules. The re-entry process is implemented on the deduplication hash table to re-enter selected fingerprint entries into the deduplication hash table based on a third set of TDev selection rules and a fourth set of track selection re-entry rules.

Abstract: Storage arrays in a mirror relationship use dynamic compression dictionary size adjustment and data buffer merging to more efficiently compress data associated with maintenance of consistency of a replicated storage object. A compression dictionary size selector selects a supported compression dictionary size based on the aggregate size of the updates enqueued for transmission in the data buffers, e.g., a compression dictionary size that is greater than or equal to, and as close as possible to, the aggregate size of the enqueued updates. A combiner selects enqueued updates based on the selected compression dictionary size, e.g., aggregate enqueued update size less than or equal to, and as close as possible to, the selected compression dictionary size. The selected updates are coalesced and compressed using the selected compression dictionary size.

Abstract: Storage arrays in a mirror relationship use dynamic compression dictionary size adjustment and data buffer merging to more efficiently compress data associated with maintenance of consistency of a replicated storage object. A compression dictionary size selector selects a supported compression dictionary size based on the aggregate size of the updates enqueued for transmission in the data buffers, e.g., a compression dictionary size that is greater than or equal to, and as close as possible to, the aggregate size of the enqueued updates. A combiner selects enqueued updates based on the selected compression dictionary size, e.g., aggregate enqueued update size less than or equal to, and as close as possible to, the selected compression dictionary size. The selected updates are coalesced and compressed using the selected compression dictionary size.

Abstract: A deduplication engine maintains a hash table containing hash values of tracks of data stored on managed drives of a storage system. The deduplication engine keeps track of how frequently the tracks are accessed by the deduplication engine using an exponential moving average for each track. Target tracks which are frequently accessed by the deduplication engine are cached in local memory, so that required byte-by-byte comparisons between the target track and write data may be performed locally rather than requiring the target track to be read from managed drives. The deduplication engine implements a Least Recently Used (LRU) cache data structure in local memory to manage locally cached tracks of data. If a track is to be removed from local memory, a final validation of the target track is implemented on the version stored in managed resources before evicting the track from the LRU cache.

Abstract: On a local data storage system, a single buffer of contiguous local memory is allocated to store host data from multiple received host data buffers in a local cache, and local internal data processing operations may be performed on the host data with local data processing resources using the single buffer of contiguous local memory in the local memory to store the host data. On a remote data storage system onto which the host data is replicated, a single buffer of contiguous remote memory is allocated to store host data from multiple transmission units used to convey the host data to the remote data storage system, and remote internal data processing operations may be performed on the host data by one or more remote data processing resources using the single buffer of contiguous remote memory to store the host data from the received transmission units.