Abstract: The meta data containing count and key fields of CKD records are reversibly decoupled from the user data of the data field so that the data can be deduplicated. Multiple CKD records may be coalesced into a larger size CKD track. The coalesced meta data is compressed and stored in a CKD hash table. The user data is hashed, and the hash is used as a hash key that is associated with the compressed meta data in the CKD hash table. When the hash of user data associated with a CKD write IO matches the hash key of an existing entry in the table, data duplication is indicated. The compressed meta data is added to the entry and the user data is deduplicated by creating storage system meta data that points to the pre-existing copy of the user data. The storage system metadata includes unique information that enables the corresponding compressed metadata to be subsequently located in the hash table to reassemble the CKD records.

Abstract: Aspects of the present disclosure relate to data deduplication (dedup) techniques for storage arrays. In embodiments, a sequence of input/output (IO) operations in an IO stream received from one or more host devices by a storage array are identified. Additionally, a determination can be made as to whether a set of previously received IO operations match the identified IO sequence based on a time series relationship between the identified IO sequence and the previously received IO operations. Further, one or more data deduplication (dedup) techniques can be performed on the matching IO sequence.

Abstract: The meta data containing count and key fields of CKD records are reversibly decoupled from the user data of the data field so that the data can be deduplicated. Multiple CKD records may be coalesced into a larger size CKD track. The coalesced meta data is compressed and stored in a CKD hash table. The user data is hashed, and the hash is used as a hash key that is associated with the compressed meta data in the CKD hash table. When the hash of user data associated with a CKD write IO matches the hash key of an existing entry in the table, data duplication is indicated. The compressed meta data is added to the entry and the user data is deduplicated by creating storage system meta data that points to the pre-existing copy of the user data. The storage system metadata includes unique information that enables the corresponding compressed metadata to be subsequently located in the hash table to reassemble the CKD records.

Abstract: A task generator runs on a data storage system in which multiple compute nodes allocate portions of local memory to a remotely accessible shared memory. The task generator is responsive to a primary task to be performed on a storage object to generate separate, per-compute node secondary tasks corresponding to the primary task. Each of the separate secondary tasks specifies at least one attribute of metadata associated with the storage object and at least one task logic function. Each of the plurality of compute nodes performs the separate secondary task generated for that compute node by scanning the local portion of the shared memory based on the attribute to identify matching pages of the metadata associated with the storage object and performs the task logic function on the identified matching pages of the metadata associated with the storage object to generate a local result. The task generator combines the local results to perform the primary task.

Abstract: A Ransomware Activity Detection System (RADS) characterizes historic read/write IO activity on a storage volume, and also characterizes historic data characteristics of the storage volume, such as the percentage reducibility of the data held in the storage volume. The RADS monitors the storage volume to identify differences between current read/write IO activity and historic read/write IO activity, as well as difference between current data characteristics of the storage volume and historic data characteristics of the storage volume. When the RADS detects a significant difference in read/write IO activity on a storage volume, that is coupled with a significant changes to the data characteristics of the storage volume, the RADS protects the storage volume and generates an alert of the possible occurrence of a ransomware attack. Protection may occur prior in connection with any bulk read operation to proactively protect storage volumes against ransomware attacks.

Abstract: A deduplication engine maintains a deduplication hash table containing a plurality of fingerprint entries identifying a track of current data and a hash value computed from the data currently stored at the identified track. A hash table manager implements a dynamic deduplication hash table management process, that includes a fingerprint entry purge process and a fingerprint entry re-entry process. The purge process is implemented on the deduplication hash table to remove selected fingerprint entries of tracks of data currently stored on the managed drives from the deduplication hash table based on a first set of Thin Device (TDev) selection rules and a second set of track selection purge rules. The re-entry process is implemented on the deduplication hash table to re-enter selected fingerprint entries into the deduplication hash table based on a third set of TDev selection rules and a fourth set of track selection re-entry rules.

Abstract: Aspects of the present disclosure relate to data deduplication (dedup) techniques for storage arrays. In embodiments, a sequence of input/output (IO) operations in an IO stream received from one or more host devices by a storage array are identified. Additionally, a determination can be made as to whether a set of previously received IO operations match the identified IO sequence based on a time series relationship between the identified IO sequence and the previously received IO operations. Further, one or more data deduplication (dedup) techniques can be performed on the matching IO sequence.

Abstract: A hardware offload includes a hash engine that performs hashing for a block-based storage system. The hash engine calculates multiple hash values for each input buffer provided by the storage system. The hash values may be calculated with variably offset and overlapping portions of the input buffer, wherein each portion is larger than the native block size of the storage system. The hardware offload may also include a compression engine that performs compression on the input buffer using the entire input buffer and/or chunks as compression domains.

Abstract: A method of determining and depicting an effective storage capacity of a storage system includes determining a storage-based limit of effective storage capacity and a metadata-based limit of effective storage capacity. The storage-based limit is based on an amount of unused capacity of a set of managed drives and a data reduction ratio achieved when host data is reduced prior to storage on the set of managed drives. Data reduction may include compression, deduplication, and pattern detection operations. The metadata-based limit is based on a volume of metadata that has been generated by the data, and the data reduction operations, in connection with writing the data to the set of managed drives, and based on an amount of memory allocated to storing the metadata. The effective storage capacity, actual storage usage, and data reduction ratio are graphically depicted over time to enable changes to these parameters to be visualized.

Abstract: Protection and recovery is provided for an allocation unit of data that is in-flight between a source device and a target device of a storage system during a systemic failure by selecting between corresponding source device and target device metadata based on consistency and reciprocal linkage. If logical volume metadata for the allocation unit of the data in volatile memory is not consistent with logical volume metadata for the allocation unit in non-volatile storage, then the allocation unit is recovered from the source device. If target device metadata for the data in volatile memory is not consistent with target device metadata for the data in non-volatile storage, then the allocation unit is recovered from the source device. If linkage between logical volume metadata for the allocation unit and target device metadata for the data is not reciprocal, then the allocation unit is recovered from the source device.

Abstract: A method and system for sharing data reduction metadata with storage systems. Specifically, the disclosed method and system entail communicating, to a storage system, information known to host devices from which data (submitted to-be-written to the storage system) may originate. This a priori reduction-pertinent information, which may include the potential to improve storage system efficiency and/or performance at least with respect to data reduction processing of the data submitted to-be-written, had previously been considered incommunicable to the storage system. The disclosed method and system, however, lift this previous limitation and enable communication of any storage system performance-improving information, applicable to the data submitted to-be-written, to the storage system.

Abstract: Within a storage array, allocation of physical storage capacity within a storage array may be managed in standard size allocation units of uncompressed data, e.g. 128kb tracks, while smaller sub-allocation unit compression domains, e.g. 32kb quarter tracks, are used for compressed data. The data within a sub-allocation unit may be compressed to a size that is less than the capacity of the sub-allocation unit. Data associated with sub-allocation units that are not required to service a read or write may not need to be compressed or decompressed in order to service the read or write. Consequently, resource usage may be more efficient.

Abstract: Protection and recovery is provided for an allocation unit of data that is in-flight between a source device and a target device of a storage system during a systemic failure by selecting between corresponding source device and target device metadata based on consistency and reciprocal linkage. If logical volume metadata for the allocation unit of the data in volatile memory is not consistent with logical volume metadata for the allocation unit in non-volatile storage, then the allocation unit is recovered from the source device. If target device metadata for the data in volatile memory is not consistent with target device metadata for the data in non-volatile storage, then the allocation unit is recovered from the source device. If linkage between logical volume metadata for the allocation unit and target device metadata for the data is not reciprocal, then the allocation unit is recovered from the source device.

Abstract: A method and system for sharing data reduction metadata with storage systems. Specifically, the disclosed method and system entail communicating, to a storage system, information known to host devices from which data (submitted to-be-written to the storage system) may originate. This a priori reduction-pertinent information, which may include the potential to improve storage system efficiency and/or performance at least with respect to data reduction processing of the data submitted to-be-written, had previously been considered incommunicable to the storage system. The disclosed method and system, however, lift this previous limitation and enable communication of any storage system performance-improving information, applicable to the data submitted to-be-written, to the storage system.

Abstract: A hardware offload includes a hash engine that performs hashing for a block-based storage system. The hash engine calculates multiple hash values for each input buffer provided by the storage system. The hash values may be calculated with variably offset and overlapping portions of the input buffer, wherein each portion is larger than the native block size of the storage system. The hardware offload may also include a compression engine that performs compression on the input buffer using the entire input buffer and/or chunks as compression domains.

Abstract: Compressing data of a storage device includes selecting a portion of data for data compression that is predicted to be unlikely to be accessed, selecting a particular one of a plurality of data compression algorithms to apply to the portion according to a frequency value associated with each of the data compression algorithms, and adjusting the frequency value of the particular one of a plurality of data compression algorithms according to performance of the particular one of a plurality of data compression algorithms. The performance may vary according to a ratio of amount of compression achieved to processing cycles of a processor used to compress the portion of data. The processor may perform storage device functions that are separate from data compression. The portion of data may be selected from a plurality of logical devices used in connection with a single application that accesses the storage device.

Abstract: Techniques are described for performing data storage optimization. A first I/O workload for a first data portion of a first snapshot of a first logical device is tracked. First processing is performed by a data storage optimizer to determine a set of one or more data movement optimizations. The first processing uses the first I/O workload for the first snapshot. The set of one or more data movement optimizations include a first data movement that is any of a promotion to move data included in the first data portion from a first storage tier to a higher performance storage tier and a demotion to move data included in the first data portion from the first storage tier to a lower performance storage tier. The first data movement is performed.

Abstract: Described are techniques for obtaining configuration information and conditionally executing a system call in accordance with a specified configuration state. A host issues a request for configuration information from a data storage system. The data storage system maintains a separate table of configuration information representing a configuration state of the data storage system. The host receives a response including a custom value indicating the current configuration state. The host may issue a request to the data storage system to conditionally execute a call if the data storage system is in a configuration state corresponding to the custom value.

Abstract: A computer system may include one or more hosts and a plurality of data storage devices for providing multihop system calls. The data storage devices are interconnected and also connected to the one or more hosts. The connections may be direct or indirect. Each data storage device classifies a data operation as a system call, a remote system call, or a multihop system call. If an operation is classified as a multihop system call by a first data storage device, a portion of the data associated with the multihop system call is removed and a second portion of the data is forwarded to a second data storage device as specified by a connection identifier included in the first portion. A multi-execute command may be issued by a host system causing one or more commands to be executed on one or more processors at points in a communication path in a forward and/or reverse direction when traversing the communication path.

Abstract: Controlling access to a storage device includes defining a plurality of groups that access the storage device, defining a plurality of pools of devices of the storage device, and, for at least one of the groups, determining access rights with respect to at least one of the pools. The pools of devices may include communication ports of the storage device. The access rights may indicate whether system calls are allowed on the communication ports. In some embodiments, restricting access to a storage device includes coupling each of a plurality of host systems to the storage device by one of a plurality of ports provided for the storage device and selectively determining, for each of the ports, whether system calls are allowed, where, for the ports in which system calls are not allowed, a system call by the host systems coupled thereto causes the storage device to indicate that the system call was not performed.