Abstract: Techniques lock a storage area in a storage system including a first storage node and a second storage node. Such techniques involve adjusting a value of at least one of a first counter at the first node and a second counter at the second node in response to receiving a data access request for the storage area at at least one of the first node and the second node. Such techniques further involve allocating a lock flag to the first node in response to a ratio of the value of the first counter to the value of the second counter being greater than a first threshold, the lock flag indicating that the storage area is locked by the first node. Accordingly, storage resources required by nodes that are accessed more frequently in statistics are effectively reduced, and the efficiency of applying stripe lock reduction are improved.

Abstract: Techniques for managing disks involve: determining, from multiple disks, a first disk set for providing redundant storage, the number of disks included in the first disk set being not less than a first threshold number that corresponds to the number of disks required to be included in a redundant array of independent disks; determining, based on the multiple disks, a candidate solution in which at least the first disk set is used to provide redundant storage; determining an unavailable capacity that the candidate solution will have; and determining, based on the unavailable capacity and the candidate solution, a target solution for providing the redundant storage. Accordingly, such techniques can promote efficient management of disks.

Abstract: Data units of a dataset may be compressed by clustering the data units into clusters, selecting a reference unit for each unit cluster, and compressing data units of each unit cluster using the reference unit of the unit cluster as a dictionary. The computational efficiency of the clustering algorithm may be improved by not applying it to data units themselves, but rather to hash values of the data units, where the hash values have a much smaller size than the data units. The hash function may be a locality-sensitive hash (LSH) function. The reference unit of a cluster may be determined in any of a variety of ways, for example, by selecting a centroid or exemplar of the cluster. Clusters, including their references values, may be indexed in a cluster index (e.g., a Faiss index), which may be searched to assign future added or modified data units to clusters.

Abstract: Techniques for managing storage blocks involve: according to a first request for allocating a target number of consecutive storage blocks from a storage device, querying an index table that is used to record index information of a plurality of groups of consecutive idle storage blocks in the storage device to determine a first group of consecutive idle storage blocks from the plurality of groups of consecutive idle storage blocks; allocating the target number of consecutive storage blocks from the first group of consecutive idle storage blocks; updating a bitmap that is used to record states of a plurality of storage blocks in the storage device; and updating first index information of the first group of consecutive idle storage blocks recorded in the index table. Accordingly, such techniques may reduce storage complexity and improve system performance.

Abstract: Techniques for managing disks involve: determining, from multiple disks, a first disk set for providing redundant storage, the number of disks included in the first disk set being not less than a first threshold number that corresponds to the number of disks required to be included in a redundant array of independent disks; determining, based on the multiple disks, a candidate solution in which at least the first disk set is used to provide redundant storage; determining an unavailable capacity that the candidate solution will have; and determining, based on the unavailable capacity and the candidate solution, a target solution for providing the redundant storage. Accordingly, such techniques can promote efficient management of disks.

Abstract: Managing pool memory in a data storage system includes maintaining free lists for corresponding object sizes. For a memory-consuming request (e.g., host write) an allocation operation is performed and the request data is stored. The allocation operation includes (1) selecting a memory object at least as large as the request size and removing all pages of the selected memory object from the corresponding free list, and (2) selecting pages of the selected memory object to store the request data and marking the selected pages as non-free, and leaving any leftover pages as free pages. For a memory-freeing request (e.g., destaging), a deallocation operation is performed that includes (1) marking the request pages free, and (2) based on neighboring pages being free, merging the request pages and neighboring pages into a corresponding memory object and adding the merged pages to the corresponding free list.

Abstract: Techniques for managing storage blocks involve: according to a first request for allocating a target number of consecutive storage blocks from a storage device, querying an index table that is used to record index information of a plurality of groups of consecutive idle storage blocks in the storage device to determine a first group of consecutive idle storage blocks from the plurality of groups of consecutive idle storage blocks; allocating the target number of consecutive storage blocks from the first group of consecutive idle storage blocks; updating a bitmap that is used to record states of a plurality of storage blocks in the storage device; and updating first index information of the first group of consecutive idle storage blocks recorded in the index table. Accordingly, such techniques may reduce storage complexity and improve system performance.

Abstract: Managing pool memory in a data storage system includes maintaining free lists for corresponding object sizes. For a memory-consuming request (e.g., host write) an allocation operation is performed and the request data is stored. The allocation operation includes (1) selecting a memory object at least as large as the request size and removing all pages of the selected memory object from the corresponding free list, and (2) selecting pages of the selected memory object to store the request data and marking the selected pages as non-free, and leaving any leftover pages as free pages. For a memory-freeing request (e.g., destaging), a deallocation operation is performed that includes (1) marking the request pages free, and (2) based on neighboring pages being free, merging the request pages and neighboring pages into a corresponding memory object and adding the merged pages to the corresponding free list.

Abstract: Data units of a dataset may be compressed by clustering the data units into clusters, selecting a reference unit for each unit cluster, and compressing data units of each unit cluster using the reference unit of the unit cluster as a dictionary. The computational efficiency of the clustering algorithm may be improved by not applying it to data units themselves, but rather to hash values of the data units, where the hash values have a much smaller size than the data units. The hash function may be a locality-sensitive hash (LSH) function. The reference unit of a cluster may be determined in any of a variety of ways, for example, by selecting a centroid or exemplar of the cluster. Clusters, including their references values, may be indexed in a cluster index (e.g., a Faiss index), which may be searched to assign future added or modified data units to clusters.

Abstract: Methods and apparatus are provided for simulating asynchronous modifications of shared data objects by a number of distributed processes within a distributed application. An exemplary method comprises obtaining a plurality of simulation parameters that define parameters of a simulated distributed application, parameters of a plurality of distributed compute nodes on which the simulated distributed application executes and a storage system employed by the simulated distributed application; and applying the plurality of simulation parameters to a simulation engine, wherein the simulation engine simulates both synchronous and asynchronous modifications of at least one shared data object by a plurality of distributed processes within the simulated distributed application.

Abstract: An apparatus in one embodiment comprises a storage system having at least first and second storage tiers each comprising a plurality of storage devices. The storage system implements a plurality of data mover modules for controlling movement of data objects between the first and second storage tiers. A given one of the data objects is transferred between the first and second storage tiers utilizing a distributed multi-part data movement protocol in which different portions of the given data object are transferred between the first and second storage tiers by different ones of the data mover modules. For example, each of the different portions may be transferred between the first and second storage tiers by its corresponding data mover module as a plurality of distinct non-overlapping parts. In such an arrangement, each of the plurality of distinct non-overlapping parts of each of the different portions may have a corresponding unique tag identifier.

Abstract: A cluster file system comprises a parallel file system and a plurality of burst buffers, with the burst buffers collectively configured to provide an interface between the parallel file system and compute nodes of a computer system. Each of the burst buffers comprises a virtual file system component and a metadata server component. The virtual file system components of respective ones of the burst buffers collectively implement a virtual file system. Metadata relating portions of the virtual file system to portions of the parallel file system is stored in a distributed manner over the metadata server components of the respective burst buffers. Each burst buffer independently manages its own flash memory, but the burst buffers are configured such that data consistency is ensured across their flash memories. The virtual file system illustratively comprises a parallel log-structured file system (PLFS) and the metadata server components illustratively comprise respective TO forwarding servers.

Abstract: Embodiments of the present disclosure relate to a method and apparatus for accessing data by receiving a data read request for reading data corresponding to a logical block number; determining a first physical block corresponding to the first logical block number on the disk when a first cache page corresponding to the first logical block number does not exist in a cache; and reading data in the second cache when a second cache corresponding to the second physical block exists in the cache, wherein the content of the second physical block is identical to the content of the first physical block. Some embodiments of the present disclosure may prevent storing mass redundant data, and thereby enhance data reading rate.