Abstract: Techniques are provided for caching data during an on-demand restore using a cloud block map. A client may be provided with access to an on-demand volume during a restore process that copies backup data from a snapshot within a remote object store to the on-demand volume stored within local storage. In response to receiving a request from the client for a block of the backup data not yet restored from the snapshot to the on-demand volume, the block may be retrieved from the snapshot in the remote object store. The block may be cached within a cloud block map stored within the local storage as a cached block. The client may be provided with access to the cached block.

Abstract: Systems and methods for automated tuning of Quality of Service (QoS) settings of volumes in a distributed storage system are provided. According to one embodiment, one or more characteristics of a workload of a client to which a storage node of multiple storage nodes of the distributed storage system is exposed are monitored. After a determination has been made that a characteristic meets or exceeds a threshold, (i) information regarding multiple QoS settings assigned to a volume of the storage node utilized by the client is obtained, (ii) a new value of a burst IOPS setting of the multiple QoS settings is calculated by increasing a current value of the burst IOPS setting by a factor dependent upon a first and a second QoS setting of the multiple QoS settings, and (iii) the new value of the burst IOPS setting is assigned to the volume for the client.

Abstract: Approaches for setting file attributes in a distributed file system using a multipart file structure are described. A request to set attributes for one or more parts of a multipart file is received. In response to the request, a rectify indicator is set to indicate the attributes for the multipart file that are to be set. In response to the request, an entry corresponding to the request is created in a rectify database. The attributes for the one or more parts of the multipart files are set using at least the entry in the rectify database.

Abstract: Approaches for providing a non-disruptive file move are disclosed. A request to move a target file from the first constituent to the second constituent is received. The file has an associated file handle. The target file in the first constituent is converted to a multipart file in the first constituent with a file location for the new file in the first constituent. A new file is created in the second constituent. Contents of the target file are moved to a new file on the second constituent while maintaining access via the associated file handle via access to the multipart file. The target file is deleted from the first constituent.

Abstract: Systems and methods for preserving storage efficiency during restoration of data from the cloud are provided. In one embodiment, a CBMAP is maintained that maps cloud block numbers (CBNs) to respective corresponding block numbers of a volume of a data storage system in which previously restored data has been stored by a previously restored file. By making use of the CBMAP during the restoration process, storage of duplicate file data blocks on the volume may be avoided by sharing with a current file being restored a reference to the corresponding file data block previously stored on the volume and associated with the previously restored file. In addition to preserving storage efficiency, use of the CBMAP facilitates avoidance of repeated GET operations for data associated with CBNs previously retrieved from the cloud and stored to the volume, thereby reducing data access costs as well as latency of the restore operation.

Abstract: Systems and methods for reducing read application in a virtual storage system are provided. According to one embodiment, heuristic data may be tracked and utilized in real-time by a file system of the virtual storage system at the level of granularity of a volume, thereby allowing a fast path flag to be enabled/disabled at a volume level during various phases of operation of a workload. The heuristic data for a given volume may be indicative of a correlation between (i) data blocks stored on the given volume being located within a compressible zone of a zoned checksum scheme and (ii) the respective data blocks containing compressed data and a corresponding checksum. Based on the heuristic data, read requests may be selectively directed to the read path (e.g., a fast path or a slow path) expected to mitigate read amplification when data compression is enabled for a zoned checksum scheme.

Abstract: Techniques are provided for multi-tier write allocation. A storage system may store data within a multi-tier storage environment comprising a first storage tier (e.g., storage devices maintained by the storage system), a second storage tier (e.g., a remote object store provided by a third party storage provider), and/or other storage tiers. A determination is made that data (e.g., data of a write request received by the storage system) is to be stored within the second storage tier. The data is stored into a staging area of the first storage tier. A second storage tier location identifier, for referencing the data according to a format utilized by the second storage tier, is assigned to the data and provided to a file system hosting the data. The data is then destaged from the staging area into the second storage tier, such as within an object stored within the remote object store.

Abstract: Systems and methods for supporting dynamic disk growth within a virtual storage appliance are provided. According to one embodiment, a portion of a logical size of respective hyperscale disks provided by a hyperscaler are provisioned for use by a virtual storage system as backing for respective file system disks. To accommodate growth, block numbers for the file system disks are pre-allocated within a sparse space of a contiguous sequence of block numbers corresponding to a number of blocks represented by the logical size. Metadata is maintained for the file system disks regarding a range of the pre-allocated block numbers that are available for use. Responsive to a triggering condition, the provisioned portion of a hyperscale disk is increased and subsequently, responsive to detecting a change in a size of the hyperscale disk by the virtual storage system, a size of the corresponding file system disk is updated within the metadata.

Abstract: Techniques are provided for dynamically implementing quality of service policies using a configurable quality of service provider pipeline. A quality of service policy is defined for throttling I/O operations received by a node based upon whether resources of the node have become over utilized. The quality of service policy is used to dynamically construct a quality of service provider pipeline with select quality of service providers that improve the ability to efficiently utilize resources compared to conventional static polices that cannot adequately react to changing considerations and resource utilization/saturation. With conventional static policies, an administrator manually defines a minimum amount of guaranteed resources and/or a maximum resource usage cap that could be set to values that result in inefficient operation and resource starvation. Dynamically constructing and utilizing the quality of service provider pipeline results in more efficient operation and mitigates resource starvation.

Abstract: Methods, non-transitory machine readable media, and computing devices that dynamically throttle non-priority workloads to satisfy minimum throughput service level objectives (SLOs) are disclosed. With this technology, a determination is made when a number of detection intervals with a violation within a detection window exceeds a threshold, when a current one of the detection intervals is outside an observation area. The detection intervals are identified a violated based on an average throughput for priority workloads within the detection intervals exceeding a minimum throughput SLO. A throttle is then set to rate-limit non-priority workloads, when the number of violated detection intervals within the detection window exceeds the threshold.

Abstract: Techniques are provided for compacting indirect blocks. For example, an object is represented as a structure including data blocks within which data of the object is stored and indirect blocks including block numbers of where the data blocks are located in storage. Block numbers within a set of indirect blocks are compacted into a compacted indirect block including a base block number, a count of additional block numbers after the base block number in the compacted indirect block, and a pattern of the block numbers in the compacted indirect block. The compacted indirect block is stored into memory for processing access operations to the object. Storing compacted indirect blocks into memory allows for more block numbers to be stored within memory.

Abstract: Systems and methods include negotiating a primary bias state for primary and secondary storage sites when a mediator is temporarily unavailable for a multi-site distributed storage system. In one example, a computer-implemented method comprises detecting, with the primary storage site having a primary storage cluster, a temporary loss of connectivity to a mediator or a failure of the mediator. The computer-implemented method includes negotiating the primary bias state and setting the primary bias state on a secondary storage cluster of the secondary storage site when the secondary storage cluster detects a temporary loss of connectivity to the mediator, determining whether the primary storage cluster receives a confirmation of the secondary storage cluster setting the primary bias state, and setting the primary bias state on the primary storage cluster when the primary storage cluster receives the confirmation.

Abstract: According to an example, a computer-implemented method comprises initiating a first process for atomically setting the primary bias state with a first node of a primary storage cluster of a multi-site distributed storage system due to a temporary loss of connectivity to a mediator or a temporary mediator failure, releasing an atomic lock for the first process on the first node of the primary storage cluster, sending the first process and an associated first generation indicator to a first node of a secondary storage cluster of the multi-site distributed storage system to handle the first process for setting the primary bias state, and initiating a second process for atomically clearing a primary bias state with the first node or any node of the primary storage cluster based on detecting a connection to the mediator or detecting that the mediator is available.

Abstract: Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time for processing of input/output (I/O) operations during an automatic unplanned failover (AUFO). A first storage cluster includes a first set of consistency groups (CGs) and a second storage cluster includes a second mirrored set of CGs. A computer-implemented method includes prefetching, with a user space of the second storage cluster, configuration information from a replicated database prior to starting the AUFO workflow, sending the configuration information to a kernel space of the second storage cluster on a per CG level while queuing the AUFO workflow, and determining if any in progress workflows conflict with the AUFO workflow.

Abstract: One or more techniques and/or computing devices are provided for replicating virtual machine disk clones. For example, a first storage controller, hosting first storage, may have a synchronous replication relationship with a second storage controller hosting second storage. A virtual machine, within the first storage, may be specified as having synchronous replication protection. Accordingly, virtual machine disk clones of a virtual machine disk of the virtual machine may be replicated from the first storage to the second storage. For example, virtual machine disk clones may be synchronous replicated, replicated by a resync process invoked by a hypervisor agent, and/or stored and replicated from a clone backup directory.

Abstract: Approaches and mechanisms for cloning a file are described. A first node requests a clone of a file at a time when it also requests an exclusive delegation of the original file from a second node where the original file is stored. The second node marks the original file as delegated to the first node and the second node records an intent to create the clone file and a delegation record for the clone file. The second node creates the clone file. The delegation of and the identity of the clone file are returned to the first node. The first node marks in the delegation record that the clone file was committed in response to modification. If the clone file was committed the delegation is cleared and the clone file is kept, and if the clone file was not committed, the delegation is cleared, and the clone file is deleted.

Abstract: One or more techniques and/or computing devices are provided for resilient replication of storage operations. For example, a first storage controller may host first storage having a replication relationship with second storage hosted by a second storage controller. To improve resiliency against transient network issues of a network between the storage controllers, the first storage controller may implement a queue and retry mechanism to retry replication operations not acknowledge back by the second storage controller within a threshold time. The second storage controller may maintain a cumulative sequence number of a latest replication operation performed in order, an operation response map of replication operations performed out of order, and an operation finder map identifying currently implemented replication operations, which may be used to process incoming replication operations. Single write semantics, write order consistency, and reduction of write amplification may be provided.

Abstract: Systems and methods for sampling a set of block IDs to facilitate estimating an amount of data stored in a data set of a storage system having one or more characteristics are provided. According to an example, metadata (e.g., block headers and block IDs) may be maintained regarding multiple data blocks of the data set. When one or more metrics relating to the data set are desired, an efficiency set, representing a subset of the block IDs of the data set, may be created to facilitate efficient calculation of the metrics by sampling the block IDs of the data set. Finally, the metrics may be estimated based on the efficiency set by analyzing one or more of the metadata (e.g., block headers) and the data contained in the data blocks corresponding to the subset of the block IDs and extrapolating the metrics for the entirety of the data set.

Abstract: The technology disclosed herein enables access to a file system by a portable executable program. In a particular example, a method includes, in a host executing the portable executable program, recognizing the portable executable program is executing on the one or more processing systems and determining the portable executable program is configured to access the file system. The method also includes directing the portable executable program to create a module therein for file system access and creating an abstraction layer with which the module exchanges file system commands. In the abstraction layer, the method includes translating the file system commands to translated commands for the file system and exchanging translated commands between the abstraction layer and the file system.

Abstract: Presented herein are methods, non-transitory computer readable media, and devices for optimizing thread assignment to schedulers, avoid starvation of individual data partitions, and maximize parallelism in the presence of hierarchical data partitioning are disclosed, which include: partitioning, by a network storage server, a scheduler servicing a data partitioned system into a plurality of autonomous schedulers; determining what fraction of thread resources in the data partitioned system at least one of the plurality of autonomous schedulers is to receive; and determining, with minimal synchronization, when it is time to allow the at least one of the plurality of autonomous schedulers servicing a coarse hierarchy to run.