Abstract: Systems and methods for preserving storage efficiency during restoration of data from the cloud are provided. In one embodiment, a CBMAP is maintained that maps cloud block numbers (CBNs) to respective corresponding block numbers of a volume of a data storage system in which previously restored data has been stored by a previously restored file. By making use of the CBMAP during the restoration process, storage of duplicate file data blocks on the volume may be avoided by sharing with a current file being restored a reference to the corresponding file data block previously stored on the volume and associated with the previously restored file. In addition to preserving storage efficiency, use of the CBMAP facilitates avoidance of repeated GET operations for data associated with CBNs previously retrieved from the cloud and stored to the volume, thereby reducing data access costs as well as latency of the restore operation.

Abstract: Systems, methods, and machine-readable media for monitoring a storage system and assigning performance service levels to workloads running on nodes within a cluster are disclosed. A performance manager may estimate the performance demands of each workload within the cluster and assign a performance service level to each workload according to the performance requirements of the workload, and further taking into account an overall budgeting framework. The estimates are performed using historical performance data for each workload. A performance service level may include a service level object, a service level agreement, and latency parameters. These parameters may provide a ceiling to the number of operations per second that a workload may use without guaranteeing the use of the operations per second, a guaranteed number of operations per second that a workload may use before being throttled, and define the permitted delay in completing a request to the workload.

Abstract: The technology disclosed herein enables accelerated data transmission between producers and consumers. In a particular example, a method includes receiving a first request from a producer-connector component of a producer component to store a payload to a storage repository. In response to the first request, the method includes providing a unique identifier to the connector component. The connector component provides the unique identifier to the distributed-clustered application. The method further includes storing the payload in association with the unique identifier to the storage repository. The method also includes retrieving the payload from the storage repository using the unique identifier to identify the payload in the storage repository. The method includes receiving a second request from a consumer-connector component of the consumer component to retrieve the payload. In response to the second request, the method includes supplying the payload to the consumer component.

Abstract: Systems and methods for multiple device consumption of shared namespaces of ephemeral storage devices by a consumer of a virtual storage system are provided. In an example, multiple namespaces of respective ephemeral storage devices are shared among multiple of consumers of a virtual storage system by creating multiple partitions within each of the namespaces for use by respective consumers of the multiple consumers. Corresponding partitions of respective shared namespace may then be treated as a stripe set to facilitate multiple device consumption for a subsystem (e.g., operation log journaling) of the virtual storage system by striping data associated with input/output (I/O) requests of a consumer (e.g., a journaling driver) across one or more stripe units of one or more stripes within the stripe set.

Abstract: Systems and methods for sharing a namespace of an ephemeral storage device by multiple consumers are provided. In an example, an NVMe driver of a virtual storage system deployed within a compute instance of a cloud environment facilitates sharing of the namespace by exposing an API through which the multiple consumers access an ephemeral storage device associated with the compute instance. During initialization processing performed by each consumer, for example, during boot processing of the virtual storage system, the consumers may share the namespace by reserving for their own use respective partitions within the namespace via the API and thereafter restrict their usage of the namespace to their respective partitions, thereby retaining the functionality provided by the multiple consumers when the host on which the compute instance is deployed has fewer ephemeral storage devices than consumers that rely on the availability of vNVRAM backed by ephemeral storage.

Abstract: Intelligent self-adjusting metric collection is described. A first rule set is distributed that describes a first set of one or more metrics corresponding to operation of elements of the receiving entities. One or more metrics based on the first rule set are received. A second rule set is generated in response to an indication of a condition change. The second rule set can be generated using machine learning techniques. The second rule set that describes a second set of one or more metrics is distributed. Metrics based on the second rule set are received.

Abstract: Systems and methods for automated tuning of Quality of Service (QoS) settings of volumes in a distributed storage system are provided. According to one embodiment, one or more characteristics of a workload of a client to which a storage node of multiple storage nodes of the distributed storage system is exposed are monitored. After a determination has been made that a characteristic meets or exceeds a threshold, (i) information regarding multiple QoS settings assigned to a volume of the storage node utilized by the client is obtained, (ii) a new value of a burst IOPS setting of the multiple QoS settings is calculated by increasing a current value of the burst IOPS setting by a factor dependent upon a first and a second QoS setting of the multiple QoS settings, and (iii) the new value of the burst IOPS setting is assigned to the volume for the client.

Abstract: Techniques are provided for caching data during an on-demand restore using a cloud block map. A client may be provided with access to an on-demand volume during a restore process that copies backup data from a snapshot within a remote object store to the on-demand volume stored within local storage. In response to receiving a request from the client for a block of the backup data not yet restored from the snapshot to the on-demand volume, the block may be retrieved from the snapshot in the remote object store. The block may be cached within a cloud block map stored within the local storage as a cached block. The client may be provided with access to the cached block.

Abstract: Techniques are provided for splitting a clone volume from a parent volume whose data is stored within objects of an object store. A transfer map is used to track mappings of selectively created child object identifiers used to subsequently copy the one or more parent objects to create child objects corresponding to the child object identifiers. A consistency point phase is performed. For each child object identifier processed during the consistency point phase, an object state for a corresponding child object is set to a copy pending state. A reverse map is populated with a reverse map entry. The transfer map is traversed to copy the one or more parent objects as the child objects for splitting the clone volume from the parent volume. The reverse map is used to verify that the child objects are successfully created with valid data.

Abstract: Approaches for setting file attributes in a distributed file system using a multipart file structure are described. A request to set attributes for one or more parts of a multipart file is received. In response to the request, a rectify indicator is set to indicate the attributes for the multipart file that are to be set. In response to the request, an entry corresponding to the request is created in a rectify database. The attributes for the one or more parts of the multipart files are set using at least the entry in the rectify database.

Abstract: Systems and methods for preserving storage efficiency during restoration of data from the cloud are provided. In one embodiment, a CBMAP is maintained that maps cloud block numbers (CBNs) to respective corresponding block numbers of a volume of a data storage system in which previously restored data has been stored by a previously restored file. By making use of the CBMAP during the restoration process, storage of duplicate file data blocks on the volume may be avoided by sharing with a current file being restored a reference to the corresponding file data block previously stored on the volume and associated with the previously restored file. In addition to preserving storage efficiency, use of the CBMAP facilitates avoidance of repeated GET operations for data associated with CBNs previously retrieved from the cloud and stored to the volume, thereby reducing data access costs as well as latency of the restore operation.

Abstract: Approaches for providing a non-disruptive file move are disclosed. A request to move a target file from the first constituent to the second constituent is received. The file has an associated file handle. The target file in the first constituent is converted to a multipart file in the first constituent with a file location for the new file in the first constituent. A new file is created in the second constituent. Contents of the target file are moved to a new file on the second constituent while maintaining access via the associated file handle via access to the multipart file. The target file is deleted from the first constituent.

Abstract: Techniques are provided for multi-tier write allocation. A storage system may store data within a multi-tier storage environment comprising a first storage tier (e.g., storage devices maintained by the storage system), a second storage tier (e.g., a remote object store provided by a third party storage provider), and/or other storage tiers. A determination is made that data (e.g., data of a write request received by the storage system) is to be stored within the second storage tier. The data is stored into a staging area of the first storage tier. A second storage tier location identifier, for referencing the data according to a format utilized by the second storage tier, is assigned to the data and provided to a file system hosting the data. The data is then destaged from the staging area into the second storage tier, such as within an object stored within the remote object store.

Abstract: Systems and methods for reducing read application in a virtual storage system are provided. According to one embodiment, heuristic data may be tracked and utilized in real-time by a file system of the virtual storage system at the level of granularity of a volume, thereby allowing a fast path flag to be enabled/disabled at a volume level during various phases of operation of a workload. The heuristic data for a given volume may be indicative of a correlation between (i) data blocks stored on the given volume being located within a compressible zone of a zoned checksum scheme and (ii) the respective data blocks containing compressed data and a corresponding checksum. Based on the heuristic data, read requests may be selectively directed to the read path (e.g., a fast path or a slow path) expected to mitigate read amplification when data compression is enabled for a zoned checksum scheme.

Abstract: Techniques are provided for dynamically implementing quality of service policies using a configurable quality of service provider pipeline. A quality of service policy is defined for throttling I/O operations received by a node based upon whether resources of the node have become over utilized. The quality of service policy is used to dynamically construct a quality of service provider pipeline with select quality of service providers that improve the ability to efficiently utilize resources compared to conventional static polices that cannot adequately react to changing considerations and resource utilization/saturation. With conventional static policies, an administrator manually defines a minimum amount of guaranteed resources and/or a maximum resource usage cap that could be set to values that result in inefficient operation and resource starvation. Dynamically constructing and utilizing the quality of service provider pipeline results in more efficient operation and mitigates resource starvation.

Abstract: Systems and methods for supporting dynamic disk growth within a virtual storage appliance are provided. According to one embodiment, a portion of a logical size of respective hyperscale disks provided by a hyperscaler are provisioned for use by a virtual storage system as backing for respective file system disks. To accommodate growth, block numbers for the file system disks are pre-allocated within a sparse space of a contiguous sequence of block numbers corresponding to a number of blocks represented by the logical size. Metadata is maintained for the file system disks regarding a range of the pre-allocated block numbers that are available for use. Responsive to a triggering condition, the provisioned portion of a hyperscale disk is increased and subsequently, responsive to detecting a change in a size of the hyperscale disk by the virtual storage system, a size of the corresponding file system disk is updated within the metadata.

Abstract: Methods, non-transitory machine readable media, and computing devices that dynamically throttle non-priority workloads to satisfy minimum throughput service level objectives (SLOs) are disclosed. With this technology, a determination is made when a number of detection intervals with a violation within a detection window exceeds a threshold, when a current one of the detection intervals is outside an observation area. The detection intervals are identified a violated based on an average throughput for priority workloads within the detection intervals exceeding a minimum throughput SLO. A throttle is then set to rate-limit non-priority workloads, when the number of violated detection intervals within the detection window exceeds the threshold.

Abstract: Techniques are provided for compacting indirect blocks. For example, an object is represented as a structure including data blocks within which data of the object is stored and indirect blocks including block numbers of where the data blocks are located in storage. Block numbers within a set of indirect blocks are compacted into a compacted indirect block including a base block number, a count of additional block numbers after the base block number in the compacted indirect block, and a pattern of the block numbers in the compacted indirect block. The compacted indirect block is stored into memory for processing access operations to the object. Storing compacted indirect blocks into memory allows for more block numbers to be stored within memory.

Abstract: Systems and methods include negotiating a primary bias state for primary and secondary storage sites when a mediator is temporarily unavailable for a multi-site distributed storage system. In one example, a computer-implemented method comprises detecting, with the primary storage site having a primary storage cluster, a temporary loss of connectivity to a mediator or a failure of the mediator. The computer-implemented method includes negotiating the primary bias state and setting the primary bias state on a secondary storage cluster of the secondary storage site when the secondary storage cluster detects a temporary loss of connectivity to the mediator, determining whether the primary storage cluster receives a confirmation of the secondary storage cluster setting the primary bias state, and setting the primary bias state on the primary storage cluster when the primary storage cluster receives the confirmation.

Abstract: According to an example, a computer-implemented method comprises initiating a first process for atomically setting the primary bias state with a first node of a primary storage cluster of a multi-site distributed storage system due to a temporary loss of connectivity to a mediator or a temporary mediator failure, releasing an atomic lock for the first process on the first node of the primary storage cluster, sending the first process and an associated first generation indicator to a first node of a secondary storage cluster of the multi-site distributed storage system to handle the first process for setting the primary bias state, and initiating a second process for atomically clearing a primary bias state with the first node or any node of the primary storage cluster based on detecting a connection to the mediator or detecting that the mediator is available.