Abstract: A system, method, and machine-readable storage medium for providing a quality of service (QoS) recommendation to a client to modify a QoS setting are provided. In some embodiments, a set of volumes of a plurality of volumes may be determined. Each volume of the set of volumes may satisfy a first QoS setting assigned to the volume and a second QoS setting assigned to the volume. The plurality of volumes may reside in a common cluster and may be accessed by the client. Additionally, a subset of the set of volumes may be determined. Each volume of the subset may satisfy an upper bound of a range based on a minimum IOPS setting of the volume. A QoS recommendation to the client to modify the first QoS setting may be transmitted for one or more volumes of the subset.

Abstract: A request to generate a storage system model is received. The storage system model represents at least a portion of a storage system. In response to receiving the request, a storage system interface configuration is loaded. The storage system interface configuration comprises an attribute of an entity model. The attribute corresponds to an attribute of a storage system entity of the storage system. Further in response to receiving the request, the entity model is identified as representing the storage system entity. In response to identifying the entity model as representing the storage system entity, the entity model is instantiated.

Abstract: Systems and methods are described for a non-disruptive planned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system. According to an example, a planned failover feature of a multi-site distributed storage system provides an order of operations such that a primary copy of a first data center continues to serve I/O operations until a mirror copy of a second data center is ready. This planned failover feature improves functionality and efficiency of the distributed storage system by providing non-disruptiveness during planned failover—even if various failures occur. The planned failover feature also includes a persistent fence to avoid serving I/O operations during a timing window when both primary data storage and secondary data storage are attempting to have a master role to serve I/O operations and this avoids a split-brain situation.

Abstract: Systems and methods are described for using a Deep Reinforcement Learning (DRL) agent to automatically tune Quality of Service (QoS) settings of a distributed storage system (DSS). According to one embodiment, a DRL agent is trained in a simulated environment to select QoS settings (e.g., a value of one or more of a minimum IOPS parameter, a maximum IOPS parameter, and a burst IOPS parameter). The training may involve placing the DRL agent into every feasible state representing combinations of QoS settings, workload conditions, and system metrics for a period of time for multiple iterations, and rewarding the DRL agent for selecting QoS settings that minimize an objective function based on a selected measure of system load. The trained DRL agent may then be deployed to one or more DSSs to constantly update QoS settings so as to minimize the selected measure of system load.

Abstract: Systems and methods for facilitating performance of data tiering by detecting and persisting types of backing storage of cloud volumes that are available for use by a virtual storage system are provided. In one example, during a boot phase of the virtual storage system within a compute instance of a cloud environment, information regarding a type of backing storage for each of multiple storage devices associated with the compute instance that is not available via hypervisor APIs is obtained via an API exposed by the cloud environment. The multiple storage devices may then be initialized to facilitate subsequent use of a subset thereof in connection with performing data tiering by persisting information indicative of the type of backing storage for the storage device within a reserved header region on each storage device.

Abstract: Systems and methods for managing data storage using a distributed file system are provided. In one example, a file system instance is deployed virtually in a node of a distributed storage system. The file system instance has a dynamic configuration including a set of services corresponding to a cluster management subsystem and a storage management subsystem. The storage management subsystem operates independently of a data management subsystem of the distributed storage system as a result of disaggregation from the data management subsystem. The data management subsystem performs storage and block management functions based on requests received from an application layer. An additional service corresponding to either the data management subsystem or the storage management subsystem is deployed virtually to meet the demand for the additional service in response to determining the presence of a demand for the additional service and availability a set of resources corresponding to the additional service.

Abstract: One or more techniques and/or computing devices are provided for managing an arbitrary set of storage items using a granset. For example, a storage controller may host a plurality of storage items and/or logical unit numbers (LUNs). A subset of the storage items are grouped into a consistency group. A granset is created for tracking, managing, and/or providing access to the storage items within the consistency group. For example, the granset comprises application programming interfaces (APIs) and/or properties used to provide certain levels of access to the storage items (e.g., read access, write access, no access), redirect operations to access either data of an active file system or to a snapshot, fence certain operations (e.g., rename and delete operations), and/or other properties that apply to each storage item within the consistency group. Thus, the granset provides a persistent on-disk layout used to manage an arbitrary set of storage items.

Abstract: Techniques are provided for synchronous replication based cutover. An asynchronous replication process is executed to perform asynchronous incremental transfers of data of a storage object from a first computing environment to a replicated storage object at a second computing environment until a cutover criteria is met. A synchronous replication process is executed to synchronously replicate operations, targeting the storage object, to the replicated storage object based upon the cutover criteria being met. A cutover is performed to direct operations from targeting the storage object to targeting the replicated storage object based upon the synchronous replication process reaching a steady state of synchronous replication for sub-objects of the storage object, where operations are committed to both the storage object and the replicated storage object.

Abstract: Techniques are provided for block allocation for persistent memory during aggregate transition. In a high availability pair including first and second nodes, the first node makes a determination that control of a first aggregate is to transition from the first node to the second node. A portion of available free storage space is allocated from a first persistent memory of the first node as allocated pages within the first persistent memory. Metadata information for the allocated pages is updated with an identifier of the first aggregate to create updated metadata information reserving the allocated pages for the first aggregate. The updated metadata information is mirrored to the second node, so that the second node also reserves those pages. Control of the first aggregate is transitioned to the second node. As a result, the nodes do not attempt allocating the same free pages to different aggregates during a transition.

Abstract: Techniques are provided for offloading the management of sensor data and generating custom views of sensor data. Sensor data received from a data network through a message is stored within storage managed by a computing device. A handle is generated to identify the sensor data. The sensor data within the message is replaced with the handle, and the message is transmitted to a device within the data network. The device may use handles of sensor data to request custom views of sensor data.

Abstract: A volume rehost tool migrates a storage volume from a source virtual server within a distributed storage system to a destination storage server within the distributed storage system. The volume rehost tool can prevent client access to data on the volume through the source virtual server until the volume has been migrated to the destination virtual server. The tool identifies a set of storage objects associated with the volume, removes configuration information for the set of storage objects, and removes a volume record associated with the source virtual server for the volume. The tool can then create a new volume record associated with the destination virtual server, apply the configuration information for the set of storage objects to the destination virtual server, and allow client access to the data on the volume through the destination virtual server.

Abstract: Systems and methods that make use of cluster-level redundancy within a distributed storage management system to address various node-level error scenarios are provided. According to one embodiment, a KV store of a node of a cluster of a distributed storage management system manages storage of data blocks as values and corresponding block IDs as keys. Data integrity errors are reported to the first node in the form of a list of missing block IDs that are in use but missing from the KV store. A metadata resynchronization process may then be caused to be performed, including for each block ID in the list of missing block IDs: (i) reading a data block corresponding to the block ID from another node of the cluster that maintains redundant information relating to the block ID; and (ii) restoring the block ID within the KV store by writing the data block to the node.

Abstract: Systems and methods for reducing delays between the time at which a need for a resynchronization of data replication between a volume of a local CG and its peer volume of a remote CG is detected and the time at which the resynchronization is triggered (Reseed Time Period) are provided. According to an example, information indicative of the direction of data replication between the volume and the peer volume is maintained within a cache of a node. Responsive to a disruptive operation (e.g., relocation of the volume from a first node to a second node), the Reseed Time Period is lessened by proactively adding a passive cache entry to a cache within the second node at the time the CG relationship is created when the second node represents an HA partner of the first node and prior to the volume coming online when the second node represents a non-HA partner.

Abstract: One or more techniques and/or computing devices are provided for utilizing snapshots for data integrity validation and/or faster application recovery. For example, a first storage controller, hosting first storage, has a synchronous replication relationship with a second storage controller hosting second storage. A snapshot replication policy rule is defined to specify that a replication label is to be used for snapshot create requests, targeting the first storage, that are to be replicated to the second storage. A snapshot creation policy is created to issue snapshot create requests comprising the replication label. Thus a snapshot of the first storage and a replication snapshot of the second storage are created based upon a snapshot create request comprising the replication label. The snapshot and the replication snapshot may be compared for data integrity validation (e.g., determine whether the snapshots comprise the same data) and/or quickly recovering an application after a disaster.

Abstract: Methods, non-transitory machine readable media, and computing devices that dynamically throttle non-priority workloads to satisfy minimum throughput service level objectives (SLOs) are disclosed. With this technology, a determination is made when a number of detection intervals with a violation within a detection window exceeds a threshold, when a current one of the detection intervals is outside an observation area. The detection intervals are identified a violated based on an average throughput for priority workloads within the detection intervals exceeding a minimum throughput SLO. A throttle is then set to rate-limit non-priority workloads, when the number of violated detection intervals within the detection window exceeds the threshold.

Abstract: Techniques are provided for utilizing a log to free pages from persistent memory. A log is maintained to comprise a list of page block numbers of pages within persistent memory of a node to free. A page block number, of a page, within the log is identified for processing. A reference count, corresponding to a number of references to the page block number, is identified. In response to the reference count being greater than 1, the reference count is decremented and the page block number is removed from the log. In response to the reference count being 1, the page is freed from the persistent memory and the page block number is removed from the log.

Abstract: Techniques are provided for implementing write ordering for persistent memory. A set of actions are identified for commitment to persistent memory of a node for executing an operation upon the persistent memory. An episode is created to comprise a first subset of actions of the set of actions that can be committed to the persistent memory in any order with respect to one another such that a consistent state of the persistent memory can be reconstructed in the event of a crash of the node during execution of the operation. The first subset of actions within the episode are committed to the persistent memory and further execution of the operation is blocked until the episode completes.

Abstract: Systems and methods for managing data storage using a distributed file system are provided. In one example, a relocation event is detected that indicates a relocation is to be initialized. The relocation is initialized by identifying a destination node of a distributed storage system for the relocation of a set of objects in a cluster database, including a logical block device, a corresponding logical aggregate, and a corresponding file system volume. A state of each of the set of objects is changed to offline. The set of objects are then relocated from an originating node of the distributed storage system to the destination node in which the corresponding logical aggregate is relocated after the logical block device and the corresponding file system volume is relocated after the logical aggregate. Finally, the state of each of the set of objects is changed to online.

Abstract: Systems and methods for making use of non-persistent storage as the journaling storage media for a virtual storage system are provided. According to one embodiment, in order to meet the needs of Extreme Low Latency Workloads while also seeking to provide predictable performance and the lowest possible latency, ephemeral storage of the virtual storage system is used to preserve state information (e.g., in the form of boot arguments and an operation log journal) across a host failure recovery scenario in which the virtual storage system is expected to be redeployed within a compute instance brought up by a cloud environment of a hyperscaler on the same host, thereby providing improved data durability (fewer host failure scenarios that result in lost data) as compared to the use of ephemeral memory of the compute instance and lower write latency than the use of persistent storage provided by the cloud environment.

Abstract: In various examples, data storage is managed using a distributed storage management system that is resilient. Data blocks of a logical block device may be distributed across multiple nodes in a cluster. The logical block device may correspond to a file system volume associated with a file system instance deployed on a selected node within a distributed block layer of a distributed file system. Each data block may have a location in the cluster identified by a block identifier associated with each data block. Each data block may be replicated on at least one other node in the cluster. A metadata object corresponding to a logical block device that maps to the file system volume may be replicated on at least another node in the cluster. Each data block and the metadata object may be hosted on virtualized storage that is protected using redundant array independent disks (RAID).