Abstract: Systems and techniques that facilitate compact representation of object store workloads are provided. In various embodiments, a system can access a stream of object-storage operation requests associated with an object store. In various aspects, the system can identify a set of repeating object-storage operation sequences, based on the stream of object-storage operation requests. In various instances, the system can generate a set of distribution ensembles that quantify variation of first attributes associated with respective ones of the set of repeating object-storage operation sequences. In various cases, the stream of object-storage operation requests can be considered as fully and/or bulkily representing the workload experienced by the object store. In contrast, the set of distribution ensembles can be considered as compactly representing the workload experienced by the object store (e.g.

Abstract: Various embodiments of the present technology generally relate to systems and methods for providing a waypoint prediction engine and its related functions. In an aspect, a waypoint prediction engine may determine navigation data associated with a client device and a source model. Then a decompression-side of the waypoint prediction engine may generate a predicted waypoint based on the source model and the navigation data for the client device as the client device travels along a navigation route. The decompression-side may receive, from a compression-side of the waypoint prediction engine, a correction factor for the predicted waypoint. Responsive to receiving the correction factor, the decompression-side of the waypoint prediction engine may store the correction factor as associated with the source model, where the correction factor and the source model allow for recreation of the navigation route of the client device.

Abstract: Techniques are provided for performing a resync transfer to recover from a storage site failure. During normal operation of a first site hosting a first volume, data is replicated to a second volume hosted by a second site. If the first site fails, when clients are redirected to the second volume at the second site. When the first site recovers, data modifications made to the second volume are resynced back to the first volume. As part of synchronizing the first volume, a data warehouse is rebuilt at the first site in order to track the location of blocks present on the replication destination. Typically, the data modifications are transferred after the data warehouse is rebuilt, which results in significantly long resync times. The techniques provided herein decrease the resync time by either rebuilding the data warehouse in parallel with resyncing the data modifications or circumvent the need for rebuild.

Abstract: Techniques are provided for coupled compute and storage resource autoscaling. Applications may be hosted within an application hosting environment (e.g., containerized applications hosted within Kubernetes) that allocations certain amounts of compute resources (e.g., processor and memory resources) to the applications for execution. The applications may store data within persistent of a backend storage platform separate from the application hosting environment. An autoscaler monitors the health of storage resources allocated to an application during deployment and runtime of the application so that issues with the storage resources can be preemptively identified and resolved for non-disruptive operation of the application. The autoscaler scales both the compute resources and the storage resources assigned to the application so that the application can continue operation in a non-disruptive manner.

Abstract: Techniques are provided for implementing write ordering for persistent memory. A set of actions are identified for commitment to persistent memory of a node for executing an operation upon the persistent memory. An episode is created to comprise a first subset of actions of the set of actions that can be committed to the persistent memory in any order with respect to one another such that a consistent state of the persistent memory can be reconstructed in the event of a crash of the node during execution of the operation. The first subset of actions within the episode are committed to the persistent memory and further execution of the operation is blocked until the episode completes.

Abstract: A computer implemented method includes maintaining information indicative of whether a data replication relationship between a dataset associated with the local CG and a mirror copy of the dataset stored on a remote CG of a remote distributed storage system is in an in-synchronization (InSync) state or an out-of-synchronization (OOS) state, initiating a storage expansion process including a source expansion to create a new source volume as a member of the local CG and a destination expansion to create a new destination volume as a member of the remote CG, and initiating a data replication relationship expand to establish a data replication relationship for the new source volume and the new destination volume.

Abstract: Systems and methods that make use of cluster-level redundancy within a distributed storage management system to address various node-level error scenarios are provided. According to one embodiment, an instance of a key-value (KV) store of a first node of a plurality of nodes of a cluster of a distributed storage system manages storage of data blocks as values and corresponding block identifiers (IDs) as keys. A list of missing block IDs that are in use for one or more volumes associated with the first node but that are missing from the instance of the KV store are identified by performing a data integrity check on the instance of the KV store. After identifying the list of missing block IDs, instead of treating the first node as failed, restoring the missing block IDs by writing redundant data blocks retrieved from other nodes within the cluster to the first node.

Abstract: Systems and methods for improving latency and throughput for metadata-heavy workloads and/or workloads including metadata bursts by decoupling data journal records and metadata-only journal records are provided. According to one embodiment, expedited and independent space reclamation is facilitated by differentiating between various types of journal records chains of which should be retained until different conditions are met. For example, data journal records may be added to data journal record chains within a persistent KV store and metadata-only journal records may be added to metadata-only journal record chains within the persistent KV store. Reclamation of spaced utilized by a data journal record chain may be reclaimed after both remote node data flush has been completed and the completion of a local CP for all records in the chain, whereas records of a metadata-only journal chain may be freed independently upon completion of a local CP for all records.

Abstract: Techniques are provided for distributed lock and authorities management. An origin may host a resource, such as a file, that may be accessed by clients through a plurality of caches. The origin may receive a first retrieve request from a first endpoint to obtain a first authority for a resource. The origin may delegate the first authority for the resource to the first endpoint. The origin may delegate the second authority for the resource to the second endpoint in response to determining that the delegated authority information and a set of locks held by the origin do not conflict with a second retrieve request from a second endpoint to obtain a second authority.

Abstract: One or more techniques and/or computing devices are provided for resynchronization. For example, a request may be received to create pseudo snapshots of a first consistency group, hosted by a first storage controller, and a second consistency group, hosted by a second storage controller, having a synchronous replication relationship with the first consistency group. Incoming client write requests are logged within an intercept tracking log at the first storage controller. After a first drain without hold of incoming write requests is performed, a first pseudo common snapshot of the second consistency group is created. After a second drain without hold of incoming write operations is performed, a second pseudo common snapshot of the first consistency group and the intercept tracking log is created. The pseudo snapshots and the intercept tracking log (e.g., indicating a delta between the pseudo snapshots) are used to resynchronize the first and second consistency groups.

Abstract: Systems and methods for reducing delays between the time at which a need for a resynchronization of data replication between a volume of a local CG and its peer volume of a remote CG is detected and the time at which the resynchronization is triggered (Reseed Time Period) are provided. According to an example, information indicative of the direction of data replication between the volume and the peer volume is maintained within a cache of a node. Responsive to a disruptive operation (e.g., relocation of the volume from a first node to a second node), the Reseed Time Period is lessened by proactively adding a passive cache entry to a cache within the second node at the time the CG relationship is created when the second node represents an HA partner of the first node and prior to the volume coming online when the second node represents a non-HA partner.

Abstract: Systems and methods are provided for implementation and use of disaggregated storage by a distributed storage system. In various examples described herein, the association of blocks of storage space within a storage pod with a given dynamically extensible file system (DEFS) may be in large chunks of one or more gigabytes (GB), which may be referred to herein as “allocation areas” (AAs) that each include multiple RAID stripes. The use of large, multi-GB chunks, as the unit of space allocation/assignment to DEFSs of a distributed storage system facilitates ease of management and independence of write allocation. The use of AAs described herein further allows disk space to be used more fluidly across individual (nodes) storage systems of a distributed storage system (cluster of nodes working together), thereby eliminating silos of storage; and processing resource (e.g., CPU) load may be distributed across the cluster.

Abstract: In one example, a computer-implemented method includes establishing bi-directional synchronous replication between one or more members of a first consistency group (CG1) of a primary storage site and one or more members of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO), initiating a non-disruptive planned failover (PFO) to change a role for the secondary storage site and change a role for the primary storage site, initiating, with the primary storage site and/or secondary storage site, a PFO out of synchronization (OOS) event that is sent to a mediator agent with no indication of a role for serving IO, and starting a configuration independent unplanned failover if a disaster or site failure occurs during the PFO.

Abstract: A technique provides efficient management of policies for objects of a distributed storage architecture configured to service storage requests issued by one or more clients of a storage cluster. The objects may include volumes for storing data served by storage nodes of the cluster and the policies may include quality of service (QoS) policies. The technique enables dynamic grouping of the volumes as management domains and applying attributes, such as performance settings of the QoS policies, to the management domains. A group of volumes may be organized as a management domain and a QoS policy may be applied to the domain. If membership of the management domain is modified, the QoS policy is automatically applied to the added volume or stripped from the removed volume. If a performance setting of the policy is modified, the modification is atomically applied and propagated to each volume of the management domain.

Abstract: A computer-implemented method includes establishing bi-directional synchronous replication between one or more members of a first consistency group (CG1) of a primary storage site and one or more members of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access. The method includes detecting a disruption in a data replication session from one or more members of the CG1 to one or more members of the CG2 due to a disaster event, initiating a consensus establishment request to be sent to a mediator agent of the primary storage site, rejecting with the mediator agent the consensus establishment request, and initiating a role flip process for primary and secondary roles in serving I/O Operations in response to the rejection of the consensus establishment request to reduce an application input/output (I/O) resumption time due to the disaster event.

Abstract: Improved techniques for using multicore processors to process storage operations for multithreaded applications. Operation type concurrency is considered for storage operations in a storage system in addition to application concurrency. A storage controller of the storage system identifies a storage operation associated with an application and an available worker thread for processing the storage operation. The storage controller determines whether to use the available worker thread to execute the storage operation based on an application concurrency limit for the application and an operation type concurrency limit for the operation type of the storage operation. The storage controller allocates a worker thread to a storage operation only where the storage operation does not satisfy either limit.

Abstract: Methods and systems for mitigating stranded data objects in storage resources used by tenants of an application service. Techniques are disclosed for determining maximum allowable retain-until-dates for objects that have object lock features enabled such that any retain-until-date selection will not result in a stranded object. A request to configure an object lock retention period for an object or for a structure containing data objects is received. The request is associated with a tenant account, based on which a maximum value for the retain-until-date can be determined. The maximum retain-until-date is the greatest retention period that the object or structure can have while not exceeding the expiration of a service relationship associated with the tenant. Service relationships for the tenant are identified and the maximum retain-until-date is determined. Based on the maximum retain-until-date, the retain-until-date for the object can be set without the risk of the objecting becoming stranded.

Abstract: Techniques are provided for virtual machine hosting and serverless disaster recovery. A virtual machine is hosted by a first hypervisor that may be located on-premise. Snapshots of virtual machine disks of the virtual machine are backed up to a cloud storage environment. The snapshots are used to on-demand host a new instance of the virtual machine within a destination environment such as within the cloud storage environment through a second hypervisor. The new instance of the virtual machine is hosted for various reasons such as part of a disaster recovery operation if the virtual machine fails, load balancing of I/O operations, migration to a different hosting environment (e.g., a cheaper or more performant environment), development testing, etc.

Abstract: Systems and methods for scaling application and/or storage system functions of a distributed storage system based on a heterogeneous resource pool are provided. According to one embodiment, the distributed storage system has a composable, service-based architecture that provides scalability, resiliency, and load balancing. The distributed storage system includes a cluster of nodes each potentially having differing capabilities in terms of processing, memory, and/or storage. The distributed storage system takes advantage of different types of nodes by selectively instating appropriate services (e.g., file and volume services and/or block and storage management services) on the nodes based on their respective capabilities. Furthermore, disaggregation of these services, facilitated by interposing a frictionless layer (e.g.

Abstract: The disclosure describes systems, devices, and methods for automating the reintegration of a failed virtual storage block into a file system service upon recovery of the virtual storage block. In an example embodiment, a method performed by a storage controller in a file system service is provided. In performing the method, the storage controller determines that a virtual storage volume of an array of virtual storage volumes has become unavailable, and in response to determining that the virtual storage volume has become unavailable, pauses use of the virtual storage volume and monitors for the virtual storage volume to return to an available state. Upon the virtual storage volume returning to the available state, the storage controller tests multiple portions of the virtual storage volume. After successfully testing the multiple portions of the virtual storage volume, the storage controller commences with the use of the virtual storage volume.