Abstract: Systems and methods for multiple device consumption of shared namespaces of ephemeral storage devices by a consumer of a virtual storage system are provided. In an example, multiple namespaces of respective ephemeral storage devices are shared among multiple consumers of a virtual storage system by creating multiple partitions within each of the namespaces for use by respective consumers of the multiple consumers. Corresponding partitions of respective shared namespace may then be treated as a stripe set to facilitate multiple device consumption for a subsystem (e.g., operation log journaling) of the virtual storage system by striping data associated with input/output (I/O) requests of a consumer (e.g., a journaling driver) across one or more stripe units of one or more stripes within the stripe set.

Abstract: A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can obtain compression information for a compressed segment generated from a segment of storage system monitoring data and generate a predicted portion of storage system monitoring data using the compression information. The monitoring system can obtain an additional portion of storage system monitoring data, the additional portion contiguous to the segment of storage system monitoring data, and determine that the predicted portion matches the additional portion and combining the additional portion and the compressed segment. In response to a user query, the monitoring system can perform at least one of: reconstructing and providing the additional portion using the compressed segment; or providing the compressed segment for reconstruction of the additional portion.

Abstract: Various embodiments of the present technology generally relate to systems and methods for providing a resource distribution engine. In an example, a method includes identifying, by an instance management service, a request from a cluster orchestrator to acquire computing resources associated with deployment of an application. The method may also include determining, by the instance management service, markets containing reclaimable resources suitable for the computing resources, and performing, by a resource distribution engine of the instance management service, a resource distribution process to determine a resource distribution of the reclaimable resources within the markets. The resource distribution of the reclaimable resources may include a distribution of instance type of the reclaimable resources across the markets. The method may also include securing, by the instance management service, the reclaimable resources identified by the resource distribution as provisioned reclaimable resources.

Abstract: Systems and methods for use of information relating to units of storage space movement and ownership to facilitate performance of disaggregated storage system workflows are provided. In various examples, the unit of disaggregated storage space for purposes of movement and ownership is an allocation area (AA). AA ownership information relating to AAs owned by dynamically extensible file systems (DEFSs) of a cluster may be maintained within two different persistent data sources, including an AA label region and DEFS AA owner metafiles. A node-scoped cache may be implemented to cache the AA ownership information in-memory for DEFSs hosted by a given node to provide a multiprocessing (MP)-safe cache for AA ownership information, a mechanism to coordinate between AA movement and other subsystems needing to know whether an AA is “local” or “remote,” and a mechanism to serialize access and Input/Output (IO) operations to the persistent AA ownership information data sources.

Abstract: Systems and methods for use of information relating to units of storage space movement and ownership to facilitate performance of disaggregated storage system workflows are provided. In various examples, the unit of disaggregated storage space for purposes of movement and ownership is an allocation area (AA). AA ownership information relating to AAs owned by dynamically extensible file systems (DEFSs) of a cluster may be maintained within two different persistent data sources, including an AA label region and DEFS AA owner metafiles. A node-scoped cache may be implemented to cache the AA ownership information in-memory for DEFSs hosted by a given node to provide a multiprocessing (MP)-safe cache for AA ownership information, a mechanism to coordinate between AA movement and other subsystems needing to know whether an AA is “local” or “remote,” and a mechanism to serialize access and Input/Output (IO) operations to the persistent AA ownership information data sources.

Abstract: Systems and methods for use of information relating to units of storage space movement and ownership to facilitate performance of disaggregated storage system workflows are provided. In various examples, the unit of disaggregated storage space for purposes of movement and ownership is an allocation area (AA). AA ownership information relating to AAs owned by dynamically extensible file systems (DEFSs) of a cluster may be maintained within two different persistent data sources, including an AA label region and DEFS AA owner metafiles. A node-scoped cache may be implemented to cache the AA ownership information in-memory for DEFSs hosted by a given node to provide a multiprocessing (MP)-safe cache for AA ownership information, a mechanism to coordinate between AA movement and other subsystems needing to know whether an AA is “local” or “remote,” and a mechanism to serialize access and Input/Output (IO) operations to the persistent AA ownership information data sources.

Abstract: Systems and methods for performing space balancing heuristics within a distributed storage system that makes use of disaggregated storage are provided. In various examples described herein, the unit of storage space assignment used to associate blocks of storage space within a storage pod with a given dynamically extensible file system (DEFS) is an AA, representing a large chunk of one or more gigabytes (GB). The use of AAs described herein allows disk space to be used more elastically across individual nodes of a storage cluster. Each DEFS periodically reports its AA usage information by persisting it to a respective heuristic block within the storage pod. An AA director operable within the cluster periodically monitors the reported AA usage information, performs a heuristic algorithm to determine donor and recipient DEFSs at a given instance, and triggers AA movement accordingly by matching one or more donor DEFSs to each recipient DEFS.

Abstract: Methods and systems for a networked storage environment are provided. One method includes generating a first and a second batch of entries in response to scanning a source file system, each batch of entries associated with one or more directories of the source file system and indicating a path to a file associated with each entry; determining, by a first worker process, a first checksum for data associated with the first batch of entries loaded in a first buffer; appending, the first buffer contents processed by the first worker process to a first archive file; and generating an archive data structure having a manifest file storing metadata for the first batch and a second batch entries with a first checksum determined by the first worker process and a second checksum determined by a second worker process, and data from the first archive file and a second archive file.

Abstract: The disclosure describes systems, devices, and methods for managing data storage environments. In an example implementation, a method of operating a controller in a data storage environment is provided. In performing the method, the controller identifies a change to a layout of drives in the data storage environment, and in response to identifying the change, takes a lock on instances of layout metadata stored on the drives. The controller then updates the instances of the layout metadata to reflect the change to the layout and releases the lock.

Abstract: The disclosure describes a workflow management service that dynamically provisions workflow documents across multiple accounts to avoid reaching document limits imposed by a compute provider. The workflow management service initially allocates workflow documents to a central account, which manages workflow execution and distribution. The workflow management service also provisions shard accounts dedicated to hosting and executing workflow documents. When the number of workflow documents in the central account or an existing shard account approaches the compute provider's imposed limit, the workflow management service provisions a new shard account and redirects future workflow documents to the new shard account.

Abstract: Systems and methods for preserving storage efficiency during restoration of data from the cloud are provided. In one embodiment, a CBMAP is maintained that maps cloud block numbers (CBNs) to respective corresponding block numbers of a volume of a data storage system in which previously restored data has been stored by a previously restored file. By making use of the CBMAP during the restoration process, storage of duplicate file data blocks on the volume may be avoided by sharing with a current file being restored a reference to the corresponding file data block previously stored on the volume and associated with the previously restored file. In addition to preserving storage efficiency, use of the CBMAP facilitates avoidance of repeated GET operations for data associated with CBNs previously retrieved from the cloud and stored to the volume, thereby reducing data access costs as well as latency of the restore operation.

Abstract: The disclosure describes systems, devices, and methods for automating the reintegration of a failed virtual storage block into a file system service upon recovery of the virtual storage block. In an example embodiment, a method performed by a storage controller in a file system service is provided. In performing the method, the storage controller determines that a virtual storage volume of an array of virtual storage volumes has become unavailable, and in response to determining that the virtual storage volume has become unavailable, pauses use of the virtual storage volume and monitors for the virtual storage volume to return to an available state. Upon the virtual storage volume returning to the available state, the storage controller tests multiple portions of the virtual storage volume. After successfully testing the multiple portions of the virtual storage volume, the storage controller commences with the use of the virtual storage volume.

Abstract: Techniques are provided for implementing a distributed control plane to facilitate communication between a container orchestration platform and a distributed storage architecture. The distributed storage architecture hosts worker nodes that manage distributed storage that can be made accessible to applications within the container orchestration platform through the distributed control plane. The distributed control plane includes control plane controllers that are each paired with a single worker node of the distributed storage architecture. The distributed control plane is configured to selectively route commands to control plane controllers that are paired with worker nodes that are current owners of objects targeted by the commands. If ownership of an object has changed from one worker node to another worker node, then subsequent commands will be re-routed to a control plane controller paired with the other worker node now owning the object.

Abstract: Techniques are provided for microservices management and orchestration. A chart package is selectively retrieved from a chart repository based upon the chart package corresponding to a set of services to host within a cluster and dependencies amongst the set of services. A set of container images may be retrieved from a container repository based upon the set of container images corresponding to the set of services. A cluster may be created within a computing environment. The set of services may be deployed as resources of the computing environment within the cluster and the dependencies may be configured using the chart package and the set of container images.

Abstract: Systems and techniques that facilitate compact representation of object store workloads are provided. In various embodiments, a system can access a stream of object-storage operation requests associated with an object store. In various aspects, the system can identify a set of repeating object-storage operation sequences, based on the stream of object-storage operation requests. In various instances, the system can generate a set of distribution ensembles that quantify variation of first attributes associated with respective ones of the set of repeating object-storage operation sequences. In various cases, the stream of object-storage operation requests can be considered as fully and/or bulkily representing the workload experienced by the object store. In contrast, the set of distribution ensembles can be considered as compactly representing the workload experienced by the object store (e.g.

Abstract: Various embodiments of the present technology generally relate to systems and methods for providing a waypoint prediction engine and its related functions. In an aspect, a waypoint prediction engine may determine navigation data associated with a client device and a source model. Then a decompression-side of the waypoint prediction engine may generate a predicted waypoint based on the source model and the navigation data for the client device as the client device travels along a navigation route. The decompression-side may receive, from a compression-side of the waypoint prediction engine, a correction factor for the predicted waypoint. Responsive to receiving the correction factor, the decompression-side of the waypoint prediction engine may store the correction factor as associated with the source model, where the correction factor and the source model allow for recreation of the navigation route of the client device.

Abstract: Techniques are provided for performing a resync transfer to recover from a storage site failure. During normal operation of a first site hosting a first volume, data is replicated to a second volume hosted by a second site. If the first site fails, when clients are redirected to the second volume at the second site. When the first site recovers, data modifications made to the second volume are resynced back to the first volume. As part of synchronizing the first volume, a data warehouse is rebuilt at the first site in order to track the location of blocks present on the replication destination. Typically, the data modifications are transferred after the data warehouse is rebuilt, which results in significantly long resync times. The techniques provided herein decrease the resync time by either rebuilding the data warehouse in parallel with resyncing the data modifications or circumvent the need for rebuild.

Abstract: Techniques are provided for coupled compute and storage resource autoscaling. Applications may be hosted within an application hosting environment (e.g., containerized applications hosted within Kubernetes) that allocations certain amounts of compute resources (e.g., processor and memory resources) to the applications for execution. The applications may store data within persistent of a backend storage platform separate from the application hosting environment. An autoscaler monitors the health of storage resources allocated to an application during deployment and runtime of the application so that issues with the storage resources can be preemptively identified and resolved for non-disruptive operation of the application. The autoscaler scales both the compute resources and the storage resources assigned to the application so that the application can continue operation in a non-disruptive manner.

Abstract: Techniques are provided for implementing write ordering for persistent memory. A set of actions are identified for commitment to persistent memory of a node for executing an operation upon the persistent memory. An episode is created to comprise a first subset of actions of the set of actions that can be committed to the persistent memory in any order with respect to one another such that a consistent state of the persistent memory can be reconstructed in the event of a crash of the node during execution of the operation. The first subset of actions within the episode are committed to the persistent memory and further execution of the operation is blocked until the episode completes.

Abstract: A computer implemented method includes maintaining information indicative of whether a data replication relationship between a dataset associated with the local CG and a mirror copy of the dataset stored on a remote CG of a remote distributed storage system is in an in-synchronization (InSync) state or an out-of-synchronization (OOS) state, initiating a storage expansion process including a source expansion to create a new source volume as a member of the local CG and a destination expansion to create a new destination volume as a member of the remote CG, and initiating a data replication relationship expand to establish a data replication relationship for the new source volume and the new destination volume.