Abstract: Systems and methods that make use of cluster-level redundancy within a distributed storage management system to address various node-level error scenarios are provided. Rather than using a generalized one-size-fits-all approach to reduce complexity, an approach tailored to the node-level error scenario at issue may be performed to avoid doing more than necessary. According to one embodiment, responsive to identifying a missing branch of a tree implemented by a KV store of a first node of a cluster of a distributed storage management system, a branch resynchronization process may be performed, including, for each block ID in the range of block IDs of the missing branch (i) reading a data block corresponding to the block ID from a second node of the cluster that maintains redundant information relating to the block ID; and (ii) restoring the block ID within the KV store by writing the data block to the first node.

Abstract: Systems and methods are described for performing persistent inflight tracking of operations (Ops) within a cross-site storage solution. According to one embodiment, a method comprises maintaining state information regarding a data synchronous replication status for a first storage object of a primary storage cluster and a second storage object of a secondary storage cluster. The state information facilitates automatic triggering of resynchronization for data replication between the first storage object and the second storage object. The method includes performing persistent inflight tracking of I/O operations with a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster, establishing and comparing Op ranges for the first and second Op logs, and determining a relation between the Op range of the first Op log and the Op range of the second Op log to prevent divergence of Ops in the first and second Op logs and to support parallel split of the Ops.

Abstract: A method, a computing device, and a non-transitory machine-readable medium for managing modes of operation for volumes in a node. A first portion of a plurality of volumes in a node is selected to operate in an active mode. A second portion of the plurality of volumes in the node is selected to operate in a passive mode. The second portion of the volumes that operates in the passive mode consumes fewer resources than the first portion of the volumes that operates in the active mode. The first portion of the plurality of volumes and the second portion of the plurality of volumes are adjusted over time based on activity of each volume of the plurality of volumes.

Abstract: Various embodiments are generally directed an apparatus and method for receiving information to write on a clustered system comprising at least a first cluster and a second cluster, determining that a failure event has occurred on the clustered system creating unsynchronized information, the unsynchronized information comprising at least one of inflight information and dirty region information, and performing a resynchronization operation to synchronize the unsynchronized information on the first cluster and the second cluster based on log information in at least one of an inflight tracker log for the inflight information and a dirty region log for the dirty region information.

Abstract: Systems, methods, and machine-readable media for creating, deleting, and restoring volume snapshots in a remote data store are disclosed. A storage volume and a storage operating system are implemented in a software container. Through a user interface, a user may create a snapshot of the volume to a cloud storage. A user may also delete individual snapshots from the cloud storage. Further, deletion of a most recent snapshot may occur by awaiting deletion (though marking as deleted to the user) until a next snapshot is received. Snapshots in the cloud storage are manipulatable even after destruction of the source volume (by destruction of the container, for example). A controller outside the container is used by implementing the same API as the controller in the container had. Full restores of snapshots in the cloud are also possible even when the original container and volume have been destroyed.

Abstract: One or more techniques and/or computing devices are provided for cross-platform replication. For example, a replication relationship may be established between a first storage endpoint and a second storage endpoint, where at least one of the storage endpoints, such as the first storage endpoint, lacks or has incompatible functionality to perform and manage replication because the storage endpoints have different storage platforms that store data differently, use different control operations and interfaces, etc. Accordingly, replication destination workflow, replication source workflow, and/or a proxy representing the first storage endpoint may be implemented at the second storage endpoint comprising the replication functionality.

Abstract: A configuration for a component of a primary node is synchronized with a configuration for a component of a partner node in a different cluster by replicating the primary node configuration with the partner node. A baseline configuration replication comprises a snapshot of a component configuration on the primary. The baseline configuration can be generated by traversing through the configuration objects, capturing their attributes and encapsulating them in a package. The baseline package can then be transferred to the partner node. The configuration objects can be applied on the partner node in the order in which they were captured on the primary node. Attributes of the configuration objects are identified that are to be transformed. Values for the identified attributes are transformed from a name space in the primary node to a name space in the partner node.

Abstract: Various mechanisms and workflows are described that can utilize power and/or carbon footprint-based metrics to manage storage unit usage and/or configuration, which can provide a more efficient and environmentally friendly computing environment. In some example configurations, storage system management mechanisms collect power consumption for storage units (e.g., individual drives, storage shelfs, nodes, clusters) and can utilize the power consumption information with other storage unit characteristics to generate power and carbon footprint metrics.

Abstract: Systems, methods, and machine-readable media for predicting interruptions to the use of spare cloud resources and rebalancing based on those predictions are disclosed. A computing platform collects data for customers over time. The computing platform runs a machine learning algorithm on the historical data to generate a prediction classifier. The prediction classifier relates to a time window for prediction into the future, on the order of minutes or hours. The prediction classifier is run on monitored data from ongoing activity with a cloud provider to generate a risk score. Each risk score may identify an amount of risk that a spare cloud resource related to new resource metrics data will be interrupted within the future time frame corresponding to that prediction classifier. If predicted to be interrupted, the customer may be assisted in rebalancing to other resources. As a result, interruptions can be predicted hours into the future.

Abstract: A request to perform a coordinated action is received. The coordinated action comprises a first plurality of operations. In response to reception of the request to perform the coordinated action, a second plurality of operations is performed in preparation for performing the coordinated action. A response to the request to perform the coordinated action is sent. The response to the request to perform the coordinated action comprises an indication that the first device will perform the coordinated action. An indication that the coordinated action is to be performed is received. In response to reception of the indication that the coordinated action is to be performed, the first plurality of operations is performed.

Abstract: Techniques are provided for storing immutable snapshot copes in write once read many (WORM) storage. A snapshot of a volume may be stored into one or more objects formatted according to an object format. An expiry time may be assigned to the snapshot and the one or more objects based upon a creation time of the snapshot and a retention time. The one or more objects may be stored within a remote object store. The one or more objects are retained in an immutable state and cannot be deleted until expiration of the expiry time. In response to identifying an existing object within the remote object store comprising shared snapshot data referenced by the snapshot, an assigned expiry time of the existing object may be modified based upon the expiry time of the snapshot to create a modified expiry time for the existing object.

Abstract: Methods and systems for co-locating journaling and data storage are provided. Separate journal and volume partitions may be maintained within each logical storage unit (e.g., Logical Unit Number (LUN)) of a distributed storage system. Journaling of metadata associated with write requests received from one or more clients may be distributed by identifying a destination logical storage unit to which data associated with a given write request is to be stored and causing the data and metadata to be persisted to disk by journaling the metadata and the data to respective portions of an active log within the journal partition of the destination logical storage unit. By using the same logical storage unit for both journaling of write requests and writing the data associated with such write requests, the bottleneck due to there being only a single device or storage unit handling all metadata for all write requests can be avoided.

Abstract: Redistribution of files in a containerized distributed file system is disclosed. Containers each have an engine and a scanner and each of the containers stores files and parameters for characteristics of files stored on the container. A first engine in a first container monitors characteristics of files stored on the first container and, responsive to determining that the parameters for files on the first container exceed one or more predetermined thresholds, communicates with a second engine in a second container to determine a destination container for one or more files from the first container. The second engine in the second container indicates to the first engine in the first container whether the second container is available to receive one or more files from the first container. The first engine triggers file system scanning by the scanner of the first container to identify files to be moved to the second container.

Abstract: Approaches and mechanisms for cloning a file are described. A first node requests a clone of a file at a time when it also requests an exclusive delegation of the original file from a second node where the original file is stored. The second node marks the original file as delegated to the first node and the second node records an intent to create the clone file and a delegation record for the clone file. The second node creates the clone file. The delegation of and the identity of the clone file are returned to the first node. The first node marks in the delegation record that the clone file was committed in response to modification. If the clone file was committed the delegation is cleared and the clone file is kept, and if the clone file was not committed, the delegation is cleared, and the clone file is deleted.

Abstract: Redistribution of files in a containerized distributed file system is disclosed. An indication of at least one remote container to which files from the local container are to be transferred is received from a rebalancing engine in the local container. One or more transfer parameters for use in selecting one or more files in the local container to be transferred the at least one remote container are received from the rebalancing engine. The local container is scanned to identify files that satisfy the one or more transfer parameters. An indication of the identified files to a file transfer mechanism is provided. Operation of the scanner is terminate until triggered in response to a query by the engine of the local container or for a predetermined period of time.

Abstract: Techniques are provided for timestamp consistency. An operation targeting a first storage object having a synchronous replication relationship with a second storage object is intercepted. A timestamp is assigned to the operation. A replication operation is created as a replication of the operation. The same timestamp is assigned to the replication operation. The operation is implemented upon the first storage object and the replication operation is implemented upon the second storage object.

Abstract: Techniques are provided for volume group backup, volume group restore, and volume group garbage collection for volume groups backed up to an object store. A volume group workflow is implemented to orchestrate individual consistent volume workflows that are separately and individually implemented by nodes hosting constituent volumes of a volume group. The volume group workflow and the individual consistent volume workflows are performed to back up the volume group to the object store, restore a volume group backup from the object store to a restore destination, and/or perform garbage collection on slots of objects storing data unique to a volume group backup to delete.

Abstract: In one embodiment, a method comprises maintaining state information regarding a data replication status for a storage object of the storage node of a primary storage cluster with the storage object being replicated to a replicated storage object of a secondary storage cluster, temporarily disallowing input/output (I/O) operations when the storage object has a connection loss or failure. The method further includes initiating a resynchronization between the storage object and the replicated storage object including initiating asynchronous persistent inflight tracking and replay of any missing I/O operations that are missing from one of a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster, and allowing new I/O operations to be handled with the storage object of the primary storage cluster without waiting for completion of the asynchronous persistent inflight tracking and replay at the secondary storage cluster.

Abstract: Systems, methods, and machine-readable media are disclosed for isolating and reporting a volume placement error for a request to place a volume on a storage platform. A volume placement service requests information from a database using an optimized database query to determine an optimal location to place a new volume. The database returns no results. The volume placement service deconstructs the optimized database query to extract a plurality of queries. The volume placement service iterates over the plurality queries, combining queries in each iteration, to determine a cause for the database to return no results. The volume placement service determines based on the results of each iterative database request a cause the database to return an empty result. The volume placement service provides an indication of the cause for returning an empty result.

Abstract: Methods, non-transitory machine readable media, and computing devices that more efficiently and effectively manage storage quota enforcement are disclosed. With this technology, a quota ticket comprising a tally generation number (TGN) and a local allowed usage amount (AUA) are obtained. The local AUA comprises a portion of a global AUA associated with a quota rule. The local AUA is increased following receipt of another portion of the global AUA in a response from a cluster peer, when another TGN in the response matches the TGN and the local AUA is insufficient to execute a received storage operation associated with the quota rule. The local AUA is decreased by an amount corresponding to, and following execution of, the storage operation, when the increased local AUA is sufficient to execute the storage operation.