Abstract: The technology disclosed herein enables access to a file system by a portable executable program. In a particular example, a method includes, in a host executing the portable executable program, recognizing the portable executable program is executing on the one or more processing systems and determining the portable executable program is configured to access the file system. The method also includes directing the portable executable program to create a module therein for file system access and creating an abstraction layer with which the module exchanges file system commands. In the abstraction layer, the method includes translating the file system commands to translated commands for the file system and exchanging translated commands between the abstraction layer and the file system.

Abstract: Various mechanisms and workflows are described that can utilize power and/or carbon footprint-based metrics to manage storage unit usage and/or configuration, which can provide a more efficient and environmentally friendly computing environment. In some example configurations, storage system management mechanisms collect power consumption for storage units (e.g., individual drives, storage shelfs, nodes, clusters) and can utilize the power consumption information with other storage unit characteristics to generate power and carbon footprint metrics.

Abstract: Techniques are provided for persistent memory file system reconciliation. As part of the persistent memory file system reconciliation, high level file system metadata associated with a persistent memory file system of persistent memory is reconciled. Client access to the persistent memory file system is inaccessible until reconciliation of the high level file system metadata has completed. A first scanner is executed to traverse pages of the persistent memory in order to fix local inconsistencies associated with the pages. A local inconsistency of a first set of metadata or data of a page is fixed using a second set of metadata or data of the page. The first scanner is executed asynchronously in parallel with processing client I/O directed to the persistent memory file system.

Abstract: Described herein are systems, methods, and software to provide ransomware detection using variable levels of encryption. In one implementation, a computing device identifies a set of files, wherein the set of files each comprise a label indicative of whether the file is representative of a safe file or a file attacked by ransomware, and wherein the set of files comprises unencrypted files, partially encrypted files, and fully encrypted file. The computing device further identifies features associated with the set of files and generates a machine learning model that outputs a determination of whether a new file has been attacked based at least on the features in relation to whether a file in the set of files was labeled as attacked.

Abstract: Systems and methods for coalescing writes to facilitate generation of larger compression groups for use during inline compression are provided. According to one embodiment, inline compression performed by a storage system is improved by temporarily staging writes to in-memory data structures (e.g., inline storage efficiency (ISE) index nodes (inodes)) and performing coalescing in a deferred manner to generate larger compression groups for use during performance of inline compression. In one example, all files may be treated in the same manner, for example, by staging writes within a staging area and then processing the staged data by an inline compression workflow. In another example, the staging processing for small and large file may be different. For instance, the data blocks associated with small files may be staged separately from data blocks associated with large files and/or data blocks of multiple small files may be staged within the same ISE inode.

Abstract: Systems and methods are described for a cross-site high availability distributed storage system. According to one embodiment, a computer implemented method includes providing a remote direct memory access (RDMA) request for a RDMA stream, and generating, with an interconnect (IC) layer of the first storage node, multiple IC channels and associated IC requests for the RDMA request. The method further includes mapping an IC channel to a group of multiple transport layer sessions to split data traffic of the IC channel into multiple packets for the group of multiple transport layer sessions using an IC transport layer of the first storage node and assigning, with the IC transport layer, a unique transaction identification (ID) to each IC request and assigning a different data offset to each packet of a transport layer session.

Abstract: Systems, methods, and software are disclosed herein for identifying duplicate blocks of a storage system and deduplicating the storage system. In one example, a method of operating a computing device includes scanning first metadata of blocks of a container file of a virtual volume to generate a first log file including records of virtual volume block numbers (VVBNs) and fingerprints of the blocks; scanning second metadata of blocks of an active file system of the virtual volume to generate a second log file including records of VVBNs and file block numbers (FBNs) of the blocks; generating tuples based on merging the records of the first log file and the second records of the second log file according to the VVBNs; identifying duplications among the blocks based on the tuples; and deduplicating the blocks based on the duplications in the active file system identified based on the tuples.

Abstract: Techniques are provided for failing over an aggregate from one file system instance to a different file system instance of a distributed scale-out storage system. The aggregate may be stored within distributed storage that is accessible to a plurality of file system instances of the distributed scale-out storage system. When the aggregate is failed over from a first file system instance to a second file system instance, the first file system instance may still have a valid read lease that allows the first file system instance to serve client I/O, directed to the aggregate, using a cache. In order to prevent the first file system instance from serving stale data from the cache before the read lease expires, state machines and a set of control data are used to ensure that the second file system instance attaches to the aggregate only after the read lease has expired.

Abstract: A computer implemented method includes maintaining information indicative of whether a data replication relationship between a dataset associated with the local CG (CG1) and a mirror copy of the dataset stored on a remote CG (CG2) of a remote distributed storage system is in an in-synchronization (InSync) state or an out-of-synchronization (OOS) state, determining whether a CG storage expansion process is in progress, and in response to determining the OOS state and whether the primary storage site has a failure, initiating an automatic unplanned failover (AUFO) workflow without manual intervention on the original volumes in the CG1 of a first storage cluster of the primary storage site and the original volumes in the CG2 of a second storage cluster when the CG storage expansion process is in progress with a new source volume to be a member of CG1 and a new destination volume to be a member of CG2.

Abstract: Network address migration using a destination compute instance to update network configuration information in a cloud environment is disclosed. A network interface either using a private address within a subnet corresponding to the created network interface or using a floating address outside of the subnet corresponding to the created network interface is created. A first node of the HA pair with a service provider interface. The first node is an active data server of the HA pair, and the second node is a backup node of the HA pair. Requests are serviced the first node using the created network interface. Upon failure of the first node, the second node performs a failover, wherein if the first node was utilizing a floating address, the second node registers the second node with the service provider interface by adding an address of the second node to the route table.

Abstract: Multi-site distributed storage systems and computer-implemented methods are described for providing an automatic unplanned failover (AUFO) feature to guarantee non-disruptive operations (e.g., operations of business enterprise applications, operations of software application) even in the presence of failures including, but not limited to, network disconnection between multiple data centers and failures of a data center or cluster.

Abstract: One or more techniques and/or computing devices are provided for managing an arbitrary set of storage items using a granset. For example, a storage controller may host a plurality of storage items and/or logical unit numbers (LUNs). A subset of the storage items are grouped into a consistency group. A granset is created for tracking, managing, and/or providing access to the storage items within the consistency group. For example, the granset comprises application programming interfaces (APIs) and/or properties used to provide certain levels of access to the storage items (e.g., read access, write access, no access), redirect operations to access either data of an active file system or to a snapshot, fence certain operations (e.g., rename and delete operations), and/or other properties that apply to each storage item within the consistency group. Thus, the granset provides a persistent on-disk layout used to manage an arbitrary set of storage items.

Abstract: The disclosure describes a system for developing a forensic projection for data lost in a cyberattack. After identifying a cyberattack causing a loss of data in the data volume, the system identifies a snapshot of the portion of the data volume affected by the cyberattack. The system estimates, based on the snapshot, an amount of lost data caused by the cyberattack. The system then determines based at least on the amount of lost data, a data loss metric.

Abstract: Techniques are provided for caching data during an on-demand restore using a cloud block map. A client may be provided with access to an on-demand volume during a restore process that copies backup data from a snapshot within a remote object store to the on-demand volume stored within local storage. In response to receiving a request from the client for a block of the backup data not yet restored from the snapshot to the on-demand volume, the block may be retrieved from the snapshot in the remote object store. The block may be cached within a cloud block map stored within the local storage as a cached block. The client may be provided with access to the cached block.

Abstract: Techniques are provided for implementing garbage collection and bin synchronization for a distributed storage architecture of worker nodes managing distributed storage composed of bins of blocks. As the distributed storage architecture scales out to accommodate more storage and worker nodes, garbage collection used to free unused blocks becomes unmanageable and slow. Accordingly garbage collection is improved by utilizing heuristics to dynamically speed up or down garbage collection and set sizes for subsets of a bin to process instead of the entire bin. This ensures that garbage collection does not use stale information about what blocks are in-use, and ensures garbage collection does not unduly impact client I/O processing or conversely falls behind on garbage collection. Garbage collection can be incorporated into a bin sync process to improve the efficiency of the bin sync process so that unused blocks are not needlessly copied by the bin sync process.

Abstract: Techniques are provided for metadata management for enabling automated switchover in accordance with a configuration of storage solution that expresses a preference for either maintaining availability (e.g., a non-zero RPO mode) of the storage solution or avoiding data loss (e.g., a zero RPO mode). In one example, responsive to detecting a switchover trigger event, a node of a local cluster of a cross-site storage solution determines whether performance of an automated switchover from a failed cluster to a surviving cluster of the cross-site storage solution is enabled. Responsive to an affirmative determination, the node selectively proceeds with the automated switchover based on the configuration.

Abstract: Various embodiments of the present disclosure relate to compression techniques that can be used for data storage, indexing, and retrieval. In an example embodiment, a controller of or in communication with a data storage system can obtain a request to store a file in the data storage system. The controller identifies time-series data associated with input/output of the file and metrics of the file, generates a feature vector for the file, and compresses the feature vector using a compression technique, resulting in a compressed feature vector having a compression ratio. The controller performs a hash algorithm using the compression ratio to determine a storage device at which to store the compressed feature vector, the compression ratio associated with the file, and an indication of the compression technique. Upon receiving a query for a file, the controller compares compression ratios to identify a storage device storing the requested file.

Abstract: Techniques are provided for a snapshot difference interface integrated into an object store data management container. The snapshot difference interface is capable of interpreting an object format and snapshot file system format of snapshots backed up to an object store within objects formatted according to the object format. The snapshot difference interface can identify differences between snapshots, such as files that changed between the snapshots, while the snapshots are still resident within the object store. Because the snapshot difference interface does not retrieve the snapshots from the object store, security is improved, resource and network consumption is reduced, there is less of an impact upon client I/O processing, and a catalog of the snapshots can be more efficiently built and recovered in the event of corruption.

Abstract: Techniques are provided for performing a resync transfer to recover from a storage site failure. During normal operation of a first site hosting a first volume, data is replicated to a second volume hosted by a second site. If the first site fails, when clients are redirected to the second volume at the second site. When the first site recovers, data modifications made to the second volume are resynced back to the first volume. As part of synchronizing the first volume, a data warehouse is rebuilt at the first site in order to track the location of blocks present on the replication destination. Typically, the data modifications are transferred after the data warehouse is rebuilt, which results in significantly long resync times. The techniques provided herein decrease the resync time by either rebuilding the data warehouse in parallel with resyncing the data modifications or circumvent the need for rebuild.

Abstract: Techniques are provided for a data format for efficient management of checkpoint support. The data format corresponds to a base metafile and a set of instance metafiles used to track storage operations such as a directory restore operation. The base metafile and the set of instance metafiles can be used to resume the storage operation from where the storage operation left off in the event of a failure. The base metafile and the set of instance metafiles can be used to track progress of the storage operation processing objects stored within an object store of a cloud storage environment.