Abstract: A system, method, and machine-readable storage medium for retrieving data are provided. In some embodiments, a cache may receive a request for data from a client. The cache may determine that a first subset of the data is stored on a storage device and that a second subset of the data is stored at a cloud address located at a cloud storage endpoint. The cache may also receive from the storage device the first subset of data. The cache further receives from the cloud storage endpoint the second subset of data in response to transmitting a request for the second subset of data stored at the cloud address to the cloud storage endpoint. The cache then transmits to the client the first and second subsets of data from the various sources in response to the data request.

Abstract: A system, method, and machine-readable storage medium for resolving a candidate community are provided. In some embodiments, a method includes obtaining a candidate community and a neighbor set for the candidate community, the neighbor set including zero or more stable communities. The method also includes resolving the candidate community as being a new stable community if the neighbor set is empty. The method further includes resolving the candidate community as being part of a matching stable community if a hash value of the candidate community matches a hash value of one or more stable communities included in the neighbor set. The method also includes resolving the candidate community as being a new stable community if an entropy value is greater than a threshold, the entropy value being based on the candidate community and the neighbor set.

Abstract: A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can obtain storage system monitoring data and generate a segment by applying the storage system monitoring data to a machine learning model trained to segment the storage system monitoring data. The monitoring system can generate a compressed segment by applying a specified compression technique to the segment. In response to user query, the user query specifying a portion of the storage system monitoring data; the monitoring system can perform at least one of: reconstructing and providing the portion using the compressed segment; or providing the compressed segment for reconstruction of the portion.

Abstract: In one embodiment, a computer implemented method includes comprises storing objects in a first bucket and files in a second bucket of a first storage cluster of the distributed storage system, initiating an audit job on the first storage cluster, synchronously replicating audit configuration data and mirroring audit data (e.g., audit files, logs) from the first storage cluster to the second storage cluster, performing a switchover process from the first storage cluster to the second storage cluster, and initiating an audit job on the second storage cluster based on the audit configuration during the switchover process. The first storage cluster initially handles input/output operations for a software application before the switchover process.

Abstract: A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can obtain compression information for a compressed segment generated from a segment of storage system monitoring data and generate a predicted portion of storage system monitoring data using the compression information. The monitoring system can obtain an additional portion of storage system monitoring data, the additional portion contiguous to the segment of storage system monitoring data, and determine that the predicted portion matches the additional portion and combining the additional portion and the compressed segment. In response to a user query, the monitoring system can perform at least one of: reconstructing and providing the additional portion using the compressed segment; or providing the compressed segment for reconstruction of the additional portion.

Abstract: In one embodiment, a computer implemented method includes storing objects in a first bucket and storing files in a second bucket of a first storage cluster of the distributed storage system, synchronously replicating data of the objects into a third mirrored bucket of a second storage cluster of the distributed storage system, synchronously replicating OSP configuration data from the first storage cluster to the second storage cluster during the synchronous replication, and providing non-disruptive operations with zero recovery time objective (RTO), and ensuring consistency between the objects in the first bucket and the objects in the third bucket for a software application that is accessing one or more objects and files using the OSP. The objects and files are accessible through an object storage protocol (OSP).

Abstract: A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can obtain a segment of storage system monitoring data and generate a compressed segment from the segment and a reconstructed segment from the compressed segment. The monitoring system can identify locations in the segment based on a comparison of the segment and the reconstructed segment and determine values for the identified locations. In response to a user query received from a user system, the user query indicating a portion of the segment, the monitoring system can perform at least one of: reconstructing and providing the portion using the identified locations, the determined values for the identified locations, and the compressed segment; or providing the identified locations, the determined values for the identified locations, and the compressed segment for reconstruction of the portion.

Abstract: The technology disclosed herein enables worker nodes to reestablish connections when the connection protocols fail to successfully reconnect on their own. In a particular example, a method includes establishing a plurality of TCP-based connections (e.g., iSCSI and/or NVMe) between a plurality of worker nodes and a storage system and determining status of the plurality of TCP-based connections. When the status indicates a first connection of the plurality of TCP-based connections has failed, the method includes determining credentials used to establish the first connection are invalid. In response to determining the credentials are invalid, the method includes requesting new credentials from a controller node and reestablishing the first connection using the new credentials.

Abstract: Techniques are provided for backing up virtual machines from a computing environment to a storage environment. A virtual machine agent is utilized to generate a snapshot of the virtual machine. Metadata comprising a snapshot identifier of the snapshot and virtual disk information of virtual disks captured by snapshot is generated at the computing environment. The metadata is retrieved and used to create a metafile that is transferred to the storage environment within which snapshots of the virtual machine are to be stored. The snapshot is retrieved from the computing environment and is packaged into a snapshot package having a protocol format used by the storage environment. The snapshot package is transferred to the storage environment.

Abstract: Techniques are provided for modifying a read path to process read operations during a storage operation, such as an operation to migrate objects storing data of a volume. The objects are stored across a storage tier and capacity tier of a source object store. As part of migrating the volume to a destination object store, the objects are migrated to the destination cluster. Directly copying the objects involves multiple read operations to the source object store and a write operation at the destination object store. The techniques provided herein improve the efficiency of the migration by initially sending metadata from the source object store to the destination object store for performing backend block copy operations to migrate the volume.

Abstract: In one embodiment, a computer implemented method includes comprises storing objects in a first bucket and files in a second bucket of a first storage cluster of the distributed storage system, initiating an audit job on the first storage cluster, synchronously replicating audit configuration data and mirroring audit data (e.g., audit files, logs) from the first storage cluster to the second storage cluster, performing a switchover process from the first storage cluster to the second storage cluster, and initiating an audit job on the second storage cluster based on the audit configuration during the switchover process. The first storage cluster initially handles input/output operations for a software application before the switchover process.

Abstract: Data is replicated on a backup node, where the granularity of the replication can be less than a full volume. A data consistency group comprising a subset of data for a volume is defined for a primary node. A set of differences for the data consistency group is sent to a backup node. The backup node creates change logs in response to receiving the set of differences. In response to receiving a request to access a file having data in the data consistency group, the backup node creates a clone of the file. The backup node determines whether an update to a data block of the file exists in the change logs. In response to determining that the update to the data block exists in the change logs, the backup node updates a copy of the data block for the cloned file with data in the change logs.

Abstract: Approaches for providing a non-disruptive file move are disclosed. A request to move a target file from the first constituent to the second constituent is received. The file has an associated file handle. The target file in the first constituent is converted to a multipart file in the first constituent with a file location for the new file in the first constituent. A new file is created in the second constituent. Contents of the target file are moved to a new file on the second constituent while maintaining access via the associated file handle via access to the multipart file. The target file is deleted from the first constituent.

Abstract: Techniques are provided for data management across a persistent memory tier and a file system tier. A block within a persistent memory tier of a node is determined to have up-to-date data compared to a corresponding block within a file system tier of the node. The corresponding block may be marked as a dirty block within the file system tier. Location information of a location of the block within the persistent memory tier is encoded into a container associated with the corresponding block. In response to receiving a read operation, the location information is obtained from the container. The up-to-date data is retrieved from the block within the persistent memory tier using the location information for processing the read operation.

Abstract: In various examples, data storage is managed using a distributed storage management system that is resilient. Data blocks of a logical block device may be distributed across multiple nodes in a cluster. The logical block device may correspond to a file system volume associated with a file system instance deployed on a selected node within a distributed block layer of a distributed file system. Each data block may have a location in the cluster identified by a block identifier associated with each data block. Each data block may be replicated on at least one other node in the cluster. A metadata object corresponding to a logical block device that maps to the file system volume may be replicated on at least another node in the cluster. Each data block and the metadata object may be hosted on virtualized storage that is protected using redundant array independent disks (RAID).

Abstract: Systems and methods for flushing an operation log journal to both ephemeral storage and persistent storage during a shutdown sequence of a virtual storage system to minimize data-loss scenarios are provided. According to one embodiment, the shutdown or reboot scenarios that result in loss of data are minimized by using persistent storage as a backup to ephemeral storage when the scenario results in rehosting of virtual storage system. For example, responsive to an event indicative of an imminent shutdown or reboot of the virtual storage system, vNVRAM memory may be flushed to both ephemeral storage and persistent storage (e.g., a boot disk). In this manner, when the virtual storage system is rehosted after an unplanned shutdown or reboot resulting from an unrecoverable host error (other than an unrecoverable hardware failure), the operation log journal may be recovered from persistent storage to facilitate vNVRAM replay and avoid data loss.

Abstract: Disclosed are systems, computer-readable mediums, and methods for managing client performance in a storage system. According to one embodiment, a total Input/Output Operations per Second (IOPS) pool and a read/write IOPS pool are managed for clients to ensure their write requests can be accommodated by both pools. In one example, a write request is received from a client by the storage system. A requested number of write IOPS is determined for a time period to accommodate the request. Based on the requested number of write IOPS exceeding a number of allocated write IOPS to the client for the time period, a target total IOPS for the client during the time period is determined by subtracting the number of allocated write IOPS from a number of allocated total IOPS to the client. At least a portion of the request is performed by executing the target total IOPS during the time period.

Abstract: Systems and methods for multiple device consumption of shared namespaces of ephemeral storage devices by a consumer of a virtual storage system are provided. In an example, multiple namespaces of respective ephemeral storage devices are shared among multiple consumers of a virtual storage system by creating multiple partitions within each of the namespaces for use by respective consumers of the multiple consumers. Corresponding partitions of respective shared namespace may then be treated as a stripe set to facilitate multiple device consumption for a subsystem (e.g., operation log journaling) of the virtual storage system by striping data associated with input/output (I/O) requests of a consumer (e.g., a journaling driver) across one or more stripe units of one or more stripes within the stripe set.

Abstract: Techniques are provided for journal replay optimization. A distributed storage architecture can implement a journal within memory for logging write operations into log records. Latency of executing the write operations is improved because the write operations can be responded back to clients as complete once logged within the journal without having to store the data to higher latency disk storage. If there is a failure, then a replay process is performed to replay the write operations logged within the journal in order to bring a file system up-to-date. The time to complete the replay of the write operations is significantly reduced by caching metadata (e.g., indirect blocks, checksums, buftree identifiers, file block numbers, and consistency point counts) directly into log records. Replay can quickly access this metadata for replaying the write operations because the metadata does not need to be retrieved from the higher latency disk storage into memory.

Abstract: Systems and methods are described for a cross-site high availability distributed storage system. According to one embodiment, a computer implemented method includes providing a remote direct memory access (RDMA) request for a RDMA stream, and generating, with an interconnect (IC) layer of the first storage node, multiple IC channels and associated IC requests for the RDMA request. The method further includes mapping an IC channel to a group of multiple transport layer sessions to split data traffic of the IC channel into multiple packets for the group of multiple transport layer sessions using an IC transport layer of the first storage node and assigning, with the IC transport layer, a unique transaction identification (ID) to each IC request and assigning a different data offset to each packet of a transport layer session.