Abstract: A computer-implemented method comprising: receiving, at an endpoint management engine, input arguments that indicate a plurality of endpoint types to be involved in a data movement operation, wherein the plurality of endpoint types includes a source endpoint type associated with a first type of data storage system (DSS), and a destination endpoint type associated with a second type of DSS; in response to the input arguments, the endpoint management engine making at least one call to an API; in response to the at least one call to the API: associating a source endpoint of the source endpoint type with a source DSS, wherein the source DSS is the first type of DSS; associating a destination endpoint of the destination endpoint type with a destination DSS, wherein the destination DSS is the second type of DSS; and the endpoint management engine causing the data movement operation to be performed between the source DSS and the destination DSS.

Abstract: A technique preserves efficiency for replication of data between a source node of a source cluster (“source”) and a destination node of a destination cluster (“destination”) of a clustered network. Replication in the clustered network may be effected by leveraging global in-line deduplication at the source to identify and avoid copying duplicate data from the source to the destination. To ensure that the copy of the data on the destination is synchronized with the data received at the source, the source creates a snapshot of the data for use as a baseline copy at the destination. Thereafter, new data received at the source that differs from the baseline snapshot are transmitted and copied to the destination. In addition, the source and destination nodes negotiate to establish a mapping of name-to-data when transferring data (i.e., an extent) between the clusters.

Abstract: A technique provides memory efficient caching of metadata managed by a volume layer of a storage input/output stack executing on one or more nodes of a cluster. Efficient caching of the metadata in a memory of a node may be realized through the use of a caching data structure, i.e., a page cache, configured to store a key-value pair, wherein the key is an extent key and the value is a metadata page containing the index entries. The page cache illustratively includes two data structures configured to maintain the properties of Least Recently Used (LRU) and Least Frequently Used (LFU) for the cache. The first data structure is a hash table that stores a dense tree metadata page (value) indexed by the extent key. The second data structure is a recycle queue that controls the metadata page stored in the hash table based on spatial and temporal locality of the page.

Abstract: A technique paces and balances a flow of messages related to processing of input/output (I/O) requests between subsystems, such as layers of a storage input/output (I/O) stack, of one or more nodes of a cluster. The I/O requests may be directed to externally-generated user data, e.g., write requests generated by a host coupled to the cluster, and internally-generated metadata, e.g., write and delete requests generated by a volume layer of the storage I/O stack. The user data (and metadata) may be organized as an arbitrary number of variable-length extents of one or more host-visible logical units (LUNs) served by the nodes. The metadata may include mappings from host-visible logical block address ranges (i.e., offset ranges) of a LUN to extent keys, which reference locations of the extents stored on storage devices, such as solid state drivers (SSDs), of a storage array coupled to the nodes.

Abstract: A technique schedules processing of metadata managed by a volume layer of a storage input/output (I/O) stack executing on a node of cluster in a manner that reduces bursty activity associated with metadata processing and maintains smooth, i.e., bounded, processing latency on the node. Operations on the metadata managed by the volume layer manifest as modifications to metadata entries of data structures, i.e., dense trees, at offset ranges of the regions. The operations are dense tree merge operations that are processed by threads of execution, i.e., a uni-processor services, on one or more central processing units of the node. The scheduling technique distributes the bursty activity of the dense tree merge operations by (i) controlling concurrency of the merge operations, (ii) distributing initiation of the merge operations (i.e., staggering the merge operations), and (iii) pacing execution of merge messages to limit the continuous runtime of the merge operations.

Abstract: A technique provides memory efficient caching of metadata managed by a volume layer of a storage input/output stack executing on one or more nodes of a cluster. Efficient caching of the metadata in a memory of a node may be realized through the use of a caching data structure, i.e., a page cache, configured to store a key-value pair, wherein the key is an extent key and the value is a metadata page containing the index entries. The page cache illustratively includes two data structures configured to maintain the properties of Least Recently Used (LRU) and Least Frequently Used (LFU) for the cache. The first data structure is a hash table that stores a dense tree metadata page (value) indexed by the extent key. The second data structure is a recycle queue that controls the metadata page stored in the hash table based on spatial and temporal locality of the page.

Abstract: A technique preserves efficiency for replication of data between a source node of a source cluster (“source”) and a destination node of a destination cluster (“destination”) of a clustered network. Replication in the clustered network may be effected by leveraging global in-line deduplication at the source to identify and avoid copying duplicate data from the source to the destination. To ensure that the copy of the data on the destination is synchronized with the data received at the source, the source creates a snapshot of the data for use as a baseline copy at the destination. Thereafter, new data received at the source that differs from the baseline snapshot are transmitted and copied to the destination. In addition, the source and destination nodes negotiate to establish a mapping of name-to-data when transferring data (i.e., an extent) between the clusters.

Abstract: Techniques for predicting data replication performance degradation are described. According to one embodiment, a method starts with receiving periodically data replication characteristics from a source storage and a target storage, the periodically received data replication characteristics being extracted from a replication process of the source storage replicating data to the target storage. The periodically received data replication characteristics are analyzed at a replication management server communicatively coupled with the source storage and the target storage, and a predictive modeling are applied. When the possibility of degradation of replication performance reaches a predetermined threshold, a notification is sent out.

Abstract: Techniques for automatically detecting data replication performance degradation are described. According to one embodiment, a method starts with receiving periodically data replication characteristics from a source storage and a target storage, the data replication characteristics being extracted from a replication process of the source storage replicating data to the target storage. The received data replication characteristics are analyzed at a replication management server communicatively coupled with the source storage, and the target storage and data replication performance degradation is detected based on analyzing the received data replication characteristics and applying one or more rules of data replication performance degradation. When the one or more applied rules indicating that a data replication performance degradation has been detected, a notification is sent out.

Abstract: A system and method for verifying the consistency of mirrored data sets between source and destination storage systems is provided. A destination verification module sends version information to the source storage system where the source verification module determines whether the source and destination storage systems are utilizing compatible versions of the verification module. If the destination verification module receives an acceptable version from the source, the destination module then determines a base PCPI to utilize for verification and sends a PCPI identifier of the base PCPI to the source. The source verification module generates and transmits a data stream consisting of comprising of checksum information. The destination verification module compares the received data stream with checksum information retrieved from disk. If there are mismatches, i.e. the received data differs from the data retrieved from disk, the destination verification module logs the data mismatches in a verification log file.

Abstract: A system and method for transparent data replication of one or more data containers over migrating virtual servers (vfilers) operating on physical storage systems is provided. A mirroring (or other replication) procedure associated with a vfiler generates an entry in a mirror table maintained by one or more appropriate mirroring applications executing on the physical storage systems. The mirror table identifies source and destination “owners” of a replicated data container by vfiler instead of only by physical storage system. Thus, when a vfiler is migrated from a source physical storage system (“source”) to a destination physical storage system (“destination”), the mirroring application may continue to perform mirroring operations on the data containers at a per vfiler level to thereby improve the utility of migration of the vfiler among the storage systems.

Abstract: A method for backing up a computer file system is disclosed. A first snapshot of the file system is generated at a first time, the first snapshot having first pointers to data blocks of the file system at the first time. A network adapter receives storage operations to write new data to the file system. A second snapshot of the file system is generated at a second time, the second snapshot having second pointers to data blocks of the file system at the second time, the data blocks at the second time including the new data. The first pointers to the second pointers are compared. In response to the second pointers differing from the first pointers, data of the file system which have been changed between the first snapshot and the second snapshot are copied to a backup copy of the file system.

Abstract: A method for transferring data from a source computer to a destination computer, and restarting the source computer transmission after a halt of the transmission, has the following steps. The source computer inserts a checkpoint number into a data stream, the data flowing in the data stream. The checkpoint number is stored at the source computer. The data stream is halted in response to an error in a path of the data stream. The destination computer locates the last checkpoint number successfully received before halting the data stream. The last checkpoint number successfully received before halting the data stream is transferred to the source computer; and the source computer compares it with checkpoint numbers stored in the source computer to determine where in the data stream to resume transfer of the data. Transfer of the data is resumed in response to the last checkpoint number.

Abstract: A system and method for remote asynchronous replication or mirroring of changes in a source file system snapshot in a destination replica file system using a scan (via a scanner) of the blocks that make up two versions of a snapshot of the source file system, which identifies changed blocks in the respective snapshot files based upon differences in volume block numbers identified in a scan of the logical file block index of each snapshot. Trees of blocks associated with the files are traversed, bypassing unchanged pointers between versions and walking down to identify the changes in the hierarchy of the tree. These changes are transmitted to the destination mirror or replicated snapshot. This technique allows regular files, directories, inodes and any other hierarchical structure to be efficiently scanned to determine differences between versions thereof.

Abstract: A system and method for verifying the consistency of mirrored data sets between source and destination storage systems is provided. A destination verification module sends version information to the source storage system where the source verification module determines whether the source and destination storage systems are utilizing compatible versions of the verification module. If the destination verification module receives an acceptable version from the source, the destination module then determines a base PCPI to utilize for verification and sends a PCPI identifier of the base PCPI to the source. The source verification module generates and transmits a data stream consisting of comprising of checksum information. The destination verification module compares the received data stream with checksum information retrieved from disk. If there are mismatches, i.e. the received data differs from the data retrieved from disk, the destination verification module logs the data mismatches in a verification log file.

Abstract: Differences between data objects stored on a mass storage device can be identified quickly and efficiently by comparing block numbers stored in data structures that describe the data objects. Bit-by-bit or byte-by-byte comparisons of the objects' actual data need only be performed if the block numbers are different. Objects that share many data blocks can be compared much faster than by a direct comparison of all the objects' data. The fast comparison techniques can be used to improve storage server mirrors and database storage operations, among other applications.

Abstract: In one embodiment, the present invention provides a method for mirroring data representing a file system. The data is stored on a primary storage server and is mirrored on a secondary storage server. In the method the file system is mirrored without comparing blocks used to that represent the file system at the primary storage server, and the secondary storage server.

Abstract: A synchronous data mirroring technique includes a log forwarding process and a consistency point (CP) process. During log forwarding, a source storage server receives write requests from clients, creates a log entry for each request in nonvolatile memory, and transmits each log entry to a destination storage server at a mirror site. The destination storage server writes each log entry to a file. If a primary volume becomes inaccessible due to a failure, the file is used to produce an updated mirror volume. The CP process includes updating a primary volume based on the write requests, and updating the mirror volume to reflect the updated primary volume. The mirror volume is updated by transmitting CP data from the source storage server to the destination storage server, and using the CP data to update the mirror volume through a network administration layer and a storage layer in the destination storage server.

Abstract: A system and method for inserting checkpoints into a data stream and for restarting an asynchronous transmission of a data stream from a source file system to a destination file system is provided. The data stream can be a set of changes between a base snapshot and incremental snapshot of the source file system for update of a replicated file system on the destination. State information relating to the progress of the source in processing and transmitting the data stream is stored at regular intervals, and a checkpoint number associated with each stored segment of the state information is inserted into the data stream. The destination tracks the fall commitment of each segment of the data stream to persistent storage on the replicated file system. If an error or communication loss requires the data transfer to be restarted, the destination sends the checkpoint number associated with the last fully committed segment of the data stream.

Abstract: A system and method for remote asynchronous replication or mirroring of changes in a source file system snapshot in a destination replica file system using a scan (via a scanner) of the blocks that make up two versions of a snapshot of the source file system, which identifies changed blocks in the respective snapshot files based upon differences in volume block numbers identified in a scan of the logical file block index of each snapshot. Trees of blocks associated with the files are traversed, bypassing unchanged pointers between versions and walking down to identify the changes in the hierarchy of the tree. These changes are transmitted to the destination mirror or replicated snapshot. This technique allows regular files, directories, inodes and any other hierarchical structure to be efficiently scanned to determine differences between versions thereof.