Abstract: Technologies are described herein for providing networked RAID in a virtualized storage cluster. The storage capacity of a storage cluster having two or more storage nodes is organized into tiers. A portion of the available storage capacity is allocated to one tier that is organized using chained declustering. Another portion of the available storage capacity is allocated to another tier that is organized using a networked RAID configuration. The storage cluster monitors the frequency at which data in the storage cluster is accessed. Frequently used data that is stored in the network RAID-configured tier is promoted to the chained declustered-configured tier. Infrequently used data that is stored in the chained declustered-configured tier is demoted to the network RAID-configured tier. The zone size for zones stored in the chained declustered-configured tier may be different than for zones stored in the network RAID-configured tier.

Abstract: Technologies are provided for integrating a caching module into a data storage system architecture. A request to perform an I/O operation is received at the caching module from an initiator. The caching module provides a notification to a storage driver that the request will be delivered to the storage driver in the future. The storage driver then determines whether the request can be satisfied. If the storage driver determines that it will be able to satisfy the I/O request, the storage driver can optimize the allocation and placement of the I/O request before receiving the actual I/O request. When the caching module subsequently sends the actual I/O request to the storage driver, the storage driver utilizes the previously allocated space and optimized placement to perform the requested I/O operation.

Abstract: Technologies are described herein for providing assisted asynchronous replication between a first storage server having a first current state and a second storage server having a second current state. An instruction to roll back a first storage server from the first current state to a previous state according to a previous snapshot is received. The first storage server is then rolled back from the first current state to the previous state according to the previous snapshot. A determination is made as to whether the previous snapshot is available on a second storage server. Assistance data is then retrieved. A last consistency point common to the first storage server and the second storage server based on the assistance data is identified, where the last consistency point is at or prior to the previous snapshot. The second storage server is rolled back from the second current state to the last consistency point.

Abstract: Technologies are described for implementing container space management (CSM) in a data storage system. Performance may be improved by ensuring that the maximum number of drive spindles is made available to I/O loads. This may be accomplished by distributing data equally amongst all drive spindles, even if the drive spindles are in different logical drives. A workflow analysis may be performed to determine when the I/O load on the storage system is low and thus identify preferred times for background data re-striping. Data may be re-striped in a minimum number of movements using a data placement algorithm and data placement maps to select data to be re-striped. The data may be re-striped such that the most frequently accessed data within a tier is moved first. Such re-striping can be performed in a manner which optimizes the performance of the storage system both during, and after, the re-striping process.

Abstract: A method, system, apparatus, and computer-readable medium are provided for taking snapshots of a storage volume. According to aspects of one method, each snapshot is represented as a unique sequence number. Every fresh write access to a volume in a new snapshot lifetime is allocated a new section in the disk, called a provision, which is labeled with the sequence number. Read-modify-write operations are performed on a sub-provision level at the granularity of a chunk. Because each provision contains chunks with valid data and chunks with invalid data, a bitmap is utilized to identify the valid and invalid chunks with each provision. Provisions corresponding to different snapshots are arranged in a linked list. Branches from the linked list can be created for storing writable snapshots. Provisions may also be deleted and rolled back by manipulating the contents of the linked lists.

Abstract: Technologies are described for implementing a migration mechanism in a data storage system containing multiple tiers of storage with each tier having different cost and performance parameters. Access statistics can be collected for each territory, or storage entity, within the storage system. Data that is accessed more frequently can be migrated toward higher performance storage tiers while data that is accessed less frequently can be migrated towards lower performance storage tiers. Each tier can be associated with a range of ILM statistics referred to as the bucket for that tier. A pivot table may be provided that relates the tiers and the buckets. Operations on the pivot table can provide counts of how many territories may be promoted or demoted between any two pairs of tiers.

Abstract: Technologies are described herein for achieving data consistency during a failover from a primary node to a secondary node in a storage cluster with a shared RAID array in a degraded mode. When the primary storage node receives a write I/O operation, a volume module on the primary node reads the missing data strip data from the failed disk belonging to the stripe targeted by the write I.O operation. The primary storage node communicates the missing data strip to the secondary storage node, which writes the missing strip data to an outstanding strip log. Upon the failure of the primary node, the secondary storage node reads the missing data strip from the outstanding strip log and writes the missing data strip to the shared RAID array, thus restoring data consistency to the stripe of the RAID array containing the missing data strip.

Abstract: Techniques for maintaining mirrored storage cluster data consistency on systems with write back cache can employ write-intent logging and periodic log clearing. The techniques can track any outstanding I/Os, data in caches, and data that has gone out of sync between mirrored nodes due to link failures. The techniques can ensure that a power failure on any of the storage nodes does not result in inconsistent data among the storage nodes. The techniques can use gate bitmaps to log outstanding I/Os. The gate bitmaps may be periodically cleared. The clearing period may be selected as to allow enough time for all caches to be flushed and for all synchronizing I/Os to complete. Efficient queuing of I/O requests in conjunction with the efficient operation of multiple gate bitmaps can reduce performance impacts of the gating or logging operations.

Abstract: A user interface is provided through which a user may specify a container for data recovery. In response to the specification of a container for recovery, another user interface is provided through which the user can select a consistency group or a journal file to be utilized for recovery. In response to receiving a selection of a journal file for data recovery, a user interface is provided through which a user may specify a point in time, an event, or a snapshot at which time recovery for the corresponding volume should occur. In response to receiving a selection of a consistency group for recovery, a user interface is provided through which a user may specify a snapshot performed across two or more volumes of the consistency group that should be utilized for recovery.

Abstract: Technologies are described for implementing a system that schedules background workflows in a storage system. Historical load statistics can be analyzed to estimate the preferred times in the future during which to schedule background workflows. After collecting load statistics related to the storage system, natural cyclical patterns in the load data can be determined as the predominant periodicities of the load data are selected from peaks in an autocorrelation of the collected data. A linear predictor may be formed as a weighted sum of the historical data sampled at the determined periodicities. Error minimization, such as the method of least squares, may be used to determine the values of the weighting in order to minimize an error between the load predictor and the measured load. Predicting future loads can improve the scheduling of background operations by avoiding conflicts with normal system operations.

Abstract: Technologies for high capacity storage servers with thin provisioning can support an increased storage capacity and an increased number of snapshots within a data storage system while maintaining a reduced memory footprint. Flexible virtual address translation can support both direct, and indirect, translation from a virtual address to an address in physical storage. A data structure, referred to as a volume table, may be provided for supporting the virtual to physical address translation. Multiple volume tables for the various volumes within a data storage system can be stored together in a global volume table. Granularities of storage allocation units, such as territories, provisions, and chunks can be reduced to improve efficiencies in the operation of the storage system. Processes for handling volume and snapshot I/O operations with various data structures can contribute to improved efficiencies while supporting increased storage capacities and an increased number of snapshots.

Abstract: A method, system, apparatus, and computer-readable medium are provided for taking snapshots of a storage volume. According to aspects of one method, each snapshot is represented as a unique sequence number. Every fresh write access to a volume in a new snapshot lifetime is allocated a new section in the disk, called a provision, which is labeled with the sequence number. Read-modify-write operations are performed on a sub-provision level at the granularity of a chunk. Because each provision contains chunks with valid data and chunks with invalid data, a bitmap is utilized to identify the valid and invalid chunks with each provision. Provisions corresponding to different snapshots are arranged in a linked list. Branches from the linked list can be created for storing writable snapshots. Provisions may also be deleted and rolled back by manipulating the contents of the linked lists.

Abstract: Technologies for efficient synchronous replication across heterogeneous storage nodes can provide the performance of high-speed storage units while leveraging low-cost and high-capacity backup storage units within the same system. The performance of low-cost, high-capacity hard disks may be improved by initially storing data into sequential physical locations. A sequential journal of I/Os may be used in a replicated secondary node to allow for rapid completion of I/Os. A separate background process can later scatter the sequentially logged I/O data into its proper location for storage. A programmable n-way router can be configured to route I/Os as needed to improve overall performance of the storage unit. A secondary node log device can also be used to provide continuous data protection (CDP). Lastly, packetizing together I/Os prior to delivery to a secondary node may reduce interrupts and context switches in the primary node, thereby improving performance of the storage system.

Abstract: Techniques for maintaining mirrored storage cluster data consistency can employ write-intent logging. The techniques can be scaled to any number of mirror nodes. The techniques can keep track of any outstanding I/Os, data in caches, and data that has gone out of sync between mirrored nodes due to link failures. The techniques can ensure that a power failure on any of the storage nodes does not result in inconsistent data among the storage nodes. The techniques may keep track of outstanding I/Os using a minimal memory foot-print and having a negligible impact on the I/O performance. Properly choosing the granularity of the system for tracking outstanding I/Os can result in a minimal amount of data requiring transfer to synchronize the mirror nodes. The capability to vary the granularity based on physical and logical parameters of the storage volumes may provide performance benefits.

Abstract: Boot consolidation based on writeable volume snapshots can leverage flexible mappings of snapshot dependencies. Such flexible mappings can be established by relating sequence numbers associated with various volume snapshots and can provide for an arbitrarily complex mapping to support a potentially unlimited number of writable snapshots in potentially any dependency configuration. Additionally, the pinning of volumes can provide support for chains of snapshots to depend from snapshots within other chains of snapshots. Merge agents can operate with writable snapshots and boot consolidation to provide system-wide updates where security patches, upgrades, and application software may be applied to a group of servers or a group of users in an efficient fashion.

Abstract: Technologies are described herein for high-performance and space efficient journaling mechanisms. A journal queue can buffer incoming I/O operations. The queue can be read by a journal write module. The journal write module can form metadata headers to store into the journal along with each I/O operation. Compound metadata headers may be formed for multiple I/O operations and the combination of multiple I/O operations may be efficiently stored to the journal as one journal write. Multiple journal entries may be written into the journal in parallel while maintaining the sequential ordering of the journal. A queue depth threshold can be established for determining if journal entries should be combined into compound entries or not. Multiple concurrent readers can support continuous data protection, and various data replication features. Each reader can be represented as a reader pointer and a journal writer can invalidate reader pointers when related data is overwritten.

Abstract: A method, system, apparatus, and computer-readable medium are provided for storing data at a virtual tape library (“VTL”) computer or server. According to one method, a VTL computer maintains one or more storage volumes for use by initiators on an array of mass storage devices. Space on each of the volumes is allocated using thin provisioning. The VTL computer may also include a cache memory that is at least the size of a full stripe of the array. Write requests received at the VTL computer are stored in the cache memory until a full stripe of data has been received. Once a full stripe of data has been received, the full stripe of data is written to the array at once. The array utilized by the VTL computer may include a hot spare mass storage device. When a failed mass storage device is identified, only the portions of the failed device that have been previously written are rebuilt onto the hot spare. The array may be maintained using RAID-5.

Abstract: Techniques for continuous data protection can include creating snapshots of one or more underlying storage volumes upon specific file system events. Generating snapshots upon every file close event can protect the files in a storage system by keeping a snapshot of every version or modification of each file. Removal of redundant snapshots can mitigate the impact on storage capacity associated with creating these large numbers of volume snapshots upon each file close event. Additionally, file closure lists can be employed to allow generating snapshots only when a previously closed file is reopened. Such an approach can protect the previous version of a file prior to the opening of a new version of the file. Such an approach can also mitigate storage capacity impact without the creation of redundant snapshots.

Abstract: Techniques for maintaining mirrored storage cluster data consistency can employ write-intent logging. The techniques can be scaled to any number of mirror nodes. The techniques can keep track of any outstanding I/Os, data in caches, and data that has gone out of sync between mirrored nodes due to link failures. The techniques can ensure that a power failure on any of the storage nodes does not result in inconsistent data among the storage nodes. The techniques may keep track of outstanding I/Os using a minimal memory foot-print and having a negligible impact on the I/O performance. Properly choosing the granularity of the system for tracking outstanding I/Os can result in a minimal amount of data requiring transfer to synchronize the mirror nodes. The capability to vary the granularity based on physical and logical parameters of the storage volumes may provide performance benefits.

Abstract: Technologies are described for implementing a system that schedules background workflows in a storage system. The background workflows may be automated maintenance tasks that require data movement, or any other tasks that would otherwise compete with normal input/output operations. Historical load statistics can be analyzed to estimate the preferred times in the future during which to schedule background workflows. After collecting data access statistics and usage patterns for several days or weeks, a number of averages can be calculated from the statistics at various natural periods of system utilization. For example, daily, weekly, or other periods. A weighted average can be established where weighting can place more emphasis on recent data. Such averages can serve as predictors of future loads having similar periodic relationships to the historical load data. Predicting future loads can improve the scheduling of background operations by avoiding conflicts with normal system operations.