Abstract: Systems, methods, and computer-readable media for intelligent distribution of data in a storage cluster are described herein. An example method can include striping a volume across a plurality of storage nodes. For example, a plurality of logical blocks of the volume can be distributed to the storage nodes in relation to respective sizes of the storage nodes. Additionally, the method can include maintaining a cluster volume table (CVT) storing information regarding distribution of the logical blocks across the storage nodes. The CVT can include a plurality of entries, where each of the entries in the CVT can include information identifying a respective owner storage node of a respective logical block. Optionally, a factor can be used to determine a number of logical blocks distributed to a storage node for each stripe when striping the volume across the storage nodes.

Abstract: An example computer-implemented method for performing efficient patrol read operations in a storage system including a plurality of disks organized in a RAID array can include determining an I/O load on the storage system, identifying at least one portion of an available storage capacity containing valid data and reading the portion of the available storage capacity containing the valid data. Optionally, the portion of the available storage capacity containing the valid data is the only portion read during the patrol read operations. The method can also include determining whether a medium error exists in the portion of the available storage capacity containing the valid data. If a medium error exists, the method can include fixing the medium error. Additionally, the above operations can optionally be performed only when the I/O load on the storage system is less than a predetermined threshold.

Abstract: Technologies are described for implementing a migration mechanism in a storage system containing multiple tiers of storage with each tier having different cost and performance parameters. Access statistics can be collected for each territory, or storage entity, within the storage system. Data that is accessed more frequently can be migrated toward higher performance storage tiers while data that is accessed less frequently can be migrated towards lower performance storage tiers. The placement of data may be governed first by the promotion of territories with higher access frequency to higher tiers. Secondly, data migration may be governed by demoting territories to lower tiers to create room for the promotion of more eligible territories from the next lower tier. In instances where space is not available on the next lower tier, further demotion may take place to an even lower tier in order to make space for the first demotion.

Abstract: A first RAID module is added to a first RAID controller and a second RAID module is added to a second RAID controller. An array of physical disks is partitioned into two partitions across the array of physical disks. The first partition is assigned to the first RAID module and the second partition is exposed to the second RAID module. Each of the RAID modules exposes their respective partitions to their associated RAID controller as a single array. Each RAID module further receives I/O from its respective RAID controller, and translates the I/O to access its associated partition.

Abstract: Disclosed is a policy-based I/O dispatcher that intercepts I/O requests from several individual volumes made to a shared storage system (such as a RAID array) and manages these I/O requests so that no single volume monopolizes the I/O bandwidth and excludes the other volumes from timely completing their own I/O requests. Also disclosed are a plurality of attributes that may be assigned to each volume sharing a disk array to effectively manage the sharing of the disk array among the volumes. The I/O policy manager may handle I/O requests from several sources including volume-based iSCSI I/O requests as well as the file-share-based XFS I/O requests.

Abstract: Technologies are described for implementing a migration mechanism in a data storage system containing multiple tiers of storage with each tier having different cost and performance parameters. Access statistics can be collected for each territory, or storage entity, within the storage system. Data that is accessed more frequently can be migrated toward higher performance storage tiers while data that is accessed less frequently can be migrated towards lower performance storage tiers. Each tier can be associated with a range of ILM statistics referred to as the bucket for that tier. A pivot table may be provided that relates the tiers and the buckets. Operations on the pivot table can provide counts of how many territories may be promoted or demoted between any two pairs of tiers.

Abstract: Technologies are described herein for providing efficient backup of multiple versions of a data file using data de-duplication. A CDP module copies blocks of data of a specific block size from a current version of the data file to a master file maintained in a backup location for the data file. Only blocks of data not matching a unique data block already in the master file are copied. The CDP module then creates an anterior file in the backup location for the current version of the data file containing an ordered list of references to the unique data blocks in the master file. The order of the list of references is based on the order of occurrence of each unique data block in the modified version of the data file. Finally, the CDP module creates a posterior file in the backup location for the current version of the data file containing any remaining block of data in the data file that is less than the specific block size.

Abstract: A method, system, and apparatus for expanding the storage capacity of a data storage system are provided. According to one method, the physical storage devices that store the data and metadata for one or more logical disks are organized into a container. A logical disk segment is created within the container for each logical disk. When storage capacity is added through the addition of logical disks, a new logical disk segment is added to the end of the container for the new logical disk. When storage capacity is added through the addition of capacity to an existing logical disk, a new logical disk segment is added to the end of the container for the new logical disk segment within the logical disk. The devices within a container may be migrated between server computers or between CPUs with their associated mass storage controllers.

Abstract: Dynamic provisioning of available space in a data storage system without having to configure partitions at system startup is presented. A system table may be maintained with entries corresponding to provisions within the available physical capacity of a data storage system. A volume table may be maintained that includes entries corresponding to territories within a logical data storage volume. When a data write operation is requested, a determination may be made as to whether physical space has been allocated for the territory in the volume that is to be written. If physical space has not yet been allocated, the necessary physical space may be allocated for the territory within the logical volume that is to receive the written data and the requested write operation may be performed. Metadata can be written to disk as to simplify recover from system crashes and unclean shutdowns.

Abstract: Technologies are described for implementing a migration mechanism in a storage system containing multiple tiers of storage with each tier having different cost and performance parameters. Access statistics can be collected for each territory, or storage entity, within the storage system. Data that is accessed more frequently can be migrated toward higher performance storage tiers while data that is accessed less frequently can be migrated towards lower performance storage tiers. The placement of data may be governed first by the promotion of territories with higher access frequency to higher tiers. Secondly, data migration may be governed by demoting territories to lower tiers to create room for the promotion of more eligible territories from the next lower tier. In instances where space is not available on the next lower tier, further demotion may take place to an even lower tier in order to make space for the first demotion.

Abstract: Technologies are described herein for skipping snapshots during a snapshot replication process in order to maintain an established recovery point objective. A first snapshot is selected after a previously replicated snapshot. A creation time of the first snapshot is retrieved. A difference between a current time and the creation time of the first snapshot is computed. A determination is made as to whether the difference is less than a recovery point objective. If the difference is less than a recovery point objective, then the first snapshot is replicated in the snapshot replicated process. If the difference is not less than a recovery point objective, then the first snapshot is skipped through the snapshot replication process.

Abstract: Technologies are described for implementing a migration mechanism in a data storage system containing multiple tiers of storage with each tier having different cost and performance parameters. Access statistics can be collected for each territory, or storage entity, within the storage system. Data that is accessed more frequently can be migrated toward higher performance storage tiers while data that is accessed less frequently can be migrated towards lower performance storage tiers. Each tier can be associated with a range of ILM statistics referred to as the bucket for that tier. A pivot table may be provided that relates the tiers and the buckets. Operations on the pivot table can provide counts of how many territories may be promoted or demoted between any two pairs of tiers.

Abstract: Technologies are described for implementing a system that schedules background workflows in a storage system. Historical load statistics can be analyzed to estimate the preferred times in the future during which to schedule background workflows. After collecting load statistics related to the storage system, natural cyclical patterns in the load data can be determined as the predominant periodicities of the load data are selected from peaks in an autocorrelation of the collected data. A linear predictor may be formed as a weighted sum of the historical data sampled at the determined periodicities. Error minimization, such as the method of least squares, may be used to determine the values of the weighting in order to minimize an error between the load predictor and the measured load. Predicting future loads can improve the scheduling of background operations by avoiding conflicts with normal system operations.

Abstract: Technologies for high capacity storage servers with thin provisioning can support an increased storage capacity and an increased number of snapshots within a data storage system while maintaining a reduced memory footprint. Flexible virtual address translation can support both direct, and indirect, translation from a virtual address to an address in physical storage. A data structure, referred to as a volume table, may be provided for supporting the virtual to physical address translation. Multiple volume tables for the various volumes within a data storage system can be stored together in a global volume table. Granularities of storage allocation units, such as territories, provisions, and chunks can be reduced to improve efficiencies in the operation of the storage system. Processes for handling volume and snapshot I/O operations with various data structures can contribute to improved efficiencies while supporting increased storage capacities and an increased number of snapshots.

Abstract: Technologies for efficient synchronous replication across heterogeneous storage nodes can provide the performance of high-speed storage units while leveraging low-cost and high-capacity backup storage units within the same system. The performance of low-cost, high-capacity hard disks may be improved by initially storing data into sequential physical locations. A sequential journal of I/Os may be used in a replicated secondary node to allow for rapid completion of I/Os. A separate background process can later scatter the sequentially logged I/O data into its proper location for storage. A programmable n-way router can be configured to route I/Os as needed to improve overall performance of the storage unit. A secondary node log device can also be used to provide continuous data protection (CDP). Lastly, packetizing together I/Os prior to delivery to a secondary node may reduce interrupts and context switches in the primary node, thereby improving performance of the storage system.

Abstract: Technologies are described for implementing a system that schedules background workflows in a storage system. The background workflows may be automated maintenance tasks that require data movement, or any other tasks that would otherwise compete with normal input/output operations. Historical load statistics can be analyzed to estimate the preferred times in the future during which to schedule background workflows. After collecting data access statistics and usage patterns for several days or weeks, a number of averages can be calculated from the statistics at various natural periods of system utilization. For example, daily, weekly, or other periods. A weighted average can be established where weighting can place more emphasis on recent data. Such averages can serve as predictors of future loads having similar periodic relationships to the historical load data. Predicting future loads can improve the scheduling of background operations by avoiding conflicts with normal system operations.

Abstract: Technologies are described for implementing a migration mechanism in a data storage system containing multiple tiers of storage with each tier having different cost and performance parameters. Access statistics can be collected for each territory, or storage entity, within the storage system. Data that is accessed more frequently can be migrated toward higher performance storage tiers while data that is accessed less frequently can be migrated towards lower performance storage tiers. Each tier can be associated with a range of ILM statistics referred to as the bucket for that tier. A pivot table may be provided that relates the tiers and the buckets. Operations on the pivot table can provide counts of how many territories may be promoted or demoted between any two pairs of tiers.

Abstract: A method, system, and apparatus for expanding the storage capacity of a data storage system are provided. According to one method, the physical storage devices that store the data and metadata for one or more logical disks are organized into a container. A logical disk segment is created within the container for each logical disk. When storage capacity is added through the addition of logical disks, a new logical disk segment is added to the end of the container for the new logical disk. When storage capacity is added through the addition of capacity to an existing logical disk, a new logical disk segment is added to the end of the container for the new logical disk segment within the logical disk. The devices within a container may be migrated between server computers or between CPUs with their associated mass storage controllers.

Abstract: Technologies are described for implementing locks to control I/O operations in a storage system. The lock implementations can have reduced resource requirements for memory and processing. Through the utilization of the technologies and concepts presented herein, an I/O process preparing to acquire a write lock can advertise its need for the lock for a pre-determined time before the lock is granted. The time period for advertisement can be specified so that all I/O operations that were initiated before the advertisement are guaranteed to have completed by the time the advertisement period is over. As such, a lock controller may only need to track outstanding I/Os that begin once the advertisement starts. This can reduce the typical requirement to maintain and process a lock structure for every unit of storage in the system at all times.

Abstract: A method, system, apparatus, and computer-readable medium are provided for dynamically provisioning available space in a data storage system without having to configure partitions at system startup. According to one method, a system table is maintained that includes entries corresponding to provisions within the available physical capacity of a data storage system. A volume table is also maintained that includes entries corresponding to territories within a logical data storage volume. When a data write operation is requested, a determination is made as to whether physical space has been allocated for the territory in the volume that is to be written. If physical space has not yet been allocated, the necessary physical space is allocated for the territory within the logical volume that is to receive the written data by creating a pointer to an available entry in the system table in an entry in the volume table corresponding to the territory to be written.