Abstract: A method, system, apparatus, and computer-readable medium are provided for performing read-ahead operations for sequential read operations. A method includes maintaining a bitmap including a plurality of bits, each bit corresponding to a sector of the disk cache and containing data indicating whether the corresponding sector is valid and can be used to satisfy read requests. The method includes receiving a request to read a sector of the disk cache, and in response, identifying a bit in the bitmap that corresponds to the requested sector. Further, the method includes determining whether the disk cache contains valid data for a sector previous to the requested sector by examining a bit in the bitmap previous to the bit that corresponds to the requested sector, and in response, reading sequentially into the disk cache sectors of the disk cache corresponding to bits in the bitmap following the bit corresponding to the requested sector.

Abstract: Technologies are described herein for boot caching to accelerate boot access in a data storage system providing boot consolidation. Boot caching may be provided to improve boot access for multiple clients concurrently booting from a common storage server. Boot statistics may be collected per user or client. The boot statistics can determine common patterns within the boot procedures of the clients and may be used to compute boot caching tables. These tables may then be used during boot caching to accelerate booting of clients. A boot caching mechanism can be implemented to support improved booting performance by caching data blocks common to multiple booting volumes. This approach can leverage the condition that much of the data loaded by clients at boot time may be substantially identical, because booting can involve loading the same operating system components into each client.

Abstract: Technologies for eliminating duplicate data within a storage system can efficiently identify and eliminate duplication by remapping borrower regions to share physical storage space with lender regions. Block-level de-duplication can co-exist with storage architectures for thin provisioning and snapshot management. Lending maps can track redirected pointers from borrower regions to shared physical storage from lender regions. The lending maps can track the freed status of regions to support efficient write I/O operations without defaulting to unnecessary read-modify-write cycles to complete data writes. Redundancy of de-duplicated data can maintain one or more copies to support recovery from media errors. Candidate regions for de-duplication can be identified by monitoring the times and patterns of data access operations. A sampled mechanism for calculating and comparing signatures of data blocks can support the efficient identification of duplicated data within the storage system.

Abstract: Technologies are described herein for performing a multi-protocol data transfer. A first protocol that guarantees reliability and in-order delivery of data packets is utilized to transmit control information for a data transfer. Once the control information has been transmitted and received at the destination, a second protocol is utilized to transfer the actual data. The second protocol does not guarantee data reliability or in-order delivery. If any data packets are not delivered to the destination using the second protocol, either the first protocol, the second protocol, or a combination can be utilized to retransmit the missing packets.

Abstract: Technologies for an automated testing environment (ATE) framework can support the creation and execution of various automated tests on data storage systems. The ATE framework can support various configurations of multiple client systems exercising multiple storage target systems. A connectionless communication model can be supported among elements within the ATE. Complex tests can be subdivided into smaller jobs that can be executed sequentially, in parallel, or any combination thereof. Sophisticated scheduling mechanisms can include dependency relationships that can require one job to finish before another job begins. Multiple storage targets and multiple initiators can be involved in various configurations and combinations for specific testing tasks. The ATE framework can support extensibility, test tracking, reporting of test results, and fault injection.

Abstract: A method, system, apparatus, and computer-readable medium are provided for implementing caching in a data storage system. According to aspects of the invention, a cache module is provided that utilizes cache lines sized according to a snapshot chunk size or an allocation unit size. The cache module utilizes cache header data structures corresponding to cache lines, each of which is assigned a device and logical block access range when active. The active headers are arranged in a set of hash queues. A free queue corresponds to the list of unused cache headers and a dirty queue corresponding to a list of unflushed cache headers. The cache header contains sector-level bitmaps of the cache line, specifying on a per sector granularity the bits that are dirty and valid. Flushing is performed by copying the dirty bitmap into a temporary memory location and flushing the bits set in it, while resetting the dirty bitmap and allowing writes to it.

Abstract: A method, system, apparatus, and computer-readable medium are described for the background movement of data between nodes in a storage cluster. According to one method, exception tables are generated that include data identifying the areas on each node that need to be migrated to another node and the areas on each node that are to be migrated from another node. The exception tables may be generated in response to the failure of a node in the storage cluster or in other circumstances. A background resynchronization thread utilizes the contents of the exception tables to migrate data between the nodes of the cluster, thereby synchronizing the data stored on the nodes. Input/output operations directed to an area that is to be migrated from another node are redirected to the other node in accordance with a timeout period associated with the input/output operations.

Abstract: A method, system, apparatus, and computer-readable medium are described for providing distributed hot-spare storage in a redundant storage cluster. According to one method, a portion of the unutilized space on the storage cluster is utilized as a distributed hot-spare storage node. Through this mechanism, a redundant storage cluster with N storage nodes may be contracted to a redundant array with N?1 nodes. Thin provisioning and intelligent data placement may also be utilized to implement the distributed hot-spare storage node. Through repeated application of such methods and systems, the failure of any storage node or the sequential failure of multiple storage nodes within a redundant storage cluster results in the recreation of the cluster as a redundant storage array with one fewer node, but with the same redundancy.

Abstract: Technologies are described herein for boot caching to accelerate boot access in a data storage system providing boot consolidation. Boot caching may be provided to improve boot access for multiple clients concurrently booting from a common storage server. Boot statistics may be collected per user or client. The boot statistics can determine common patterns within the boot procedures of the clients and may be used to compute boot caching tables. These tables may then be used during boot caching to accelerate booting of clients. A boot caching mechanism can be implemented to support improved booting performance by caching data blocks common to multiple booting volumes. This approach can leverage the condition that much of the data loaded by clients at boot time may be substantially identical, because booting can involve loading the same operating system components into each client.

Abstract: A method, system, apparatus, and computer-readable medium are provided for taking snapshots of a storage volume. According to aspects of one method, each snapshot is represented as a unique sequence number. Every fresh write access to a volume in a new snapshot lifetime is allocated a new section in the disk, called a provision, which is labeled with the sequence number. Read-modify-write operations are performed on a sub-provision level at the granularity of a chunk. Because each provision contains chunks with valid data and chunks with invalid data, a bitmap is utilized to identify the valid and invalid chunks with each provision. Provisions corresponding to different snapshots are arranged in a linked list. Branches from the linked list can be created for storing writable snapshots. Provisions may also be deleted and rolled back by manipulating the contents of the linked lists.

Abstract: A method, system, apparatus, and computer-readable medium are provided for taking snapshots of a storage volume. According to aspects of one method, each snapshot is represented as a unique sequence number. Every fresh write access to a volume in a new snapshot lifetime is allocated a new section in the disk, called a provision, which is labeled with the sequence number. Read-modify-write operations are performed on a sub-provision level at the granularity of a chunk. Because each provision contains chunks with valid data and chunks with invalid data, a bitmap is utilized to identify the valid and invalid chunks with each provision. Provisions corresponding to different snapshots are arranged in a linked list. Branches from the linked list can be created for storing writable snapshots. Provisions may also be deleted and rolled back by manipulating the contents of the linked lists.

Abstract: Technologies for efficient synchronous replication across heterogeneous storage nodes can provide the performance of high-speed storage units while leveraging low-cost and high-capacity backup storage units within the same system. The performance of low-cost, high-capacity hard disks may be improved by initially storing data into sequential physical locations. A sequential journal of I/Os may be used in a replicated secondary node to allow for rapid completion of I/Os. A separate background process can later scatter the sequentially logged I/O data into its proper location for storage. A programmable n-way router can be configured to route I/Os as needed to improve overall performance of the storage unit. A secondary node log device can also be used to provide continuous data protection (CDP). Lastly, packetizing together I/Os prior to delivery to a secondary node may reduce interrupts and context switches in the primary node, thereby improving performance of the storage system.

Abstract: Technologies are described herein for high-performance and space efficient journaling mechanisms. A journal queue can buffer incoming I/O operations. The queue can be read by a journal write module. The journal write module can form metadata headers to store into the journal along with each I/O operation. Compound metadata headers may be formed for multiple I/O operations and the combination of multiple I/O operations may be efficiently stored to the journal as one journal write. Multiple journal entries may be written into the journal in parallel while maintaining the sequential ordering of the journal. A queue depth threshold can be established for determining if journal entries should be combined into compound entries or not. Multiple concurrent readers can support continuous data protection, and various data replication features. Each reader can be represented as a reader pointer and a journal writer can invalidate reader pointers when related data is overwritten.

Abstract: A method, system, apparatus, and computer-readable medium are provided for storing data at a virtual tape library (“VTL”) computer or server. According to one method, a VTL computer maintains one or more storage volumes for use by initiators on an array of mass storage devices. Space on each of the volumes is allocated using thin provisioning. The VTL computer may also include a cache memory that is at least the size of a full stripe of the array. Write requests received at the VTL computer are stored in the cache memory until a full stripe of data has been received. Once a full stripe of data has been received, the full stripe of data is written to the array at once. The array utilized by the VTL computer may include a hot spare mass storage device. When a failed mass storage device is identified, only the portions of the failed device that have been previously written are rebuilt onto the hot spare. The array may be maintained using RAID-5.

Abstract: Techniques for continuous data protection can include creating snapshots of one or more underlying storage volumes upon specific file system events. Generating snapshots upon every file close event can protect the files in a storage system by keeping a snapshot of every version or modification of each file. Removal of redundant snapshots can mitigate the impact on storage capacity associated with creating these large numbers of volume snapshots upon each file close event. Additionally, file closure lists can be employed to allow generating snapshots only when a previously closed file is reopened. Such an approach can protect the previous version of a file prior to the opening of a new version of the file. Such an approach can also mitigate storage capacity impact without the creation of redundant snapshots.

Abstract: A method, system, apparatus, and computer-readable medium are described for the background movement of data between nodes in a storage cluster. According to one method, exception tables are generated that include data identifying the areas on each node that need to be migrated to another node and the areas on each node that are to be migrated from another node. The exception tables may be generated in response to the failure of a node in the storage cluster or in other circumstances. A background resynchronization thread utilizes the contents of the exception tables to migrate data between the nodes of the cluster, thereby synchronizing the data stored on the nodes. Input/output operations directed to an area that is to be migrated from another node are redirected to the other node in accordance with a timeout period associated with the input/output operations.

Abstract: A method, system, apparatus, and computer-readable medium are described for providing distributed hot-spare storage in a redundant storage cluster. According to one method, a portion of the unutilized space on the storage cluster is utilized as a distributed hot-spare storage node. Through this mechanism, a redundant storage cluster with N storage nodes may be contracted to a redundant array with N?1 nodes. Thin provisioning and intelligent data placement may also be utilized to implement the distributed hot-spare storage node. Through repeated application of such methods and systems, the failure of any storage node or the sequential failure of multiple storage nodes within a redundant storage cluster results in the recreation of the cluster as a redundant storage array with one fewer node, but with the same redundancy.

Abstract: A method, system, apparatus, and computer-readable medium are provided for taking snapshots of a storage volume. According to aspects of one method, each snapshot is represented as a unique sequence number. Every fresh write access to a volume in a new snapshot lifetime is allocated a new section in the disk, called a provision, which is labeled with the sequence number. Read-modify-write operations are performed on a sub-provision level at the granularity of a chunk. Because each provision contains chunks with valid data and chunks with invalid data, a bitmap is utilized to identify the valid and invalid chunks with each provision. Provisions corresponding to different snapshots are arranged in a linked list. Branches from the linked list can be created for storing writable snapshots. Provisions may also be deleted and rolled back by manipulating the contents of the linked lists.

Abstract: A method, system, apparatus, and computer-readable medium are described for the background movement of data between nodes in a storage cluster. According to one method, exception tables are generated that include data identifying the areas on each node that need to be migrated to another node and the areas on each node that are to be migrated from another node. The exception tables may be generated in response to the failure of a node in the storage cluster or in other circumstances. A background resynchronization thread utilizes the contents of the exception tables to migrate data between the nodes of the cluster, thereby synchronizing the data stored on the nodes. Input/output operations directed to an area that is to be migrated from another node are redirected to the other node. The methods, systems, apparatus, and computer-readable medium may be utilized with operations including the rebuilding, expansion, contraction, and compaction of a storage cluster.

Abstract: A method, system, apparatus, and computer-readable medium are provided for taking snapshots of a storage volume. According to aspects of one method, each snapshot is represented as a unique sequence number. Every fresh write access to a volume in a new snapshot lifetime is allocated a new section in the disk, called a provision, which is labeled with the sequence number. Read-modify-write operations are performed on a sub-provision level at the granularity of a chunk. Because each provision contains chunks with valid data and chunks with invalid data, a bitmap is utilized to identify the valid and invalid chunks with each provision. Provisions corresponding to different snapshots are arranged in a linked list. Branches from the linked list can be created for storing writable snapshots. Provisions may also be deleted and rolled back by manipulating the contents of the linked lists.