Abstract: Technologies are described herein for providing file-level CDP of files stored on a remote storage system. A client module executing on a client computer monitors for modifications made at the client computer to files or folders stored on a virtual storage volume provided by the remote storage system. Upon detecting a modification of a monitored file, the client module sends a message to the remote storage system identifying the file. A server module executing in the remote storage system receives the message identifying the modified file and copies a version of the file containing the modification from the virtual storage volume to a backup volume located in the remote storage system.

Abstract: Technologies are described herein for achieving data consistency during a failover from a primary node to a secondary node in a storage cluster with a shared RAID array in a degraded mode. When the primary storage node receives a write I/O operation, a volume module on the primary node reads the missing data strip data from the failed disk belonging to the stripe targeted by the write I/O operation. The primary storage node communicates the missing data strip to the secondary storage node, which writes the missing strip data to an outstanding strip log. Upon the failure of the primary node, the secondary storage node reads the missing data strip from the outstanding strip log and writes the missing data strip to the shared RAID array, thus restoring data consistency to the stripe of the RAID array containing the missing data strip.

Abstract: Technologies are described herein for providing information lifecycle management (ILM)-assisted synchronous replication between a first storage server having a first current state and a second storage server having a second current state. For example, a notification is received at the first storage server that indicates the second storage server has resumed operations following a failure that occurred at a failure time. At this time, the first storage server is in the first current state and the second storage server is in the second current state that is different than the first current state. Then, ILM data for the first storage server is retrieved, and at least one block of data on the first storage server that was accessed after the failure time is identified. After identifying the at least one block of data, the identified block of data is re-synchronized between the first storage server and the second storage server.

Abstract: Technologies are described herein for providing bi-directional communication between redundant storage controllers in a storage system. One of the storage controllers operates in the initiator role over a point-to-point communication channel connecting the controllers, while the other operates in the target role. To transmit a function call from a storage module executing on the initiator storage controller to a corresponding module executing on the target storage controller, the initiator storage controller issues a write command to the second storage controller over the point-to-point communication channel. In order to transmit a function call from a storage module executing on the target storage controller to a corresponding module executing on the initiator storage controller, the target storage controller responds to a pending anticipatory read command sent from the initiator storage controller over the point-to-point communication channel.

Abstract: A system, method, apparatus, and computer-readable medium are described for providing redundancy in a storage system. According to one method, maps are generated and stored that define stripe patterns for storing data on the storage nodes of a storage cluster. The maps are defined such that when a new storage node is added to the cluster, no movement of data occurs between two storage nodes that existed in the cluster prior to the addition of the new storage node during re-striping, and such that the data stored on each storage node is mirrored on another storage node. Storage nodes may also be designated as an owner or a peer for each storage zone. Input/output operations received at an owner node are fielded directly and mirrored to the peer node, while input/output operations received at a peer node are redirected to the owner node for processing.

Abstract: Technologies are described herein for providing efficient backup of multiple versions of a data file using data de-duplication. A CDP module copies blocks of data of a specific block size from a current version of the data file to a master file maintained in a backup location for the data file. Only blocks of data not matching a unique data block already in the master file are copied. The CDP module then creates an anterior file in the backup location for the current version of the data file containing an ordered list of references to the unique data blocks in the master file. The order of the list of references is based on the order of occurrence of each unique data block in the modified version of the data file. Finally, the CDP module creates a posterior file in the backup location for the current version of the data file containing any remaining block of data in the data file that is less than the specific block size.

Abstract: Techniques for maintaining mirrored storage cluster data consistency can employ write-intent logging. The techniques can be scaled to any number of mirror nodes. The techniques can keep track of any outstanding I/Os, data in caches, and data that has gone out of sync between mirrored nodes due to link failures. The techniques can ensure that a power failure on any of the storage nodes does not result in inconsistent data among the storage nodes. The techniques may keep track of outstanding I/Os using a minimal memory foot-print and having a negligible impact on the I/O performance. Properly choosing the granularity of the system for tracking outstanding I/Os can result in a minimal amount of data requiring transfer to synchronize the mirror nodes. The capability to vary the granularity based on physical and logical parameters of the storage volumes may provide performance benefits.

Abstract: Technologies for providing block-level continuous data protection can operate without additional external devices. I/O operations to a storage volume may be logged to a sequential journal volume. The logging can occur in parallel to the I/O operation thereby having little, or no, impact on the performance of the storage system. Previous data need not be backed up; instead only new I/O operations may be recorded in the journal or log. Snapshot events may also be recorded to the logging journal. When a volume is to be recovered, a snapshot can be mounted and I/O operations after the snapshot creation, but prior to the recovery point, can be played back onto the snapshot. Operators may be provided with a flexible mechanism for reviewing and recovering data after a data loss. Using snapshots and I/O journals, a volume can be rolled back to a desired point nearly instantaneously.

Abstract: A method, system, apparatus, and computer-readable medium are provided for implementing caching in a data storage system. According to aspects of the invention, a cache module is provided that utilizes cache lines sized according to a snapshot chunk size or an allocation unit size. The cache module utilizes cache header data structures corresponding to cache lines, each of which is assigned a device and logical block access range when active. The active headers are arranged in a set of hash queues. A free queue corresponds to the list of unused cache headers and a dirty queue corresponding to a list of unflushed cache headers. The cache header contains sector-level bitmaps of the cache line, specifying on a per sector granularity the bits that are dirty and valid. Flushing is performed by copying the dirty bitmap into a temporary memory location and flushing the bits set in it, while resetting the dirty bitmap and allowing writes to it.

Abstract: Technologies for an automated testing environment (ATE) framework can support the creation and execution of various automated tests on data storage systems. The ATE framework can support various configurations of multiple client systems exercising multiple storage target systems. A connectionless communication model can be supported among elements within the ATE. Complex tests can be subdivided into smaller jobs that can be executed sequentially, in parallel, or any combination thereof. Sophisticated scheduling mechanisms can include dependency relationships that can require one job to finish before another job begins. Multiple storage targets and multiple initiators can be involved in various configurations and combinations for specific testing tasks. The ATE framework can support extensibility, test tracking, reporting of test results, and fault injection.

Abstract: A method, system, apparatus, and computer-readable medium are described for the background movement of data between nodes in a storage cluster. According to one method, exception tables are generated that include data identifying the areas on each node that need to be migrated to another node and the areas on each node that are to be migrated from another node. The exception tables may be generated in response to the failure of a node in the storage cluster or in other circumstances. A background resynchronization thread utilizes the contents of the exception tables to migrate data between the nodes of the cluster, thereby synchronizing the data stored on the nodes. Input/output operations directed to an area that is to be migrated from another node are redirected to the other node in accordance with a timeout period associated with the input/output operations.

Abstract: Technologies are provided for reclaiming a file system coupled to a block device. In these technologies, a dummy file is created. The dummy file is expanded to fill a volume in the block device. A mapping of the dummy file to locations in the volume is retrieved. When the mapping of the dummy file to the locations in the volume is retrieved, corresponding space in the volume is reclaimed. This space may include the locations in the volume corresponding to the mapping of the dummy file. When this space in the volume is reclaimed, the dummy file is deleted.

Abstract: A method, system, and apparatus for expanding the storage capacity of a data storage system are provided. According to one method, the physical storage devices that store the data and metadata for one or more logical disks are organized into a container. A logical disk segment is created within the container for each logical disk. When storage capacity is added through the addition of logical disks, a new logical disk segment is added to the end of the container for the new logical disk. When storage capacity is added through the addition of capacity to an existing logical disk, a new logical disk segment is added to the end of the container for the new logical disk segment within the logical disk. The devices within a container may be migrated between server computers or between CPUs with their associated mass storage controllers.

Abstract: Techniques for maintaining mirrored storage cluster data consistency on systems with two-node, highly available storage solutions can employ an initiator-side agent operable to prevent split-brain scenarios. Split brain syndrome can be avoided, information identifying changes of synchronization states can be maintained, and both graceful and ungraceful shutdowns (or failures) of either one of the nodes or of the intelligent initiator itself can be mitigated. Technology presented herein supports load balancing and hot failover/failback in systems that may feature redundant network connectivity. Moreover, a method is supported for communicating storage cluster status between the storage nodes and the initiator.

Abstract: A method, system, apparatus, and computer-readable medium are described for providing distributed hot-spare storage in a redundant storage cluster. According to one method, a portion of the unutilized space on the storage cluster is utilized as a distributed hot-spare storage node. Through this mechanism, a redundant storage cluster with N storage nodes may be contracted to a redundant array with N?1 nodes. Thin provisioning and intelligent data placement may also be utilized to implement the distributed hot-spare storage node. Through repeated application of such methods and systems, the failure of any storage node or the sequential failure of multiple storage nodes within a redundant storage cluster results in the recreation of the cluster as a redundant storage array with one fewer node, but with the same redundancy.

Abstract: Dynamic provisioning of available space in a data storage system without having to configure partitions at system startup is presented. A system table may be maintained with entries corresponding to provisions within the available physical capacity of a data storage system. A volume table may be maintained that includes entries corresponding to territories within a logical data storage volume. When a data write operation is requested, a determination may be made as to whether physical space has been allocated for the territory in the volume that is to be written. If physical space has not yet been allocated, the necessary physical space may be allocated for the territory within the logical volume that is to receive the written data and the requested write operation may be performed. Metadata can be written to disk as to simplify recover from system crashes and unclean shutdowns.

Abstract: Technologies are described for implementing a migration mechanism in a storage system containing multiple tiers of storage with each tier having different cost and performance parameters. Access statistics can be collected for each territory, or storage entity, within the storage system. Data that is accessed more frequently can be migrated toward higher performance storage tiers while data that is accessed less frequently can be migrated towards lower performance storage tiers. The placement of data may be governed first by the promotion of territories with higher access frequency to higher tiers. Secondly, data migration may be governed by demoting territories to lower tiers to create room for the promotion of more eligible territories from the next lower tier. In instances where space is not available on the next lower tier, further demotion may take place to an even lower tier in order to make space for the first demotion.

Abstract: Technologies are described herein for caching variably sized access control lists (ACLs) in a data storage system utilizing page object caching. A request to access a file is received. An inode number of the requested file is identified. A hash key based on the inode number and a predefined hash value is determined. A determination is made as to whether the hash key is contained in a hash list. In response to determining that the hash key is contained in the hash list, access to a page object in the memory is provided. The page object contains an access control list (ACL) associated with the requested file. The ACL is retrieved from the page object.

Abstract: Technologies are described herein for boot caching to accelerate boot access in a data storage system providing boot consolidation. Boot caching may be provided to improve boot access for multiple clients concurrently booting from a common storage server. Boot statistics may be collected per user or client. The boot statistics can determine common patterns within the boot procedures of the clients and may be used to compute boot caching tables. These tables may then be used during boot caching to accelerate booting of clients. A boot caching mechanism can be implemented to support improved booting performance by caching data blocks common to multiple booting volumes. This approach can leverage the condition that much of the data loaded by clients at boot time may be substantially identical, because booting can involve loading the same operating system components into each client.

Abstract: Technologies are described herein for skipping snapshots during a snapshot replication process in order to maintain an established recovery point objective. A first snapshot is selected after a previously replicated snapshot. A creation time of the first snapshot is retrieved. A difference between a current time and the creation time of the first snapshot is computed. A determination is made as to whether the difference is less than a recovery point objective. If the difference is less than a recovery point objective, then the first snapshot is replicated in the snapshot replicated process. If the difference is not less than a recovery point objective, then the first snapshot is skipped through the snapshot replication process.