Abstract: Asymmetric distributed replicated data storage systems and methods are described herein. The storage system includes zones that are independent, and autonomous. The zones include nodes that are independent and autonomous. The nodes include storage devices. When a data item is stored, it is partitioned into a plurality of data objects and a plurality of parity objects using erasure coding. The data objects and parity objects are spread across all nodes and zones in the storage system asymmetrically such that a first zone includes all of the data objects and no parity objects while the remaining zones include subsets of the data objects and all of the parity objects. The systems and methods provide for data resiliency while keeping the amount of storage space required relatively low.

Abstract: A resilient distributed replicated data storage system is described herein. The storage system includes zones that are independent, and autonomous from each other. The zones include nodes that are independent and autonomous. The nodes include storage devices. When a data item is stored, it is partitioned into a plurality of data objects and a plurality of parity objects are calculated. Reassembly instructions are created for the data item. The data objects, parity objects and reassembly instructions are spread across nodes and zones in the storage system according to a policy for the data item. When a zone is inaccessible, a virtual zone is created and used until the intended zone is available. When a read request is received, the data item is prepared from the lowest latency nodes according to the reassembly instructions, and a virtual zone is accessed in place of a real zone when the real zone is inaccessible.

Abstract: A control processor manages the power budget in a drive enclosure and is within the drive enclosure which monitors in real time a redundantly configured power supply unit, drives, interposers, and temperature sensors, and determines the power settings for each drive to avoid overload and overheating in the system. The control processor dynamically adjusts the mode of operation as needed during operation through the SAS interposer. A localized monitoring and control mechanism eliminates the need for extraneous coordination of information across various entities that access the storage. Data tunneling takes place directly between the compute nodes and target drives through the SAS expander and interposer, and does not need buffering the pending IO requests in the DRAM.

Abstract: A system for data migration between high performance computing architectures and data storage disks includes an intermediate data migration handling system which has an intermediate data storage module coupled to the computer architecture to store data received, and a data controller module which includes data management software supporting the data transfer activity between the intermediate data storage module and the disk drives in an orderly manner independent of the random I/O activity of the computer architecture. RAID calculations are performed on the data prior to storage in the intermediate storage module, as well as when reading data from it for assuring data integrity, and carrying out reconstruction of corrupted data. The data transfer to the disk drives is actuated in sequence determined by the data management software based on minimization of seeking time, tier usage, predetermined time since the previous I/O cycle, or fullness of the intermediate data storage module.

Abstract: Data storage systems and methods for storing data are described herein. The storage system includes a first storage node is configured to issue a first delivery request to a first set of other storage nodes in the storage system, the first delivery request including a first at least one data operation for each of the first set of other storage nodes and issuing at least one other delivery request, while the first delivery request remains outstanding, the at least one other delivery request including a first commit request for each of the first set of other storage nodes. The first node causes the first at least one data operation to be made active within the storage system in response to receipt of a commit indicator along with a delivery acknowledgement regarding one of the at least one other delivery request.

Abstract: Systems and methods for reducing metadata in a write-anywhere storage system are disclosed herein. The system includes a plurality of clients coupled with a plurality of storage nodes, each storage node having a plurality of primary storage devices coupled thereto. A memory management unit including cache memory is included in the client. The memory management unit serves as a cache for data produced by the clients before the data is stored in the primary storage. The cache includes an extent cache, an extent index, a commit cache and a commit index. The movement of data and metadata is by an interval tree. Methods for reducing data in the interval tree increase data storage and data retrieval performance of the system.

Abstract: Data storage systems and methods for storing data are described herein. The storage system may be integrated with or coupled with a compute cluster or super computer having multiple computing nodes. A plurality of nonvolatile memory units may be included with computing nodes, coupled with computing nodes or coupled with input/output nodes. The input/output nodes may be included with the compute cluster or super computer, or coupled thereto. The nonvolatile memory units store data items provided by the computing nodes, and the input/output nodes maintain where the data items are stored in the nonvolatile memory units via a hash table distributed among the input/output nodes. The use of a distributed hash table allows for quick access to data items stored in the nonvolatile memory units even as the computing nodes are writing large amounts of data to the storage system quickly in bursts.

Abstract: A latency reduction method for read operations of an array of N solid-state storage devices having n solid-state storage devices for data storage and p solid-state storage devices for storing parity data is provided. Utilizing the parity generation engine fault tolerance for a loss of valid data from at least two of the N solid-state storage devices, the integrity of the data is determined when N?1 of the solid-state storage devices have completed executing a read command. If the data is determined to be valid, the missing data of the Nth solid-state storage device is reconstructed and the data transmitted to the requesting processor. By that arrangement the time necessary for the Nth solid-state storage device to complete execution of the read command is saved, thereby improving the performance of the solid-state memory system.

Abstract: A data migration system between computing architectures and disk drives includes an intermediate data migration handling system having an intermediate data storage module coupled to the computer architecture, and a data controller module including data management software supporting the data transfers between the intermediate data storage module and the disk drives independent of the random I/O activity of the computer architecture. RAID calculations are performed on the data prior to storage in the intermediate storage module, as well as when read therefrom for assuring data integrity, and reconstruction of corrupted data. The data transfer to the disk drives is actuated in a sequence determined by the data management software based on predetermined criteria. The RAID capability is complimented with a real time adaptive RAID stripe size selection depending on the overall capacity of “healthy” components capable of supporting data striping both for RAID 5 and RAID 6 configurations.

Abstract: Realigning storage devices arranged as storage arrays when one of the storage arrays enters a critical state after failure of a storage device is disclosed. The method is particularly useful for RAID groups of storage devices. The method may be used with hard disk drives, solid-state drives, and other storage devices arranged as groups. The method includes identifying that a storage array of a plurality of storage arrays is in a critical condition. A critical condition storage array and a healthy storage array are identified. Both the critical condition storage array and the healthy storage array are rebuilt. The rebuilding includes configuring the critical condition storage array to include a storage device from the healthy storage array and configuring the healthy storage array to function with one less storage device. The method may be implemented in hardware, firmware, software, or a combination thereof.

Abstract: A method for auto-correction of errors in an array of solid-state storage devices having a plurality of storage channels dedicated to storing parity data to provide fault tolerance for a loss of at least two of the plurality of storage channels. A read operation from the storage channels transfers data to a plurality of channel memories. The data in the channel memories is checked to confirm the data is valid. Responsive to detection of invalid data, the data may be tested to identify the storage channel in error, including sequentially excluding data read form a different one of the plurality of channel memories from a parity check and determining the validity of data from remaining channel memories. If valid data is obtained, the storage channel from which the data was excluded is identified as the storage channel in error.

Abstract: A latency reduction method for read operations of an array of N disk storage devices (210) having n disk storage devices (210A-210H) for data storage and p of disk storage devices (210I, 210J) for storing parity data is provided. Utilizing the parity generation engine's (230) fault tolerance of for a loss of valid data from at least two of the N disk storage devices (210A-210J), the integrity of the data is determined when N?1 of the disk storage devices (210A-210J) have completed executing a read command. If the data is determined to be valid, the missing data of the Nth disk storage device is reconstructed and the data transmitted to the requesting processor (10). By that arrangement the time necessary for the Nth disk storage device to complete execution of the read command is saved, thereby improving the performance of memory system (200).

Abstract: A system for data migration between a compute cluster and disk drives by inclusion of a buffer node coupled to the compute cluster to store data received therefrom in a random fashion. The buffer node signals the computer nodes to promptly return from the I/O cycle to the computing state to improve the duty cycle of the device. The system further includes a storage controller which is coupled between the buffer node and the disk drives to schedule data transfer activity between them. The data transfers are actuated in the sequence determined based on minimization of seeking time and tier usage, and harvest priority, when the buffer node either reaches a predetermined storage space minimal level or a predetermined time has elapsed since the previous I/O cycle. The storage controller deactivates the disk drives which are not needed for the data transfer.

Abstract: A method for auto-correction of errors in an array of disk storage devices having a plurality of disk storage devices dedicated to storing parity data to provide fault tolerance for a loss of at least two of the plurality of disk storage devices. A read operation from the storage channels transfers data to a plurality of disk channel memories. The data in the disk channel memories is checked to confirm the data is valid. Responsive to detection of invalid data, the data may be tested to identify the disk storage channel in error, including sequentially excluding data read from a different one of the plurality of disk channel memories from a parity check and determining the validity of data from remaining disk channel memories. If valid data is obtained, the disk storage channel from which the data was excluded is identified as the disk storage channel in error.

Abstract: Realigning storage devices arranged as storage arrays when one of the storage arrays enters a critical state after failure of a storage device is disclosed. The method is particularly useful for RAID groups of storage devices. The method may be used with hard disk drives, solid-state drives, and other storage devices arranged as groups. The method includes identifying when a storage array of a plurality of storage arrays is in a critical condition. A critical condition storage array and a healthy storage array are identified. Both the critical condition storage array and the healthy storage array are rebuilt. The rebuilding includes configuring the critical condition storage array to include a storage device from the healthy storage array and configuring the healthy storage array to function with one less storage device. The method may be implemented in hardware, firmware, software, or a combination thereof.

Abstract: Rebuilding a storage device after failure of a storage device is disclosed. The method is particularly useful for RAID groups of hard disks. The method may also apply to other storage media arranged as a group. The method includes rebuilding a hard disk in a non-linear fashion according to a heuristic analysis of logical units of the failed hard disk. The method may be implemented in hardware, firmware, software, or a combination thereof.

Abstract: In order to detect problematic drives in random arrays of independent disks, the system measures the latency of executing command sets which are broadcast to all disks in the data storage system and the results are compared to identify which disks take substantially longer to complete the requests. Disks that take longer to complete requests are likely to be problematic and are candidates for further examination and replacement. The disks in each tier group are compared to determine if any disk in that group exhibits problems. Also, counters for each tier group are compared to determine if the problem is with the disk or with the channel of the tier group. The latency of each disk in the tier group is saved in a table to provide a histogram of the latency of the disks in the tier group. Histograms of the disks in a single tier group are compared to determine if a specific disk is problematic.

Abstract: In a data storage system, failed disk drives are switched temporarily off-line to be quickly rebuilt by executing a journaling/rebuild algorithm which tracks the updates to the failed disk drive into a journal structure created in a non-volatile memory. The journal information is used to update those data sections of the disk drive affected by updates after the disk drive is failed. The journal information is stored in bit maps indicating which portions of the disk drive have been updated with new data while the disk was failed. As an option, the system permits verification of data consistency on the data section of the disk drive which have not been affected by the updates. The journaling/rebuild of failed disks is applicable, among others, to RAID data storage systems.

Abstract: An improved duty cycle, increased effective bandwidth, and minimized power consumption are attained in a system for data migration between a compute cluster and disk drives by inclusion of a buffer node coupled to the compute cluster to store data received therefrom in a random fashion. The buffer node signals the computer nodes to promptly return from the I/O cycle to the computing state to improve the duty cycle of the device. The system further includes a storage controller which is coupled between the buffer node and the disk drives to schedule data transfer activity between them in an optimal orderly manner. The data transfers are actuated in the sequence determined based on minimization of seeking time and tier usage, and harvest priority, when the buffer node either reaches a predetermined storage space minimal level or a predetermined time has elapsed since the previous I/O cycle. The storage controller deactivates the disk drives which are not needed for the data transfer.

Abstract: In a data storage system, failed disk drives are switched temporarily off-line to be quickly rebuilt by executing a journaling/rebuild algorithm which tracks the updates to the failed disk drive into a journal structure created in a non-volatile memory. The journal information is used to update those data sections of the disk drive affected by updates after the disk drive is failed. The journal information is stored in bit maps indicating which portions of the disk drive have been updated with new data while the disk was failed. As an option, the system permits verification of data consistency on the data section of the disk drive which have not been affected by the updates. The journaling/rebuild of failed disks is applicable, among others, to RAID data storage systems.