Abstract: Technology is disclosed for improving the storage efficiency and communication efficiency for a storage client device by maximizing the cache hit rate and minimizing data requests to the storage server. The storage server provides a duplication list to the storage client device. The duplication list contains references (e.g. storage addresses) to data blocks that contain duplicate data content. The storage client uses the duplication list to improve the cache hit rate. The duplication list is pruned to contain references to data blocks relevant to the storage client device. The storage server can prune the duplication list based on a working set of storage objects for a client. Alternatively, the storage server can prune the duplication list based on content characteristics, e.g. duplication degree and access frequency. Duplicate blocks to which the client does not have access can be excluded from the duplication list.

Abstract: A generic RAID engine system accepts an access request, accepts a metadata input comprising a layout description and, optionally, a plurality of resource optimization objectives, accepts a dynamic input comprising a dynamic state of an I/O stack comprising the generic RAID engine and a fault configuration of a plurality of storage devices in the I/O stack, and accepts RAID code input comprising information about the RAID code used by the I/O stack. The metadata input, the dynamic input, and the RAID code input are utilized to transform the access request into individual device reads and individual device writes such that RAID code relationships for the storage devices are maintained at all times. An optional optimizer module selects strategies that meet the resource optimization objectives.

Abstract: The embodiments of the invention provide a method, apparatus, etc. for a cache arrangement for improving RAID I/O operations. More specifically, a method begins by partitioning a data object into a plurality of data blocks and creating one or more parity data blocks from the data object. Next, the data blocks and the parity data blocks are stored within storage nodes. Following this, the method caches data blocks within a partitioned cache, wherein the partitioned cache includes a plurality of cache partitions. The cache partitions are located within the storage nodes, wherein each cache partition is smaller than the data object. Moreover, the caching within the partitioned cache only caches data blocks in parity storage nodes, wherein the parity storage nodes comprise a parity storage field. Thus, caching within the partitioned cache avoids caching data blocks within storage nodes lacking the parity storage field.

Abstract: A fault-tolerant and efficient way of deducing a set of inconsistent stripes for a network RAID protocol, wherein clients forward input/output (I/O) to a particular controller device called the coordinator, which executes RAID logic and which sends out device IOs to the relevant storage devices. If the coordinator fails then a new coordinator reconstructs its state from the storage devices.

Abstract: The embodiments of the invention provide a method, apparatus, etc. for a cache arrangement for improving RAID I/O operations. More specifically, a method begins by partitioning a data object into a plurality of data blocks and creating one or more parity data blocks from the data object. Next, the data blocks and the parity data blocks are stored within storage nodes. Following this, the method caches data blocks within a partitioned cache, wherein the partitioned cache includes a plurality of cache partitions. The cache partitions are located within the storage nodes, wherein each cache partition is smaller than the data object. Moreover, the caching within the partitioned cache only caches data blocks in parity storage nodes, wherein the parity storage nodes comprise a parity storage field. Thus, caching within the partitioned cache avoids caching data blocks within storage nodes lacking the parity storage field.

Abstract: The embodiments of the invention provide a method, apparatus, etc. for a cache arrangement for improving RAID I/O operations. More specifically, a method begins by partitioning a data object into a plurality of data blocks and creating one or more parity data blocks from the data object. Next, the data blocks and the parity data blocks are stored within storage nodes. Following this, the method caches data blocks within a partitioned cache, wherein the partitioned cache includes a plurality of cache partitions. The cache partitions are located within the storage nodes, wherein each cache partition is smaller than the data object. Moreover, the caching within the partitioned cache only caches data blocks in parity storage nodes, wherein the parity storage nodes comprise a parity storage field. Thus, caching within the partitioned cache avoids caching data blocks within storage nodes lacking the parity storage field.

Abstract: A generic RAID engine system accepts an access request, accepts a metadata input comprising a layout description and, optionally, a plurality of resource optimization objectives, accepts a dynamic input comprising a dynamic state of an I/O stack comprising the generic RAID engine and a fault configuration of a plurality of storage devices in the I/O stack, and accepts RAID code input comprising information about the RAID code used by the I/O stack. The metadata input, the dynamic input, and the RAID code input are utilized to transform the access request into individual device reads and individual device writes such that RAID code relationships for the storage devices are maintained at all times. An optional optimizer module selects strategies that meet the resource optimization objectives.

Abstract: A fault-tolerant and efficient way of deducing a set of inconsistent stripes for a network RAID protocol, wherein clients forward input/output (I/O) to a particular controller device called the coordinator, which executes RAID logic and which sends out device IOs to the relevant storage devices. If the coordinator fails then a new coordinator reconstructs its state from the storage devices.

Abstract: A generic RAID engine system accepts an access request, accepts a metadata input comprising a layout description and, optionally, a plurality of resource optimization objectives, accepts a dynamic input comprising a dynamic state of an I/O stack comprising the generic RAID engine and a fault configuration of a plurality of storage devices in the I/O stack, and accepts RAID code input comprising information about the RAID code used by the I/O stack. The metadata input, the dynamic input, and the RAID code input are utilized to transform the access request into individual device reads and individual device writes such that RAID code relationships for the storage devices are maintained at all times. An optional optimizer module selects strategies that meet the resource optimization objectives.

Abstract: A fault-tolerant and efficient way of deducing a set of inconsistent stripes for a network RAID protocol, wherein clients forward input/output (I/O) to a particular controller device called the coordinator, which executes RAID logic and which sends out device IOs to the relevant storage devices. If the coordinator fails then a new coordinator reconstructs its state from the storage devices.

Abstract: A client node of a distributed storage system adaptive determines on an operation-by-operation basis whether to process a network-RAID IO operation (IO request) locally at the client node or to forward the IO request to a storage server of the system for processing. The client node determines to process the requested network-RAID operation at the client node when a local estimator for a type of the requested network-RAID operation is less than a central estimator for the type of the requested network-RAID operation. Otherwise, the client node determines to process the requested network-RAID operation at a storage server when the local estimator for the type of the requested network-RAID operation is greater than the central estimator for the type of the requested network-RAID operation.

Abstract: A collective storage system and method for restoring data in the system after a failure in the system. The system includes multiple storage nodes that are interconnected by a network and store data as extents. There are also a set of Data Service (DS) agents for managing the extents, a set of Metadata Service (MDS) agents for managing metadata relating to the nodes and the extents, and a Cluster Manager (CM) agent in each node. After a node failure is detected by one of the CM agents, the agents responsible for coordinating the data restoring are notified of the failure. The agents generate a plan to restore the data extents affected by the failure, and then collectively restoring the affected extents based on the generated plan. The coordinating agents might be the MDS agents or DS agents. The failure might be a node failure or a disk failure.

Abstract: A network scheduler for real time applications is described. The network scheduler supports scheduling of network packets in a multi-threaded operating system to insure correct scheduling order and to preclude kernel preemption of active threads. The network scheduler resides between a kernel (and device drivers) and multiple sessions at the user-level. The network scheduler uses input/output control primitives such as HOLD and RELEASE to control servicing of the queues of packets at the kernel or network device driver. The control primitives serve as a middleware interface to the underlying kernel or network device driver. The network scheduler functions as a proxy scheduler to manage the queues of packets at the kernel or network device driver.

Abstract: A collective storage system and method for restoring data in the system after a failure in the system. The system includes multiple storage nodes that are interconnected by a network and store data as extents. There are also a set of Data Service (DS) agents for managing the extents, a set of Metadata Service (MDS) agents for managing metadata relating to the nodes and the extents, and a Cluster Manager (CM) agent in each node. After a node failure is detected by one of the CM agents, the agents responsible for coordinating the data restoring are notified of the failure. The agents generate a plan to restore the data extents affected by the failure, and then collectively restoring the affected extents based on the generated plan. The coordinating agents might be the MDS agents or DS agents. The failure might be a node failure or a disk failure.

Abstract: A distributed block-based programming model for control applications such as multimedia based real-time applications is described. According to one aspect of the invention, user program specifies interconnections between a plurality of program blocks distributed across a plurality of processing nodes. The user program is automatically translated into a system-level program having a program fragment for each one of the processing nodes. Each one of the program fragments comprise the program blocks located on the processing node and one or more system-level blocks for establishing connections between the processing node and a different processing node across a network.

Abstract: Disclosed is a system for updating and destaging data in a storage system. To overcome the limitations in the prior art described above, preferred embodiments of the present invention disclose a system for updating a data block in a storage device. A first processing unit receives an update to a data block in a storage device. The first processing unit sends a first message including the update and information indicating data blocks that were recently updated to a second processing unit. The second processing unit stores the update in a second storage area and sends a second message to the first processing unit after receiving the first message. The first processing unit stores the update in a first storage area and transfers the update from the first storage area to the data block in the storage device. The first processing unit transfers the update from the first storage area to the data block in the storage device.

Abstract: Disclosed is a system for altering the structure of parity groups, e.g., altering the RAID level or number of storage devices included in the RAID array. A parity group consists of a set of data blocks and parity data for the data blocks. The data and parity data are stored in at least two storage devices. The first processing unit alters the parity structure of an Nth set of parity groups, wherein N is an integer value greater than or equal to zero. A set of parity groups includes at least one parity group. The second processing unit determines an Nth range of parity groups including the Nth set of parity groups. After detecting the failure of the first processing unit while altering the parity structure of the parity groups in the Nth set, the second processing unit processes the parity groups in the determined Nth range to determine a parity group whose parity structure was being altered when the first processing unit failed.