Abstract: Data storage systems and methods perform error correction on a single physical storage disk. The technique includes arranging a plurality of addressable blocks on the single physical storage disk into error correction groups, wherein each error correction group includes N data blocks and M coding blocks. M is determined in accordance with a desired failure tolerance of the error correction groups and an error-correcting code. For each error correction group, error-correcting code data is computed across the N data blocks in the error correction group. The computed error-correcting coding data is stored in the M coding blocks in the error correcting group. The arranging, computing and storing steps are performed by a hardware or software component external to the single physical storage disk.

Abstract: A data storage system includes a storage device divided into a plurality of blocks for storing data for a plurality of volumes, and a processor to execute instructions for maintaining a block map corresponding to the data stored on the storage device. The storage system may be part of a storage system network. The block map stores reference data indicating which of the volumes reference which blocks on the storage device, and which blocks on the storage device are unallocated. The reference data may include, for groups of one or more blocks, a first value identifying the oldest volume in which the group of blocks was allocated and a second value identifying the newest volume in which the group of blocks was allocated. The volumes may include one or more snapshots.

Abstract: Parallel reconstruction of file components following a failure of one or more of the storage devices is implemented in the context of a storage system that includes a plurality of storage devices for storing file components and a plurality of metadata managers. A storage device having one or more unrecoverable read errors requiring reconstruction is identified. A metadata manager which will serve as a scheduler, and a plurality of metadata managers which serve as a plurality of workers, are identified. The plurality of workers includes metadata managers other than the scheduler. A scheduler service running on the metadata manager identified as the scheduler is used to construct a list of file components from the storage device affected by the one or more unrecoverable read errors requiring reconstruction. The scheduler service assembles a work list corresponding to each of a plurality of the workers. The work list for each worker includes a subset of file components from the list requiring reconstruction.

Abstract: A data storage system where a RAID-X format is implemented at a file level includes storage devices, and one or more nodes that access files from the storage devices. N represents a maximum limit of failures beyond which reconstruction of a file stored using the file level implementation is not possible. A directory is encoded for each of a plurality of files, and a map is encoded that includes a list of storage devices where components of the directory associated with the map reside. The encoding of the directory is performed using a RAID-X format that can tolerate at least N+1 failures prior to being unable to reconstruct the directory.

Abstract: A system includes storage devices where a RAID-X format is implemented at a file level. In response to a failure, a map is retrieved. The map includes a list of devices where components of at least one file reside. The map is used to determine whether a number of lost or damaged components associated with the file exceed a maximum limit beyond which reconstruction of a file stored using the file level implementation of the RAID-X format is not possible. If the maximum limit has not been exceeded, the file is reconstructed. The system continues to service data storage and access requests, with reconstruction being performed as a background operation, after a number of device failures has occurred which would have exceeded the maximum limit of device failures for reconstruction had the file been stored using a volume level implementation of the RAID-X format.

Abstract: A data storage system includes a storage device divided into a plurality of blocks for storing data for a plurality of volumes, and a processor to execute instructions for maintaining a block map corresponding to the data stored on the storage device. The storage system may be part of a storage system network. The block map stores reference data indicating which of the volumes reference which blocks on the storage device, and which blocks on the storage device are unallocated. The reference data may include, for groups of one or more blocks, a first value identifying the oldest volume in which the group of blocks was allocated and a second value identifying the newest volume in which the group of blocks was allocated. The volumes may include one or more snapshots.

Abstract: Data storage systems and methods perform error correction on a single physical storage disk. The technique includes arranging a plurality of addressable blocks on the single physical storage disk into error correction groups, wherein each error correction group includes N data blocks and M coding blocks. M is determined in accordance with a desired failure tolerance of the error correction groups and an error-correcting code. For each error correction group, error-correcting code data is computed across the N data blocks in the error correction group. The computed error-correcting coding data is stored in the M coding blocks in the error correcting group. The arranging, computing and storing steps are performed by a hardware or software component external to the single physical storage disk.

Abstract: A hybrid quorum/consensus and primary-backup fault-tolerance model in an object-based distributed data storage system. When a primary manager fails, a hierarchy of network entities is established in which a group of realm managers first authorizes a failure-handling event through quorum/consensus and a backup manager (for the failing primary manager) then executes the decision of the quorum of realm managers. The realm managers, operating by consensus, determine whether (a) the primary manager can indeed be asserted to be down, and (b) whether there is a quorum of realm managers in agreement on this decision. If both are true, a master realm manager instructs the backup manager to proceed to execute the necessary steps to become the primary manager and function as the primary until the original primary manager is brought back into service. The hybrid fault-tolerance approach handles both single unit failures and network partitions in a unified way, without creating a single cluster out of the fault domain.

Abstract: A hybrid quorum/consensus and primary-backup fault-tolerance model in an object-based distributed data storage system. When a primary manager fails, a hierarchy of network entities is established in which a group of realm managers first authorizes a failure-handling event through quorum/consensus and a backup manager (for the failing primary manager) then executes the decision of the quorum of realm managers. The realm managers, operating by consensus, determine whether (a) the primary manager can indeed be asserted to be down, and (b) whether there is a quorum of realm managers in agreement on this decision. If both are true, a master realm manager instructs the backup manager to proceed to execute the necessary steps to become the primary manager and function as the primary until the original primary manager is brought back into service. The hybrid fault-tolerance approach handles both single unit failures and network partitions in a unified way, without creating a single cluster out of the fault domain.

Abstract: A scalable method and system for easily scheduling data transfer for a video-on-demand server system is provided. A strategy for locating available bandwidth and allocating available bandwidth to subscribers as needed is provided. The input/output bandwidth of storage devices that store data for video image sequences is modelled as a series of bandwidth units. Each bandwidth unit has sufficient capacity to service an individual subscriber during a scheduling cycle. Video image sequences or other data sequences are striped across storage devices such that data for a data sequence is forwarded to the subscriber by reading from a sequence of storage devices in consecutive time frames.