Abstract: In a distributed object-based storage system that includes a plurality of object storage devices and one or more clients that access distributed, object-based files from the object storage devices, each of the files being comprised of a plurality of object components residing on different object storage device, systems and methods that use pointers stored as object attributes for file analysis and monitoring.

Abstract: In a distributed object-based storage system that includes a plurality of object storage devices and one or more clients that access distributed, object-based files from the object storage devices, each of the files being comprised of a plurality of object components residing on different object storage device, systems and methods that use pointers stored as object attributes for file analysis and monitoring.

Abstract: A method for allowing client computers in a distributed object-based data storage system to interpret higher revision data structures. In one embodiment, clients request layout maps of objects to be accessed from respective storage managers in the system and then construct and execute directed acyclic graphs (DAG) to accomplish desired I/O operations for data access. When a client computer running an older software revision is not able to interpret a particular data organization supplied by the storage manager that was designed after that client's software was installed, the client sends to the storage manager a description of the desired data access operation. The storage manager, in turn, interprets a data layout scheme for a client on a per-access basis and returns to that client a DAG containing a data structure that describes the sequence of I/O operations necessary to effect the data access. The client can then execute the storage manager-supplied DAG to carry out the data access.

Abstract: A data storage system includes a storage device divided into a plurality of blocks for storing data for a plurality of volumes, and a processor to execute instructions for maintaining a block map corresponding to the data stored on the storage device. The storage system may be part of a storage system network. The block map stores reference data indicating which of the volumes reference which blocks on the storage device, and which blocks on the storage device are unallocated. The reference data may include, for groups of one or more blocks, a first value identifying the oldest volume in which the group of blocks was allocated and a second value identifying the newest volume in which the group of blocks was allocated. The volumes may include one or more snapshots.

Abstract: Parallel reconstruction of file components following a failure of one or more of the storage devices is implemented in the context of a storage system that includes a plurality of storage devices for storing file components and a plurality of metadata managers. A storage device having one or more unrecoverable read errors requiring reconstruction is identified. A metadata manager which will serve as a scheduler, and a plurality of metadata managers which serve as a plurality of workers, are identified. The plurality of workers includes metadata managers other than the scheduler. A scheduler service running on the metadata manager identified as the scheduler is used to construct a list of file components from the storage device affected by the one or more unrecoverable read errors requiring reconstruction. The scheduler service assembles a work list corresponding to each of a plurality of the workers. The work list for each worker includes a subset of file components from the list requiring reconstruction.

Abstract: A data storage system where a RAID-X format is implemented at a file level includes storage devices, and one or more nodes that access files from the storage devices. N represents a maximum limit of failures beyond which reconstruction of a file stored using the file level implementation is not possible. A directory is encoded for each of a plurality of files, and a map is encoded that includes a list of storage devices where components of the directory associated with the map reside. The encoding of the directory is performed using a RAID-X format that can tolerate at least N+1 failures prior to being unable to reconstruct the directory.

Abstract: A system includes storage devices where a RAID-X format is implemented at a file level. In response to a failure, a map is retrieved. The map includes a list of devices where components of at least one file reside. The map is used to determine whether a number of lost or damaged components associated with the file exceed a maximum limit beyond which reconstruction of a file stored using the file level implementation of the RAID-X format is not possible. If the maximum limit has not been exceeded, the file is reconstructed. The system continues to service data storage and access requests, with reconstruction being performed as a background operation, after a number of device failures has occurred which would have exceeded the maximum limit of device failures for reconstruction had the file been stored using a volume level implementation of the RAID-X format.

Abstract: A system and method for recovering from an I/O error in a distributed object-based storage system that includes a plurality of object storage devices for storing object components, a manager coupled to each of the object storage devices, wherein the object storage devices coordinate with the file manager, and one or more clients that access and store distributed, object-based files on the object storage devices. A client attempts to perform an operation selected from the group consisting of: a data read operation from an object storage device, a data write operation to an object storage device, a set attribute operation to an object storage device, a get attribute operation from an object storage device and a create object operation to an object storage device. Upon failure of the operation, the client sends a message from the client to the manager that includes information representing a description of the failure.

Abstract: A methodology wherein two different types of locks are used by a storage manager when multiple clients wish to access a particular redundantly-stored file. Simple byte-range based mutual exclusion (or mutex) locks are granted by the storage manager for data writes/updates to the file when the file is in the fault-free state, and individual readers/writers (R/W) locks are granted by the storage manager when the file is in the degraded state. No read locks are required of clients when the file object is in the fault-free state. During the fault-free state of the file object, when exactly one client is writing to the file object, the storage manger grants that file object a whole-file lock valid over the entire file object. Each client may have a client lock manager that interacts with appropriate storage manager lock manager to request and obtain necessary locks. These various locking mechanisms reduce lock-related network traffic in a data storage system.

Abstract: A hybrid quorum/consensus and primary-backup fault-tolerance model in an object-based distributed data storage system. When a primary manager fails, a hierarchy of network entities is established in which a group of realm managers first authorizes a failure-handling event through quorum/consensus and a backup manager (for the failing primary manager) then executes the decision of the quorum of realm managers. The realm managers, operating by consensus, determine whether (a) the primary manager can indeed be asserted to be down, and (b) whether there is a quorum of realm managers in agreement on this decision. If both are true, a master realm manager instructs the backup manager to proceed to execute the necessary steps to become the primary manager and function as the primary until the original primary manager is brought back into service. The hybrid fault-tolerance approach handles both single unit failures and network partitions in a unified way, without creating a single cluster out of the fault domain.

Abstract: In a file system recovery (FSRC) process, an FSRC module performs dynamic checks on file system objects to verify parent-child relationships among various objects in the system. The FSRC module stores a list of objects in the system and relevant object-related information for each object in a memory. A dynamic check on each listed object is then performed using the object-related information to verify parent-child relationships for that object. After the dynamic check is over, the object is removed from the list and its object-related information is deleted from the memory.

Abstract: A system and method to perform a system-wide file system image without time smear in a distributed object-based data storage system. A realm manager is elected as an image master using the Distributed Consensus Algorithm to execute image-taking. All pending write capabilities are invalidated prior to taking the system-wide file system image so as to quiesce the realm and prepare the storage system for the system-wide image. Once the system is quiesced, the image master instructs each storage manager in the system to clone each live object group contained therein without explicitly cloning any objects contained in such object group. In one embodiment, a file manager copies an object in the system before a write operation is performed on that object after the image is taken. Neither the cloning operation nor the copying operation update any directory objects in the system. At run time, a client application may use a mapping scheme to access objects contained in the system-wide image.

Abstract: A data storage methodology wherein fault tolerance is accomplished by generation of multiple canonical names for each primary manager active in the data storage system and then balancing the primary manager's failure-induced workload across the multiple backup managers using the canonical names. A qualified name for each backup manager is derived from a canonical name assigned to the primary manager. During fault-free state of operation, all the qualified names are assigned to the primary manager, which serves all its qualified names. When that primary manager fails, the work it was supporting is distributed among its backup managers using the qualified names for those backups. The distribution of primary manager's workload among multiple backups eliminates the severe performance bottlenecks present in the traditional approach of keeping one backup manager for a primary manager and keeping that backup manager idle during the fault-free state of its primary manager.

Abstract: A system and method to perform a system-wide file system image without time smear in a distributed object-based data storage system. A realm manager is elected as an image master using the Distributed Consensus Algorithm to execute image-taking. All pending write capabilities are invalidated prior to taking the system-wide file system image so as to quiesce the realm and prepare the storage system for the system-wide image. Once the system is quiesced, the image master instructs each storage manager in the system to clone each live object group contained therein without explicitly cloning any objects contained in such object group. In one embodiment, a file manager copies an object in the system before a write operation is performed on that object after the image is taken. Neither the cloning operation nor the copying operation update any directory objects in the system. At run time, a client application may use a mapping scheme to access objects contained in the system-wide image.

Abstract: A data storage methodology wherein a hashing algorithm is applied to break a directory object experiencing frequent concurrent accesses from a number of client or manager applications into a predetermined number of hash component objects and a hash master object that manages the component objects. The hash master object and the hash components, together, constitute a hash directory, which replaces the original non-hashed directory object. Each hash component object contains a portion of the entries contained in the original directory object. Each hash component is managed by only one file manager. The entries in the original directory object are distributed among the hash component objects using a predefined hashing algorithm. The creation of hash components and the hash master allows more than one client application or file manager to concurrently write corresponding hash components without the need for access coordination on each access.

Abstract: A data storage methodology wherein a data file is initially stored in a format consistent with RAID-1 and RAID-X and then migrated to a format consistent with RAID-X and inconsistent with RAID-1 when the data file grows in size beyond a certain threshold. Here, RAID-X refers to any non-mirrored storage scheme employing XOR-based error correction coding (e.g., a RAID-5 configuration). Each component object (including the data objects and the parity object) for the data file is configured to be stored in a different stripe unit per object-based secure disk. Each stripe unit may store, for example, 64 KB of data. So long as the data file does not grow beyond the size threshold of a stripe unit (e.g., 64 KB), the parity stripe unit contains a mirrored copy of the data stored in one of the data stripe units because of the exclusive-ORing of the input data with “all zeros” assumed to be contained in empty or partially-filled stripe units.

Abstract: A data storage methodology wherein fault tolerance is accomplished by generation of multiple canonical names for each primary manager active in the data storage system and then balancing the primary manager's failure-induced workload across the multiple backup managers using the canonical names. A qualified name for each backup manager is derived from a canonical name assigned to the primary manager. During fault-free state of operation, all the qualified names are assigned to the primary manager, which serves all its qualified names. When that primary manager fails, the work it was supporting is distributed among its backup managers using the qualified names for those backups. The distribution of primary manager's workload among multiple backups eliminates the severe performance bottlenecks present in the traditional approach of keeping one backup manager for a primary manager and keeping that backup manager idle during the fault-free state of its primary manager.