Abstract: A system, method, and computer program product for a block-based backing up a storage device to an object storage service is provided. This includes the generation of a data object that encapsulates a data of a data extent. The data extent covers a block address range of the storage device. The data object is named with a base name that represents a logical block address (LBA) of the data extent. The base name is appended with an identifier that deterministically identifies a recovery point that the data object is associated with. The base name combined with the identifier represents a data object name for the data object. The named data object is then transmitted to the object storage service for backup of the data extent. At an initial backup, the full storage device is copied. In incremental backups afterwards, only those data extents that changed are backed up.

Abstract: A system, method, and computer program product for a block-based backing up a storage device to an object storage service is provided. This includes the generation of a data object that encapsulates a data of a data extent. The data extent covers a block address range of the storage device. The data object is named with a base name that represents a logical block address (LBA) of the data extent. The base name is appended with an identifier that deterministically identifies a recovery point that the data object is associated with. The base name combined with the identifier represents a data object name for the data object. The named data object is then transmitted to the object storage service for backup of the data extent. At an initial backup, the full storage device is copied. In incremental backups afterwards, only those data extents that changed are backed up.

Abstract: A system and method for managing distributed coherent datasets using a hierarchical change log is provided. In some embodiments, a distributed storage system is provided that includes a primary storage device containing a primary dataset and a mirror storage device containing a mirror dataset. The mirror dataset includes a coherent copy of the primary dataset. The distributed storage system further includes a hierarchical change log tracking a coherence state for the mirror dataset. The hierarchical change log includes a first sub-log and a second sub-log, and a block range of the first sub-log overlaps a block range of the second sub-log. The hierarchical change log may define a priority relationship between the first sub-log and the second sub-log governing the overlap. The first sub-log and the second sub-log may be independently configured and may be different in one of a representation and a block size.

Abstract: A system and method for improving storage system performance by maintaining data integrity during bulk export to a cloud system is provided. A backup host reads a selected volume from the storage system via an I/O channel. The storage system remains online during bulk export and tracks I/O to the selected volume in a tracking log. The backup host compresses, encrypts, and calculates a checksum for each data block of the volume before writing a corresponding data object to export devices and sending a checksum data object to the cloud system. The devices are shipped to the cloud system, which imports the data objects and calculates a checksum for each. The storage system compares the imported checksums with the checksums in the checksum data object, and adds data blocks to the tracking log when errors are detected. An incremental backup is performed based on the contents of the tracking log.

Abstract: A system and method for improving storage system performance by maintaining data integrity during bulk export to a cloud system is provided. A backup host reads a selected volume from the storage system via an I/O channel. The storage system remains online during bulk export and tracks I/O to the selected volume in a tracking log. The backup host compresses, encrypts, and calculates a checksum for each data block of the volume before writing a corresponding data object to export devices and sending a checksum data object to the cloud system. The devices are shipped to the cloud system, which imports the data objects and calculates a checksum for each. The storage system compares the imported checksums with the checksums in the checksum data object, and adds data blocks to the tracking log when errors are detected. An incremental backup is performed based on the contents of the tracking log.

Abstract: A system and method for improving storage system performance by maintaining data integrity during bulk export to a cloud system is provided. A backup host reads a selected volume from the storage system via an I/O channel. The storage system remains online during bulk export and tracks I/O to the selected volume in a tracking log. The backup host compresses, encrypts, and calculates a checksum for each data block of the volume before writing a corresponding data object to export devices and sending a checksum data object to the cloud system. The devices are shipped to the cloud system, which imports the data objects and calculates a checksum for each. The storage system compares the imported checksums with the checksums in the checksum data object, and adds data blocks to the tracking log when errors are detected. An incremental backup is performed based on the contents of the tracking log.

Abstract: A system and method for improving storage system performance by maintaining data integrity during bulk export to a cloud system is provided. A backup host reads a selected volume from the storage system via an I/O channel. The storage system remains online during bulk export and tracks I/O to the selected volume in a tracking log. The backup host compresses, encrypts, and calculates a checksum for each data block of the volume before writing a corresponding data object to export devices and sending a checksum data object to the cloud system. The devices are shipped to the cloud system, which imports the data objects and calculates a checksum for each. The storage system compares the imported checksums with the checksums in the checksum data object, and adds data blocks to the tracking log when errors are detected. An incremental backup is performed based on the contents of the tracking log.

Abstract: A system and method for recovering data backed up to an object store are provided. In some embodiments, the method includes identifying an address space of a data set to be recovered. A set of data objects stored by an object-based system is identified that corresponds to the address space and a selected recovery point. The identified set of data objects is retrieved, and data contained in the retrieved set of data objects is stored to at least one storage device at a block address determined by the retrieved set of data objects to recreate the address space. In some embodiments, the set of data objects is retrieved by providing an HTTP request and receiving the set of data objects as an HTTP response. In some embodiments, the set of data objects are retrieved based on the data objects being the target of a data transaction.

Abstract: A system and method for managing distributed coherent datasets using a hierarchical change log is provided. In some embodiments, a distributed storage system is provided that includes a primary storage device containing a primary dataset and a mirror storage device containing a mirror dataset. The mirror dataset includes a coherent copy of the primary dataset. The distributed storage system further includes a hierarchical change log tracking a coherence state for the mirror dataset. The hierarchical change log includes a first sub-log and a second sub-log, and a block range of the first sub-log overlaps a block range of the second sub-log. The hierarchical change log may define a priority relationship between the first sub-log and the second sub-log governing the overlap. The first sub-log and the second sub-log may be independently configured and may be different in one of a representation and a block size.

Abstract: A system and method for managing distributed coherent datasets using a hierarchical change log is provided. In some embodiments, a distributed storage system is provided that includes a primary storage device containing a primary dataset and a mirror storage device containing a mirror dataset. The mirror dataset includes a coherent copy of the primary dataset. The distributed storage system further includes a hierarchical change log tracking a coherence state for the mirror dataset. The hierarchical change log includes a first sub-log and a second sub-log, and a block range of the first sub-log overlaps a block range of the second sub-log. The hierarchical change log may define a priority relationship between the first sub-log and the second sub-log governing the overlap. The first sub-log and the second sub-log may be independently configured and may be different in one of a representation and a block size.

Abstract: A system, method, and computer program product for a block-based backing up a storage device to an object storage service is provided. This includes the generation of a data object that encapsulates a data of a data extent. The data extent covers a block address range of the storage device. The data object is named with a base name that represents a logical block address (LBA) of the data extent. The base name is appended with an identifier that deterministically identifies a recovery point that the data object is associated with. The base name combined with the identifier represents a data object name for the data object. The named data object is then transmitted to the object storage service for backup of the data extent. At an initial backup, the full storage device is copied. In incremental backups afterwards, only those data extents that changed are backed up.

Abstract: A system and method for recovering data backed up to an object store are provided. In some embodiments, the method includes identifying an address space of a data set to be recovered. A set of data objects stored by an object-based system is identified that corresponds to the address space and a selected recovery point. The identified set of data objects is retrieved, and data contained in the retrieved set of data objects is stored to at least one storage device at a block address determined by the retrieved set of data objects to recreate the address space. In some embodiments, the set of data objects is retrieved by providing an HTTP request and receiving the set of data objects as an HTTP response. In some embodiments, the set of data objects are retrieved based on the data objects being the target of a data transaction.

Abstract: A system and method for managing distributed coherent datasets using a hierarchical change log is provided. In some embodiments, a distributed storage system is provided that includes a primary storage device containing a primary dataset and a mirror storage device containing a mirror dataset. The mirror dataset includes a coherent copy of the primary dataset. The distributed storage system further includes a hierarchical change log tracking a coherence state for the mirror dataset. The hierarchical change log includes a first sub-log and a second sub-log, and a block range of the first sub-log overlaps a block range of the second sub-log. The hierarchical change log may define a priority relationship between the first sub-log and the second sub-log governing the overlap. The first sub-log and the second sub-log may be independently configured and may be different in one of a representation and a block size.

Abstract: The present invention is a system for optimizing the reconstruction and copyback of data contained on a failed disk in a multi-disk mass storage system. A system in accordance with the present invention may comprise the following: a processing unit requiring mass-storage; one or more disks configured as a RAID system; an associated global hot spare disk; and interconnections linking the processing unit, the RAID and the global hot spare disk. In a further aspect of the present invention, a method for the reconstruction and copyback of a disconnected RAID disk utilizing a global hot spare disk is disclosed. The method includes: disconnecting a RAID component disk; reconstructing data from the disconnected RAID disk onto a global hot spare disk; reconnecting the disconnected RAID component disk; and copying the reconstructed data from the global hot spare disk back to the reconnected RAID component disk.

Abstract: The present invention is directed to a method of an adaptive procedure table which is capable of providing default behaviors for each procedure if a corresponding procedure is not defined or has been removed from a software build. The default behaviors for each procedure may be defined in a template file provided by a developer before a compile time of software. The present invention may permit a module that defines the implementation of a procedure to be removed from the software build without requiring source code changes. As such, the developer may be allowed to remove or add certain features from a compiled program without introducing compile time or link time errors.

Abstract: The present invention is a system for optimizing the reconstruction and copyback of data contained on a failed disk in a multi-disk mass storage system. A system in accordance with the present invention may comprise the following: a processing unit requiring mass-storage; one or more disks configured as a RAID system; an associated global hot spare disk; and interconnections linking the processing unit, the RAID and the global hot spare disk. In a further aspect of the present invention, a method for the reconstruction and copyback of a disconnected RAID disk utilizing a global hot spare disk is disclosed. The method includes: disconnecting a RAID component disk; reconstructing data from the disconnected RAID disk onto a global hot spare disk; reconnecting the disconnected RAID component disk; and copying the reconstructed data from the global hot spare disk back to the reconnected RAID component disk.

Abstract: Methods and structure for improved migration of a RAID logical volume from a higher level RAID management to a lower level. Features and aspects hereof provide for migrating a RAID logical volume by removal of one or more disk drives from the logical volume and moving or regenerating only the information of the removed drive(s) that is required for the lower level of RAID storage management. In one exemplary embodiment, a RAID level 6 volume may be migrated to a RAID level 5 volume by removing a single disk drive of the volume. Minimal movement of remaining data blocks and RAID 5 parity blocks in each stripe may be performed creating new RAID level 5 stripes devoid of RAID level 6 second redundancy blocks. The newly formed RAID level 5 volume may then be mapped according to modified mapping algorithms to reduce the need for further data movement.

Abstract: Methods and structure for improved import of RAID level 6 logical volumes into subsystems devoid of RAID level 6 hardware support. When a RAID level 6 logical volume is exported from a first storage subsystem and imported into a second storage subsystem devoid of RAID level 6 hardware support, features and aspects hereof first migrate the logical volume for use as a logical volume with a different RAID level supported by the second storage subsystem. The migration may be, for example, to a RAID level 5 logical volume performed by movement of data blocks to form new stripes and re-generation of associated parity blocks or by simpler re-mapping of existing blocks of the RAID level 6 volume to use only blocks and parity needed for RAID level 5 management. Further features and aspects allow migration to any other RAID management level supported by the second storage subsystem.

Abstract: Methods and structure for improved import/export of RAID level 6 logical volumes in subsystems supporting RAID level 5 but not level 6. When a RAID level 6 logical volume is imported into a RAID level 5 storage subsystem, features and aspects hereof re-map the logical volume for use as a RAID level 5 logical volume. Disk blocks containing the level 6 additional redundancy information are not used by the RAID level 5 storage subsystem but are skipped in the re-mapping of the logical volume. All other blocks of the logical volume are mapped to corresponding blocks of the RAID level 6 mapping of the logical volume. The logical volume may then be flagged to indicate the additional redundancy information is invalid. A RAID level 6 storage subsystem may then rebuild the additional redundancy information when the flagged logical volume is re-imported to a RAID level 6 storage subsystem.

Abstract: Methods and structure for storing volume and other configuration information on all disk drives of a volume group and for performing pre-merge operations to inform a user of the pending availability of the identified volumes. Configuration information has been historically stored in two distinct areas of configuration information on disk drives—a volume information area and an “other” information area. Aspects of the invention provide for storing all configuration information in the “other” information area. All disks of a system therefore include all configuration information relating both to volumes and to pseudo volume. A pre-merge operation is performed when each disk drive is inserted into a storage system to present new volume information to a user. When the last disk drive of a volume group is inserted, the user may select new volumes to be imported into the system.