Abstract: A technique for restoring file systems by applying full and incremental backups together while streaming them from multiple devices provides a faster system restore from a full backup where incremental backups must also be applied. A first storage device streams the full backup while at least one second storage device streams at least one incremental backup. As files are detected in the incremental backup, they are written instead of the corresponding file in the full backup. Incremental backups can be pre-merged to reduce their number to one less than a number of storage devices available to stream the backups, so that the full backup can be streamed along with the pre-merged incremental backup(s) to restore the file system.

Abstract: A technique for restoring file systems by applying full and incremental backups together while streaming them from multiple devices provides a faster system restore from a full backup where incremental backups must also be applied. A first storage device streams the full backup while at least one second storage device streams at least one incremental backup. As files are detected in the incremental backup, they are written instead of the corresponding file in the full backup. Incremental backups can be pre-merged to reduce their number to one less than a number of storage devices available to stream the backups, so that the full backup can be streamed along with the pre-merged incremental backup(s) to restore the file system.

Abstract: Embodiments relate to scheduling operations to perform on objects. A method for scheduling operations to perform on objects is provided. The method identifies a plurality of operations to perform on a plurality of objects each having at least one attribute. At least one of the operations has scheduling dependency on another operation. The method generates a numeric value for each of the objects from the attribute of the object. The method schedules the operations to perform on the objects based on the numeric values of the objects and the scheduling dependency.

Abstract: Provided are a computer program product, system, and method for using a metadata image of a file system and archive instance to restore files in the file system. A metadata image of the file system for a point-in-time backup as of a point-in-time includes information on files and directories in the file system as of the point-in-time and an archive instance including a copy of database records in the backup database for the files in the point-in-time backup. A restore request is received. A file representation is created of each file to restore in the directory structure of the file system from the metadata image, wherein at least one of the created file representations indicates that the file is stored off-line and has an external identifier used to access information on the file in the database records in the archive instance for the point-in-time backup.

Abstract: Managing a shared file system comprising a directory and files stored on a multiple storage devices shared by plural processing nodes, is provided. A plurality of parallel directory traversal records are partitioned into a plurality of ranges to allow for the records in each range to be written independently in parallel by plural processing nodes during parallel directory traversal. Parallel operations are performed comprising parallel directory traversal of all directory paths and files in the shared file system.

Abstract: Provided are a computer program product, system, and method for using a metadata image of a file system and archive instance to backup files in the file system. A point-in-time backup is performed of files in the file system as of a point-in-time. A metadata image includes information on files and directories in the file system as of the point-in-time. Files in the point-in-time backup are backed up to a backup storage. A backup database has records on the files backed-up from the file system. Each record in the backup database has a unique identifier for each backed-up file and a location of the backed-up file in the backup storage. An archive instance is generated including a copy of the database records for the files in the point-in-time backup. The metadata image and the archive instance are associated for the point-in-time backup.

Abstract: Provided are a computer program product, system, and method for using a metadata image of a file system and archive instance to restore files in the file system. A metadata image of the file system for a point-in-time backup as of a point-in-time includes information on files and directories in the file system as of the point-in-time and an archive instance including a copy of database records in the backup database for the files in the point-in-time backup. A restore request is received. A file representation is created of each file to restore in the directory structure of the file system from the metadata image, wherein at least one of the created file representations indicates that the file is stored off-line and has an external identifier used to access information on the file in the database records in the archive instance for the point-in-time backup.

Abstract: Provided are a computer program product, system, and method for using a metadata image of a file system and archive instance to backup files in the file system. A point-in-time backup is performed of files in the file system as of a point-in-time. A metadata image includes information on files and directories in the file system as of the point-in-time. Files in the point-in-time backup are backed up to a backup storage. A backup database has records on the files backed-up from the file system. Each record in the backup database has a unique identifier for each backed-up file and a location of the backed-up file in the backup storage. An archive instance is generated including a copy of the database records for the files in the point-in-time backup. The metadata image and the archive instance are associated for the point-in-time backup.

Abstract: A mechanism performs a metadata image backup of a hierarchical storage management (HSM) file system. Concurrent with performing the metadata image backup of the HSM file system, the mechanism identifies a set of resident files that are not yet stored off-line in the HSM file system. The mechanism identifies immutable files within the set of resident files to form an optimized set of resident files and passes the optimized set of resident files to a traditional backup system. The traditional backup system backs up file data for the set of resident files. Responsive to detecting an inability to restore a migrated managed state of a file, the mechanism unsets an immutable bit of the file attribute, restores file data for the file using the traditional backup system, and sets the immutable bit attribute of the file.

Abstract: A mechanism performs a metadata image backup of a hierarchical storage management (HSM) file system. Concurrent with performing the metadata image backup of the HSM file system, the mechanism identifies a set of resident files that are not yet stored off-line in the HSM file system. The mechanism identifies immutable files within the set of resident files to form an optimized set of resident files and passes the optimized set of resident files to a traditional backup system. The traditional backup system backs up file data for the set of resident files. Responsive to detecting an inability to restore a migrated managed state of a file, the mechanism unsets an immutable bit of the file attribute, restores file data for the file using the traditional backup system, and sets the immutable bit attribute of the file.

Abstract: A technique for restoring file systems by applying full and incremental backups together while streaming them from multiple devices provides a faster system restore from a full backup where incremental backups must also be applied. A first storage device streams the full backup while at least one second storage device streams at least one incremental backup. As files are detected in the incremental backup, they are written instead of the corresponding file in the full backup. Incremental backups can be pre-merged to reduce their number to one less than a number of storage devices available to stream the backups, so that the full backup can be streamed along with the pre-merged incremental backup(s) to restore the file system.

Abstract: Managing a shared file system comprising a directory and files stored on a multiple storage devices shared by plural processing nodes, is provided. A plurality of parallel directory traversal records are partitioned into a plurality of ranges to allow for the records in each range to be written independently in parallel by plural processing nodes during parallel directory traversal. Parallel operations are performed comprising parallel directory traversal of all directory paths and files in the shared file system.

Abstract: A technique for restoring file systems by applying full and incremental backups together while streaming them from multiple devices provides a faster system restore from a full backup where incremental backups must also be applied. A first storage device streams the full backup while at least one second storage device streams at least one incremental backup. As files are detected in the incremental backup, they are written instead of the corresponding file in the full backup. Incremental backups can be pre-merged to reduce their number to one less than a number of storage devices available to stream the backups, so that the full backup can be streamed along with the pre-merged incremental backup(s) to restore the file system.

Abstract: A technique for restoring file systems by applying full and incremental backups together while streaming them from multiple devices provides a faster system restore from a full backup where incremental backups must also be applied. A first storage device streams the full backup while at least one second storage device streams at least one incremental backup. As files are detected in the incremental backup, they are written instead of the corresponding file in the full backup. Incremental backups can be pre-merged to reduce their number to one less than a number of storage devices available to stream the backups, so that the full backup can be streamed along with the pre-merged incremental backup(s) to restore the file system.

Abstract: A selective restore technique for restoring file systems within computer systems provides on-demand access during the restore process, while preventing users from slowing the restore process by generating random accesses to files that have not yet been restored, early in the restore process. The restored files are grouped into multiple groups of files, which correspond to multiple phases of the restore process. On-demand requests may be locked out for some or all users during the first one or more phases of the restore, while they are permitted for the subsequent phases, ensuring that critical files needed by all users will be available before the backup process is interrupted by on-demand accesses. The groups are generally organized by elapsed time since last access or modification, and may be organized at restore time or during periodic back-up operations so that the files may be similarly organized in the backup media.

Abstract: One aspect of the present invention provides an operation for recovering lost space and maintaining data consistency in a storage environment by providing a two-way orphan reconciliation method for extremely large file systems, such as a Hierarchical Storage Management (HSM) system which manages billions of files. This highly scalable and parallelizable orphan identification process may be used in a HSM controlled environment to enable the execution of a two-way file orphan check. In one embodiment, this orphan check identifies file system client orphans and server object orphans in a single pass by comparing a queue containing a list of migrated files on a storage repository server with a queue containing a list of stub files on a file system. If the queue elements do not match, a file system orphan or server object orphan can appropriately be identified.

Abstract: A method is provided for managing the storage of a file that has been selected for migration from a first storage level within a file system to a second storage level. The method comprises copying each of one or more data blocks of the selected file that are stored in the first storage level at a physical disk address maintained in a corresponding reference of an inode for the selected file from the first storage level to the second storage level; maintaining a logical ditto address in each reference of an inode for each snapshot file in the file system that refers to one of the one or more data blocks of the selected file that were copied to the second storage level; and updating the file system to include a managed region for controlling access to the one or more data blocks through the inode for the selected file.

Abstract: A technique for restoring file systems by applying full and incremental backups together while streaming them from multiple devices provides a faster system restore from a full backup where incremental backups must also be applied. A first storage device streams the full backup while at least one second storage device streams at least one incremental backup. As files are detected in the incremental backup, they are written instead of the corresponding file in the full backup. Incremental backups can be pre-merged to reduce their number to one less than a number of storage devices available to stream the backups, so that the full backup can be streamed along with the pre-merged incremental backup(s) to restore the file system.

Abstract: A selective restore technique for restoring file systems within computer systems provides on-demand access during the restore process, while preventing users from slowing the restore process by generating random accesses to files that have not yet been restored, early in the restore process. The restored files are grouped into multiple groups of files, which correspond to multiple phases of the restore process. On-demand requests may be locked out for some or all users during the first one or more phases of the restore, while they are permitted for the subsequent phases, ensuring that critical files needed by all users will be available before the backup process is interrupted by on-demand accesses. The groups are generally organized by elapsed time since last access or modification, and may be organized at restore time or during periodic back-up operations so that the files may be similarly organized in the backup media.

Abstract: Space is allocated on data storage devices in proportion to weights associated with the storage devices. The weights can be dynamically adjusted at any time in order to accommodate changes in the system and to better utilize the storage devices. The technique used to perform the allocating is independent of the weights used by the allocating. Further, the allocation technique can accommodate general purpose data streams having varying lengths and/or varying access patterns, as well as special purpose data streams, such as video streams.