Abstract: Data structures are provided for file systems to facilitate backup processes that are especially useful for large and/or sparse data files. In one aspect of the invention, these data structures include time stamp information that is accessible for use by a system user at the application program level. These data structures also include indications of current validity that reduce the need to perform I/O operations which are naturally very resource intensive for large files. The ability to incorporate efficiencies accorded to files having blocks designated as being sparse is also provided. The incorporation of these data structures in the file system itself permits the backup process to be not only incremental in nature but also to be directed at the file level as opposed to, say, the disk level.

Abstract: File system restoration is made more flexible through the use of indicators of individual file restoration status throughout the restoration process. Such indicators are particularly useful in the restoration of large file systems where, for example, the number of files may range upwards of several hundred million. Since file system restoration for large systems takes such a relatively long time, the present invention provides various degrees of individual file access even as the restoration process is being carried out. During file system restoration times, file access may be full or limited to a file's attributes. Most advantageously, the present invention is capable of providing full, dynamically driven (on-demand) file access even during file system restoration operations.

Abstract: File system backups are carried out by first generating a list of inodes including associated inode numbers in inode number order that have changed since the last backup operation. A table which has file names and inode numbers for all of the files currently in the file system is also generated. This list is sorted by inode number and the table and list are merged to provide a structure for determining which files are to be backed up. This means that relevant inodes and file names are now provided in a single entity. It is also noted that the structure that results from the merge operation is particularly suitable for being read in blocks which thus permits the backup operation to be carried out in parallel. The task of backing up files is also preferably partitioned by file size or other criteria as opposed to being partitioned simply by the number of files assigned to be backed up by any one processor in a distributed or parallel data processing system.

Abstract: Space is allocated on data storage devices in proportion to weights associated with the storage devices. The weights can be dynamically adjusted at any time in order to accommodate changes in the system and to better utilize the storage devices. The technique used to perform the allocating is independent of the weights used by the allocating. Further, the allocation technique can accommodate general purpose data streams having varying lengths and/or varying access patterns, as well as special purpose data streams, such as video streams.

Abstract: A system, method and computer readable medium for providing a snapshot of a subset of a file system. A first snapshot of a first set of source files in a file system is generated. The first snapshot includes an inode corresponding to each source file in the first set of files. Stored in each inode is a first identifier associated with the first set of files and a second identifier associated with the time of the first snapshot. Next, a second snapshot of a second set of source files is taken. The second snapshot includes an inode corresponding to each source file in the second set of files. Stored in each inode are a first identifier and a second identifier. Subsequent snapshots are taken every first period and every second period for the first set of files and the second set of files, respectively.

Abstract: Concurrent access to data is managed through concurrency control techniques. Various types of techniques are employed to manage the access, including locking-based techniques and non-locking-based techniques. A dynamic switch from one type of concurrency control technique (e.g., a locking-based technique) to a different type of concurrency control technique (e.g., a non-locking-based technique) is enabled. This switching is based on access patterns and/or application requirements for each file. The switching allows enhanced performance for both coarse-grain sharing and fine-grain sharing of data.

Abstract: A system, method and computer readable medium for deferring copy-on-write of a snapshot is disclosed. The method includes the generation of snapshot of a source file. Upon modification of a first data block referenced by the source file, the first data block is referenced by the snapshot and a second data block is allocated for the source file. Then, a first variable associated with the source file is set to a value indicating an incomplete source file data block and a second variable associated with the source file is set to a value indicating the valid portion of the second data block. Any portion of the second data block that is overwritten is considered valid. The second data block is then modified and the second variable is changed to reflect the modification. Upon reception of a read request, the corresponding portion of the second data block is retrieved.

Abstract: A method is provided for limiting access to disks and files in the event of a system partitioning occurring as the result of a node failure in a data distributed processing system in which files are shared amongst a plurality of independently running nodes. The nodes in the system are provided with limited duration leases which permit data access and which have to be periodically renewed. Each partition is also associated with a quorum which is also a necessary element in providing access at proper times. The lease and quorum concepts work together to insure that recovery operations do not occur until recovery is possible. The lease concept thus is also seen as further operating in close cooperation with the quorum concept to prevent long delays in the recovery process that could otherwise occur.

Abstract: The present invention is directed to a method for controlling file access in a multinode, shared storage, data processing system in which a one set of nodes is capable of acting as metadata controllers and a second set of nodes are tuned for running user applications. By providing two kinds of locks operative at the file level of granularity, a mechanism is produced which permits application programs to also run on their set of metadata controller nodes. One of the locks permits read and write access to existing data blocks but precludes operations such as file deletion or file truncation. The other lock is of lower authority and permits metadata controller nodes, which can provide time limited permissions to the application nodes, to withhold locks which authorize access when requests for access are made from application nodes.

Abstract: The present invention is directed to the operation of a multinode, shared storage data processing system in which a first set of nodes acts as metadata controller nodes. In normal operation one of these nodes acts to provide access to an individual file contained within the shared storage. This access actually occurs directly not from the metadata controller node but from an application node which has time limited access to the file but which does not act as a metadata controller for that file. In the event of node failure, lock recovery is initiated for all files without metadata controller locks after which lock recovery is carried out for the files with metadata controller locks with due consideration now being given to the existence of file access authorization for limited periods of time at nodes which are not metadata controller nodes.

Abstract: A method for data access in a multinode, shared storage data processing environment is provided by enhancing the file locking mechanism structure so as to permit nodes not normally designated as metadata controller nodes to fulfill that function for limited times thus at least temporarily eliminating the need for continual node-to-node coordination. This is particularly advantageous in the access and transfer of large files such as video files or files containing complex visualization data.

Abstract: This invention allows a related set of coordinator services to migrate from one node to another without disrupting applications on any of the nodes in the system. Unlike other methods, this invention allows a set of interdependent services to be brought to “quiescence” and migrated together. Since service operations depend upon the results and/or data of other service operations, any particular operation can only complete properly when those other operations return data necessary for the completion of the dependent operation. Therefore, this invention permits the completion of non-disruptive migration by phasing the “quiescence” of the services. Operations that are most dependent upon other operations are suspended before those other operations; then the process waits for any current operations to complete.

Abstract: A system, method and computer readable medium for providing dynamic links to file system snapshots is disclosed. A link corresponding to a file system inode is generated. Upon activation of the link, the snapshot identifier of the most recent snapshot is retrieved and the most recent snapshot is identified. If the most recent snapshot includes an inode corresponding to the file system inode and the file system inode includes a disk address, then the data block referenced by the disk address is retrieved. If the most recent snapshot includes an inode corresponding to the file system inode and the file system inode includes a ditto disk address (or the inode corresponding to the file system inode is not included), then the inode corresponding to the file system inode indicated in the link is retrieved and the data block corresponding to the file system inode is retrieved.

Abstract: A system, method and computer readable medium for providing a file system snapshot with ditto address feature is disclosed. The method includes generating a file system snapshot wherein the snapshot is substantially empty. When metadata associated with a source file included in the snapshot is modified, the inode corresponding to the source file is copied to a shadow inode in the snapshot. When information is appended to a data block associated with a source file included in the snapshot, the inode corresponding to the source file is also copied to a shadow inode in the snapshot. When information in a data block associated with a source file included in the snapshot is overwritten or deleted, the inode corresponding to the source file is copied to a shadow inode in the snapshot and the original data block corresponding to the source file is copied to the snapshot.

Abstract: A system, method and computer readable medium for providing a snapshot of a subset of a file system. A first snapshot of a first set of source files in a file system is generated. The first snapshot includes an inode corresponding to each source file in the first set of files. Stored in each inode is a first identifier associated with the first set of files and a second identifier associated with the time of the first snapshot. Next, a second snapshot of a second set of source files is taken. The second snapshot includes an inode corresponding to each source file in the second set of files. Stored in each inode are a first identifier and a second identifier. Subsequent snapshots are taken every first period and every second period for the first set of files and the second set of files, respectively.

Abstract: A system, method and computer readable medium for providing a standby file system with snapshot feature is disclosed. A first snapshot of a primary file system is generated. Next, a standby file system restores the first snapshot of the primary file system in the standby file system and a snapshot of the standby file system is taken. Then, a second snapshot of the primary file system is taken and a data set capturing the differences between the first snapshot of the primary file system and a previous data set of the primary file system is generated. Subsequently, the data set is applied to the standby file system. Upon unavailability of the primary file system, the standby file system is initialized as the new primary file system. Upon recovery of the old primary file system, the old primary file system is initialized as a new standby file system.

Abstract: A system, method and computer readable medium for providing a writable file system snapshot with ditto address feature is disclosed. In an embodiment of the present invention, the method includes accessing a first file system snapshot in a set of file system snapshots. A snapshot includes data contents comprising at least one shadow inode or at least one shadow inode and at least one data block corresponding to the shadow inode. The data contents of the first snapshot are copied and written to the next oldest file system snapshot. Subsequently, the data contents of the first snapshot are modified in accordance with modifications to at least one source file corresponding to the first snapshot. Then, a next most recent file system snapshot is accessed. The data contents of the next most recent snapshot are copied and written to the first file system snapshot.

Abstract: A system, method and computer readable medium for deferring copy-on-write of a snapshot is disclosed. The method includes the generation of snapshot of a source file. Upon modification of a first data block referenced by the source file, the first data block is referenced by the snapshot and a second data block is allocated for the source file. Then, a first variable associated with the source file is set to a value indicating an incomplete source file data block and a second variable associated with the source file is set to a value indicating the valid portion of the second data block. Any portion of the second data block that is overwritten is considered valid. The second data block is then modified and the second variable is changed to reflect the modification. Upon reception of a read request, the corresponding portion of the second data block is retrieved.

Abstract: One or more conflicting tokens are revoked using a revocation capability that eliminates the need for the holders of the conflicting tokens to communicate with the token server. Instead, information used to revoke the one or more conflicting tokens is provided by the token holders to a requester of a token that is in conflict with the conflicting tokens. The requester then forwards this information to the token server in a message already being sent to the server. Thus, additional messages between the requester and the token server are not needed.

Abstract: Asynchronous messages are used to prefetch and/or relinquish tokens used in providing locking of shared resources. A message is sent to prefetch one or more tokens, and prior to receiving a reply for the message, another message (e.g., an acquire) is sent for at least one of those tokens. Similarly, a message is sent indicating a voluntary relinquish of one or more tokens and without having acknowledgment that the relinquish request was processed, a further message (e.g., an acquire) is sent for at least one of the tokens. Multiple tokens may be acquired and/or relinquished in a single message to the token server. This reduces message delays and overall message traffic.