Abstract: Aspects of the subject matter described herein relate to file system filters and transactions. In aspects, a filter may enlist to receive notification of events associated with a transaction. Afterwards, the filter may receive notification of a transaction event for which it has enlisted. In response to receiving notification of the transaction the filter may perform an action as appropriate. Aspects of the subject matter described herein may be applied to stacked and managed filters.

Abstract: File system metadata regarding states of a file system affected by transactions is tracked consistently even in the face of dirty shutdowns which might cause rollbacks in transactions which have already been reflected in the metadata. In order to only request time- and resource-heavy rebuilding of metadata for metadata which may have been affected by rollbacks, reliability information is tracked regarding metadata items. When a metadata item is affected by a transaction which may not complete properly in the case of a problematic shutdown or other event, that metadata item's reliability information indicates that it may not be reliable in case of such a problematic (“dirty” or“abnormal”) event. In addition to flag information indicating unreliability, timestamp information tracking a time of the command which has made a metadata item unreliable is also maintained.

Abstract: A generic disk driver filter may be used to accelerate performance when writing to a disk. The generic disk driver filter may be connected between a file system and a disk driver and may be configured to be extensible and compatible with a variety of different file systems and different disk drivers. The generic disk driver filter has a filter component that intercepts and filters raw sector write commands from the file system before they are received by the disk driver. The generic disk driver filter may also have a cache memory component that stores a checksum for each sector which is written to the disk. The generic disk driver filter may also have a scavenger thread component that detects and removes latent checksum entries from the cache memory so as to preserve memory availability and reduce memory requirements.

Abstract: Object rundown protection that scales with the number of processors in a shared-memory computer system is disclosed. Prior to object rundown, a cache-aware reference count data structure is used to prevent cache-pinging that would otherwise result from data sharing across processors in a multiprocessor computer system. In this data structure, a counter of positive references and negative dereferences, aligned on a particular cache line, is maintained for each processor. When an object is to be destroyed, a rundown wait process is begun, during which new references on the object are prohibited, and the total number of outstanding references is added to an on-stack global counter. Destruction is delayed until the global reference count is reduced to zero. In an implementation on non-uniform memory access multiprocessor machines, each processor's reference count is additionally allocated in a region of main memory that is physically close to that processor.

Abstract: A system and method for object rundown protection that scales with the number of processors in a shared-memory computer system is disclosed. In an embodiment of the present invention, prior to object rundown, a cache-aware reference count data structure is used to prevent cache-pinging that would otherwise result from data sharing across processors in a multiprocessor computer system. In this data structure, a counter of positive references and negative dereferences, aligned on a particular cache line, is maintained for each processor. When an object is to be destroyed, a rundown wait process is begun, during which new references on the object are prohibited, and the total number of outstanding references is added to an on-stack global counter. Destruction is delayed until the global reference count is reduced to zero.

Abstract: A system and method for object rundown protection that scales with the number of processors in a shared-memory computer system is disclosed. In an embodiment of the present invention, prior to object rundown, a cache-aware reference count data structure is used to prevent cache-pinging that would otherwise result from data sharing across processors in a multiprocessor computer system. In this data structure, a counter of positive references and negative dereferences, aligned on a particular cache line, is maintained for each processor. When an object is to be destroyed, a rundown wait process is begun, during which new references on the object are prohibited, and the total number of outstanding references is added to an on-stack global counter. Destruction is delayed until the global reference count is reduced to zero.

Abstract: A technique is provided whereby a stream of data located in a first storage location is registered for migration administration and according to preset criteria, portion(s) of the stream of data that are suited to another storage location are migrated to maximize system storage efficiencies. The file system may have use of facilities that enable the monitoring of files/streams that have been registered for migration administration or files/streams may be polled according to preset criteria. If the stream of data has portions to migrate to another storage location, the hierarchical storage management (HSM) system migrates the data to the other storage location, such as to long term or off-line storage, and preserves the data relationships of the stream via metadata. The technique allows for multiple volumes to be spanned by the migration target location, and multiple file systems may service the source and target.

Abstract: A model in which filter drivers are managed to receive callbacks for I/O requests in which the filter drivers have registered an interest. Per-volume instances of filter drivers register with a filter manager for pre-callbacks (for I/O to the file system) and post-callbacks (for I/O from the file system), and identify which I/O requests (e.g., create, read, write) they are registering to receive callbacks. The filter manager orders the instances for callbacks. When an I/O request is received, the filter manager converts the I/O request to callback data and calls the interested filters in the callback order, whereby the filter instances can process the I/O data. As the request returns from the file system, filters desiring post callbacks are called in the reverse order. Efficient context management for the filters and other functions, such as non-reentrant file I/O, are also provided by the model.

Abstract: An extensible file system format for portable storage media is provided. The extensible file system format includes the specification of primary and secondary directory entry types that may be custom defined. The primary and secondary directory entry types can be further classified as critical and benign directory entries.

Abstract: A system and method for organizing and sorting targets received in a referral response and for realizing a target fail-back and a target priority policy in a distributed file system is provided. In one embodiment, a sorting method includes receiving a referral response in the form of a list of targets that are sorted into bounded sets. Having a sorted referral response in bounded sets provides a basis for implementing a target fail-back and a target priority policy. The computer system may select a target from a sorted list of targets sorted according to site-cost and/or target priority. Then, the computer system may determine if the set target is associated with a more preferred target when compared to all available targets in the sorted list and if not, switch back to a more preferred target.

Abstract: A method and system for limiting resource usage of a version store. A limit indicates a total amount of resources that may be consumed by in-progress transactions. A running total of the amount of remaining resources is maintained. Before a new transaction is started, a determination is made as to whether there are enough resources remaining to complete the new transaction. If so, the transaction starts and the running total is adjusted. If not, the transaction waits until more resources are available. When a transaction completes, if it is the oldest in-progress transaction, resources are released. Otherwise, the resources associated with the transaction are not released until all transactions that started before the transaction have completed. This ensures that a client of a version store does not exceed the limit of resources that may be consumed by in-progress transactions.

Abstract: A model in which filter drivers are managed to receive callbacks for I/O requests in which the filter drivers have registered an interest. Per-volume instances of filter drivers register with a filter manager for pre-callbacks (for I/O to the file system) and post-callbacks (for I/O from the file system), and identify which I/O requests (e.g., create, read, write) they are registering to receive callbacks. The filter manager orders the instances for callbacks. When an I/O request is received, the filter manager converts the I/O request to callback data and calls the interested filters in the callback order, whereby the filter instances can process the I/O data. As the request returns from the file system, filters desiring post callbacks are called in the reverse order. Efficient context management for the filters and other functions, such as non-reentrant file I/O, are also provided by the model.

Abstract: A technique is provided whereby a stream of data located in a first storage location is registered for migration administration and according to preset criteria, portion(s) of the stream of data that are suited to another storage location are migrated to maximize system storage efficiencies. The file system may have use of facilities that enable the monitoring of files/streams that have been registered for migration administration or files/streams may be polled according to preset criteria. If the stream of data has portions to migrate to another storage location, the hierarchical storage management (HSM) system migrates the data to the other storage location, such as to long term or off-line storage, and preserves the data relationships of the stream via metadata. The technique allows for multiple volumes to be spanned by the migration target location, and multiple file systems may service the source and target.

Abstract: A technique for recalling data objects stored on media. A queue is created for each medium on which data objects are located, where each request to recall a data object is placed on the queue corresponding to the medium on which the data object is located. A queue is “active” when its corresponding medium is mounted and being used for recall; otherwise the queue is “non-active.” A thread is created for each active queue, where the thread retrieves from a medium the requested items on the active queue. When plural drives are available for mounting and reading media, plural queues may be active concurrently, so that the plural queues' respective threads may recall items from the plural media in parallel. Preferably, the requests on each queue are organized in an order such that the offset locations of the requested items form two monotonically increasing sequences.

Abstract: A technique for recalling data objects stored on media. A queue is created for each medium on which data objects are located, where each request to recall a data object is placed on the queue corresponding to the medium on which the data object is located. A queue is “active” when its corresponding medium is mounted and being used for recall; otherwise the queue is “non-active.” A thread is created for each active queue, where the thread retrieves from a medium the requested items on the active queue. When plural drives are available for mounting and reading media, plural queues may be active concurrently, so that the plural queues' respective threads may recall items from the plural media in parallel. Preferably, the requests on each queue are organized in an order such that the offset locations of the requested items form two monotonically increasing sequences.

Abstract: A technique is provided whereby a stream of data located in a first storage location is registered for migration administration and according to preset criteria, portion(s) of the stream of data that are suited to another storage location are migrated to maximize system storage efficiencies. The file system may have use of facilities that enable the monitoring of files/streams that have been registered for migration administration or files/streams may be polled according to preset criteria. If the stream of data has portions to migrate to another storage location, the hierarchical storage management (HSM) system migrates the data to the other storage location, such as to long term or off-line storage, and preserves the data relationships of the stream via metadata. The technique allows for multiple volumes to be spanned by the migration target location, and multiple file systems may service the source and target.

Abstract: Systems and methods are described that facilitate management of file system filters, comprising a filter manager that maps altitudes of minifilters to legacy filter order groups. An aspect of the present invention provides for minifilters that can dynamically load and unload, thus facilitating increased flexibility in filter system management.

Abstract: A method for concurrent data migration includes classifying files to be migrated into plural jobs, selecting media to which to migrate each job, and using plural drives concurrently to write the jobs to the media. The selection of a medium is performed in a way that prevents the number of writeable media from exceeding the number of available drives, unless no allocated medium has sufficient space to store any files in a migration job. A medium is preferentially selected that has already been allocated for writing, has space to store at least one file in the job, is not in use for another job, and can be robotically mounted on a drive. If such a medium does not exist, then the set of available media is canvassed to locate an alternative medium.

Abstract: The invention provides a system and method for dynamically unloading file system filters in a stacked-call-back model where filters are stacked one on top of the other to form a filter stack. A filter manager keeps track of the progress of each I/O operation and calls each filter in turn with the filter returning after it has completed processing the given operation. The filter manager dynamically unloads a filter (or an instance of a filter) that is positioned at any position in the filter stack in a reasonable amount of time while I/O operations are actively being processed. The filter/filter instance can be unloaded with outstanding I/O operations on the filter either hosted by the filter or pended by other filters. I/O operations are canceled, completed or drained in order to unload the filter or filter instance. A filter may veto the unloading of the filter.

Abstract: A method for concurrent data migration includes classifying files to be migrated into plural jobs, selecting media to which to migrate each job, and using plural drives concurrently to write the jobs to the media. The selection of a medium is performed in a way that prevents the number of writeable media from exceeding the number of available drives, unless no allocated medium has sufficient space to store any files in a migration job. A medium is preferentially selected that has already been allocated for writing, has space to store at least one file in the job, is not in use for another job, and can be robotically mounted on a drive. If such a medium does not exist, then the set of available media is canvassed to locate an alternative medium.