Abstract: Embodiments are directed to a file system engine that provides a file system with parent objects associated with child objects. The file system engine provides a replication snapshot associated with an epoch of the file system such that each child object modified during the epoch and each associated parent object is associated with the replication snapshot. A root object of a portion of the file system included in the replication snapshot may be provided. A replication engine may traverse the portion of file system starting from the root object such that the parent objects that are not associated with the replication snapshot are omitted from the traversal. The replication engine may determine replication objects based on the traversal such that each replication object was modified during the epoch. Then the replication engine may execute a replication job that copies replication objects to a target file system.

Abstract: Managing data in a file system with a verification engine that may obtain a user identifier associated with a user, an object identifier, and a target object. The verification engine may determine target identities associated with two or more file system protocols based on the user identifier. The verification engine may determine permission entries associated with the two or more file system protocols associated and the target object. The verification engine may employ the target identities and the permission entries to directly verify access rights to the target object for the user. Accordingly, the verification engine may provide a report that includes the target identities, the permission entries, or the access rights.

Abstract: Embodiments are directed to managing data in a file system. A pre-fetch engine may receive requests from a client the file system that includes a pre-fetch storage tier and a file storage tier of storage devices. The pre-fetch engine determines a pre-fetch policy based on the requests such that the pre-fetch policy determines which blocks to copy to the pre-fetch storage tier. And, the pre-fetch policy may be associated with a score model that includes score rules where one of the rules may be associated with a client score. The pre-fetch engine may obtain scores associated with the score rules such that the scores are based on previous requests made by the client. In response to scores exceeding a threshold value, the pre-fetch engine may copy the blocks to the pre-fetch storage tier. The pre-fetch engine may update the scores based on the performance of the pre-fetch policy.

Abstract: Embodiments are directed to file systems. A replication engine may establish a secure communication channel between a source file system and a target file system. The replication engine may: instantiate a replication job associated with rules; determine changes in the source file system; determine characteristics of the replication job that may be based on the changes; compare the to the characteristics and a black-out schedule; execute the replication job to communicate the changes in the source file system to the target file system based on a result of the comparison. Upon completion of the replication job, the replication engine may automatically instantiating a next replication job to copy subsequent changes in the source file system to the target file system.

Abstract: A facility for snapshot space accounting for a storage system, such as a filesystem is disclosed. The facility enables users to quickly and easily determine the amount of storage space that would be released or recovered if a snapshot were to be purged. The facility may work in conjunction with, or as part of, a snapshot service. The facility maintains an expiration data structure and a count data structure and uses these data structures in implementing the disclosed snapshot space accounting techniques. The expiration data structure represents the life cycle of each snapshot element maintained by the facility while the count data structure represents, for pairs of snapshots, the size of the information stored in the snapshot data that expired and that spans the corresponding pair of snapshots.

Abstract: Embodiments are directed to a file system engine that provides a file system that includes a plurality of blocks on a file storage tier with portion of the blocks associated with a cache storage tier. A cache engine provides heat extents that include a heat score and a run length value such that the heat extents are represent activities associated with the blocks. Heat extents associated with file system activities are determined based on a position of the blocks that corresponds with the run length values of the heat extents and sectors on the file storage tier. Heat scores associated with the heat extents may be modified based on the activities and distribution models to conform the heat extents with the distribution models. Blocks associated with heat scores that are less than a threshold value provided by the distribution models may be disassociated from the cache storage tier.

Abstract: Embodiments are directed to managing data in a file system. An engine may receive read requests associated with a file system that includes pre-fetch storage and file storage. The engine may employ a pre-fetch policy selected from a plurality of pre-fetch policies to copy blocks to pre-fetch storage. The engine may evaluate the selected pre-fetch policy and also evaluate simulated performances for each unselected pre-fetch policy based on subsequent read requests. The engine may provide a performance score based on the evaluation of the selected pre-fetch policy and it may provide other performance scores based on the simulated evaluation of each unselected pre-fetch policy If one of the other performance scores for one of the unselected pre-fetch policies exceeds the performance score of the selected pre-fetch policy, the engine may select that unselected pre-fetch policies as a new selected pre-fetch policy.

Abstract: A software and hardware facility persistently maintains metrics on directories at different levels within a tree of a filesystem. The facility hierarchically aggregates attributes of files contained by directories and stores them as metric values in each directory within a tree. The stored values represent summed or otherwise aggregated data from the descendant directories and files. The metric values represent aggregated attributes such as total space consumed by a directory and all the descendant directories and files, total number of files within a directory, total data blocks used by a directory and its descendant directories and data files, etc.

Abstract: Providing a statistical analysis of all files in a file system based on random sampling of data blocks to identify individual user consumption of file system resources and characteristics of the files stored in the file system. In one or more of the various embodiments, the file system is based on information for a plurality of cylinder groups. Also, each cylinder group may include at one or more known locations at least three types of data structures that enable reverse mapping of data blocks to root directories.

Abstract: Embodiments are directed to managing data in a file system. A file system engine provides a file system that includes file system objects that may be accessible by two or more clients that use different native permission schemes. And, a permissions engine may determine map rules based on characteristics of a request to access request a file. The permission engine may execute the one or more map rules to provide platform permission values based on the native permission values. The permission engine may compare requested platform permission values to the platform permission values associated with the file. The permission engine may provide the access rights to the file based on an affirmative result of the comparison.

Abstract: A facility comprising methods and systems for analyzing and visualizing trends in the usage of data within a shared storage filesystem is disclosed. The facility analyzes the rate at which the filesystem or a portion thereof is used by periodically generating manifests of the usage of the filesystem and comparing one manifest to another manifest. Furthermore, the facility may store additional information relevant to the manifest, such as the time at which the manifest was taken; for each directory, the sum of all of the values determined for items in that directory (including any subdirectories); and so on. In this manner, the facility collects and stores information relevant to developing trend information for each item in the filesystem. The trends analysis and visualizations described herein provide quick insight into the changes deemed most interesting or significant between two times.

Abstract: Providing a statistical analysis of all files in a file system based on random sampling of data blocks to identify individual user consumption of file system resources and characteristics of the files stored in the file system. In one or more of the various embodiments, the file system is based on information for a plurality of cylinder groups. Also, each cylinder group may include at one or more known locations at least three types of data structures that enable reverse mapping of data blocks to root directories.

Abstract: A facility comprising systems and method for automatically triggering the collection of comprehensive monitoring information in a distributed processing system. The facility compares the overall performance of distributed processing system to one or more performance metrics and, in response to determining that one or more performance metrics is not satisfied, triggers one or more of the nodes within the distributed processing system to increase one or more of its monitoring rate or its distribution rate. The facility collects and analyzes the collected information to provide resources that can be used to assess and diagnose failures within the distributed processing system. In this manner, the facility reacts to performance anomalies by triggering nodes within in the system to provide comprehensive performance information over a trigger period for diagnostic purposes.

Abstract: A facility comprising methods and systems for analyzing and visualizing trends in the usage of data within a shared storage filesystem is disclosed. The facility analyzes the rate at which the filesystem or a portion thereof is used by periodically generating manifests of the usage of the filesystem and comparing one manifest to another manifest. Furthermore, the facility may store additional information relevant to the manifest, such as the time at which the manifest was taken; for each directory, the sum of all of the values determined for items in that directory (including any subdirectories); and so on. In this manner, the facility collects and stores information relevant to developing trend information for each item in the filesystem. The trends analysis and visualizations described herein provide quick insight into the changes deemed most interesting or significant between two times.

Abstract: Embodiments are directed to managing data in a file system. A file system engine may provide a file system that includes file system objects accessible by two or more clients that use different native permission schemes. The file system engine may receive an access request from a client to access the file system objects that includes native permission values. A permissions engine provides platform permission values from a platform permission scheme associated with the file system objects. The permissions engine provides requested platform permission values based on the platform permission scheme and the native permission values included in the access request. The permissions engine compares the requested platform permission values to the platform permission values associated with the file system objects. The permissions engine provides access rights to the file system objects based on an affirmative result of the comparison.

Abstract: A software and hardware facility persistently maintains metrics on directories at different levels within a tree of a filesystem. The facility hierarchically aggregates attributes of files contained by directories and stores them as metric values in each directory within a tree. The stored values represent summed or otherwise aggregated data from the descendant directories and files. The metric values represent aggregated attributes such as checksums, MIN and/or MAX aggregates, sameness bits, bloom filter aggregates, queryable user tags, moving average aggregates, b-tree aggregates, etc. The facility provides mechanisms for efficiently sampling activities performed by the filesystem and visually representing these activities.

Abstract: A transactional block storage system is provided which is capable of supporting a single-phase commit for data writes specifying a protected storage unit. The storage system includes a data storage map that logically links the protected data storage unit to two or more block storage units associated with a layer of the protected data storage unit. The storage system also includes an address abstraction layer which translates write requests to the block storage units and resolves whether those write requests are atomically committed to the storage system in a single phase transaction. The address abstraction layer is further configured to detected when a block storage unit becomes unavailable during a transaction and create a cleaning kit for that block in order to prevent data loss. Additionally, the address abstraction layer facilitates moving, copying, and merging of block storage units without global locking in the storage system.

Abstract: A facility for managing filesystem object storage quotas (i.e., size limits) in a storage environment is disclosed. The facility enables users to establish, modify, and remove quotas on directories and files within a filesystem. Each quota acts as a soft limit on the size of the associated filesystem object, including any child objects of the filesystem object. The facility improves the speed at which the system can test for and identify violations of quotas established for individual filesystem objects by using aggregation and reconciliation techniques rather than constantly traversing a filesystem in its entirety to test for violations of quotas.

Abstract: Embodiments are directed to file systems. A replication engine may establish a secure communication channel between a source file system and a target file system. The replication engine may: instantiate a replication job associated with rules; determine changes in the source file system; determine characteristics of the replication job that may be based on the changes; compare the to the characteristics and a black-out schedule; execute the replication job to communicate the changes in the source file system to the target file system based on a result of the comparison. Upon completion of the replication job, the replication engine may automatically instantiating a next replication job to copy subsequent changes in the source file system to the target file system.

Abstract: A software and hardware facility persistently maintains metrics on directories at different levels within a tree of a filesystem. The facility hierarchically aggregates attributes of files contained by directories and stores them as metric values in each directory within a tree. The stored values represent summed or otherwise aggregated data from the descendant directories and files. The metric values represent aggregated attributes such as checksums, MIN and/or MAX aggregates, sameness bits, bloom filter aggregates, queryable user tags, moving average aggregates, b-tree aggregates, etc. The facility provides mechanisms for efficiently sampling activities performed by the filesystem and visually representing these activities.