Abstract: Embodiments are directed to managing data in a file system. An object from the file system that has parent objects may be provided. A snapshot in a current epoch may be generated such that the first snapshot may be based on the object and descendants of the object. A coverage set may be provided for the object that references the snapshot and also references each other snapshot that includes the parent objects. Read-only access to a version of the object may be provided based on a correspondence of the version and a snapshot referenced by the coverage set of the object. A coverage engine compares the update to a coverage update epoch (cue) associated with the parent objects. The coverage engine updates coverage sets of the parent objects based on grandparents of the object.

Abstract: Embodiments are directed to managing data in a file system. The file system that includes storage nodes that may be associated with storage volumes that may have a different capacity for storing data. A storage capacity of the file system may be determined based on a number of stripes of data that fit in the file system such that each stripe may be comprised of chunks that have a same chunk storage capacity. Slots in the file system that each match the chunk storage capacity may be determined based on the storage volumes. The chunks may be assigned to the slots in the file system based on the capacity of the storage nodes such that a number of chunks allocated to a same storage volume or a same storage node may be based on protection factor information.

Abstract: Embodiments are directed to a file system engine that provides a file system that includes a plurality of blocks on a file storage tier with portion of the blocks associated with a cache storage tier. A cache engine provides heat extents that include a heat score and a run length value such that the heat extents are represent activities associated with the blocks. Heat extents associated with file system activities are determined based on a position of the blocks that corresponds with the run length values of the heat extents and sectors on the file storage tier. Heat scores associated with the heat extents may be modified based on the activities and distribution models to conform the heat extents with the distribution models. Blocks associated with heat scores that are less than a threshold value provided by the distribution models may be disassociated from the cache storage tier.

Abstract: Embodiments are directed to managing file systems over a network. Jobs may be provided to a storage computer in a file system. Control models may be associated with the jobs. Scores may be generated based on the control models. Each job may be associated with a score provided by its associated control model. And, each job that may be behind its corresponding schedule may be associated with a higher score value than each other job that may be either on its corresponding other schedule or ahead of its corresponding other schedule. Commands may be selected for execution based on the commands being associated with a job that may be associated with the higher score value that may be greater than score values associated with other jobs. The jobs may be ranked based on the updated scores. Subsequent commands may be selected and executed based on the ranking of the jobs.

Abstract: Embodiments are directed to managing data in a file system that includes a plurality of storage nodes and a plurality of storage volumes such that each storage node may be a compute instance provided by a cloud computing environment (CCE) and each storage volume may be based on blobs provided by the CCE. Write requests that include write data may be provided. A lowest range of addresses in the storage volume may be determined for storing the write data. Buckets may be determined based on the range of addresses and the write data assigned to the buckets. If portions of the buckets remain unassigned, other data may be provided from a cache volume or the blobs. The write data and the other data may be written to the portion of the blobs.

Abstract: A facility for managing filesystem object storage quotas (i.e., size limits) in a storage environment is disclosed. The facility enables users to establish, modify, and remove quotas on directories and files within a filesystem. Each quota acts as a soft limit on the size of the associated filesystem object, including any child objects of the filesystem object. The facility improves the speed at which the system can test for and identify violations of quotas established for individual filesystem objects by using aggregation and reconciliation techniques rather than constantly traversing a filesystem in its entirety to test for violations of quotas.

Abstract: Embodiments are directed to traversing a file system to determine file system objects to copy to an object store. In response to visiting a document object in the file system, performing further actions, including: determining a hierarchical file path of the document object that corresponds to a location in the file system based on the file system objects that are ancestor file system objects of the document object; generating an object key for the document object that encodes the hierarchical file path such that each portion of the object key corresponds to an ancestor file system object; copying the document object and the object key to the object store such that the document object may be stored in the object store as an object store object and such that the object store object may be indexed using an unordered index based on the object key; or the like.

Abstract: Embodiments are directed to managing data in a file system over a network. A source file system that includes a plurality of objects may be provided. A replication job that copies each object associated with a source replication snapshot to a target file system may be executed. The replication job may be associated with a job identifier. Recovery point information that includes the job identifier, a source snapshot number that corresponds to the source replication snapshot, a target snapshot number that corresponds to a target replication snapshot may be generated. The recovery point information may be stored on the source file system and a copy of the recovery point information may be stored on the target file system. The recovery point information or the copy of the recovery point information may be employed to recover from errors detected during execution of a next replication job.

Abstract: Embodiments are directed to a replication engine that provides a root node for a file that may be associated with a replication snapshot where the file is comprised of the root node, internal nodes, and data nodes. The replication engine may start at the root node and traverse the file system. If a visited internal node has an epoch value that matches the epoch, the replication engine may continue the traversal by visiting a next unvisited adjacent internal node. If the visited internal node has an epoch value that mismatches the epoch, the replication engine may backtrack to a nearest adjacent internal node that matches the epoch. And, in response to visiting a data node the replication engine may be arranged to determine blocks that are associated with the data node and communicate a copy of the blocks to a target file system.

Abstract: A software and hardware facility persistently maintains metrics on directories at different levels within a tree of a filesystem. The facility hierarchically aggregates attributes of files contained by directories and stores them as metric values in each directory within a tree. The stored values represent summed or otherwise aggregated data from the descendant directories and files. The metric values represent aggregated attributes such as checksums, MIN and/or MAX aggregates, sameness bits, bloom filter aggregates, queryable user tags, moving average aggregates, b-tree aggregates, etc.

Abstract: Embodiments are directed to managing data in a file system that includes storage nodes and storage volumes in a cloud computing environment (CCE). A backup job may be executed to perform further actions, including: providing a backup identifier that is associated with the backup job; determining the storage nodes based on the file system and the CCE; disabling the file system by disabling the storage nodes, wherein each disabled storage node is associated with a shutdown timestamp; determining storage volumes associated with each disabled storage node; and generating storage images that archive a copy of information from the storage volumes; associating the storage images with the backup identifier; confirming that the file system is consistent based on an absence of file system activity occurring subsequent to each shutdown timestamp; and enabling the file system by enabling the storage nodes.

Abstract: Embodiments are directed to managing data in a file system. A query may be provided to determine characteristics associated with a file system and a sampling model. The sampling model may be employed to determine objects in the file system based on a traversal of the file system. The sampling model may be employed to determine metrics associated with the alternate data streams (ADSs) included in the objects such that each of the objects that includes a separate main data stream and the ADSs may be associated with data blocks and the ADSs may each be associated with other data blocks. The sampling model and the query may be employed to sample the objects to provide partial results such that the sampling may be distributed between the objects based on the sampling model and the metrics. A result to the query may be provided based on the partial results.

Abstract: Embodiments are direct to managing file systems. Alternate data streams (ADSs) of an object in a file system may be modified based on a request from a client and metrics associated with the ADSs may be modified. Values for reconcile fields associated with the object may be set based on the modified metrics such that the object with the modified ADSs may be included in a reconcile list. And, such that an acknowledgment of the request may be provided to the client. A reconcile engine may determine a parent object of the object. The object may be reconciled and the parent object may be placed on the reconcile list for reconciling in turn.

Abstract: A software and hardware facility persistently maintains metrics on directories at different levels within a tree of a filesystem. The facility hierarchically aggregates attributes of files contained by directories and stores them as metric values in each directory within a tree. The stored values represent summed or otherwise aggregated data from the descendant directories and files. The metric values represent aggregated attributes such as checksums, MIN and/or MAX aggregates, sameness bits, bloom filter aggregates, queryable user tags, moving average aggregates, b-tree aggregates, etc. The facility provides mechanisms for efficiently sampling activities performed by the filesystem and visually representing these activities.

Abstract: Embodiments are directed to managing file systems over a network. Jobs may be provided to a storage computer in a file system. Control models may be associated with the jobs. Scores may be generated based on the control models. Each job may be associated with a score provided by its associated control model. And, each job that may be behind its corresponding schedule may be associated with a higher score value than each other job that may be either on its corresponding other schedule or ahead of its corresponding other schedule. Commands may be selected for execution based on the commands being associated with a job that may be associated with the higher score value that may be greater than score values associated with other jobs. The jobs may be ranked based on the updated scores. Subsequent commands may be selected and executed based on the ranking of the jobs.

Abstract: Embodiments are directed to managing data in a file system over a network. A file system that includes a master node and client nodes may be provided. A master quorum value and a client quorum value having the same value may be provided. A profile catalog associated with the master node that includes parameters may be provided. Profile registers associated with the client nodes may be provided. Change actions may be executed on each client node based on change information provided by the master node. In response to errors, access to the file system may be suspended; the master quorum value may be modified to indicate that the file system is out of quorum; recovery actions may be executed; the client quorum value at each client node may be modified to correspond with the master quorum value; and access to the file system may be restored.

Abstract: A transactional block storage system is provided which is capable of supporting a single-phase commit for data writes specifying a protected storage unit. The storage system includes a data storage map that logically links the protected data storage unit to two or more block storage units associated with a layer of the protected data storage unit. The storage system also includes an address abstraction layer which translates write requests to the block storage units and resolves whether those write requests are atomically committed to the storage system in a single phase transaction. The address abstraction layer is further configured to detected when a block storage unit becomes unavailable during a transaction and create a cleaning kit for that block in order to prevent data loss. Additionally, the address abstraction layer facilitates moving, copying, and merging of block storage units without global locking in the storage system.

Abstract: Embodiments are directed to managing file systems over a network. A core specification that defines characteristics of a portion of a file system and parameters may be provided. The characteristics may include one or more storage device characteristics. And, the parameters may include one or more of a cluster size, or a protection level. File system models may be generated based on the core specification and the parameters. Each file system model may correspond to a file system that conforms to a core specification and specific values for the parameters. A simulation engine may be employed to provide simulation results based on the file system models. Deployment models may be generated based on the simulation results. The deployment models may be employed to provision file systems that conform to the file system performance characteristics associated with its corresponding deployment model.

Abstract: Embodiments are directed to managing data in a file system. A source file system may be mirrored on a target file system by executing one or more replication jobs. A root object of the source file system may be associated with a current read-only identifier corresponds to a current update epoch of the source file system. A replication job that copies objects from the source file system to the target file system may be executed. A write request directed to the source file system may be received. Write objects associated with the write request may be determined The source file system may be traversed upwards from the write objects. A read-only identifier associated with each visited object may be determined based on the read-only identifier matching the current read-only identifier. The write request may be enabled based on the mismatch of the read-only identifier and current read-only identifier.

Abstract: A facility comprising systems and method for automatically triggering the collection of comprehensive monitoring information in a distributed processing system. The facility compares the overall performance of distributed processing system to one or more performance metrics and, in response to determining that one or more performance metrics is not satisfied, triggers one or more of the nodes within the distributed processing system to increase one or more of its monitoring rate or its distribution rate. The facility collects and analyzes the collected information to provide resources that can be used to assess and diagnose failures within the distributed processing system. In this manner, the facility reacts to performance anomalies by triggering nodes within in the system to provide comprehensive performance information over a trigger period for diagnostic purposes.