Abstract: Embodiments are directed to in-place erasure code transcoding for distributed file systems. A file system may be divided into a first partition associated with a first erasure code and a second partition second partition is associated with a second erasure code. If the second partition has sufficient storage space to store protection groups further actions may be performed, including: determining block stores in the first partition associated with the protection groups; transcoding contents of the block stores into other block stores based on the second erasure code; storing the other block stores in the second partition; deleting the block stores from the first partition; shifting another portion of the storage space from the first partition to the second partition such that the shifted other portion increases a size of the second partition to provide sufficient storage space for other protection groups; or the like.

Abstract: Embodiments are directed to tiered data store with persistent layers. A write tier in the file system for storing in a file system. A value for a performance metric that corresponds to write requests to the file system may be predicted based on characteristics of the write requests such that the performance metric may be determined based on a plurality of interactions with the write tier. The predicted value that exceeds a threshold value of the performance metric may be employed to cause performance of further actions, including: queuing a portion of the write requests in a memory buffer based on the predicted value and the threshold value; combining the queued portion of the write requests into s; storing the data segments in the write tier such that a measured value of the performance metric may be less than the threshold value.

Abstract: Embodiments are directed to sharing namespaces across file system clusters. A file in a spoke file system may be determined based on a command provided to the spoke file system such that the file may be associated with an inode mask. Data blocks associated with the file may be determined based on the command and the inode mask such that the data blocks may be absent from the spoke file system. Requests for leases may be generated based on the absent data blocks such that each lease request corresponds to a portion of the absent data blocks Employing the leases provided by a hub file system to: copy the absent data blocks from the hub file system where each portion of absent data blocks may be associated with a lease; updating the inode mask to include the leases and each copied portion of absent blocks.

Abstract: Embodiments are directed to managing data in a file system over a network. A file system that includes objects may be provided. A capacity value of the file system may be provided based on a storage size of the objects. A threshold value may be determined based on the capacity value and a metric partition count. Traversals of the file system may determine metric partitions such that each metric partition may be associated with some objects Metrics associated with activities that are associated with the file system may be monitored. Metric partition values may be determined based on metric values associated with the metric partitions such that each metric partition value may be based on a portion of the metric values associated with a portion of the activities associated with a corresponding metric partition. Visualizations may be generated based on the metric partition values.

Abstract: Embodiments are directed to managing file systems over a network. A core specification that defines characteristics of a portion of a file system and parameters may be provided. The characteristics may include one or more storage device characteristics. And, the parameters may include one or more of a cluster size, or a protection level. File system models may be generated based on the core specification and the parameters. Each file system model may correspond to a file system that conforms to a core specification and specific values for the parameters. A simulation engine may be employed to provide simulation results based on the file system models. Deployment models may be generated based on the simulation results. The deployment models may be employed to provision file systems that conform to the file system performance characteristics associated with its corresponding deployment model.

Abstract: Embodiments are directed to bandwidth management in distributed file systems. A request stored in a queue may be dequeued. A work credit value may be determined based on a duration since a last job and a work rate associated with the file system such that the work credit value conforms to a maximum threshold value. A job may be initialized based on the dequeued request such that the work credit value may be reduced based on a size of the job. In response to the reduced work credit value being a negative value further actions may be performed, including: determining a time delay based on the reduced work credit value, the work rate, and the size of the job; delaying an execution of the job until the expiry of the time delay such that upon expiry of the time delay, the job is executed to satisfy the request.

Abstract: Embodiments are directed to error resistant logging. A write-ahead log (WAL) for preserving a verifiable record of file system activity may be provided such that the WAL may include a sequence of log blocks that each may include a cyclic redundancy check (CRC) value of a next adjacent log block in the WAL. In response to executing a file system operation associated with payload data, further actions may be performed, including: generating log blocks based on the payload data; generating a log segment that includes the log blocks; modifying a portion of the contents of a head-block of the log segment such that the modified head-block of the log segment has a CRC value that matches a CRC value of a tail-block of the WAL; copying the contents of the modified head-block of the log segment into the tail-block of the WAL.

Abstract: A file system with distributed resource caching that includes cache volumes and agents that may be associated with clients of the file system may be provided. A cache allocation for each agent may be determined based on a capacity of the cache volumes and a number of the agents such that each cache allocation is associated with tokens that each represents a reserved portion of free space in the cache volumes. Storage jobs may be provided to the agents. Data associated with the storage jobs may be stored in the cache volumes. The cache allocation for each agent may be reduced based on the data stored for each agent.

Abstract: Embodiments are directed to managing data in a file system. A source bucket that contains a plurality of objects on a cloud computing environment may be determined. The source bucket may be associated with a target directory in the file system based on an integration relationship. Object keys from the object store may be provided based on a query and filter rules such that each object key corresponds to an object in the source bucket that conforms to the filter rules. Source objects in the source bucket may be determined based on the object keys such that each source object corresponds to one of the object keys. Target files that correspond to the source objects may be determined based on the integration relationship. In response to determining changed target files or changed source objects, copying the changed source objects or the changed target files to the target directory.

Abstract: Embodiments are directed to managing data in a file system that includes a plurality of storage nodes and a plurality of storage volumes in a cloud computing environment. Metrics associated with each storage volume may be monitored. In response to the metrics exceeding a threshold value, performing further actions, including: determining storage volumes that are unhealthy based on the metrics that exceed the threshold value; updating metadata associated with the storage volumes to indicate that the storage volumes are unhealthy; decoupling the unhealthy storage volumes from storage nodes coupled to the unhealthy storage volumes; determining replacement storage volumes based on the metadata associated with the unhealthy storage volumes; updating other metadata associated with the replacement storage volumes to indicate that the replacement storage volumes are healthy storage volumes; and coupling the healthy storage volumes with the storage nodes that were coupled to the unhealthy storage volumes.

Abstract: Embodiments are directed to managing data in a file system. A file system that includes a file storage tier and a cache storage tier may be provided. An amount of hot blocks present in the cache storage tier that are associated with a heat score that matches a maximum heat score value may be determined. In response to the amount of hot blocks exceeding an amount threshold value further actions may be performed, including: determining each cooldown block in the cache storage tier based on each heat score associated with each block in the cache storage tier; and decrementing the heat score associated with each cooldown block. In response to one or more blocks in the cache storage tier being read, the heat score associated with the one or more blocks being read may be set to the maximum heat score value.

Abstract: A software and hardware facility persistently maintains metrics on directories at different levels within a tree of a filesystem. The facility hierarchically aggregates attributes of files contained by directories and stores them as metric values in inode date of each directory within a tree. The stored metric values represent summed or otherwise aggregated data from the descendant directories and files. The metric values are utilized to provide representative compositional samplings of data in the filesystem tree structure. Based on the metric value in its inode data, each directory is weighted in a manner that determines the likelihood that each file sample is taken from that directory.

Abstract: Embodiments are directed to managing data in a file system that includes storage nodes and storage volumes in a cloud computing environment (CCE). A backup job may be executed to perform further actions, including: providing a backup identifier that is associated with the backup job; determining the storage nodes based on the file system and the CCE; disabling the file system by disabling the storage nodes, wherein each disabled storage node is associated with a shutdown timestamp; determining storage volumes associated with each disabled storage node; and generating storage images that archive a copy of information from the storage volumes; associating the storage images with the backup identifier; confirming that the file system is consistent based on an absence of file system activity occurring subsequent to each shutdown timestamp; and enabling the file system by enabling the storage nodes.

Abstract: Embodiments are directed to traversing a file system to determine file system objects to copy to an object store. In response to visiting a document object in the file system, performing further actions, including: determining a hierarchical file path of the document object that corresponds to a location in the file system based on the file system objects that are ancestor file system objects of the document object; generating an object key for the document object that encodes the hierarchical file path such that each portion of the object key corresponds to an ancestor file system object; copying the document object and the object key to the object store such that the document object may be stored in the object store as an object store object and such that the object store object may be indexed using an unordered index based on the object key; or the like.

Abstract: Embodiments are directed to managing data in a file system over a network. A source file system that includes a plurality of objects may be provided. A replication job that copies each object associated with a source replication snapshot to a target file system may be executed. The replication job may be associated with a job identifier. Recovery point information that includes the job identifier, a source snapshot number that corresponds to the source replication snapshot, a target snapshot number that corresponds to a target replication snapshot may be generated. The recovery point information may be stored on the source file system and a copy of the recovery point information may be stored on the target file system. The recovery point information or the copy of the recovery point information may be employed to recover from errors detected during execution of a next replication job.

Abstract: Embodiments are directed to managing data in a file system. An object from the file system that has parent objects may be provided. A snapshot in a current epoch may be generated such that the first snapshot may be based on the object and descendants of the object. A coverage set may be provided for the object that references the snapshot and also references each other snapshot that includes the parent objects. Read-only access to a version of the object may be provided based on a correspondence of the version and a snapshot referenced by the coverage set of the object. A coverage engine compares the update to a coverage update epoch (cue) associated with the parent objects. The coverage engine updates coverage sets of the parent objects based on grandparents of the object.

Abstract: Embodiments are directed to managing data in a file system. The file system that includes storage nodes that may be associated with storage volumes that may have a different capacity for storing data. A storage capacity of the file system may be determined based on a number of stripes of data that fit in the file system such that each stripe may be comprised of chunks that have a same chunk storage capacity. Slots in the file system that each match the chunk storage capacity may be determined based on the storage volumes. The chunks may be assigned to the slots in the file system based on the capacity of the storage nodes such that a number of chunks allocated to a same storage volume or a same storage node may be based on protection factor information.

Abstract: Embodiments are directed to a file system engine that provides a file system that includes a plurality of blocks on a file storage tier with portion of the blocks associated with a cache storage tier. A cache engine provides heat extents that include a heat score and a run length value such that the heat extents are represent activities associated with the blocks. Heat extents associated with file system activities are determined based on a position of the blocks that corresponds with the run length values of the heat extents and sectors on the file storage tier. Heat scores associated with the heat extents may be modified based on the activities and distribution models to conform the heat extents with the distribution models. Blocks associated with heat scores that are less than a threshold value provided by the distribution models may be disassociated from the cache storage tier.

Abstract: Embodiments are directed to managing data in a file system that includes a plurality of storage nodes and a plurality of storage volumes such that each storage node may be a compute instance provided by a cloud computing environment (CCE) and each storage volume may be based on blobs provided by the CCE. Write requests that include write data may be provided. A lowest range of addresses in the storage volume may be determined for storing the write data. Buckets may be determined based on the range of addresses and the write data assigned to the buckets. If portions of the buckets remain unassigned, other data may be provided from a cache volume or the blobs. The write data and the other data may be written to the portion of the blobs.

Abstract: Embodiments are directed to managing file systems over a network. Jobs may be provided to a storage computer in a file system. Control models may be associated with the jobs. Scores may be generated based on the control models. Each job may be associated with a score provided by its associated control model. And, each job that may be behind its corresponding schedule may be associated with a higher score value than each other job that may be either on its corresponding other schedule or ahead of its corresponding other schedule. Commands may be selected for execution based on the commands being associated with a job that may be associated with the higher score value that may be greater than score values associated with other jobs. The jobs may be ranked based on the updated scores. Subsequent commands may be selected and executed based on the ranking of the jobs.