Abstract: Embodiments are directed towards managing the distribution of tasks in a storage system. An execution path for tasks may be generated based on the type of the task and characteristics of the storage system such that the execution path includes storage computers in a storage system. The tasks may be provided to each storage computer in the execution path. A working set of intermediate results may be generated on the storage computer in the execution path. If there is more than one storage computer in the execution path, working sets may be iteratively communicated to a next storage computer in the execution path such that the next storage computer employs a previously generated working set to generate a next working set until each storage computer in the execution path has been employed to generate a working set. The results may be stored on the storage computers.

Abstract: Embodiments are directed to storing data in a storage system. Data portions may be provided from blocks of write data provided by client computers. A storage location identifier may be associated with each data portion based on the content of the each data portion. Data portions may be stored in a memory buffer that may correspond to the storage location identifiers. If an amount of data stored in the memory buffer exceeds a threshold value, or if a maximum latency time expires, the data portions are stored on a storage location that corresponds to the memory buffer, such that, the maximum latency time is selected to prevent a collision caused by simultaneously storing of data portions in the same storage location. A write acknowledgement message may be provided to the client computers indicating that the blocks of write data are stored to stable storage.

Abstract: Embodiments are directed to storing data in a storage system. Data portions may be provided from blocks of write data provided by client computers. A storage location identifier may be associated with each data portion based on the content of the each data portion. Data portions may be stored in a memory buffer that may correspond to the storage location identifiers. If an amount of data stored in the memory buffer exceeds a threshold value, or if a maximum latency time expires, the data portions are stored on a storage location that corresponds to the memory buffer, such that, the maximum latency time is selected to prevent a collision caused by simultaneously storing of data portions in the same storage location. A write acknowledgement message may be provided to the client computers indicating that the blocks of write data are stored to stable storage.

Abstract: Embodiments are directed towards managing designations of a plurality of files across multiple clusters. Files may be designated as primary files or secondary files. Primary files at one cluster may be asynchronously replicated at another cluster as secondary files. Secondary files may be synchronously validated with remote primary files based on local file access requests. If a write request for a local secondary file is provided, then the local file may be designated as the primary file and the remote file may be designated as the secondary file. Local access may then be provided to the local primary file. If a read request for a local secondary file is provide, then local access may be provided to the local secondary file without changing file designations. In some embodiments, file designations may be preemptively changed based on a file change designation schedule for clusters at different geographic locations.

Abstract: Embodiments are directed towards storing data in a storage system. A data controller may obtain a write request and write data from a client computer. A write message may be generated and provided to a data coordinator computer. The data coordinator may communicate the write message to a plurality of L-node computers. The data coordinator may obtain write confirmation messages from the L-node computers that indicate that the write data is stored. If enough write confirmation messages are obtained to indicate that a quorum is reached, the data coordinator may communicate a save confirmation message to the data controller. The data controller may generate a write acknowledgement message based on the save confirmation message provided by the data coordinator. The data controller may provide the write acknowledgement message to the client computer that made the original write request.

Abstract: Embodiments are directed towards managing data storage that may experience a data failure. If a repair event is associated with a data storage failure, a new repair task may be generated and added to a task list. A priority value for each repair task in the task list may be determined based in part on the mean-time-to-data-loss (MTTDL) value associated with each repair task in the task list such that a lower MTTDL may indicate a higher priority value over a lower MTTDL. One or more repair tasks may be promoted to become active repair tasks based on the priority value the repair tasks such that the promoted repair tasks have a higher priority that than other repair tasks in the task list, if any. Each active repair task may be executed to repair one or more associated the storage failures.

Abstract: Embodiments are directed towards managing the distribution of tasks in a storage system. An execution path for tasks may be generated based on the type of the task and characteristics of the storage system such that the execution path includes storage computers in a storage system. The tasks may be provided to each storage computer in the execution path. A working set of intermediate results may be generated on the storage computer in the execution path. If there is more than one storage computer in the execution path, working sets may be iteratively communicated to a next storage computer in the execution path such that the next storage computer employs a previously generated working set to generate a next working set until each storage computer in the execution path has been employed to generate a working set. The results may be stored on the storage computers.

Abstract: Embodiments are directed towards cloning files in a distributed file system. In response to a request to create file clones, a file cloning application can be employed to generate file clones based on a source file. The file cloning application can generate a new inode for each new file clone and at least one new shadow store. Data blocks from the source file can be transferred to the shadow store. The transferred data blocks can be replaced with reference blocks that point to the data blocks transferred to the shadow store. Likewise, additional reference blocks pointing to data blocks in the shadow store can be generated for the new file clone inodes. Further, the shadow store can maintain reference counters that track the number of reference blocks pointing to each shadow store block. Also, file clones can be generated based on a point-in-time snapshot of the source file.

Abstract: Embodiments are directed towards managing the distribution of tasks in a storage system. An execution path for tasks may be generated based on the type of the task and characteristics of the storage system such that the execution path includes storage computers in a storage system. The tasks may be provided to each storage computer in the execution path. A working set of intermediate results may be generated on the storage computer in the execution path. If there is more than one storage computer in the execution path, working sets may be iteratively communicated to a next storage computer in the execution path such that the next storage computer employs a previously generated working set to generate a next working set until each storage computer in the execution path has been employed to generate a working set. The results may be stored on the storage computers.

Abstract: Embodiments are directed towards managing data storage that may experience a data failure. If a repair event is associated with a data storage failure, a new repair task may be generated and added to a task list. A priority value for each repair task in the task list may be determined based in part on the mean-time-to-data-loss (MTTDL) value associated with each repair task in the task list such that a lower MTTDL may indicate a higher priority value over a lower MTTDL. One or more repair tasks may be promoted to become active repair tasks based on the priority value the repair tasks such that the promoted repair tasks have a higher priority that than other repair tasks in the task list, if any. Each active repair task may be executed to repair one or more associated the storage failures.

Abstract: Embodiments are directed towards managing data storage that may experience a data failure. If a repair event is associated with a data storage failure, a new repair task may be generated and added to a task list. A priority value for each repair task in the task list may be determined based in part on the mean-time-to-data-loss (MTTDL) value associated with each repair task in the task list such that a lower MTTDL may indicate a higher priority value over a lower MTTDL. One or more repair tasks may be promoted to become active repair tasks based on the priority value the repair tasks such that the promoted repair tasks have a higher priority that than other repair tasks in the task list, if any. Each active repair task may be executed to repair one or more associated the storage failures.

Abstract: A treewalk for splitting a file directory is disclosed for parallel execution of work items over a filesystem. The given work item is assigned to a worker. Thereafter, a request is sent to split the file directory to share a portion of the file directory with another worker. The worker examines the file directory for a next splittable directory entry key after a current in-progress location within a key space for the file directory. If no key is detected, the file directory is identified as unsplittable. If an entry is detected, the key space is progressively searched for a next entry between a next available directory entry key and an upper boundary. The worker identifies the file directory as splittable if the next available entry is found, providing a range of entries within the file directory for which parallel execution of the work item.

Abstract: Embodiments are directed towards storing data in a storage system using an extensible data path. Data files may be provided to a caching tier in a storage system. If data files remain in the caching tier longer than a time limit, those data files may be removed from the caching tier and provided to a processing pipeline. The processing pipeline may be coupled to a capacity tier of the storage system. Filters to include in the processing pipeline may be determined based on the type of the data files. The data files may be updated based on applying each filter, such that each update corresponding to each filter may be cumulatively applied to each data file. Each updated data file may be stored in the capacity tier of the storage system after each filter in the processing pipeline has been applied.

Abstract: A treewalk for splitting a file directory is disclosed for parallel execution of work items over a filesystem. The given work item is assigned to a worker. Thereafter, a request is sent to split the file directory to share a portion of the file directory with another worker. The worker examines the file directory for a next splittable directory entry key after a current in-progress location within a key space for the file directory. If no key is detected, the file directory is identified as unsplittable. If an entry is detected, the key space is progressively searched for a next entry between a next available directory entry key and an upper boundary. The worker identifies the file directory as splittable if the next available entry is found, providing a range of entries within the file directory for which parallel execution of the work item.