Abstract: The described technology is generally directed towards data movement from data storage clusters. A data movement job can be split into data movement tasks, and the data movement tasks can be distributed among multiple nodes of a data storage cluster. In order to distribute the tasks, a data store that identifies the tasks can be split into multiple segments, and different groups of nodes can be assigned affinities to process tasks identified in the different segments. Furthermore, the nodes of each group of nodes can be assigned different start positions within their storage segment, to reduce the likelihood of contention between nodes in connection with claiming a same task. Some of the tasks can be generated in an already claimed state, which also reduces contention and allows for more efficient operation.

Abstract: The described technology is generally directed towards data movement from data storage clusters. A data movement job can be split into data movement tasks, and the data movement tasks can be distributed among multiple nodes of a data storage cluster. In order to distribute the tasks, a data store that identifies the tasks can be split into multiple segments, and different groups of nodes can be assigned affinities to process tasks identified in the different segments. Furthermore, the nodes of each group of nodes can be assigned different start positions within their storage segment, to reduce the likelihood of contention between nodes in connection with claiming a same task. Some of the tasks can be generated in an already claimed state, which also reduces contention and allows for more efficient operation.

Abstract: Systems and methods facilitating namespace representation and enhanced browsability for replicated file systems are described herein. A method as described herein can include computing, by a device operatively coupled to a processor, a first length of a first object key for a first data object, wherein the first object key comprises a directory path associated with the first data object; in response to determining that the first length is larger than a threshold, creating, by the device, a second data object having a second object key that comprises a representation of the directory path associated with the first data object; and altering, by the device, the first object key in response to the creating the second data object, resulting in an altered first object key that comprises a reference to the second data object instead of the directory path associated with the first data object.

Abstract: The described technology is generally directed towards fault tolerant cluster data handling techniques, as well as devices and computer readable media configured to perform the disclosed fault tolerant cluster data handling techniques. Nodes in a computing cluster can be configured to generate wire format resources corresponding to operating system resources. A wire format resource can comprise a cache key and a hint information to locate data, such as a file, corresponding to the operating system resource. The wire format resource can be stored in a resource cache along with a pointer that points to the operating system resource. The wire format resource can also be provided to client devices. Nodes in the computing cluster can be configured to receive and process client instructions that include wire format resources, as well as to use hint information to re-allocate data associated with a wire format resource.

Abstract: Incremental replication between foreign system dataset stores is described herein. A method as described herein can include connecting to a target storage system to facilitate an initial transfer of a dataset from the system to the target storage system, wherein the system is distinct from the target storage system; in response to determining that a first version of the dataset is present on the target storage system, comparing the first version of the dataset to a second version of the dataset stored on the system; and, in response to determining that the second version of the dataset is more recent than the first version of the dataset, transferring a differential between the first version of the dataset and the second version of the dataset to the target storage system, the differential comprising less than all of the second version of the dataset.

Abstract: The described technology is generally directed towards fault tolerant cluster data handling techniques, as well as devices and computer readable media configured to perform the disclosed fault tolerant cluster data handling techniques. Nodes in a computing cluster can be configured to generate wire format resources corresponding to operating system resources. A wire format resource can comprise a cache key and a hint information to locate data, such as a file, corresponding to the operating system resource. The wire format resource can be stored in a resource cache along with a pointer that points to the operating system resource. The wire format resource can also be provided to client devices. Nodes in the computing cluster can be configured to receive and process client instructions that include wire format resources, as well as to use hint information to re-allocate data associated with a wire format resource.

Abstract: Systems and methods facilitating separation of dataset creation from movement in file replication are described herein. A method as described herein can include scheduling, by a first system operatively coupled to a processor, a first transfer of a dataset to a second system at a first time according to a first replication schedule; scheduling, by the first system, a second transfer of the dataset to a third system at the first time according to a second replication schedule; creating, by the first system, a snapshot of the dataset at a second time that is distinct from the first time according to a snapshot schedule that is distinct from the first replication schedule and the second replication schedule; and transferring, by the first system, at least a portion of the snapshot to the second system and the third system at the first time.

Abstract: Systems and methods facilitating separation of dataset creation from movement in file replication are described herein. A method as described herein can include scheduling, by a first system operatively coupled to a processor, a first transfer of a dataset to a second system at a first time according to a first replication schedule; scheduling, by the first system, a second transfer of the dataset to a third system at the first time according to a second replication schedule; creating, by the first system, a snapshot of the dataset at a second time that is distinct from the first time according to a snapshot schedule that is distinct from the first replication schedule and the second replication schedule; and transferring, by the first system, at least a portion of the snapshot to the second system and the third system at the first time.

Abstract: Incremental replication between foreign system dataset stores is described herein. A method as described herein can include connecting to a target storage system to facilitate an initial transfer of a dataset from the system to the target storage system, wherein the system is distinct from the target storage system; in response to determining that a first version of the dataset is present on the target storage system, comparing the first version of the dataset to a second version of the dataset stored on the system; and, in response to determining that the second version of the dataset is more recent than the first version of the dataset, transferring a differential between the first version of the dataset and the second version of the dataset to the target storage system, the differential comprising less than all of the second version of the dataset.

Abstract: Systems and methods facilitating namespace representation and enhanced browsability for replicated file systems are described herein. A method as described herein can include computing, by a device operatively coupled to a processor, a first length of a first object key for a first data object, wherein the first object key comprises a directory path associated with the first data object; in response to determining that the first length is larger than a threshold, creating, by the device, a second data object having a second object key that comprises a representation of the directory path associated with the first data object; and altering, by the device, the first object key in response to the creating the second data object, resulting in an altered first object key that comprises a reference to the second data object instead of the directory path associated with the first data object.

Abstract: Implementations are provided herein for having at least two data streams associated with each file in a file system. The first, a cache overlay layer, can store additional state information on a per block basis that details whether each individual block of file data within the cache overlay layer is clean, dirty, or indicates that a write back to the storage layer is in progress. The second, a storage layer, can be a use case defined repository that can transform data using data augmentation methods as a part of tiering data to external repositories. File system operations directed to the cache overlay layer can be processed asynchronously from file system operations directed to the storage layer.

Abstract: Data storage facilities that provide data storage services frequently perform management tasks (e.g., workflows) on the data files that are being managed. These workflows have a variety of different characteristics and, as such, the data storage facility has an incentive to set different behaviors for different workflows. For example, many types of workflows performing operations on a file, once started, should be executed to completion before another workflow begins performing operations on the file. Thus, a persistent behavior for these workflows is desired such that the workflow can resume to completion after the workflow ceases executing for some reason (e.g., power loss, crash, restart, etc.). However, there is no guarantee that some workflows (e.g., user-initiated workflows) will resume after ceasing execution. For those workflows, a persistent behavior is not desired. A cookie can be employed to provide enhanced management of file workflows such as setting various behavioral parameters.

Abstract: Implementations are provided herein for cluster wide unique and persistent locking cookie that can be generated and associated with a file. An operation, like in one example, a semantic operation, can be divided into multiple sub operations and each can be associated with the locking cookie that was placed on the file at the onset of the operation. While the locking cookie is in place on a file, any operation targeted to the file can have the cookie associated with the operation matched against the locking cookie, and only operations that have a matching cookie can proceed past the lock and have the operation performed by the file system. It can be appreciated the multiple sub operations can be performed in parallel if their operation cookie matches the locking cookie for the file. In one implementation, checkpoints can be established and progress towards checkpoints can be tracked to allow restarts of operations in the event of crashes or failures.

Abstract: Implementations are provided herein for providing a consistent view of file during an extended backup process of a file system using a persistent cache layer to tier data to an external repository. A snapshot of the files that are targeted for backup can be taken. A deep write-back operation can then be processed that includes processing all outstanding write-back operations and associated convert-and-store-metadata operations for each file targeted in the backup process. After the deep write-back process finishes, a backup index of the storage layer can be generated and the backup can be performed relying on a consistent view of the storage layer being preserved throughout the backup process.

Abstract: Implementations are provided herein for having at least two data streams associated with each file in a file system. The first, a cache overlay layer, can store additional state information on a per block basis that details whether each individual block of file data within the cache overlay layer is clean, dirty, or indicates that a write back to the storage layer is in progress. The second, a storage layer, can be a use case defined repository that can transform data using data augmentation methods or store unmodified raw data in local storage. File system operations directed to the cache overlay layer can be processed asynchronously from file system operations directed to the storage layer.