Abstract: A generation identifier is employed with various systems and methods in order to identify situations where a workload has been reassigned to a new node and where a workload is still being processed by an old node during a failure between nodes. A master node may assign a workload to a worker node. The worker node sends a request to access target data. The request may be associated with a generation identifier and workload identifier that identifies the node and workload. At some point, a failure occurs between the master node and worker node. The master node reassigns the workload to another worker node. The new worker node accesses the target data with a different generation identifier, indicating to the storage system that the workload has been reassigned. The old worker node receives an indication from the storage system that the workload has been reassigned and stops processing the workload.

Abstract: Some storage systems are configured with VDL (valid data length) type controls that are implemented on a per cluster basis and, in some instances, on a sub-cluster basis, rather than simply a per file basis. In some instances, per-cluster VDL metadata for the storage clusters is stored and referenced at the edge data volume nodes of a distributed network for the storage system rather than, and/or without, storing or synchronizing the per-cluster VDL metadata at a master node that manages the corresponding storage clusters for the different data volume nodes. Sequence controls are also provided and managed by the master node and synchronized with the edge data volume nodes to further control access to data contained in the storage clusters.

Abstract: A generation identifier is employed with various systems and methods in order to identify situations where a workload has been reassigned to a new node and where a workload is still being processed by an old node during a failure between nodes. A master node may assign a workload to a worker node. The worker node sends a request to access target data. The request may be associated with a generation identifier and workload identifier that identifies the node and workload. At some point, a failure occurs between the master node and worker node. The master node reassigns the workload to another worker node. The new worker node accesses the target data with a different generation identifier, indicating to the storage system that the workload has been reassigned. The old worker node receives an indication from the storage system that the workload has been reassigned and stops processing the workload.

Abstract: Some storage systems are configured with VDL (valid data length) type controls that are implemented on a per cluster basis and, in some instances, on a sub-cluster basis, rather than simply a per file basis. In some instances, per-cluster VDL metadata for the storage clusters is stored and referenced at the edge data volume nodes of a distributed network for the storage system rather than, and/or without, storing or synchronizing the per-cluster VDL metadata at a master node that manages the corresponding storage clusters for the different data volume nodes. Sequence controls are also provided and managed by the master node and synchronized with the edge data volume nodes to further control access to data contained in the storage clusters.

Abstract: Some storage systems are configured with VDL (valid data length) type controls that are implemented on a per cluster basis and, in some instances, on a sub-cluster basis, rather than simply a per file basis. In some instances, per-cluster VDL metadata for the storage clusters is stored and referenced at the edge data volume nodes of a distributed network for the storage system rather than, and/or without, storing or synchronizing the per-cluster VDL metadata at a master node that manages the corresponding storage clusters for the different data volume nodes. Sequence controls are also provided and managed by the master node and synchronized with the edge data volume nodes to further control access to data contained in the storage clusters.

Abstract: Examples described herein generally relate to maintaining a snapshot of a multitier distributed file system. A snapshot directory volume of a directory volume can be created at least in part by copying, to the snapshot directory volume, metadata of the directory volume. Based on a file access request, it can be determined whether the file has not been snapshotted. If not, a snapshot file record of the file, and associated with the snapshot directory volume.

Abstract: Some storage systems are configured with VDL (valid data length) type controls that are implemented on a per cluster basis and, in some instances, on a sub-cluster basis, rather than simply a per file basis. In some instances, per-cluster VDL metadata for the storage clusters is stored and referenced at the edge data volume nodes of a distributed network for the storage system rather than, and/or without, storing or synchronizing the per-cluster VDL metadata at a master node that manages the corresponding storage clusters for the different data volume nodes. Sequence controls are also provided and managed by the master node and synchronized with the edge data volume nodes to further control access to data contained in the storage clusters.

Abstract: Some storage systems are configured with VDL (valid data length) type controls that are implemented on a per cluster basis and, in some instances, on a sub-cluster basis, rather than simply a per file basis. In some instances, per-cluster VDL metadata for the storage clusters is stored and referenced at the edge data volume nodes of a distributed network for the storage system rather than, and/or without, storing or synchronizing the per-cluster VDL metadata at a master node that manages the corresponding storage clusters for the different data volume nodes. Sequence controls are also provided and managed by the master node and synchronized with the edge data volume nodes to further control access to data contained in the storage clusters.

Abstract: Examples described herein generally relate to a scalable multi-tier storage system. An entry may be added and/or deleted within the storage system. To delete an entry, the storage system may determine whether the entry corresponds to the file or the directory based on directory metadata, and request deletion of the directory metadata associated with the entry from the directory volume based on determining that the entry corresponds to the directory, and further requesting deletion of the file from a file volume based on a determination that the entry corresponds to the file. To add a file, the storage system may generate directory metadata associated with the entry in the directory volume based on a determination that the entry corresponds to the directory, and may further allocate file metadata in the file volume based on a determination that the entry corresponds to the file.

Abstract: A generation identifier is employed with various systems and methods in order to identify situations where a workload has been reassigned to a new node and where a workload is still being processed by an old node during a failure between nodes. A master node may assign a workload to a worker node. The worker node sends a request to access target data. The request may be associated with a generation identifier and workload identifier that identifies the node and workload. At some point, a failure occurs between the master node and worker node. The master node reassigns the workload to another worker node. The new worker node accesses the target data with a different generation identifier, indicating to the storage system that the workload has been reassigned. The old worker node receives an indication from the storage system that the workload has been reassigned and stops processing the workload.

Abstract: Examples described herein generally relate to maintaining a snapshot of a multitier distributed file system. A snapshot directory volume of a directory volume can be created at least in part by copying, to the snapshot directory volume, metadata of the directory volume. Based on a file access request, it can be determined whether the file has not been snapshotted. If not, a snapshot file record of the file, and associated with the snapshot directory volume.

Abstract: Examples described herein generally relate to a scalable multi-tier storage system. An entry may be added and/or deleted within the storage system. To delete an entry, the storage system may determine whether the entry corresponds to the file or the directory based on directory metadata, and request deletion of the directory metadata associated with the entry from the directory volume based on determining that the entry corresponds to the directory, and further requesting deletion of the file from a file volume based on a determination that the entry corresponds to the file. To add a file, the storage system may generate directory metadata associated with the entry in the directory volume based on a determination that the entry corresponds to the directory, and may further allocate file metadata in the file volume based on a determination that the entry corresponds to the file.

Abstract: A generation identifier is employed with various systems and methods in order to identify situations where a workload has been reassigned to a new node and where a workload is still being processed by an old node during a failure between nodes. A master node may assign a workload to a worker node. The worker node sends a request to access target data. The request may be associated with a generation identifier and workload identifier that identifies the node and workload. At some point, a failure occurs between the master node and worker node. The master node reassigns the workload to another worker node. The new worker node accesses the target data with a different generation identifier, indicating to the storage system that the workload has been reassigned. The old worker node receives an indication from the storage system that the workload has been reassigned and stops processing the workload.

Abstract: Examples described herein generally relate to maintaining a snapshot of a multitier distributed file system. A snapshot directory volume of a directory volume can be created at least in part by copying, to the snapshot directory volume, metadata of the directory volume. Based on a file access request, it can be determined whether the file has not been snapshotted. If not, a snapshot file record of the file, and associated with the snapshot directory volume.

Abstract: Some storage systems are configured with VDL (valid data length) type controls that are implemented on a per cluster basis and, in some instances, on a sub-cluster basis, rather than simply a per file basis. In some instances, per-cluster VDL metadata for the storage clusters is stored and referenced at the edge data volume nodes of a distributed network for the storage system rather than, and/or without, storing or synchronizing the per-cluster VDL metadata at a master node that manages the corresponding storage clusters for the different data volume nodes. Sequence controls are also provided and managed by the master node and synchronized with the edge data volume nodes to further control access to data contained in the storage clusters.

Abstract: Examples described herein generally relate to maintaining a snapshot of a multitier distributed file system. A snapshot directory volume of a directory volume can be created at least in part by copying, to the snapshot directory volume, metadata of the directory volume. Based on a file access request, it can be determined whether the file has not been snapshotted. If not, a snapshot file record of the file, and associated with the snapshot directory volume.

Abstract: A system is configured to use a de-randomizer and budget data structure to economize I/O operations for a shared storage device while still allowing access to the device to a number of different entities. Embodiments can identify a comparatively low cost next operation as compared to other I/O operations, including a cost for seek time, for a first entity to dispatch to the storage device when the first entity has sufficient budget to have the I/O operation performed on its behalf and to identify an I/O operation for a second entity to dispatch to the storage device when there is insufficient budget for the first entity.

Abstract: Systems and methods for live migration of VMs between different nodes are provided. The system and methods utilize a heat map and an access pattern to determine the fetching order of files and/or blocks from the cache of a host node for rebuilding the cache on a destination node. The systems and methods are able to persist the cache of a host node across a crash, allowing a destination node to fetch blocks and/or files from the cache on the host node after a crash instead of having to access the VM files and/or blocks from an original source. Accordingly, the systems and methods decrease migration latency of the VM and free up network bandwidth during the live migration when compared to prior systems and methods that utilize just the heat map to determine the fetching order and/or that are not able to persist the cache across a crash.

Abstract: A request identifier is employed with various systems and methods in order to enable an application to pause execution during a failure and subsequently resume execution and communication with a storage system once the failure is resolved. An application residing on a node may attempt to access target data on a storage system. The application sends a request to access the target data. The request may be associated with a request identifier that identifies the application. When the storage system experiences a failure, the application receives a failure notification. The application pauses execution as a result of the notification. When the node reestablishes a connection to the storage system, the application resumes execution using the same request identifier. If the resume request is granted, the application is guaranteed that the target data was not modified during the failure.

Abstract: Examples described herein generally relate to a scalable multi-tier storage system. An entry may be added and/or deleted within the storage system. To delete an entry, the storage system may determine whether the entry corresponds to the file or the directory based on directory metadata, and request deletion of the directory metadata associated with the entry from the directory volume based on determining that the entry corresponds to the directory, and further requesting deletion of the file from a file volume based on a determination that the entry corresponds to the file. To add a file, the storage system may generate directory metadata associated with the entry in the directory volume based on a determination that the entry corresponds to the directory, and may further allocate file metadata in the file volume based on a determination that the entry corresponds to the file.