Abstract: A method for managing data storage using a distributed file system. A file system volume associated with a write request received at a data management subsystem is identified. A logical block device associated with the file system volume is identified. A plurality of data blocks is formed based on the write request. The plurality of data blocks is distributed across a plurality of node block stores in a distributed block layer of a storage management subsystem of the distributed file system. Each of the plurality of node block stores corresponds to a different node of a plurality of nodes in the distributed storage system. The storage management subsystem operates separately from but in communication with the data management subsystem.

Abstract: Techniques are provided for providing a storage abstraction layer for a composite aggregate architecture. A storage abstraction layer is utilized as an indirection layer between a file system and a storage environment. The storage abstraction layer obtains characteristic of a plurality of storage providers that provide access to heterogeneous types of storage of the storage environment (e.g., solid state storage, high availability storage, object storage, hard disk drive storage, etc.). The storage abstraction layer generates storage bins to manage storage of each storage provider. The storage abstraction layer generates a storage aggregate from the heterogeneous types of storage as a single storage container. The storage aggregate is exposed to the file system as the single storage container that abstracts away from the file system the management and physical storage details of data of the storage aggregate.

Abstract: A method for managing data storage using a distributed file system. A file system volume associated with a write request received at a data management subsystem is identified. A logical block device associated with the file system volume is identified. A plurality of data blocks is formed based on the write request. The plurality of data blocks is distributed across a plurality of node block stores in a distributed block layer of a storage management subsystem of the distributed file system. Each of the plurality of node block stores corresponds to a different node of a plurality of nodes in the distributed storage system. The storage management subsystem operates separately from but in communication with the data management subsystem.

Abstract: Systems and methods for managing data storage using a distributed file system are provided. In one example, a file system instance is deployed virtually in a node of a distributed storage system. The file system instance has a dynamic configuration including a set of services corresponding to a cluster management subsystem and a storage management subsystem. The storage management subsystem operates independently of a data management subsystem of the distributed storage system as a result of disaggregation from the data management subsystem. The data management subsystem performs storage and block management functions based on requests received from an application layer. An additional service corresponding to either the data management subsystem or the storage management subsystem is deployed virtually to meet the demand for the additional service in response to determining the presence of a demand for the additional service and availability a set of resources corresponding to the additional service.

Abstract: In various examples, data storage is managed using a distributed storage management system that is resilient. Data blocks of a logical block device may be distributed across multiple nodes in a cluster. The logical block device may correspond to a file system volume associated with a file system instance deployed on a selected node within a distributed block layer of a distributed file system. Each data block may have a location in the cluster identified by a block identifier associated with each data block. Each data block may be replicated on at least one other node in the cluster. A metadata object corresponding to a logical block device that maps to the file system volume may be replicated on at least another node in the cluster. Each data block and the metadata object may be hosted on virtualized storage that is protected using redundant array independent disks (RAID).

Abstract: Systems and methods for managing data storage using a distributed file system are provided. In one example, a relocation event is detected that indicates a relocation is to be initialized. The relocation is initialized by identifying a destination node of a distributed storage system for the relocation of a set of objects in a cluster database, including a logical block device, a corresponding logical aggregate, and a corresponding file system volume. A state of each of the set of objects is changed to offline. The set of objects are then relocated from an originating node of the distributed storage system to the destination node in which the corresponding logical aggregate is relocated after the logical block device and the corresponding file system volume is relocated after the logical aggregate. Finally, the state of each of the set of objects is changed to online.

Abstract: Failover methods and systems for a storage environment are provided. During a takeover operation to take over storage of a first storage system node by a second storage system node, the second storage system node copies information from a first storage location to a second storage location. The first storage location points to an active file system of the first storage system node, and the second storage location is assigned to the second storage system node for the takeover operation. The second storage system node quarantines storage space likely to be used by the first storage system node for a write operation, while the second storage system node attempts to take over the storage of the first storage system node. The second storage system node utilizes information stored at the second storage location during the takeover operation to give back control of the storage to the first storage system node.

Abstract: Failover methods and systems for a storage environment are provided. During a takeover operation to take over storage of a first storage system node by a second storage system node, the second storage system node copies information from a first storage location to a second storage location. The first storage location points to an active file system of the first storage system node, and the second storage location is assigned to the second storage system node for the takeover operation. The second storage system node quarantines storage space likely to be used by the first storage system node for a write operation, while the second storage system node attempts to take over the storage of the first storage system node. The second storage system node utilizes information stored at the second storage location during the takeover operation to give back control of the storage to the first storage system node.

Abstract: A method, non-transitory computer readable medium, and device that assists with reducing memory fragmentation in solid state devices includes identifying an allocation area within an address range to write data from a cache. Next, the identified allocation area is determined for including previously stored data. The previously stored data is read from the identified allocation area when it is determined that the identified allocation area comprises previously stored data. Next, both the write data from the cache and the read previously stored data are written back into the identified allocation area sequentially through the address range.

Abstract: A method for managing data storage using a distributed file system. A file system volume associated with a write request received at a data management subsystem is identified. A logical block device associated with the file system volume is identified. A plurality of data blocks is formed based on the write request. The plurality of data blocks is distributed across a plurality of node block stores in a distributed block layer of a storage management subsystem of the distributed file system. Each of the plurality of node block stores corresponds to a different node of a plurality of nodes in the distributed storage system. The storage management subsystem operates separately from but in communication with the data management subsystem.

Abstract: A distributed storage management system comprising nodes that form a cluster, a distributed block layer that spans the nodes in the cluster, and file system instances deployed on the nodes. Each file system instance comprises a data management subsystem and a storage management subsystem disaggregated from the data management subsystem. The storage management subsystem comprises a node block store that forms a portion of the distributed block layer and a storage manager that manages a key-value store and virtualized storage supporting the node block store. A file system volume hosted by the data management subsystem maps to a logical block device hosted by the virtualized storage in the storage management subsystem. The key-value store includes, for a data block of the logical block device, a key that comprises a block identifier for the logical block device and a value that comprises the data block.

Abstract: Techniques are provided for selectively storing data into allocation areas using streams. A set of allocation areas (e.g., ranges of block numbers such as virtual block numbers) are defined for a storage device. Data having particular characteristics (e.g., user data, metadata, hot data, cold data, randomly accessed data, sequentially accessed data, etc.) will be sent to the storage device for selective storage in corresponding allocation areas. For example, when a file system receives a write stream of hot data, the hot data may be assigned to a stream. The stream will be tagged using a stream identifier that is used as an indicator to the storage device to process data of the stream using an allocation area defined for hot data. In this way, data having different characteristics will be stored/confined within particular allocation areas of the storage device to reduce fragmentation and write amplification.

Abstract: Failover methods and systems for a storage environment are provided. During a takeover operation to take over storage of a first storage system node by a second storage system node, the second storage system node copies information from a first storage location to a second storage location. The first storage location points to an active file system of the first storage system node, and the second storage location is assigned to the second storage system node for the takeover operation. The second storage system node quarantines storage space likely to be used by the first storage system node for a write operation, while the second storage system node attempts to take over the storage of the first storage system node. The second storage system node utilizes information stored at the second storage location during the takeover operation to give back control of the storage to the first storage system node.

Abstract: A method, non-transitory computer readable medium, and device that assists with reducing memory fragmentation in solid state devices includes identifying an allocation area within an address range to write data from a cache. Next, the identified allocation area is determined for including previously stored data. The previously stored data is read from the identified allocation area when it is determined that the identified allocation area comprises previously stored data. Next, both the write data from the cache and the read previously stored data are written back into the identified allocation area sequentially through the address range.

Abstract: Techniques are provided for providing a storage abstraction layer for a composite aggregate architecture. A storage abstraction layer is utilized as an indirection layer between a file system and a storage environment. The storage abstraction layer obtains characteristic of a plurality of storage providers that provide access to heterogeneous types of storage of the storage environment (e.g., solid state storage, high availability storage, object storage, hard disk drive storage, etc.). The storage abstraction layer generates storage bins to manage storage of each storage provider. The storage abstraction layer generates a storage aggregate from the heterogeneous types of storage as a single storage container. The storage aggregate is exposed to the file system as the single storage container that abstracts away from the file system the management and physical storage details of data of the storage aggregate.

Abstract: Failover methods and systems for a storage environment are provided. During a takeover operation to take over storage of a first storage system node by a second storage system node, the second storage system node copies information from a first storage location to a second storage location. The first storage location points to an active file system of the first storage system node, and the second storage location is assigned to the second storage system node for the takeover operation. The second storage system node quarantines storage space likely to be used by the first storage system node for a write operation, while the second storage system node attempts to take over the storage of the first storage system node. The second storage system node utilizes information stored at the second storage location during the takeover operation to give back control of the storage to the first storage system node.

Abstract: Techniques are provided for providing a storage abstraction layer for a composite aggregate architecture. A storage abstraction layer is utilized as an indirection layer between a file system and a storage environment. The storage abstraction layer obtains characteristic of a plurality of storage providers that provide access to heterogeneous types of storage of the storage environment (e.g., solid state storage, high availability storage, object storage, hard disk drive storage, etc.). The storage abstraction layer generates storage bins to manage storage of each storage provider. The storage abstraction layer generates a storage aggregate from the heterogeneous types of storage as a single storage container. The storage aggregate is exposed to the file system as the single storage container that abstracts away from the file system the management and physical storage details of data of the storage aggregate.

Abstract: A method, non-transitory computer readable medium, and device that assists with reducing memory fragmentation in solid state devices includes identifying an allocation area within an address range to write data from a cache. Next, the identified allocation area is determined for including previously stored data. The previously stored data is read from the identified allocation area when it is determined that the identified allocation area comprises previously stored data. Next, both the write data from the cache and the read previously stored data are written back into the identified allocation area sequentially through the address range.

Abstract: Techniques are provided for selectively storing data into allocation areas using streams. A set of allocation areas (e.g., ranges of block numbers such as virtual block numbers) are defined for a storage device. Data having particular characteristics (e.g., user data, metadata, hot data, cold data, randomly accessed data, sequentially accessed data, etc.) will be sent to the storage device for selective storage in corresponding allocation areas. For example, when a file system receives a write stream of hot data, the hot data may be assigned to a stream. The stream will be tagged using a stream identifier that is used as an indicator to the storage device to process data of the stream using an allocation area defined for hot data. In this way, data having different characteristics will be stored/confined within particular allocation areas of the storage device to reduce fragmentation and write amplification.

Abstract: Techniques are provided for selectively storing data into allocation areas using streams. A set of allocation areas (e.g., ranges of block numbers such as virtual block numbers) are defined for a storage device. Data having particular characteristics (e.g., user data, metadata, hot data, cold data, randomly accessed data, sequentially accessed data, etc.) will be sent to the storage device for selective storage in corresponding allocation areas. For example, when a file system receives a write stream of hot data, the hot data may be assigned to a stream. The stream will be tagged using a stream identifier that is used as an indicator to the storage device to process data of the stream using an allocation area defined for hot data. In this way, data having different characteristics will be stored/confined within particular allocation areas of the storage device to reduce fragmentation and write amplification.