Abstract: Techniques are provided for block allocation for persistent memory during aggregate transition. In a high availability pair including first and second nodes, the first node makes a determination that control of a first aggregate is to transition from the first node to the second node. A portion of available free storage space is allocated from a first persistent memory of the first node as allocated pages within the first persistent memory. Metadata information for the allocated pages is updated with an identifier of the first aggregate to create updated metadata information reserving the allocated pages for the first aggregate. The updated metadata information is mirrored to the second node, so that the second node also reserves those pages. Control of the first aggregate is transitioned to the second node. As a result, the nodes do not attempt allocating the same free pages to different aggregates during a transition.

Abstract: Techniques are provided for providing a storage abstraction layer for a composite aggregate architecture. A storage abstraction layer is utilized as an indirection layer between a file system and a storage environment. The storage abstraction layer obtains characteristic of a plurality of storage providers that provide access to heterogeneous types of storage of the storage environment (e.g., solid state storage, high availability storage, object storage, hard disk drive storage, etc.). The storage abstraction layer generates storage bins to manage storage of each storage provider. The storage abstraction layer generates a storage aggregate from the heterogeneous types of storage as a single storage container. The storage aggregate is exposed to the file system as the single storage container that abstracts away from the file system the management and physical storage details of data of the storage aggregate.

Abstract: Techniques are provided for implementing write ordering for persistent memory. A set of actions are identified for commitment to persistent memory of a node for executing an operation upon the persistent memory. An episode is created to comprise a first subset of actions of the set of actions that can be committed to the persistent memory in any order with respect to one another such that a consistent state of the persistent memory can be reconstructed in the event of a crash of the node during execution of the operation. The first subset of actions within the episode are committed to the persistent memory and further execution of the operation is blocked until the episode completes.

Abstract: Failover methods and systems for a networked storage environment are provided. A filtering data structure and a metadata data structure are generated before starting a replay of a log stored in a non-volatile memory of a second storage node, during a failover operation initiated in response to a failure at a first storage node. The second storage node operates as a partner node of the first storage node to mirror at the log one or more write requests received by the first storage node prior to the failure, and data associated with the one or more write requests. The filtering data structure identifies each log entry and the metadata structure stores a metadata attribute of each log entry. The filtering data structure and the metadata structure are used for providing access to a logical storage object during the log replay from the second storage node.

Abstract: Failover methods and systems for a networked storage environment are provided. In one aspect, a read request associated with a first storage object is received, during a replay of entries of a log stored in a non-volatile memory of a second storage node for a failover operation initiated in response to a failure at a first storage node. The second storage node operates as a partner node of the first storage node. The read request is processed using a filtering data structure that is generated from the log prior to the replay and identifies each log entry. The read request is processed when the log does not have an entry associated with the read request, and when the filtering data structure includes an entry associated with the read request, the requested data is located at the non-volatile memory.

Abstract: Failover methods and systems for a networked storage environment are provided. A metadata data structure is generated, before starting a replay of entries at a log stored in a non-volatile memory of a second storage node, during a failover operation initiated in response to a failure at a first storage node. The second storage node operates as a partner node of the first storage node, and the metadata structure stores a metadata attribute of each log entry. Furthermore, the metadata attribute of each log entry is persistently stored. The persistently stored metadata attribute is used to respond to a read request received during the replay by the second storage node, while a write request metadata attribute of a write request is used for executing the write request received by the second storage node during the replay.

Abstract: Techniques are provided for utilizing a log to free pages from persistent memory. A log is maintained to comprise a list of page block numbers of pages within persistent memory of a node to free. A page block number, of a page, within the log is identified for processing. A reference count, corresponding to a number of references to the page block number, is identified. In response to the reference count being greater than 1, the reference count is decremented and the page block number is removed from the log. In response to the reference count being 1, the page is freed from the persistent memory and the page block number is removed from the log.

Abstract: Techniques are provided for implementing a persistent memory storage tier to manage persistent memory of a node. The persistent memory is managed by the persistent memory storage tier at a higher level within a storage operating system storage stack than a level at which a storage file system of the node is managed. The persistent memory storage tier intercepts an operation targeting the storage file system. The persistent memory storage tier retargets the operation from targeting the storage file system to targeting the persistent memory. The operation is transmitted to the persistent memory.

Abstract: Techniques are provided for implementing write ordering for persistent memory. A set of actions are identified for commitment to persistent memory of a node for executing an operation upon the persistent memory. An episode is created to comprise a first subset of actions of the set of actions that can be committed to the persistent memory in any order with respect to one another such that a consistent state of the persistent memory can be reconstructed in the event of a crash of the node during execution of the operation. The first subset of actions within the episode are committed to the persistent memory and further execution of the operation is blocked until the episode completes.

Abstract: Techniques are provided for implementing a file system format for persistent memory. A node, comprising persistent memory, receives an operation comprising a file identifier and file system instance information. A list of file system info objects are evaluated to identify a file system info object matching the file system instance information. An inofile, identified by the file system info object as being associated with inodes of files within an instance of the file system targeted by the operation, is traversed to identify an inode matching the file identifier. If the inode comprises an indicator that the file is tiered into the persistent memory, then the inode it utilized to facilitate execution of the operation upon the persistent memory. Otherwise, the operation is routed to a storage file system tier for execution by a storage file system upon storage associated with the node.

Abstract: Techniques are provided for atomic writes for persistent memory. In response to receiving a write operation, a new per-page structure with a new page block number is allocated. New data of the write operation is persisted to a new page of the persistent memory having the new page block number, and the new per-page structure is persisted to the persistent memory. If the write operation targets a hole after the new data and the new per-page structure have been persisted, then a new per-page structure identifier of the new per-page structure is inserted into a parent indirect page of a page comprising the new data. If the write operation targets old data after the new data and the new per-page structure have been persisted, then an old per-page structure of the old data is updated with the new page block number.

Abstract: Techniques are provided for implementing a file system format for persistent memory. A node, with persistent memory, receives an operation associated with a file identifier and file system instance information. A list of file system info objects are evaluated to identify a file system info object matching the file system instance information. An inofile, identified by the file system info object as being associated with inodes of files within an instance of the file system targeted by the operation, is traversed to identify an inode matching the file identifier. If the inode has an indicator that the file is tiered into the persistent memory, then the inode it utilized to facilitate execution of the operation upon the persistent memory. Otherwise, the operation is routed to a storage file system tier for execution by a storage file system upon storage associated with the node.

Abstract: Methods, non-transitory computer readable media, and computing devices that execute a storage operation, without journaling the storage operation in a log, and withhold from a file system a list of freed inodes including an indication of an inode freed as a result of the execution of the storage operation. A consistency point operation is then initiated that retrieves storage operations logged as journal entries in the log and commits a result of each of the storage operations to data storage devices. A list of available inodes is updated based on the list of freed inodes, when the consistency point operation is determined to be complete. This technology reduces the number of storage operations that are required to be journaled to maintain consistency of a file system, thereby reducing the runtime resources required to facilitate the journaling and replay resource required to replay the storage operations following a recovery.

Abstract: Techniques are provided for remote object store error handling. A storage system may store data within one or more tiers of storage, such as a local storage tier (e.g., solid state storage and disks maintained by the storage system), a remote object store (e.g., storage provided by a third party storage provider), and/or other storage tiers. Because the remote object store may not provide the same data consistency and guarantees that the storage system provides for clients such as through the local storage tier, additional validation is provided by the storage system for the remote object store. For example, when data is put into an object of the remote object store, a verification get operation is performed to read and validate information within a header of the object. Other verifications and checks are performed such as using a locally stored metafile to detect corrupt or lost metadata and/or objects.

Abstract: Techniques are provided for compacting indirect blocks. For example, an object is represented as a structure comprising data blocks within which data of the object is stored and indirect blocks comprising block numbers of where the data blocks are located in storage. Block numbers within a set of indirect blocks are compacted into a compacted indirect block comprising a base block number, a count of additional block numbers after the base block number in the compacted indirect block, and a pattern of the block numbers in the compacted indirect block. The compacted indirect block is stored into memory for processing access operations to the object. Storing compacted indirect blocks into memory allows for more block numbers to be stored within memory. This improves the processing of access operations because reading the block numbers from memory is faster than loading the block numbers from disk.

Abstract: Techniques are provided for compacting indirect blocks. For example, an object is represented as a structure comprising data blocks within which data of the object is stored and indirect blocks comprising block numbers of where the data blocks are located in storage. Block numbers within a set of indirect blocks are compacted into a compacted indirect block comprising a base block number, a count of additional block numbers after the base block number in the compacted indirect block, and a pattern of the block numbers in the compacted indirect block. The compacted indirect block is stored into memory for processing access operations to the object. Storing compacted indirect blocks into memory allows for more block numbers to be stored within memory.

Abstract: Techniques are provided for remote object store error handling. A storage system may store data within one or more tiers of storage, such as a local storage tier (e.g., solid state storage and disks maintained by the storage system), a remote object store (e.g., storage provided by a third party storage provider), and/or other storage tiers. Because the remote object store may not provide the same data consistency and guarantees that the storage system provides for clients such as through the local storage tier, additional validation is provided by the storage system for the remote object store. For example, when data is put into an object of the remote object store, a verification get operation is performed to read and validate information within a header of the object. Other verifications and checks are performed such as using a locally stored metafile to detect corrupt or lost metadata and/or objects.

Abstract: Techniques are provided for compacting indirect blocks. For example, an object is represented as a structure comprising data blocks within which data of the object is stored and indirect blocks comprising block numbers of where the data blocks are located in storage. Block numbers within a set of indirect blocks are compacted into a compacted indirect block comprising a base block number, a count of additional block numbers after the base block number in the compacted indirect block, and a pattern of the block numbers in the compacted indirect block. The compacted indirect block is stored into memory for processing access operations to the object. Storing compacted indirect blocks into memory allows for more block numbers to be stored within memory. This improves the processing of access operations because reading the block numbers from memory is faster than loading the block numbers from disk.

Abstract: One or more techniques and/or systems are provided for incremental transfer with unused data block reclamation. For example, source volume of a source storage filer may have a physical replication relationship with a destination volume of a destination storage filer. Snapshots of a file system of the source volume may be used to identify new data blocks (e.g., data blocks allocated or reallocated to a new snapshot) and/or unused data blocks (e.g., data blocks of a deleted snapshot) used to incrementally update the destination volume so that the destination volume is a mirrored replication of the source volume. In this way, new data blocks may be updated and unused data blocks may be punched out (e.g., deallocated) of the destination volume during execution of an incremental update.

Abstract: Methods, non-transitory computer readable media, and computing devices that execute a storage operation, without journaling the storage operation in a log, and withhold from a file system a list of freed inodes including an indication of an inode freed as a result of the execution of the storage operation. A consistency point operation is then initiated that retrieves storage operations logged as journal entries in the log and commits a result of each of the storage operations to data storage devices. A list of available inodes is updated based on the list of freed inodes, when the consistency point operation is determined to be complete. This technology reduces the number of storage operations that are required to be journaled to maintain consistency of a file system, thereby reducing the runtime resources required to facilitate the journaling and replay resource required to replay the storage operations following a recovery.