Abstract: Techniques are provided for transitioning from a hierarchical file system to an object store. A request is received to change file metadata of a hierarchical file system. In response to the request: in-cache file metadata of the hierarchical file system is modified. The in-cache file metadata includes a directory structure of the hierarchical file system and file metadata of the hierarchical file system. Additionally, an in-store metadata journal entry is generated. The in-store metadata journal entry indicates an update to in-store file metadata. The in-store file metadata includes the directory structure of the hierarchical file system and file metadata of the hierarchical file system and is stored in an object store. The in-store metadata journal entry is stored in the object store. Furthermore, in-store file metadata and in-store metadata journal entry may be used to reconstruct the in-cache file metadata on another node.

Abstract: A system and method for efficiently managing data through compression interfaces may include receiving, by a controller, data, generating, by the controller, a compressed payload based on the data, generating, by the controller, metadata describing the compressed payload, the metadata including fixed size metadata and variable size metadata, generating, by the controller, a data container comprising the uncompressed payload and the metadata, and transmitting, by the controller, the data container to an application. Some implementations of the system may include a storage media, and a storage controller executable by a processor that may include an interface processor, a controller logic, and a media processor configured to communicate with an application and the storage media to perform aspects of the method.

Abstract: Various aspects for data deduplication in a storage system are provided. For instance, a storage controller may perform operations including receiving a data chunk including a set of data blocks, determining a signature for the data chunk, and comparing the signature and a set of reference signatures to determine a match. Responsive to a match, the operations may further include identifying a reference data chunk including a set of comparison blocks associated with the matched reference signature, performing a deduplication technique on the set of data blocks based on the set of comparison blocks, and identifying a subsequent reference data chunk for an estimated next data chunk based on identification of the reference data chunk and prior to receipt of the next data chunk.

Abstract: Techniques are provided for transitioning from a hierarchical file system to an object store. A request is received to change file metadata of a hierarchical file system. In response to the request: in-cache file metadata of the hierarchical file system is modified. The in-cache file metadata includes a directory structure of the hierarchical file system and file metadata of the hierarchical file system. Additionally, an in-store metadata journal entry is generated. The in-store metadata journal entry indicates an update to in-store file metadata. The in-store file metadata includes the directory structure of the hierarchical file system and file metadata of the hierarchical file system and is stored in an object store. The in-store metadata journal entry is stored in the object store. Furthermore, in-store file metadata and in-store metadata journal entry may be used to reconstruct the in-cache file metadata on another node.

Abstract: A data storage apparatus that includes a storage device and a processor coupled to the storage device. The processor is configured to receive a read request for a first translation table entry associated with a logical block, identify a dump unit associated with the logical block using a hash function, determine a dump group associated with the dump unit, and identify a second translation table entry associated with the dump unit.

Abstract: A shared storage architecture persistently stores database files in non-volatile memories (NVMs) of a plurality of computing nodes of a multi-node DBMS. The computing nodes of the multi-node DBMS store data blocks in NVM and each computing node of the DBMS stores copies of each data block stored on the plurality of computing nodes. A computing node that disconnects and subsequently rejoins the DBMS employs an on-demand approach to resilvering stale data blocks that have been updated in other computing nodes in the DBMS while the computing node was offline. A data block may be resilvered on-demand based on an I/O request for a specific data block from a workload running on the reconnected computing node. Stale data blocks on the reconnected computing node are not resilvered unless they are accessed by the workload.

Abstract: A system and method for efficiently managing data through compression interfaces may include receiving, by a controller, data, generating, by the controller, a compressed payload based on the data, generating, by the controller, metadata describing the compressed payload, the metadata including fixed size metadata and variable size metadata, generating, by the controller, a data container comprising the uncompressed payload and the metadata, and transmitting, by the controller, the data container to an application. Some implementations of the system may include a storage media, and a storage controller executable by a processor that may include an interface processor, a controller logic, and a media processor configured to communicate with an application and the storage media to perform aspects of the method.

Abstract: A system and method for efficiently managing data through compression interfaces may include receiving, by a controller, data, generating, by the controller, a compressed payload based on the data, generating, by the controller, metadata describing the compressed payload, the metadata including fixed size metadata and variable size metadata, generating, by the controller, a data container comprising the uncompressed payload and the metadata, and transmitting, by the controller, the data container to an application. Some implementations of the system may include a storage media, and a storage controller executable by a processor that may include an interface processor, a controller logic, and a media processor configured to communicate with an application and the storage media to perform aspects of the method.

Abstract: A shared storage architecture persistently stores database files in non-volatile memories (NVMs) of a plurality of computing nodes of a multi-node DBMS. The computing nodes of the multi-node DBMS store data blocks in NVM and each computing node of the DBMS stores copies of each data block stored on the plurality of computing nodes. A computing node that disconnects and subsequently rejoins the DBMS employs an on-demand approach to resilvering stale data blocks that have been updated in other computing nodes in the DBMS while the computing node was offline. A data block may be resilvered on-demand based on an I/O request for a specific data block from a workload running on the reconnected computing node. Stale data blocks on the reconnected computing node are not resilvered unless they are accessed by the workload.

Abstract: Various aspects for managing data blocks in a storage system are provided. For instance, a method may include storing, in a buffer memory, a plurality of comparison blocks, initiating a data deduplication process utilizing the plurality of comparison blocks, and performing garbage collection in conjunction with the data deduplication process. Garbage collection may include maintaining a hit count for comparison blocks of a passive set of comparison blocks in the buffer memory and deleting the passive set from the buffer memory when the hit count is decremented to a predetermined value. The hit count may be incremented and decremented based on utilization of a comparison block in the data deduplication process.

Abstract: A system comprising a processor and a memory storing instructions that, when executed, cause the system to determine a first value of a first checkpoint associated with a first snapshot, receive a second value of a second checkpoint associated with a translation table entry from an additional source, determine whether the second value of the second checkpoint is after the first value of the first checkpoint, in response to determining that the second value of the second checkpoint is after the first value of the first checkpoint, retrieve the translation table entry associated with the second checkpoint from the additional source, and reconstruct the translation table using the translation table entry associated with the second checkpoint.

Abstract: Various aspects for managing data blocks in a storage system are provided. For instance, a method may include storing, in a buffer memory device, a comparison block library, selecting a first set of comparison blocks in the comparison block library to create an active set of comparison blocks, and utilizing the active set in conjunction with a data deduplication technique. A method may further include determining an occurrence of a predetermined event in the data deduplication technique, selecting a second set of comparison blocks in the comparison block library to create a new active set of comparison blocks in response to the predetermined event, replacing the active set with the new active set, and utilizing the new active set in conjunction with the data deduplication technique.

Abstract: A system comprising a processor and a memory storing instructions that, when executed, cause the system to receive a data stream including one or more data blocks; determine a size of the one or more data blocks; determine a number of mappings needed for a physical block based on the size of a data block and a size of the physical block, the number of mappings being variable for different physical blocks depending on the size of the one or more data blocks storing in the physical block; retrieve a dynamically sized reverse map, the dynamically sized reverse map being a dynamic tree structure; determine a starting location in the dynamically sized reverse map for mappings of the one or more data blocks; and create an entry for the physical block in the dynamically sized reverse map.

Abstract: A system comprising a processor and a memory storing instructions that, when executed, cause the system to receive a first translation table entry for a logical block, map the first translation table entry to a first dump unit, the first dump unit included in an array of dump units, identify a second translation table entry for the logical block in the first dump unit, the second translation table entry also being stored in a storage device, and generate a linked list in the storage device from the second translation table entry associated with the first dump unit, the linked list identifying previous translation table entries associated with the logical block.

Abstract: Various aspects for managing data blocks in a storage system are provided. For instance, a method may include storing, in a buffer memory, a plurality of comparison blocks, initiating a data deduplication process utilizing the plurality of comparison blocks, and performing garbage collection in conjunction with the data deduplication process. Garbage collection may include maintaining a hit count for comparison blocks of a passive set of comparison blocks in the buffer memory and deleting the passive set from the buffer memory when the hit count is decremented to a predetermined value. The hit count may be incremented and decremented based on utilization of a comparison block in the data deduplication process.

Abstract: Various aspects for data deduplication in a storage system are provided. For instance, a storage controller may perform operations including receiving a data chunk including a set of data blocks, determining a signature for the data chunk, and comparing the signature and a set of reference signatures to determine a match. Responsive to a match, the operations may further include identifying a reference data chunk including a set of comparison blocks associated with the matched reference signature, performing a deduplication technique on the set of data blocks based on the set of comparison blocks, and identifying a subsequent reference data chunk for an estimated next data chunk based on identification of the reference data chunk and prior to receipt of the next data chunk.

Abstract: A data storage apparatus that includes a storage device and a processor coupled to the storage device. The processor is configured to receive a read request for a first translation table entry associated with a logical block, identify a dump unit associated with the logical block using a hash function, determine a dump group associated with the dump unit, and identify a second translation table entry associated with the dump unit.

Abstract: Various aspects for managing data blocks in a storage system are provided. For instance, a method may include storing, in a buffer memory device, a comparison block library, selecting a first set of comparison blocks in the comparison block library to create an active set of comparison blocks, and utilizing the active set in conjunction with a data deduplication technique. A method may further include determining an occurrence of a predetermined event in the data deduplication technique, selecting a second set of comparison blocks in the comparison block library to create a new active set of comparison blocks in response to the predetermined event, replacing the active set with the new active set, and utilizing the new active set in conjunction with the data deduplication technique.

Abstract: A system comprising a processor and a memory storing instructions that, when executed, cause the system to receive a data stream including one or more data blocks; determine a size of the one or more data blocks; determine a number of mappings needed for a physical block based on the size of a data block and a size of the physical block, the number of mappings being variable for different physical blocks depending on the size of the one or more data blocks storing in the physical block; retrieve a dynamically sized reverse map, the dynamically sized reverse map being a dynamic tree structure; determine a starting location in the dynamically sized reverse map for mappings of the one or more data blocks; and create an entry for the physical block in the dynamically sized reverse map.

Abstract: A system comprising a processor and a memory storing instructions that, when executed, cause the system to receive a data stream including one or more data blocks; determine a size of the one or more data blocks; determine a number of mappings needed for a physical block based on the size of a data block and a size of the physical block, the number of mappings being variable for different physical blocks depending on the size of the one or more data blocks storing in the physical block; retrieve a dynamically sized reverse map, the dynamically sized reverse map being a dynamic tree structure; determine a starting location in the dynamically sized reverse map for mappings of the one or more data blocks; and create an entry for the physical block in the dynamically sized reverse map.