Abstract: A system can maintain a group of data sets, a vector database that comprises respective first identifiers of respective data sets of the group of data sets stored on respective data chunks, respective second identifiers in respective chunk descriptors of the respective data chunks, wherein the respective second identifiers identify the respective first identifiers in the vector database. The system can, based on determining to adjust a characteristic of a data set, determine, from a second identifier of the second identifiers that is stored in a chunk descriptor of the chunk descriptors, that the vector database indicates that a data chunk of the data chunks identifies that the data chunk stores at least part of the data set, and adjust the characteristic of the data set as applicable to at least the part of the data set that is stored in the data chunk.

Abstract: A system can maintain a filesystem across nodes of a cluster. The system can, based on receiving a first request from a first node to ingest or write data to a location on a second node, issue, by a lock manager, a write lock on the location of the second node. The system can, based on the first node receiving the write lock, write, by the first node, the data to a write back cache of the first node while refraining from writing the data to the location on the second node. The system can, before the first node flushes the write back cache, comprising writing the data to the location of the second node, respond, by the lock manager, to a second request to read the location on the second node with a read lock that indicates reading the data in the write back cache on the first node.

Abstract: A system can store data in a filesystem as blocks. The system can select a similarity hash type for a first block of the blocks from a group of similarity hash types based on a file type associated with data of the first block. The system can perform a corresponding similarity hash on the first block according to the similarity hash type, to produce a first hash. The system can identify a second block of the blocks that satisfies a similarity criterion relative to the first block based on the first hash and respective second hashes in a hash database that stores respective virtual mapping pointers to respective blocks that have a same similarity hash type. The system can perform deduplication on the first block using a dictionary associated with the second block, and incrementing a reference count of the dictionary.

Abstract: A system can maintain a group of data sets, a vector database that comprises respective first identifiers of respective data sets of the group of data sets stored on respective data chunks, respective second identifiers in respective chunk descriptors of the respective data chunks, wherein the respective second identifiers identify the respective first identifiers in the vector database. The system can, based on determining to adjust a characteristic of a data set, determine, from a second identifier of the second identifiers that is stored in a chunk descriptor of the chunk descriptors, that the vector database indicates that a data chunk of the data chunks identifies that the data chunk stores at least part of the data set, and adjust the characteristic of the data set as applicable to at least the part of the data set that is stored in the data chunk.

Abstract: A system can maintain a log filesystem in which data is stored as respective data chunks, wherein the respective data chunks correspond to respective chunk descriptors, and wherein respective inode leaf blocks, which correspond to respective files that are stored in the respective data chunks, comprise respective first virtual pointers to respective virtual chunk extents that comprise respective second virtual pointer groups to the respective data chunks. The system can, for a data chunk of the respective data chunks, store, in a corresponding chunk descriptor, a maximum temperature value and a minimum temperature value of the data chunk. The system can adjust a data tiering level of at least part of the data of the data chunk based on at least one of the maximum temperature value or the minimum temperature value.

Abstract: A system can, based on storing data for a file in a chunk of a log filesystem, store a first virtual pointer and a second physical pointer in an inode leaf block that corresponds to the file. The system can, based on determining to read the data, fetch a chunk identifier that corresponds to the data, and that comprises an uber identifier of an uber and a first uber generation identifier. The system can, based on the first uber generation identifier matching a second uber generation identifier of a garbage collection reference counter of the uber that is accessed based on the uber identifier, use a result of the reading the data from the uber using the second physical pointer. The system can, based on the first uber generation identifier differing from the second uber generation identifier, use a result of reading the data using the first virtual pointer.

Abstract: The technologies described herein are generally directed toward using differential checkpoints to maintain a data store of file system metadata. For instance, a system can, based on a comparison of a first metadata state capture of a data storage system with a second metadata state capture, identify changed metadata of a data object. The system can further, based on the changed metadata, generate a metadata checkpoint object. Further, the system can receive an indication from a consumer process that, the metadata checkpoint object has been processed to generate a data modification command configured to be received by a record system node, and to change contents of a record system, and the data modification command has been processed by the record system node to alter the contents of the record system based on the data modification command.

Abstract: The technologies described herein are generally directed toward using differential checkpoints to search non-logical file system metadata. For instance, a system can, based on a comparison of a first metadata state capture of a data storage system with a second metadata state capture, identify changed metadata of a data object. The system can further, based on the changed metadata, generate a metadata checkpoint object. Further, the system can receive an indication from a consumer process that, the metadata checkpoint object has been processed to generate a data modification command configured to be received by a record system node, and to change contents of a record system, and the data modification command has been processed by the record system node to alter the contents of the record system based on the data modification command.

Abstract: A system can maintain a data storage system, wherein the data storage system comprises persistent memory and chunk storage, wherein the persistent memory is more performant than the chunk storage, wherein the chunk storage stores data in chunks of a fixed-size, and wherein respective chunks of the chunks correspond to respective parity values. The system can, based on receiving a request associated with a storage account to write data to the data storage system, write the data to the persistent memory concurrently with writing the data to the chunk storage. The system can, after completing writing the data to the persistent memory and before completing writing the data to the chunk storage, send an indication, to the storage client, that the data has been successfully stored in the data storage system.

Abstract: A system can maintain a data storage system, wherein the data storage system comprises persistent memory and chunk storage, wherein the persistent memory is more performant than the chunk storage, wherein the chunk storage stores data in chunks of a fixed-size, and wherein respective chunks of the chunks correspond to respective parity values. The system can, based on receiving a request associated with a storage account to write data to the data storage system, write the data to the persistent memory concurrently with writing the data to the chunk storage. The system can, after completing writing the data to the persistent memory and before completing writing the data to the chunk storage, send an indication, to the storage client, that the data has been successfully stored in the data storage system.

Abstract: The described technology is generally directed towards maintaining a heat score per file based on a file's access/activity level, and using the heat score to re-tier files from faster to slower storage and vice-versa. Files on a slower tier that have more activity are thereby moved (“up-tiered”) to a faster access storage tier, while the files that have less activity on the faster access storage tier are moved (“down-tiered”) to a slower access storage tier. The heat score for a file is increased based on more input-output operation activity, and decayed over time. The decaying can be based on a relative importance value (e.g., business value) assigned to the file or a domain that includes the file. Ordering the files based on their respective heat scores allows only a fraction of the files to be considered for re-tiering before the re-tiering moves are performed.

Abstract: A method for direct addressing in virtual addressed systems includes obtaining, by a system including a processor and in response to receiving a request to access a data item stored by a storage system, address information for the data item from a data structure. The address information includes a first physical storage address of the storage system, a first generation number associated the first physical storage address in the data structure, and an address redirector. The method also includes accessing, by the system and in response to the first generation number being determined to be different from a second generation number associated with the first physical storage address, the data item at a second physical storage address of the storage system instead of the first physical storage address, the second physical storage address being determined based on the address redirector.

Abstract: A method for direct addressing in virtual addressed systems includes obtaining, by a system including a processor and in response to receiving a request to access a data item stored by a storage system, address information for the data item from a data structure. The address information includes a first physical storage address of the storage system, a first generation number associated the first physical storage address in the data structure, and an address redirector. The method also includes accessing, by the system and in response to the first generation number being determined to be different from a second generation number associated with the first physical storage address, the data item at a second physical storage address of the storage system instead of the first physical storage address, the second physical storage address being determined based on the address redirector.

Abstract: A method for efficient ingest tier references via virtual addresses includes generating, by a system including a processor, an address redirector for a data block in response to the data block being written to a first storage location, the address redirector including a reference to the first storage location. The method also includes writing, by the system, data including the data block to a second storage location in response to an amount of the data to be written to the second storage location being determined to be at least a threshold amount. The method further includes altering, by the system in response to the writing of the data to the second storage location, the address redirector to include a reference to the second storage location.

Abstract: Data lock transitioning associated with writing of files can be performed and managed. In connection with a write operation to write a portion of a file to a node of a node network, an access management component (AMC) can analyze file-related information indicative of a status associated with the portion of the file, wherein the status can relate to whether the file portion is associated with a streaming write, whether a protocol indication indicates a file handle associated with the file has been closed, or whether an advisory lock has been released by a client device. The AMC can determine whether to transition from an exclusive data lock associated with the portion of the file to a shared data lock with respect to the portion of the file, based on a result of analyzing the file-related information. The AMC can fine tune determinations or predictions of data lock transitions.

Abstract: Data lock transitioning associated with writing of files can be performed and managed. In connection with a write operation to write a portion of a file to a node of a node network, an access management component (AMC) can analyze file-related information indicative of a status associated with the portion of the file, wherein the status can relate to whether the file portion is associated with a streaming write, whether a protocol indication indicates a file handle associated with the file has been closed, or whether an advisory lock has been released by a client device. The AMC can determine whether to transition from an exclusive data lock associated with the portion of the file to a shared data lock with respect to the portion of the file, based on a result of analyzing the file-related information. The AMC can fine tune determinations or predictions of data lock transitions.

Abstract: The described technology is generally directed towards maintaining a heat score per file based on a file's access/activity level, and using the heat score to re-tier files from faster to slower storage and vice-versa. Files on a slower tier that have more activity are thereby moved (“up-tiered”) to a faster access storage tier, while the files that have less activity on the faster access storage tier are moved (“down-tiered”) to a slower access storage tier. The heat score for a file is increased based on more input-output operation activity, and decayed over time. The decaying can be based on a relative importance value (e.g., business value) assigned to the file or a domain that includes the file. Ordering the files based on their respective heat scores allows only a fraction of the files to be considered for re-tiering before the re-tiering moves are performed.

Abstract: Techniques for address mirroring of a file system journal are presented. A transactional file system update can correspond to a write operation to be performed at a first participant node of a data storage system. A journal transfer component can transfer the transactional file system update from an initiator node of the data storage system to a first journal associated with the first participant node and recovery information relating to the update to a second journal associated with a second participant node of the data storage system. The recovery information can be determined based on the update. To facilitate the update, a journal descriptor block can be generated and sent to the second participant node, such block comprising the recovery information, which can include transaction state, block addresses involved in the transaction, error correction-related information, or a mirror block address set for the block addresses where desired.

Abstract: Systems and methods facilitating fault tolerance for transaction mirroring are described herein. A method as described herein can include receiving a commit command for a data transaction from an initiator node of the system, wherein the data transaction is associated with a first failure domain, and wherein the commit command is directed to a primary participant node and a secondary participant node of the system; determining whether a response to the commit command has been received at the primary participant node from the secondary participant node in response to the receiving; and, in response to determining that the response to the commit command was not received at the primary participant node, indicating that the secondary participant node is invalid in a data store associated with a second failure domain that is distinct from the first failure domain.

Abstract: Systems and methods for mirroring a file system journal are described herein. A method as described herein can include receiving, by a system operatively coupled to a processor, transactional file system updates corresponding to a write operation to be performed at a first node of the system; transferring, by the system, the transactional file system updates from an initiator node of the system to a first journal at the first node and a second journal at a second node of the system that is logically distinct from the first node and the initiator node; and committing, by the system, the transactional file system updates to the first journal and the second journal.