Abstract: Systems, methods, and circuitries are provided for supporting distributed erasure coding in a shared file system. In one example, a method is provided to be performed by an initiator device configured to read and write data in files stored in a plurality of storage nodes that are controlled by a file management system. The method includes generating a stripe by identifying data to be stored in a storage system; dividing the data into K data segments; and performing an erasure encoding operation on the K data segments to generate a parity segment, wherein the stripe includes the K data segments and the parity segment. The method includes requesting, from the file management system, respective memory allocations in the storage devices for storing respective segments of the stripe; and transmitting, to each of the plurality of storage nodes, a respective instruction to store a respective data or parity segment in the memory allocation on the respective storage device.

Abstract: Systems, methods, and circuitries are provided for supporting distributed erasure coding in a shared file system. In one example, a method is provided to be performed by an initiator device configured to read and write data in files stored in a plurality of storage nodes that are controlled by a file management system. The method includes generating a stripe by identifying data to be stored in a storage system; dividing the data into K data segments; and performing an erasure encoding operation on the K data segments to generate a parity segment, wherein the stripe includes the K data segments and the parity segment. The method includes requesting, from the file management system, respective memory allocations in the storage devices for storing respective segments of the stripe; and transmitting, to each of the plurality of storage nodes, a respective instruction to store a respective data or parity segment in the memory allocation on the respective storage device.

Abstract: Embodiments include an apparatus for replicating a file system (FS) that stores a file in a first location in a first data storage device, the apparatus comprising a query circuit configured to generate a spatial query that identifies a primary directory tree in the FS, and to receive a response to the query from the FS, where the response identifies a range of the primary directory tree in which a changed file may be found; a scan filter circuit configured to generate a scan filter based on the response, where the scan filter defines a portion of the primary directory tree that is smaller than the primary directory tree to search for a changed file; and a replication circuit configured to scan the portion of the primary directory tree, and upon detecting a changed file, replicate the changed file from the first location to a different location.

Abstract: Embodiments include an apparatus for replicating a file system (FS) that stores a file in a first location in a first data storage device, the apparatus comprising a query circuit configured to generate a spatial query that identifies a primary directory tree in the FS, and to receive a response to the query from the FS, where the response identifies a range of the primary directory tree in which a changed file may be found; a scan filter circuit configured to generate a scan filter based on the response, where the scan filter defines a portion of the primary directory tree that is smaller than the primary directory tree to search for a changed file; and a replication circuit configured to scan the portion of the primary directory tree, and upon detecting a changed file, replicate the changed file from the first location to a different location.

Abstract: Example apparatus and methods create and maintain a functional mirror of file system metadata for an active file system. The functional mirror may be implemented as a set of feature vectors in a feature space stored in a spatial database that allow a journal functionality for information received from the active file system. Mirror creation and maintenance can operate in parallel. The functional mirror may be created primarily using a systematic, system-wide, multi-threaded crawler approach whose results are updated using an event-driven approach that accounts for changes that may obsolete information acquired during the crawler approach. Transaction identifiers may be stored and potentially obsolete information preserved to facilitate providing views and rewind features.

Abstract: Example apparatus and methods create and maintain a functional mirror of file system metadata for an active file system. The functional mirror may be implemented as a set of feature vectors in a feature space stored in a spatial database that allow a journal functionality for information received from the active file system. Mirror creation and maintenance can operate in parallel. The functional mirror may be created primarily using a systematic, system-wide, multi-threaded crawler approach whose results are updated using an event-driven approach that accounts for changes that may obsolete information acquired during the crawler approach. Transaction identifiers may be stored and potentially obsolete information preserved to facilitate providing views and rewind features.

Abstract: Example apparatus and methods create and maintain a functional mirror of file system metadata for an active file system. The functional mirror may be implemented as a set of database tables that allow a journal functionality for information received from the active file system. Example apparatus and methods also restore the file system metadata from the functional mirror. Mirror creation and maintenance can operate in parallel. Mirror maintenance and metadata restore can also operate in parallel. The functional mirror may be created primarily using a systematic, system-wide, multi-threaded crawler approach whose results are updated using an event-driven approach that accounts for changes that may make obsolete information acquired during the crawler approach. The metadata may be restored using a systematic, system-wide, multi-threaded crawler approach acting together with an event-driven approach that restores certain files out of order as they are needed.

Abstract: Example apparatus and methods create and maintain a functional mirror of file system metadata for an active file system. The functional mirror may be implemented as a set of database tables that allow a journal functionality for information received from the active file system. Example apparatus and methods also restore the file system metadata from the functional mirror. Mirror creation and maintenance can operate in parallel. Mirror maintenance and metadata restore can also operate in parallel. The functional mirror may be created primarily using a systematic, system-wide, multi-threaded crawler approach whose results are updated using an event-driven approach that accounts for changes that may make obsolete information acquired during the crawler approach. The metadata may be restored using a systematic, system-wide, multi-threaded crawler approach acting together with an event-driven approach that restores certain files out of order as they are needed.