Abstract: Example implementations relate to partial file system instances. In an example, a subset of objects of a source file system instance on a source system are replicated to a target system to form a partial file system instance on the target system comprised of the subset of objects. Each of the objects of the source file system instance is identified by a signature based on content of each of the objects and the objects exhibit a hierarchical relationship to a root object in the file system instance. An unmaterialized object is dynamically added to the partial file system instance by replicating the corresponding object from the source file system instance. The target system is asynchronously updated from the source file system instance based on a comparison of the partial file system instance to the source file system instance.

Abstract: Systems and methods are provide for automatically constructing data lineage representations for distributed data processing pipelines. These data lineage representations (which are constructed and stored in a central repository shared by the multiple data processing sites) can be used to among other things, clone the distributed data processing pipeline for quality assurance or debugging purposes. Examples of the presently disclosed technology are able to construct data lineage representations for distributed data processing pipelines by (1) generating a hash content value for universally identifying each data artifact of the distributed data processing pipeline across the multiple processing stages/processing sites of the distributed data processing pipeline; and (2) creating an data processing pipeline abstraction hierarchy for associating each data artifact to input and output events for given executions of given data processing stages (performed by the multiple data processing sites).

Abstract: Examples described herein relate to In some examples, a least recently used (LRU) list used to evict nodes from a cache is traversed, the nodes referencing data blocks storing data in a storage device, the nodes being leaf nodes in a tree representing a file system, and for each traversed node, determining a stride length as an absolute value of a difference between a block offset value of the traversed node and a block offset value of a previously traversed node, comparing the stride length to a proximity threshold, and updating a sequential access pattern counter based at least in part on the comparing; and proactively prefetching nodes from the storage device to the cache when the sequential access pattern counter indicates a detected pattern of sequential accesses to nodes.

Abstract: Examples described herein relate to In some examples, a least recently used (LRU) list used to evict nodes from a cache is traversed, the nodes referencing data blocks storing data in a storage device, the nodes being leaf nodes in a tree representing a file system, and for each traversed node, determining a stride length as an absolute value of a difference between a block offset value of the traversed node and a block offset value of a previously traversed node, comparing the stride length to a proximity threshold, and updating a sequential access pattern counter based at least in part on the comparing; and proactively prefetching nodes from the storage device to the cache when the sequential access pattern counter indicates a detected pattern of sequential accesses to nodes.

Abstract: Example implementations relate to partial file system instances. In an example, a subset of objects of a source file system instance on a source system are replicated to a target system to form a partial file system instance on the target system comprised of the subset of objects. Each of the objects of the source file system instance is identified by a signature based on content of each of the objects and the objects exhibit a hierarchical relationship to a root object in the file system instance. An unmaterialized object is dynamically added to the partial file system instance by replicating the corresponding object from the source file system instance. The target system is asynchronously updated from the source file system instance based on a comparison of the partial file system instance to the source file system instance.

Abstract: Example techniques of data streaming between datacenters are described. In an example, a delta data may be replicated at a specific time interval from a first filesystem of a first stream-processing platform implemented at a first datacenter or source datacenter to a second filesystem of a second stream-processing platform implemented at a second datacenter or a target datacenter. The delta data indicates modifications to data of the data stream stored in the first filesystem, during the specific time interval.