Abstract: In connection with a data distribution architecture, client-side “deduplication” techniques may be utilized for data transfers occurring among various file system nodes. In some examples, these deduplication techniques involve fingerprinting file system elements that are being shared and transferred, and dividing each file into separate units referred to as “blocks” or “chunks.” These separate units may be used for independently rebuilding a file from local and remote collections, storage locations, or sources. The deduplication techniques may be applied to data transfers to prevent unnecessary data transfers, and to reduce the amount of bandwidth, processing power, and memory used to synchronize and transfer data among the file system nodes. The described deduplication concepts may also be applied for purposes of efficient file replication, data transfers, and file system events occurring within and among networks and file system nodes.

Abstract: Systems and techniques for a distributed file system element collection are described herein. A node may identify a state of a file system element, which is identified in a file system element definition, from a local data store. The state corresponds to a distribution action. The file system element definition is one of a plurality of file system element definitions in a collection schema of a collection to which the node is a member. The node may, in response to identifying the state of the file system element, issue a communication to some members of the collection in furtherance of completion of the distribution action. A response may be received by the node from a participant node in the list of participant nodes. The node may complete the distribution action using content from the response.

Abstract: Disclosed in some examples are predictive storage techniques for use in a distributed data system. The predictive storage techniques may be used to manage locally stored elements of a shared data collection, such as the storage of files on nodes of the distributed data system that are limited in local storage space. The predictive storage techniques may achieve a balance between consumption of local resources and timely access of important elements in the shared data collection. For example, the predictive storage techniques may be used for keeping or pre-caching certain items of a collection that are determined as likely to be used in local storage for convenient access, and allowing access the remaining items on request over a network.

Abstract: In connection with a data distribution architecture, client-side “deduplication” techniques may be utilized for data transfers occurring among various file system nodes. In some examples, these deduplication techniques involve fingerprinting file system elements that are being shared and transferred, and dividing each file into separate units referred to as “blocks” or “chunks.” These separate units may be used for independently rebuilding a file from local and remote collections, storage locations, or sources. The deduplication techniques may be applied to data transfers to prevent unnecessary data transfers, and to reduce the amount of bandwidth, processing power, and memory used to synchronize and transfer data among the file system nodes. The described deduplication concepts may also be applied for purposes of efficient file replication, data transfers, and file system events occurring within and among networks and file system nodes.

Abstract: Disclosed in some examples is a data distribution mechanism for distributing a collection of file system elements across one or more computing devices. The system can include a plurality of nodes implemented on a machine. The data distribution mechanism may be used in connection with data synchronization, sharing, backup, archiving, and versioning operations for a plurality of connected machines on behalf of one or a plurality of users.