Abstract: A map-reduce compatible distributed file system that consists of successive component layers that each provide the basis on which the next layer is built provides transactional read-write-update semantics with file chunk replication and huge file-create rates. Containers provide the fundamental basis for data replication, relocation, and transactional updates. A container location database allows containers to be found among all file servers, as well as defining precedence among replicas of containers to organize transactional updates of container contents. Volumes facilitate control of data placement, creation of snapshots and mirrors, and retention of a variety of control and policy information. Also addressed is the use of distributed transactions in a map-reduce system; the use of local and distributed snapshots; replication, including techniques for reconciling the divergence of replicated data after a crash; and mirroring.

Abstract: A map-reduce compatible distributed file system that consists of successive component layers that each provide the basis on which the next layer is built provides transactional read-write-update semantics with file chunk replication and huge file-create rates. Containers provide the fundamental basis for data replication, relocation, and transactional updates. A container location database allows containers to be found among all file servers, as well as defining precedence among replicas of containers to organize transactional updates of container contents. Volumes facilitate control of data placement, creation of snapshots and mirrors, and retention of a variety of control and policy information. Also addressed is the use of distributed transactions in a map-reduce system; the use of local and distributed snapshots; replication, including techniques for reconciling the divergence of replicated data after a crash; and mirroring.

Abstract: A map-reduce compatible distributed file system that consists of successive component layers that each provide the basis on which the next layer is built provides transactional read-write-update semantics with file chunk replication and huge file-create rates. A primitive storage layer (storage pools) knits together raw block stores and provides a storage mechanism for containers and transaction logs. Storage pools are manipulated by individual file servers. Containers provide the fundamental basis for data replication, relocation, and transactional updates. A container location database allows containers to be found among all file servers, as well as defining precedence among replicas of containers to organize transactional updates of container contents. Volumes facilitate control of data placement, creation of snapshots and mirrors, and retention of a variety of control and policy information.

Abstract: A method involves detecting that a first storage device is inaccessible. The information stored on the first storage device is a synchronized copy of information stored on a second storage device. In response to detecting that the first storage device is inaccessible, modification logging is initiated. Modification logging involves detecting modifications to information stored on the second storage device and storing information that indicates an order in which the modifications occur to the second storage device. The stored information can indicate the order explicitly (e.g., by including a timestamp or sequence number) or indirectly (e.g., the order of entries stored in a log can indicate the order of the modifications represented by those entries). The stored information can also include the new values generated by the modifications. The stored information can then be used to resynchronize the information stored on the first and second storage devices.

Abstract: A method, system, and computer-readable medium to quickly synchronize copies of data that have diverged over time. The present invention can be used in synchronizing replicated copies of data that have become unsynchronized due to the restoration of primary data from a backup copy of the primary data. The solution enables copies of data to be synchronized without copying all of the data from one copy to another. The solution uses minimal resources to maintain data consistency and has minimal effect on performance of applications using the data and on network usage. The method includes determining that a change occurred to data in a region of a primary volume without including the region in a set of regions designated for replication to a secondary volume. The region is added to the set of regions designated for replication to the secondary volume and replicated.

Abstract: Various methods and systems are disclosed for handling I/O requests to a replica that has not yet been fully synchronized. One method involves detecting the transfer of the role of working volume from a first volume to a second volume. The second volume is configured as a replica of the first volume, but the second volume may not be fully synchronized with the first volume at the time that the role of working volume is transferred. Subsequent to the role of working volume being transferred to the second volume, a read request to the second volume is satisfied by accessing a log. The log is configured to store values that have been written to the first volume, and at least one of the values written to the first volume has not been written to the second volume.

Abstract: A method for efficiently performing range deletions in a B+ tree. The method operates to delete all keys within a range in a plurality of iterations. Each iteration may comprise: 1) deleting all of the keys in one or more leaf-nodes that lie within the range of keys; and 2) adjusting entries in the node and its neighbors to perform any required rebalancing of the tree after the deletion. Each iteration of the deletion may comprise a Walk phase identifies nodes in the range of keys to be deleted; a Prepare phase which determines operations (delete and/or node adjust operations) to be performed at one or more levels in the B+ tree based on the identified nodes; and a Delete phase which performs the operations to delete keys in the range of keys in the B+ tree.