Abstract: Techniques are disclosed for disconnected operation in a distributed database system. In an embodiment, the distributed database system implements a disconnected mode of operation allowing isolated regions of database nodes to provisionally commit transactions, with the global requirements of those transactions later satisfied by a transparent healing process after network connectivity is reestablished between the regions. The healing process, in turn, enables the distributed database system to construct a consistent global state of the database that accounts for the transactions provisionally-committed in each isolated region during the disconnected mode. Once the healing process completes, database clients “see” a healed version of the database that simulates or otherwise closely approximates a state of the database had the distributed database system performed the transactions during a normal, fully-connected, mode of operation.

Abstract: An “operate with missing region” feature allows a cluster to continue servicing reads for available regions even when some regions are missing. Upon a given node failure condition, the cluster is placed in an effective read-only mode for all regions. The node failure condition typically is one where there has been a failure of an authoritative region copy and no backup copy is then available. Typically, “read-only” means no client write or update requests succeed while the cluster is in this state. Those requests should fail with an error message that indicates failure because of the read-only status. Preferably, such requests are re-tried. In this mode, regions are only allowed to perform read operations but the cluster continues to operate with missing regions, and missing regions are entered on a region map. The cluster then automatically recovers returning missing regions, after which it leaves the read-only state.

Abstract: A redundant array of independent nodes are networked together. Each node executes an instance of an application that provides object-based storage. The nodes are grouped into systems each having multiple nodes. Objects in the systems have access control lists (ACLs) associated therewith. A method of providing a search key to each ACL in any one system comprises: selecting a region for each metadata object pertaining to the objects in the system; and generating, for each ACL associated with an object in the system, an ACL ID (identifier) as a search key which includes the region number of the region selected for the metadata object pertaining to the object, a region map level corresponding to the region selected, and a monotonically increasing counter value that is automatically incremented each time an ACL ID is generated. The ACL ID remains unique during reconfiguration via splitting/merging existing distributed portions of the database.

Abstract: An “operate with missing region” feature allows a cluster to continue servicing reads for available regions even when some regions are missing. Upon a given node failure condition, the cluster is placed in an effective read-only mode for all regions. The node failure condition typically is one where there has been a failure of an authoritative region copy and no backup copy is then available. Typically, “read-only” means no client write or update requests succeed while the cluster is in this state. Those requests should fail with an error message that indicates failure because of the read-only status. Preferably, such requests are re-tried. In this mode, regions are only allowed to perform read operations but the cluster continues to operate with missing regions, and missing regions are entered on a region map. The cluster then automatically recovers returning missing regions, after which it leaves the read-only state.

Abstract: An “operate with missing region” feature allows a cluster to continue servicing reads for available regions even when some regions are missing. Upon a given node failure condition, the cluster is placed in an effective read-only mode for all regions. The node failure condition typically is one where there has been a failure of an authoritative region copy and no backup copy is then available. Typically, “read-only” means no client write or update requests succeed while the cluster is in this state. Those requests should fail with an error message that indicates failure because of the read-only status. Preferably, such requests are re-tried. In this mode, regions are only allowed to perform read operations but the cluster continues to operate with missing regions, and missing regions are entered on a region map. The cluster then automatically recovers returning missing regions, after which it leaves the read-only state.

Abstract: A redundant array of independent nodes are networked together. Each node executes an instance of an application that provides object-based storage. The nodes are grouped into systems each having multiple nodes. Objects in the systems have access control lists (ACLs) associated therewith. A method of providing a search key to each ACL in any one system comprises: selecting a region for each metadata object pertaining to the objects in the system; and generating, for each ACL associated with an object in the system, an ACL ID (identifier) as a search key which includes the region number of the region selected for the metadata object pertaining to the object, a region map level corresponding to the region selected, and a monotonically increasing counter value that is automatically incremented each time an ACL ID is generated. The ACL ID remains unique during reconfiguration via splitting/merging existing distributed portions of the database.

Abstract: This disclosure enables a user to incrementally increase or decrease the size of the cluster and, in particular, to incrementally increase or decrease the number of map regions implemented in a metadata manager associated with the cluster while maintaining availability of the cluster. In a representative embodiment, a configurable target map size is identified, either manually (e.g., by the user) or programmatically. If the target map size is found to be within an acceptable value (given the available node configuration), a procedure to create a new map is triggered. The new map preferably is generated as follows. Initially, one or more new temporary region(s) are created either to split or join the existing regions (depending on whether the map is being increased or decreased). For each temporary region, the process also generates MDPL backup regions. Metadata from an original region (or regions) is then migrated to the temporary region(s).

Abstract: An “operate with missing region” feature allows a cluster to continue servicing reads for available regions even when some regions are missing. Upon a given node failure condition, the cluster is placed in an effective read-only mode for all regions. The node failure condition typically is one where there has been a failure of an authoritative region copy and no backup copy is then available. Typically, “read-only” means no client write or update requests succeed while the cluster is in this state. Those requests should fail with an error message that indicates failure because of the read-only status. Preferably, such requests are re-tried. In this mode, regions are only allowed to perform read operations but the cluster continues to operate with missing regions, and missing regions are entered on a region map. The cluster then automatically recovers returning missing regions, after which it leaves the read-only state.

Abstract: Cluster recovery time is reduced by an incremental refresh technique that is described herein. The goal of the technique is to salvage the portion of the redundant distributed database lost (e.g., during a fault) by performing an incremental refresh of only the updates occurring to that portion of the database since the time of loss.

Abstract: An “operate with missing region” feature of this disclosure allows the cluster to continue servicing reads for available regions even when some regions are missing. In particular, upon a given node failure condition, the cluster is placed in an effective read-only mode for all regions. The node failure condition typically is one where there has been a failure of an authoritative region copy and no backup copy is then available. As used herein, “read-only” means that no client write or update requests will succeed while the cluster is in this state. In particular, those requests should fail with an error message (or the like) that indicates failure because of the read-only status. Preferably, such requests are then re-tried. In this mode, all regions are only allowed to perform read operations. During the read-only state, the cluster continues to operate with missing regions, and missing regions are entered on the region map.

Abstract: This disclosure enables a user to incrementally increase or decrease the size of the cluster and, in particular, to incrementally increase or decrease the number of map regions implemented in a metadata manager associated with the cluster while maintaining availability of the cluster. In a representative embodiment, a configurable target map size is identified, either manually (e.g., by the user) or programmatically. If the target map size is found to be within an acceptable value (given the available node configuration), a procedure to create a new map is triggered. The new map preferably is generated as follows. Initially, one or more new temporary region(s) are created either to split or join the existing regions (depending on whether the map is being increased or decreased). For each temporary region, the process also generates MDPL backup regions. Metadata from an original region (or regions) is then migrated to the temporary region(s).

Abstract: An “operate with missing region” feature of this disclosure allows the cluster to continue servicing reads for available regions even when some regions are missing. In particular, upon a given node failure condition, the cluster is placed in an effective read-only mode for all regions. The node failure condition typically is one where there has been a failure of an authoritative region copy and no backup copy is then available. As used herein, “read-only” means that no client write or update requests will succeed while the cluster is in this state. In particular, those requests should fail with an error message (or the like) that indicates failure because of the read-only status. Preferably, such requests are then re-tried. In this mode, all regions are only allowed to perform read operations. During the read-only state, the cluster continues to operate with missing regions, and missing regions are entered on the region map.

Abstract: Cluster recovery time is reduced by an incremental refresh technique that is described herein. The goal of the technique is to salvage the portion of the redundant distributed database lost (e.g., during a fault) by performing an incremental refresh of only the updates occurring to that portion of the database since the time of loss.