Abstract: Various embodiments set forth a computer-readable media storing program instructions that, when executed by one or more processors, cause the processors to perform steps of maintaining, by a first node of a first cluster, a respective first open transport control protocol (TCP) connection with each of a plurality of second nodes in a second cluster; maintaining, by a third node of the first cluster, a second open TCP connection with the first node, wherein the third node is prevented from establishing a TCP connection with any of the plurality of second nodes; and sending, by the third node, a message to the second cluster by sending the message to the first node via the second open TCP connection, wherein the first node is configured to forward the message to one of the second nodes via a corresponding one of the respective first open TCP connections.

Abstract: Various embodiments set forth a computer-readable media storing program instructions that, when executed by one or more processors, cause the processors to perform steps of establishing, by a first node of a first cluster, a respective first connection with each of a plurality of second nodes in a second cluster; and establishing, by third node of the first cluster, a second connection with the first node. The third node is prevented from establishing a connection with any of the second nodes. The steps further include sending, by the third node, a message to the second cluster by sending the message to the first node via the second connection, where the first node is configured to forward the message to one of the second nodes via a corresponding one of the respective first connections.

Abstract: An illustrative embodiment disclosed herein is an apparatus including a processor having programmed instructions to update a versioned object by appending a latest version of the versioned object to a data structure, read a predetermined version identifier corresponding to the latest version from a pointer entry, and look up the latest version of the versioned object, in the data structure, using the predetermined version identifier corresponding to the latest version of the versioned object.

Abstract: Methods, systems and computer program products for computer system data management. A leader node that is in a leader-follower relationship with a follower node is deployed to manage metadata across a plurality of interconnected computing nodes of the computer system. Upon a failure of any sort that results in a loss of operation or a disconnection of the leader node, the followers undertake steps to bring up a new leader. Such steps to bring up a new leader include electing a new leader to replace the lost or disconnected leader and initializing the data state of the new leader node by causing the leader to request incremental snapshots from followers. Additional metadata that might have changed since closing the incremental snapshots can also be retrieved from the followers. The metadata that might have changed since the time of closing the incremental snapshot is subjected to verification using a consensus algorithm.

Abstract: Various embodiments set forth techniques for managing and/or accessing metadata associated with a vblock, systems implementing said techniques, and computer-readable media storing instructions for performing said techniques. In some embodiments, one or more computer-readable media store instructions that, when executed by one or more processors, cause the one or more processors to perform steps including receiving a request for metadata associated with a vblock; accessing a merged metadata record associated with the vblock, where the merged metadata record comprises metadata corresponding to metadata in metadata records for all but a last snapshot or a live vblock having a metadata record, and a first identifier of the last snapshot or the live vblock having a metadata record; and returning the requested metadata based on the metadata in the merged metadata record and metadata in the metadata record identified by the first identifier.

Abstract: Systems for high-performance distributed computing. The systems include techniques for managing data and metadata across multiple nodes. A method embodiment commences by storing data at a node using a first storage mechanism that is local to the node. A first set of metadata is configured to identify a storage location for the stored data. The first set of metadata is stored using the same first storage mechanism that is local to the node. For accessing the first set of metadata, a second set of metadata is configured to identify a storage location for the first set of metadata. The second set of metadata is stored using a second storage mechanism that comprises a distributed metadata storage facility that stores metadata across multiple storage locations having at least one of the multiple storage locations that is not local to the node that stores data and metadata using the first storage mechanism.

Abstract: Various embodiments set forth techniques for managing metadata associated with a vblock, In some embodiments, one or more computer-readable media store instructions that, when executed by one or more processors, cause the one or more processors to perform steps including receiving a request to write data to a live vblock, wherein the request to write data is a first write request for the live vblock; accessing a merged metadata record associated with the live vblock, wherein the merged metadata record comprises metadata corresponding to metadata in metadata records for all but a last snapshot included in a set of snapshots having a metadata record; adding metadata associated with the request to write data to a metadata record for the live vblock; merging a metadata record for the last snapshot into the merged metadata record; and updating a first identifier of the merged metadata record to identify the live vblock.

Abstract: A technique replicates an index of an operations log (oplog) from a primary node to a secondary node of a cluster in the event of failure. The oplog functions as a staging area to coalesce random write operations directed to a virtual disk (vdisk) stored on a backend storage tier. The oplog temporarily caches write data as well as metadata describing the write data. The metadata includes descriptors to the write data corresponding to offset ranges of the vdisk and are used to identify ranges of write data for the vdisk that are cached in the oplog. To facilitate fast lookup operations of whether write data is cached in the oplog, an oplog index provides a state of the latest data for offset ranges of the vdisk that enables fast failover of metadata used to construct the oplog index in memory without downtime or significant metadata replay.

Abstract: Systems for low-latency data access in distributed computing systems. A method embodiment commences upon generating a first storage area in local storage of a first computing node. Access to the first storage area is provided through the first computing node. A second storage area is generated wherein the second storage area comprises a first set of metadata that comprises local storage device locations of at least some of the local storage areas of the first storage area. A set of physical access locations of the second storage area is stored to a database that manages updates to the second set of metadata pertaining to the second storage area. Accesses to the first storage area are accomplished by querying the database to retrieve a location of the second set of metadata, and then accessing the first storage area through one or more additional levels of metadata that are node-wise collocated.

Abstract: An improved architecture is provided which enables significant convergence of the components of a system to implement virtualization. The infrastructure is VM-aware, and permits scaled out converged storage provisioning to allow storage on a per-VM basis, while identifying I/O coming from each VM. The current approach can scale out from a few nodes to a large number of nodes. In addition, the inventive approach has ground-up integration with all types of storage, including solid-state drives. The architecture of the invention provides high availability against any type of failure, including disk or node failures. In addition, the invention provides high performance by making I/O access local, leveraging solid-state drives and employing a series of patent-pending performance optimizations.

Abstract: A technique replicates an index of an operations log (oplog) from a primary node to a secondary node of a cluster in the event of a failure of the primary node. The oplog functions as a staging area to coalesce random write operations directed to a virtual disk (vdisk) stored on a backend storage tier organized as an extent store. The oplog temporarily caches data associated with the random write operations (i.e., write data) as well as metadata describing the write data. The metadata includes descriptors to the write data corresponding to virtual address regions, i.e., offset ranges, of the vdisk and are used to identify the offset ranges of write data for the vdisk that are cached in the oplog. To facilitate fast lookup operations of the offset ranges when determining whether write data io is cached in the oplog, an oplog index provides a state of the latest data for offset ranges of the vdisk.

Abstract: Various embodiments set forth techniques for managing metadata associated with a vblock, In some embodiments, one or more computer-readable media store instructions that, when executed by one or more processors, cause the one or more processors to perform steps including receiving a request to write data to a live vblock, wherein the request to write data is a first write request for the live vblock; accessing a merged metadata record associated with the live vblock, wherein the merged metadata record comprises metadata corresponding to metadata in metadata records for all but a last snapshot included in a set of snapshots having a metadata record; adding metadata associated with the request to write data to a metadata record for the live vblock; merging a metadata record for the last snapshot into the merged metadata record; and updating a first identifier of the merged metadata record to identify the live vblock.

Abstract: Various embodiments set forth techniques for managing and/or accessing metadata associated with a vblock, systems implementing said techniques, and computer-readable media storing instructions for performing said techniques. In some embodiments, one or more computer-readable media store instructions that, when executed by one or more processors, cause the one or more processors to perform steps including receiving a request for metadata associated with a vblock; accessing a merged metadata record associated with the vblock, where the merged metadata record comprises metadata corresponding to metadata in metadata records for all but a last snapshot or a live vblock having a metadata record, and a first identifier of the last snapshot or the live vblock having a metadata record; and returning the requested metadata based on the metadata in the merged metadata record and metadata in the metadata record identified by the first identifier.

Abstract: An improved architecture is provided which enables significant convergence of the components of a system to implement virtualization. The infrastructure is VM-aware, and permits scaled out converged storage provisioning to allow storage on a per-VM basis, while identifying I/O coming from each VM. The current approach can scale out from a few nodes to a large number of nodes. In addition, the inventive approach has ground-up integration with all types of storage, including solid-state drives. The architecture of the invention provides high availability against any type of failure, including disk or node failures. In addition, the invention provides high performance by making I/O access local, leveraging solid-state drives and employing a series of patent-pending performance optimizations.

Abstract: Methods, systems and computer program products for computer system data management. A leader node that is in a leader-follower relationship with a follower node is deployed to manage metadata across a plurality of interconnected computing nodes of the computer system. Upon a failure of any sort that results in a loss of operation or a disconnection of the leader node, the followers undertake steps to bring up a new leader. Such steps to bring up a new leader include electing a new leader to replace the lost or disconnected leader and initializing the data state of the new leader node by causing the leader to request incremental snapshots from followers. Additional metadata that might have changed since closing the incremental snapshots can also be retrieved from the followers. The metadata that might have changed since the time of closing the incremental snapshot is subjected to verification using a consensus algorithm.

Abstract: Systems for low-latency data access in distributed computing systems. A method embodiment commences upon generating a first storage area in local storage of a first computing node. Access to the first storage area is provided through the first computing node. A second storage area is generated wherein the second storage area comprises a first set of metadata that comprises local storage device locations of at least some of the local storage areas of the first storage area. A set of physical access locations of the second storage area is stored to a database that manages updates to the second set of metadata pertaining to the second storage area. Accesses to the first storage area are accomplished by querying the database to retrieve a location of the second set of metadata, and then accessing the first storage area through one or more additional levels of metadata that are node-wise collocated.

Abstract: Systems for high-performance distributed computing. The systems include techniques for managing data and metadata across multiple nodes. A method embodiment commences by storing data at a node using a first storage mechanism that is local to the node. A first set of metadata is configured to identify a storage location for the stored data. The first set of metadata is stored using the same first storage mechanism that is local to the node. For accessing the first set of metadata, a second set of metadata is configured to identify a storage location for the first set of metadata. The second set of metadata is stored using a second storage mechanism that comprises a distributed metadata storage facility that stores metadata across multiple storage locations having at least one of the multiple storage locations that is not local to the node that stores data and metadata using the first storage mechanism.

Abstract: An improved architecture is provided which enables significant convergence of the components of a system to implement virtualization. The infrastructure is VM-aware, and permits scaled out converged storage provisioning to allow storage on a per-VM basis, while identifying I/O coming from each VM. The current approach can scale out from a few nodes to a large number of nodes. In addition, the inventive approach has ground-up integration with all types of storage, including solid-state drives. The architecture of the invention provides high availability against any type of failure, including disk or node failures. In addition, the invention provides high performance by making I/O access local, leveraging solid-state drives and employing a series of patent-pending performance optimizations.

Abstract: Systems for low-latency data access in distributed computing systems. A method embodiment commences upon generating a first storage area in local storage of a first computing node. Access to the first storage area is provided through the first computing node. A second storage area is generated wherein the second storage area comprises a first set of metadata that comprises local storage device locations of at least some of the local storage areas of the first storage area. A set of physical access locations of the second storage area is stored to a database that manages updates to the second set of metadata pertaining to the second storage area. Accesses to the first storage area are accomplished by querying the database retrieve a location of the second set of metadata, and then accessing the first storage area through one or more additional levels of metadata that are node-wise collocated.

Abstract: Systems for high-performance distributed computing. The systems include techniques for managing data and metadata across multiple nodes. A method embodiment commences by storing data at a node using a first storage mechanism that is local to the node. A first set of metadata is configured to identify a storage location for the stored data. The first set of metadata is stored using the same first storage mechanism that is local to the node. For accessing the first set of metadata, a second set of metadata is configured to identify a storage location for the first set of metadata. The second set of metadata is stored using a second storage mechanism that comprises a distributed metadata storage facility that stores metadata across multiple storage locations having at least one of the multiple storage locations that is not local to the node that stores data and metadata using the first storage mechanism.