Abstract: A consistent time service that provides a method of maintaining deterministic clock-related operations for a group of replicas in a fault-tolerant distributed system. A consistent clock synchronization algorithm is utilized that yields a single consistent group clock for the replicas in the group, and does not require synchronization of the underlying physical hardware clocks. The consistent group clock ensures the determinism of the replicas in the group with respect to clock-related operations, is monotonically increasing, has bounded increment, skew and drift. The consistent time service provides benefits for active replication during normal operation, as well as passive replication and semi-active replication to ensure a consistent monotonically increasing clock when the primary replica fails and a backup replica takes over as the new primary replica.

Abstract: A method for checkpointing a multithreaded application program, based on the egalitarian and competitive active replication strategy. The invention enables different threads to be checkpointed at different times in such a way that the checkpoints restore a consistent state of the threads at a new or recovering replica, even though the threads operate concurrently and asynchronously. Separate checkpoints are generated for the local state of each thread and for the data that are shared between threads and are protected by mutexes. The checkpoint of the shared data is communicated in a special message that also determines the order in which the claims of mutexes are granted to the threads. A source-code preprocessor tool is described for inserting code into an application program to checkpoint the state of the thread during normal operation and to restore the state of the thread from the checkpoint subsequently.

Abstract: Method, system, apparatus and/or computer program for achieving transparent integration of high-availability services for distributed application programs. Loss-less migration of sub-programs from their respective primary nodes to backup nodes is performed transparently to a client which is connected to the primary node. Migration is performed by high-availability services which are configured for injecting registration codes, registering distributed applications, detecting execution failures, executing from backup nodes in response to failure, and other services. High-availability application services can be utilized by distributed applications having any desired number of sub-programs without the need of modifying or recompiling the application program and without the need of a custom loader. In one example embodiment, a transport driver is responsible for receiving messages, halting and flushing of messages, and for issuing messages directing sub-programs to continue after checkpointing.

Abstract: Incremental single and multiprocess checkpointing and restoration is described, which is transparent in that the application program need not be modified, re-compiled, or re-linked to gain the benefits of the invention. The processes subject to checkpointing can be either single or multi-threaded. The method includes incremental page-boundary checkpointing, as well as storage checkpointing of data files associated with applications to ensure correct restoration without the need to restore files for other application programs. Incremental and full checkpoints are asynchronously merged to ensure proper operation while reducing checkpointing delay. By way of example a user-level programming library is described for loading into the address space of the application in conjunction with a loadable kernel module (LKM) or device driver used to capture and restore process state on behalf of the application. These techniques are particularly well suited for use with high-availability (HA) protection programming.

Abstract: Methods of transparent connection failover allowing a remote computer (i.e., a client), to continue to use a network connection to communicate with one of at least two or more other computers (i.e., the backup servers) over a network, when one of the other computers (i.e., the primary server) fails. With the mechanisms of this invention, there is no need for the client to establish a new connection to a backup server when the primary server fails. The failover is preferably executed within a bridge layer between the TCP layer and the IP layer of the server's TCP/IP stack. No modifications are required to the network infrastructure, the client's TCP/IP stack, the client application or the server application. The methods support active or semi-active replication of the server application, and do not require rollback of the application during failover. The invention also provides mechanisms for bringing up new backup servers.

Abstract: A method and system for transparent consistent active replication of multithreaded application programs is described. At each replica, control messages that contain mutex ordering information indicating the order in which threads in the replicas claim mutexes are multicast, and the control messages are delivered using a multicast group communication protocol that delivers the messages in an order that determines the order in which the operating system's thread library grants the claims of mutexes to the threads in the replicas. Because the replicas receive the same messages in the same source order, the corresponding threads in the different replicas are granted their corresponding claims to the corresponding mutexes in the same order, maintaining strong replica consistency.

Abstract: A system and method for replicating a multithreaded application program using a semi-active or passive replication strategy, wherein the application program executes under the control of an operating system having a thread library. The method comprises piggybacking mutex ordering information at the Primary replica onto regular multicast messages specifying the order in which threads in the Primary replica have been granted their claims to mutexes; and receiving the multicast messages at a Backup replica containing the mutex ordering information which determines the order in which threads in the Backup replica are granted mutexes. Thread library interpositioning is preferably utilized to intercept calls to functions in the operating system's thread library, so that the system and method of the invention may be implemented transparently. The invention enforces strong replica consistency without the need to count instructions, add significant messaging overhead, or modify application code.

Abstract: A method and mechanisms for checkpointing objects, processes and other components of a multithreaded application program, based on the leader-follower strategy of semi-active or passive replication, where it is not possible to stop and checkpoint all of the threads of the object, process or other component simultaneously. Separate checkpoints are generated for the local state of each thread and for the data that are shared between threads and are protected by mutexes. The invention enables different threads to be checkpointed at different times in such a way that the checkpoints restore a consistent state of the threads between the existing replicas and a new or recovering replica, even though the threads operate concurrently and asynchronously. The checkpoint of the shared data is piggybacked onto regular messages along with ordering information that determines the order in which the mutexes are granted to the threads.