Abstract: Certain embodiments described herein are directed to methods and systems for adding one or more nodes to a first cluster including a first node in a computer system. A method performed by the first node comprises receiving a first request from a second node to join the first cluster. The method also comprises retrieving a first cluster configuration associated with the first cluster from a distributed database through a first database server (DBS) and creating a second cluster configuration using the first cluster configuration and information received from the second node as part of the request. The method further comprises populating a first one or more local trust stores of a first one or more processes executing on the first node with a second one or more security certificates of a second one or more processes executing on the second node. The method further comprises writing the second cluster configuration to the distributed database and returning the second cluster configuration to the second node.

Abstract: Distributed database management systems may maintain collections of items spanning multiple partitions. Index structures may correspond to items on one partition or to items on multiple partitions. Item collections and indexes may be replicated. Changes to the data maintained by the distributed database management system may result in updates to multiple index structures. The changes may be compiled into an instruction set applicable to the index structures. In-memory buffers may contain the instructions prior to transmission to affected partitions. Replication logs may be combined with an acknowledgment mechanism for reliable transmission of the instructions to the affected partitions.

Abstract: Some embodiments provide a method for a global manager that manages a logical network spanning multiple physical sites. The method receives a set of data describing a modification to a definition of the logical network. From multiple queues that each correspond to a different one of the physical sites spanned by the logical network, the method identifying a set of queues to which to store the received set of data. The method stores the received set of data in the identified set of queues. From each respective queue in the identified set of queues, the method sends the set of data through a respective channel between the global manager and a respective local manager at the respective physical site corresponding to the respective queue.

Abstract: Various aspects are disclosed for distributed application management using an embedded persistent queue framework. In some aspects, task execution data is monitored from a plurality of task execution engines. A task request is identified. The task request can include a task and a Boolean predicate for task assignment. The task is assigned to a task execution engine embedded in a distributed application process if the Boolean predicate is true, and a capacity of the task execution engine is sufficient to execute the task. The task is enqueued in a persistent queue. The task is retrieved from the persistent queue and executed.

Abstract: Some embodiments provide a method for a global manager that manages a logical network configuration for multiple datacenters that each have a local manager for managing the logical network configuration within the datacenter. Based on detecting that a connection to a particular local manager of a particular datacenter has been restored after a period of unavailability, the method identifies a portion of the logical network configuration that is relevant to the particular datacenter. In a series of transactions, the method transfers the identified portion of the logical network configuration to the particular local manager. During the series of transactions, the method identifies modifications to the identified portion of the logical network configuration to be included in the series of transactions. Upon completion of the series of transactions, the method transfers a notification to the particular local manager indicating completion of the series of transactions.

Abstract: A shared log system can provide a key-index structure (main map) that allows clients (users) to access the state of objects stored in the shared log system. The client can define secondary indices on the main map to provide access to the data that are tailored to the client's data needs. Each client can generate their own customized set of secondary indices different from other clients.

Abstract: Some embodiments provide a method that, at a first machine that operates at a first physical site as a primary global manager for a logical network spanning a plurality of physical sites, receives a set of ordered data describing a configuration of the logical network. The method stores, in a particular order, the received set of ordered data in a first database located at the first physical site. Through a channel between (i) the first machine and (ii) a second machine that operates at a second physical site as a secondary global manager for the logical network in case of failure of the primary global manager, the method provides the set of ordered data in the particular order for the second machine to store in a second database in the particular order. The second database located at the second physical site.

Abstract: Disclosed are a method and system for implementing a distributed persistent queue. To implement the queue, an API is defined that includes a command to enqueue an object onto the queue, to obtain the ordering of all objects in the queue and to remove any item from the queue. Enqueuing an object includes obtaining a unique ID for the object from a generator and then storing the object in a persistent shared log, which provides persistent storage for the queue. Obtaining an ordering of all objects in the persistent queue includes receiving for each object stored in the queue a pair of values, the first being the unique ID and the second being a sequence number from the persistent shared log. Regardless of the host computer system making the request for the ordering of objects in the queue the ordering provided to each host computer system is the same.

Abstract: Some embodiments provide a method for a global manager that manages a logical network configuration for multiple datacenters that each have a local manager for managing the logical network configuration within the datacenter. Based on detecting that a connection to a particular local manager of a particular datacenter has been restored after a period of unavailability, the method identifies a portion of the logical network configuration that is relevant to the particular datacenter. In a series of transactions, the method transfers the identified portion of the logical network configuration to the particular local manager. During the series of transactions, the method identifies modifications to the identified portion of the logical network configuration to be included in the series of transactions. Upon completion of the series of transactions, the method transfers a notification to the particular local manager indicating completion of the series of transactions.

Abstract: Some embodiments provide a method that, at a first machine that operates at a first physical site as a primary global manager for a logical network spanning a plurality of physical sites, receives a set of ordered data describing a configuration of the logical network. The method stores, in a particular order, the received set of ordered data in a first database located at the first physical site. Through a channel between (i) the first machine and (ii) a second machine that operates at a second physical site as a secondary global manager for the logical network in case of failure of the primary global manager, the method provides the set of ordered data in the particular order for the second machine to store in a second database in the particular order. The second database located at the second physical site.

Abstract: Some embodiments provide a method for a global manager that manages a logical network spanning multiple physical sites. The method receives a set of data describing a modification to a definition of the logical network. From multiple queues that each correspond to a different one of the physical sites spanned by the logical network, the method identifying a set of queues to which to store the received set of data. The method stores the received set of data in the identified set of queues. From each respective queue in the identified set of queues, the method sends the set of data through a respective channel between the global manager and a respective local manager at the respective physical site corresponding to the respective queue.

Abstract: Various aspects are disclosed for distributed application management using an embedded persistent queue framework. In some aspects, task execution data is monitored from a plurality of task execution engines. A task request is identified. The task request can include a task and a Boolean predicate for task assignment. The task is assigned to a task execution engine embedded in a distributed application process if the Boolean predicate is true, and a capacity of the task execution engine is sufficient to execute the task. The task is enqueued in a persistent queue. The task is retrieved from the persistent queue and executed.

Abstract: Certain aspects described herein are directed to methods and systems for updating a status of a first process. In certain aspects, a method includes periodically examining a first table stored in a consistent and distributed database, wherein the table includes a first indication of a status of the first process and a second indication of a status of a second process and the first process and the second process are clustered for implementing functions of a distributed application in a distributed manner. The method further includes determining that the first indication has not changed during a first time period and changing the status of the first process in a second table to reflect that the first process is non-operational, wherein a different share of workload associated with the distributed application is performed by the second process as compared to when the first process was operational.

Abstract: Some embodiments provide a method for a global manager that manages a logical network configuration for multiple datacenters that each have a local manager for managing the logical network configuration within the datacenter. Based on detecting that a connection to a particular local manager of a particular datacenter has been restored after a period of unavailability, the method identifies a portion of the logical network configuration that is relevant to the particular datacenter. In a series of transactions, the method transfers the identified portion of the logical network configuration to the particular local manager. During the series of transactions, the method identifies modifications to the identified portion of the logical network configuration to be included in the series of transactions. Upon completion of the series of transactions, the method transfers a notification to the particular local manager indicating completion of the series of transactions.

Abstract: A log unit provides a shared log for recording updates on data objects. Garbage collection is performed locally and in-place by the log unit. In a marking portion of the garbage collection process, the log unit identifies and marks log entries that record supersedable updates. In a deallocation portion of the process, the log unit deallocates at least portions of the marked log entries that contain supersedable updates.

Abstract: Disclosed are a method and system for implementing a distributed persistent queue. To implement the queue, an API is defined that includes a command to enqueue an object onto the queue, to obtain the ordering of all objects in the queue and to remove any item from the queue. Enqueuing an object includes obtaining a unique ID for the object from a generator and then storing the object in a persistent shared log, which provides persistent storage for the queue. Obtaining an ordering of all objects in the persistent queue includes receiving for each object stored in the queue a pair of values, the first being the unique ID and the second being a sequence number from the persistent shared log. Regardless of the host computer system making the request for the ordering of objects in the queue the ordering provided to each host computer system is the same.

Abstract: Certain embodiments described herein are directed to methods and systems for adding one or more nodes to a first cluster including a first node in a computer system. A method performed by the first node comprises receiving a first request from a second node to join the first cluster. The method also comprises retrieving a first cluster configuration associated with the first cluster from a distributed database through a first database server (DBS) and creating a second cluster configuration using the first cluster configuration and information received from the second node as part of the request. The method further comprises populating a first one or more local trust stores of a first one or more processes executing on the first node with a second one or more security certificates of a second one or more processes executing on the second node. The method further comprises writing the second cluster configuration to the distributed database and returning the second cluster configuration to the second node.

Abstract: Certain embodiments described relate to methods and systems for adding one or more nodes to a first cluster including a first node in a computer system. A method performed by the first node comprises retrieving a first cluster configuration associated with the first cluster from a distributed database through a first database server (DBS) and creating a second cluster configuration using the first cluster configuration and information received from a second node as part of a request to join the first cluster. The method further comprises populating a first one or more local trust stores of a first one or more processes executing on the first node with a second one or more security certificates of a second one or more processes executing on the second node. The method further comprises writing the second cluster configuration to the distributed database and returning the second cluster configuration to the second node.

Abstract: Some embodiments provide a method for assigning different service path identifiers to each of a set of different service paths along each of which a same set of service operations are performed on a set of packets. The method retrieves an available service path identifier from an identifier storage that stores service path identifiers for multiple service paths. The method generates a set of proposed service path identifiers based on the retrieved available service path identifier and the number of service paths in the set of service paths. The method performs a write operation on the identifier storage based on the set of proposed service path identifiers, and based on a determination that the write operation was successful, assigns the set of proposed service path identifiers to the set of service paths. Based on the set of service path identifiers, the method forwards the set of packets along the set of service paths.

Abstract: A log unit provides a shared log for recording updates on data objects. Garbage collection is performed locally and in-place by the log unit. In a marking portion of the garbage collection process, the log unit identifies and marks log entries that record mergeable updates. In a deallocation portion of the process, the log unit merges one or more mergeable log entries and deallocates at least portions of the merged log entries.