Abstract: In one embodiment, a method for adding partitions to a massively parallel processing (MPP) cluster includes determining whether a first number of available nodes of a first leaf sub-cluster (LSC) of a meta sub-cluster (MSC) of the MPP cluster is greater than or equal to a second number of partitions of a table and assigning a first node of the first LSC to a first partition when the first number of available nodes is greater than or equal to the second number of partitions. The method also includes searching for a second LSC in the MSC when the first number of available nodes is less than the second number of partitions.

Abstract: System and method embodiments are provided to implement highly scalable and high availability (HA) clusters in massively parallel processing (MPP) systems. The embodiments include a method to build a highly scalable MPP HA cluster, which provides HA to the cluster while allowing it to scale to relatively larger number of nodes. An embodiment apparatus includes a plurality of data processing nodes distributed in a plurality of corresponding sub-clusters and configured to exchange heart-beat messages between each other within limit of each of the corresponding sub-clusters to maintain sub-cluster membership integrity and detect failures in the corresponding sub-clusters. The sub-clusters are arranged in a fan-out tree hierarchy and configured to prevent heart-beat messaging between each other.

Abstract: In one embodiment, a method for adding partitions to a massively parallel processing (MPP) cluster includes determining whether a first number of available nodes of a first leaf sub-cluster (LSC) of a meta sub-cluster (MSC) of the MPP cluster is greater than or equal to a second number of partitions of a table and assigning a first node of the first LSC to a first partition when the first number of available nodes is greater than or equal to the second number of partitions. The method also includes searching for a second LSC in the MSC when the first number of available nodes is less than the second number of partitions.

Abstract: A method for performing a transaction in a massively parallel processing (MPP) database includes receiving, by a transaction manager, from a first resource manager, a first snapshot of the first resource manager and receiving, by the transaction manager, from a second resource manager, a second snapshot of the second resource manager. Also, the method includes reconciling the first snapshot of the first resource manager and the second snapshot of the second resource manager to generate a reconciled snapshot and transmitting, by the transaction manager, to the first resource manager and the second resource manager, the reconciled snapshot. Additionally, the method includes performing a transaction using the reconciled snapshot.

Abstract: An embodiment method for massively parallel processing includes initiating a management instance on an initial machine, the management instance generating an initial partition corresponding to the initial machine, determining a total number of partitions desired for processing a database, the total number of partitions including the initial partition, determining a number of additional machines available to process the database, grouping the initial machine and the additional machines together in a pod, and launching the management instance on the additional machines in the pod to generate the total number of partitions desired for the database. Additional embodiment methods and an embodiment system operable to perform such methods are also disclosed.

Abstract: An embodiment method for massively parallel processing includes assigning a primary key to a first table in a database and a foreign key to a second table in the database, the foreign key of the second table identical to the primary key of the first table, determining a number of partition groups desired for the database, partitioning the first table into first partitions based on the primary key assigned and the number of partition groups desired, partitioning the second table into second partitions based on the foreign key assigned and the number of partition groups desired, and distributing the first partitions and the second partitions to the partition groups as partitioned. An embodiment system for implementing the embodiment methods is also disclosed.

Abstract: A distributed database system providing data and space management methodology. In one embodiment a system for allocating resources amongst a plurality of database servers sharing resources comprises: an input module for receiving user input for allocating resources shared amongst a plurality of database servers and establishing thresholds for utilization of resources at each database server; a monitoring module at each database server for monitoring local resource utilization at each database server and sending a request for balancing resources to a second database server when local resource utilization reaches an established threshold at a first database server; and a load balancing module for receiving the request for balancing resources and determining whether to adjust resources allocated to the first database server and the second database server in response to the request.

Abstract: Embodiments are provided herein for an efficient database storage model, which utilizes sparse file features to efficiently store and retrieve data. The embodiments provide database algorithms that utilize the file system abstraction layer to hide the complexity of managing disk space while providing the database a linear and contiguous logical address space for holding multiple database objects. An embodiment method includes pre-allocating, in a logical sparse file, a plurality of segments fixed in size and contiguous at fixed offsets. Upon receiving a command to write database objects to the segments, the database objects are mapped to the segments in a database catalog. The method further includes interfacing with a file system to initialize storage medium space for writing the data objects to the segments at the fixed offsets.

Abstract: Embodiments are provided to provide a distributed transaction manager supporting repeatable read isolation level in Massively Parallel Processing (MPP) database systems without a centralized component. Before starting a transaction, a first node identifies a second node involved in the transaction, and requests from the second node a snapshot of current transactions at the second node. After receiving the snapshot from the second node, the first node combines into a reconciled snapshot the snapshot of transactions from the second node with current transactions at the first node. The first node then transmits the reconciled snapshot to the second node and starts the transaction using the reconciled snapshot. A branch transaction is then started at the second node in accordance with the reconciled snapshot. Upon ending the transaction and the branch transaction, the first node and the second node perform a two phase commit (2PC) protocol.

Abstract: System and method embodiments are provided to implement highly scalable and high availability (HA) clusters in massively parallel processing (MPP) systems. The embodiments include a method to build a highly scalable MPP HA cluster, which provides HA to the cluster while allowing it to scale to relatively larger number of nodes. An embodiment apparatus includes a plurality of data processing nodes distributed in a plurality of corresponding sub-clusters and configured to exchange heart-beat messages between each other within limit of each of the corresponding sub-clusters to maintain sub-cluster membership integrity and detect failures in the corresponding sub-clusters. The sub-clusters are arranged in a fan-out tree hierarchy and configured to prevent heart-beat messaging between each other.

Abstract: A method for performing a transaction in a massively parallel processing (MPP) database includes receiving, by a transaction manager, from a first resource manager, a first snapshot of the first resource manager and receiving, by the transaction manager, from a second resource manager, a second snapshot of the second resource manager. Also, the method includes reconciling the first snapshot of the first resource manager and the second snapshot of the second resource manager to generate a reconciled snapshot and transmitting, by the transaction manager, to the first resource manager and the second resource manager, the reconciled snapshot. Additionally, the method includes performing a transaction using the reconciled snapshot.

Abstract: A massively parallel processing (MPP) database can be re-partitioned/re-balanced while remaining on-line through a staged migration procedure. Staged migration may include a first stage and a second stage. During the first stage, entries in an existing partition are reallocated to the new partition, and the catalog is updated to associate the re-allocated entries with both the existing partition and the new partition such that queries for the re-allocated entries are directed toward the existing partition and the new partition. During the second stage, the re-allocated entries are migrated from the existing partition to the new partition, and after the migration is complete, the catalog is re-updated to associate the migrated entries with the new partition such that new queries are directed toward the new partition.

Abstract: A massively parallel processing (MPP) database can be re-partitioned/re-balanced while remaining on-line through a staged migration procedure. Staged migration may include a first stage and a second stage. During the first stage, entries in an existing partition are reallocated to the new partition, and the catalog is updated to associate the re-allocated entries with both the existing partition and the new partition such that queries for the re-allocated entries are directed toward the existing partition and the new partition. During the second stage, the re-allocated entries are migrated from the existing partition to the new partition, and after the migration is complete, the catalog is re-updated to associate the migrated entries with the new partition such that new queries are directed toward the new partition.

Abstract: An embodiment method for massively parallel processing includes assigning a primary key to a first table in a database and a foreign key to a second table in the database, the foreign key of the second table identical to the primary key of the first table, determining a number of partition groups desired for the database, partitioning the first table into first partitions based on the primary key assigned and the number of partition groups desired, partitioning the second table into second partitions based on the foreign key assigned and the number of partition groups desired, and distributing the first partitions and the second partitions to the partition groups as partitioned. An embodiment system for implementing the embodiment methods is also disclosed.

Abstract: An embodiment method for massively parallel processing includes initiating a management instance on an initial machine, the management instance generating an initial partition corresponding to the initial machine, determining a total number of partitions desired for processing a database, the total number of partitions including the initial partition, determining a number of additional machines available to process the database, grouping the initial machine and the additional machines together in a pod, and launching the management instance on the additional machines in the pod to generate the total number of partitions desired for the database. Additional embodiment methods and an embodiment system operable to perform such methods are also disclosed.

Abstract: System and method for assignment of unique identifiers in a distributed environment. In a distributed system having a plurality of nodes, a method for allocating identifiers comprises steps of: allocating a pool of identifiers for use; maintaining lists of free identifiers in the pool at participating nodes; obtaining at a first node permission to update the lists of free identifiers; upon receiving permission to update the lists, allocating for the first node a set of identifiers from the lists; updating the lists of free identifiers to reflect allocation of the set of identifiers for the first node; sending the updated lists from the first node to other participating nodes; upon receiving the updated lists of free identifiers at each other participating node, updating each other participating node's respective copy of the lists of free identifiers; and relinquishing the first node's permission to update the lists of free identifiers.

Abstract: A distributed database system providing data and space management methodology. In a distributed database system having an allocation page for tracking allocation and deallocation of data pages including a deallocation bit for tracking data page deallocation, a method for determining when to clear the deallocation bit during recovery comprises: during runtime operation, generating log records recording allocation and deallocation of data pages, setting the deallocation bit during deallocation of a data page by a transaction, and clearing the deallocation bit after the transaction has committed; during recovery following a failure, identifying a particular log record corresponding to the deallocation bit set during deallocation of a given data page; determining whether the deallocation of the given data page recorded in the particular log record committed prior to the failure; and if the deallocation of the given data page committed prior to the failure, clearing the deallocation bit.

Abstract: A distributed database system providing data and space management methodology. In a distributed system consisting a plurality of servers sharing access to data pages, a method for setting a timestamp indicating when modifications to data pages are made in a non-regressing fashion includes steps of: maintaining a local sequence at each server and an allocation page timestamp on an allocation page; upon allocation of a data page, applying a timestamp to the data page which is greater than the allocation page timestamp; upon modification of a given data page at a particular server, setting the timestamp on the given data page to the greater of the local sequence at the particular server and an existing timestamp already on the given data page; and when data pages are deallocated, adjusting the allocation page timestamp to a value greater than the timestamp value on any deallocated data page.

Abstract: A distributed database system providing data and space management methodology. In one embodiment, for example, a method for transferring a data structure in cache at a first database server to a second database server in a distributed database system comprises steps of: determining a first database server having a data structure in cache in response to a request for the data structure from a second database server, the request including a request for a lock on the data structure; providing the request for the data structure to the first database server; in response, sending the data structure from the first database server to the second database server; and receiving and using the data structure at the second database server without waiting for the lock request to be explicitly granted.

Abstract: A distributed database system providing data and space management methodology. In a distributed database system having an allocation page for tracking allocation and deallocation of data pages including a deallocation bit for tracking data page deallocation, a method for determining when to clear the deallocation bit during recovery comprises: during runtime operation, generating log records recording allocation and deallocation of data pages, setting the deallocation bit during deallocation of a data page by a transaction, and clearing the deallocation bit after the transaction has committed; during recovery following a failure, identifying a particular log record corresponding to the deallocation bit set during deallocation of a given data page; determining whether the deallocation of the given data page recorded in the particular log record committed prior to the failure; and if the deallocation of the given data page committed prior to the failure, clearing the deallocation bit.