Abstract: A networked database management system (DBMS) is disclosed. In particular, the disclosed DBMS includes a plurality of nodes, one of which is elected as a designated leader. The designated leader is elected using a consensus algorithm, such as tabulated random votes, RAFT or PAXOS. The designated leader is responsible for managing open coding lines, and determining when to close an open coding line.

Abstract: A method includes generating, by a processing entity of a computing system, a plurality of parity blocks from a plurality of lines of data blocks. A first number of parity blocks of the plurality of parity blocks is generated from a first line of data blocks of the plurality of lines of data blocks. The method further includes storing, by the processing entity, the plurality of lines of data blocks in data sections of memory of a cluster of computing devices of the computing system in accordance with a read/write balancing pattern and a restricted file system. The method further includes storing, by the processing entity, the plurality of parity blocks in parity sections of memory of the cluster of computing devices in accordance with the read/write balancing pattern and the restricted file system.

Abstract: A networked database management system (DBMS) is disclosed. In particular, the disclosed DBMS includes a plurality of nodes, one of which is elected as a designated leader. The designated leader is elected using a consensus algorithm, such as tabulated random votes, RAFT or PAXOS. The designated leader is responsible for managing open coding lines, and determining when to close an open coding line.

Abstract: A method includes generating, by a processing entity of a computing system, a plurality of parity blocks from a plurality of lines of data blocks. A first number of parity blocks of the plurality of parity blocks is generated from a first line of data blocks of the plurality of lines of data blocks. The method further includes storing, by the processing entity, the plurality of lines of data blocks in data sections of memory of a cluster of computing devices of the computing system in accordance with a read/write balancing pattern and a restricted file system. The method further includes storing, by the processing entity, the plurality of parity blocks in parity sections of memory of the cluster of computing devices in accordance with the read/write balancing pattern and the restricted file system.

Abstract: A method for execution by a virtual machine core includes retrieving a first pointer by accessing a first buffer of a plurality of buffers stored in allocated memory of a main memory based on assignment of the virtual machine core as a single consumer of the first buffer. First intermediate data in the allocated memory is accessed by utilizing the first pointer. Second intermediate data is generated by executing one of an ordered set of operations on the first intermediate data. The second intermediate data is written to the allocated memory. A second pointer is written to a second buffer of the plurality of buffers based on assignment of the virtual machine core as a single producer of the second buffer.

Abstract: A large highly parallel database management system includes thousands of nodes storing huge volume of data. The database management system includes multiple query optimizers for determining low cost execution plans for queries. The database management system is adapted to receive a data query. An execution plan generator component of the database management system generates an initial execution plan for the query. The initial execution plan is fed as input to more than one query optimizers. Each optimizer starts with the initial execution plan, generates alternative execution plans, and determines a satisfactory execution plan that incurs the lowest cost. The database management system compares the selected execution plans by the optimizers and selects one with the lowest cost. The multiple query optimizers run in parallel.

Abstract: A database management system operates by: obtaining a dataset from a memory of the database management system, wherein the dataset includes a set of data records; determining a set of data characteristics of the dataset; determining a list of analytical calculations that are able to be executed on the dataset; determining an analytical calculation characteristic for each analytical calculation of the list of analytical calculations to produce a set of analytical calculation characteristics, wherein the analytical calculation characteristic indicates an estimated execution time to perform the analytical calculation; ranking each analytical calculation of the list of analytical calculations based on the set of data characteristics and the set of analytical calculation characteristics to produce a ranked list of analytical calculations; selecting an analytical calculation from the ranked list of analytical calculations based on the ranking; and executing the selected analytical calculation on the dataset to prod

Abstract: A massively parallel database management system includes an index store and a payload store including a set of storage systems of different temperatures. Both the stores each include a list of clusters. Each cluster includes a set of nodes with storage devices forming a group of segments. Nodes and clusters are connected over high speed links. The list of clusters within the payload store includes clusters of different temperatures. The payload store transitions data of a segment group from a higher temperature to a segment group in a lower temperature cluster in parallel. A node moves data of a segment in the higher temperature cluster to a corresponding node's segment in the lower temperature cluster. Once the data is written in the destination segment in the lower temperature cluster, the source segment is freed to store other data. The temperatures include blazing, hot, warm and cold.

Abstract: A method for managing data storage and retrieval and operated within a database management system includes determining whether a data confidence value of a data record that is stored or is to be stored in memory of the database management system is less than a data confidence threshold, where the data confidence value includes one or more of an estimate of an accuracy of data within the data record, an estimate of the accuracy of the data record, and an estimate of a reliability level of the data. When the data confidence value is less than the data confidence threshold, the method continues by generating a confidence data record based on the data record and the data confidence value. The method continues by storing the confidence data record in memory of the database management system.

Abstract: A networked database management system (DBMS) is disclosed. In particular, the disclosed DBMS includes a plurality of nodes, one of which is elected as a designated leader. The designated leader is responsible for maintaining an inventory of coding lines. A coding line spans all nodes in a cluster, and each node is assigned either data or parity. In addition, the designated leader maintains a pattern of parity rotation.

Abstract: A large highly parallel database management system includes thousands of nodes storing huge volume of data. The database management system includes a query optimizer for optimizing data queries. The optimizer estimates the column cardinality of a set of rows based on estimated column cardinalities of disjoint subsets of the set of rows. For a particular column, the actual column cardinality of the set of rows is the sum of the actual column cardinalities of the two subsets of rows. The optimizer creates two respective Bloom filters from the two subsets, and then combines them to create a combined Bloom filter using logical OR operations. The actual column cardinality of the set of rows is estimated using a computation from the combined Bloom filter.

Abstract: Analyzing large datasets requires prioritization of analytical calculations to reduce analysis time and resource requirements. The prioritization process includes determining characteristics of a dataset, determining a list of analytical calculations, prioritizing the list of analytical calculations based on the dataset characteristics and characteristics of the analytical calculations, selecting the highest ranked analytical calculation, and applying the selected calculation on the dataset. The prioritization process of a new calculation based on a known set of calculations includes ranking the set of calculations based on their result scores from execution on a given dataset, comparing the new calculation to the list of known set of calculation, determining a similar calculation, and assigning the rank of the similar calculation to the new calculation.

Abstract: A large highly parallel database management system includes thousands of nodes storing huge volume of data. The database management system includes multiple query optimizers for determining low cost execution plans for queries. The database management system is adapted to receive a data query. An execution plan generator component of the database management system generates an initial execution plan for the query. The initial execution plan is fed as input to more than one query optimizers. Each optimizer starts with the initial execution plan, generates alternative execution plans, and determines a satisfactory execution plan that incurs the lowest cost. The database management system compares the selected execution plans by the optimizers and selects one with the lowest cost. The multiple query optimizers run in parallel.

Abstract: A massively parallel database management system includes an index store and a payload store including a set of storage systems of different temperatures. Both the stores each include a list of clusters. Each cluster includes a set of nodes with storage devices forming a group of segments. Nodes and clusters are connected over high speed links. The list of clusters within the payload store includes clusters of different temperatures. The payload store transitions data of a segment group from a higher temperature to a segment group in a lower temperature cluster in parallel. A node moves data of a segment in the higher temperature cluster to a corresponding node's segment in the lower temperature cluster. Once the data is written in the destination segment in the lower temperature cluster, the source segment is freed to store other data. The temperatures include blazing, hot, warm and cold.

Abstract: A method includes identifying, by a processing entity of a computing device, data units to read from non-volatile memory and to write into ordered buffers of volatile memory. The method further includes generating, by the processing entity, read operations regarding the data units, wherein the number of read operations equals “n”. The method further includes tagging, by the processing entity, each read operation of the read operations with a unique ordered tag value. The method further includes receiving, by the processing entity, read responses to the read operations from the non-volatile memory. The method further includes writing, by the processing entity, data units contained in the read responses into the ordered buffers in accordance with the ordered tag values. The method further includes tracking, by the processing entity, consumption of the data units from the ordered buffers.

Abstract: Data records and associated data confidence of data in each data record are both stored in a database system. Each data record includes a data confidence. The data confidence indicates an accuracy or reliability level of the data of corresponding data record. The data records with confidence information are constructed in memory before they are stored into the database system. When the data records are retrieved from the database for analysis, the data confidence is retrieved as well. The analysis of data contained in the data records further considers the data confidence. The analysis result is thus affected by the data confidence.

Abstract: A computing device includes a plurality of nodes, where a first node operates in accordance with a computing device operation system (OS) and remaining nodes operate in accordance with a custom OS. The remaining nodes include a plurality of sets of processing core resources that process a plurality of sets of threads of an application. The computing device also includes a main memory divided into a computing device memory section and a custom memory section that includes portions logically allocated as a plurality of buffers. The computing device also includes a memory access control module operable to coordinate access to the plurality of buffers by at least some of the plurality of sets of threads in accordance with the custom OS. The computing device also includes disk memory and a disk memory access control module operable to coordinate access to the disk memory in accordance with the computing device OS.

Abstract: A method includes receiving, by a host computing device of a storage cluster of computing devices, a segment group of data. The method further includes processing, by the host computing device, the segment group of data to produce data segments. The method further includes, sending, by the host computing device, the data segments to the computing devices of the storage cluster. The method further includes allocating, by a host node of the first computing device, data segment divisions of the first data segment to nodes of the first computing device. The method further includes allocating, by a host processing core resource of the first node, data segment sub-divisions of the first data segment division to processing core resources of the first node. The method further includes storing, by the first computing device, the first data segment having the data segment divisions and the first data segment division having the data segment sub-divisions.

Abstract: A method includes receiving, by a first computing entity of a database system, data that is organized in rows and columns. The method further includes determining, by the first computing entity, one or more key columns from the columns based on a desired sort criteria for the data. The method further includes sorting, by the first computing entity, other columns of the columns based on the one or more key columns to produce sorted other columns. The method further includes sending, by the first computing entity, the one or more key columns to a second computing entity of the database system for storage in a first storage location associated with the second computing entity. The method further includes sending, by the first computing entity, the sorted other columns to the second computing entity for storage in a second storage location associated with the second computing entity.

Abstract: A data input sub-system of a large scale application specific computing system receives a data set that includes a plurality of records, each with a plurality of data fields, and divides the data set into a plurality of data segments. The data input sub-system further restructures records of data segments based on a key field of the plurality of data fields to produce restructured data segments and generates storage instructions for storing the restructured data segments. A data storage and processing sub-system of the computing system interprets the storage instructions to determine resources to engage and stores the restructured data segments using engaged resources. A query and results sub-system of the computing system generates an initial query plan based on a data processing request, optimizes the initial query plan to produce an optimized query plan, and sends the optimized query plan to the data storage and processing sub-system for execution.