Abstract: A database management system operates by: obtaining a dataset from a memory of the database management system, wherein the dataset includes a set of data records; determining a set of data characteristics of the dataset; determining a list of analytical calculations that are able to be executed on the dataset; determining an analytical calculation characteristic for each analytical calculation of the list of analytical calculations to produce a set of analytical calculation characteristics, wherein the analytical calculation characteristic indicates an estimated execution time to perform the analytical calculation; ranking each analytical calculation of the list of analytical calculations based on the set of data characteristics and the set of analytical calculation characteristics to produce a ranked list of analytical calculations; selecting an analytical calculation from the ranked list of analytical calculations based on the ranking; and executing the selected analytical calculation on the dataset to prod

Abstract: A method for managing data storage and retrieval and operated within a database management system includes determining whether a data confidence value of a data record that is stored or is to be stored in memory of the database management system is less than a data confidence threshold, where the data confidence value includes one or more of an estimate of an accuracy of data within the data record, an estimate of the accuracy of the data record, and an estimate of a reliability level of the data. When the data confidence value is less than the data confidence threshold, the method continues by generating a confidence data record based on the data record and the data confidence value. The method continues by storing the confidence data record in memory of the database management system.

Abstract: A networked database management system (DBMS) is disclosed. In particular, the disclosed DBMS includes a plurality of nodes, one of which is elected as a designated leader. The designated leader is responsible for maintaining an inventory of coding lines. A coding line spans all nodes in a cluster, and each node is assigned either data or parity. In addition, the designated leader maintains a pattern of parity rotation.

Abstract: Analyzing large datasets requires prioritization of analytical calculations to reduce analysis time and resource requirements. The prioritization process includes determining characteristics of a dataset, determining a list of analytical calculations, prioritizing the list of analytical calculations based on the dataset characteristics and characteristics of the analytical calculations, selecting the highest ranked analytical calculation, and applying the selected calculation on the dataset. The prioritization process of a new calculation based on a known set of calculations includes ranking the set of calculations based on their result scores from execution on a given dataset, comparing the new calculation to the list of known set of calculation, determining a similar calculation, and assigning the rank of the similar calculation to the new calculation.

Abstract: A massively parallel database management system includes an index store and a payload store including a set of storage systems of different temperatures. Both the stores each include a list of clusters. Each cluster includes a set of nodes with storage devices forming a group of segments. Nodes and clusters are connected over high speed links. The list of clusters within the payload store includes clusters of different temperatures. The payload store transitions data of a segment group from a higher temperature to a segment group in a lower temperature cluster in parallel. A node moves data of a segment in the higher temperature cluster to a corresponding node's segment in the lower temperature cluster. Once the data is written in the destination segment in the lower temperature cluster, the source segment is freed to store other data. The temperatures include blazing, hot, warm and cold.

Abstract: Data records and associated data confidence of data in each data record are both stored in a database system. Each data record includes a data confidence. The data confidence indicates an accuracy or reliability level of the data of corresponding data record. The data records with confidence information are constructed in memory before they are stored into the database system. When the data records are retrieved from the database for analysis, the data confidence is retrieved as well. The analysis of data contained in the data records further considers the data confidence. The analysis result is thus affected by the data confidence.

Abstract: A computing device includes a plurality of nodes, where a first node operates in accordance with a computing device operation system (OS) and remaining nodes operate in accordance with a custom OS. The remaining nodes include a plurality of sets of processing core resources that process a plurality of sets of threads of an application. The computing device also includes a main memory divided into a computing device memory section and a custom memory section that includes portions logically allocated as a plurality of buffers. The computing device also includes a memory access control module operable to coordinate access to the plurality of buffers by at least some of the plurality of sets of threads in accordance with the custom OS. The computing device also includes disk memory and a disk memory access control module operable to coordinate access to the disk memory in accordance with the computing device OS.

Abstract: A highly parallel database system includes multiple clusters of nodes. Each cluster includes multiple nodes. Each node includes a set of first tier threads and a set of second tier threads. Each second tier thread determines its own current load, and communicates the load status to a corresponding first tier thread. Each first tier thread checks the load status of each corresponding second tier thread when it allocates tasks between a number of corresponding second tier threads to achieve load balance between the second tier threads.

Abstract: A cluster node within a cluster of a highly parallel database system includes at least one processing unit that runs a set of first tier threads and a set of second tier threads, a storage disk drive, and a networking interface. When a first tier thread receives a task, it divides the task into a set of subtasks. The first tier thread also assigns the set of subtasks between a subset of the set of second tier threads for execution. Each second tier thread within the subset processes the one or more subtasks it is assigned to. When the task is a work, the subtasks are work units. When the task is a work unit, the subtasks are subwork units.

Abstract: Data records and associated data confidence of data in each data record are both stored in a database system. Each data record includes a data confidence. The data confidence indicates an accuracy or reliability level of the data of corresponding data record. The data records with confidence information are constructed in memory before they are stored into the database system. When the data records are retrieved from the database for analysis, the data confidence is retrieved as well. The analysis of data contained in the data records further considers the data confidence. The analysis result is thus affected by the data confidence.

Abstract: Analyzing large datasets requires prioritization of analytical calculations to reduce analysis time and resource requirements. The prioritization process includes determining characteristics of a dataset, determining a list of analytical calculations, prioritizing the list of analytical calculations based on the dataset characteristics and characteristics of the analytical calculations, selecting the highest ranked analytical calculation, and applying the selected calculation on the dataset. The prioritization process of a new calculation based on a known set of calculations includes ranking the set of calculations based on their result scores from execution on a given dataset, comparing the new calculation to the list of known set of calculation, determining a similar calculation, and assigning the rank of the similar calculation to the new calculation.

Abstract: A networked database management system (DBMS) and supporting infrastructure is disclosed. At least one application in the disclose DBMS can directly access a pinned RDMA buffer for network reads. In addition, an application can directly access pinned DMA buffer for drive reads. The nodes of the DBMS are configured in a particular configuration to aid in high speed accesses. In addition, all data is stored in register width fields, or integer multiples thereof. Finally, at least one application in the disclosed DBMS system includes a drive access class. The drive access class includes a NVME drive access subclass and a SATA drive access subclass. The NVME drive access subclass allows the application to directly access NVME drives without making an operating system call, while the SATA drive access subclass allows the application to directly access SATA drives without making an operating system call.

Abstract: A networked database management system (DBMS) and supporting infrastructure is disclosed. At least one node in the DBMS includes a processor having a preferred data alignment. An application operating on the processor has a preferred memory alignment. The application creates a data structure having multiple fields, each of which field exists on a boundary having the preferred memory alignment. This allows the application to access or write data into the fields with a lower processing overhead than if the fields were not forced to be on the preferred memory alignment.

Abstract: A massively parallel database management system includes an index store and a payload store including a set of storage systems of different temperatures. Both the stores each include a list of clusters. Each cluster includes a set of nodes with storage devices forming a group of segments. Nodes and clusters are connected over high speed links. The list of clusters within the payload store includes clusters of different temperatures. The payload store transitions data of a segment group from a higher temperature to a segment group in a lower temperature cluster in parallel. A node moves data of a segment in the higher temperature cluster to a corresponding node's segment in the lower temperature cluster. Once the data is written in the destination segment in the lower temperature cluster, the source segment is freed to store other data. The temperatures include blazing, hot, warm and cold.

Abstract: A massively parallel database management system includes an index store and a payload store including a set of storage systems of different temperatures. Both the index store and the storage system each include a list of clusters. Each cluster includes a set of nodes with storage devices forming a group of segments. Nodes and clusters are connected over high speed links. Each cluster receives data and splits the data into data rows based on a predetermined size. The data rows are randomly and evenly distributed between all nodes of the cluster.