Abstract: A database system includes a set of processing core resources (PCRs) operable to determine a short-term storage process for temporarily storing data of a dataset based on probability of high-volume use. As the data of the dataset is being received, first and second processing core resources process, per the short-term storage process, respective portion of the data of the dataset to produce sets of respective short-term storage data. The first and second processing core resources are further operable to store first sets of the sets of respective short-term storage data in a first memory device and to store second sets of the sets of respective short-term storage data in a second memory device. The set of PCRs is further operable to determine, per the short-term storage process, a condition for deleting at least some sets of the respective short-term storage data and when the condition is met, deleting such data from temporary storage.

Abstract: Within a database system, a computing node obtains a query that includes a training query operation regarding training of a machine learning model, identifies training data, and provides the training query operation and the training data to other computing nodes. Processing core resources (PCRs) of the computing nodes receive the training query operation. The PCRs receive, in a distributed manner, the sets of the training data. The PCRs execute, in substantial parallel, the training query operation on at least a portion of the machine learning model based on respective sub-sets of the sets of the training data to produce a plurality of partial training results. The computing node compiles the plurality of partial training results to produce a training result and, when the training result is favorable, update the machine learning model based on the training result.

Abstract: A node of a database system is operable to receive a plurality of data blocks from a set of source nodes via a set of relay nodes. At least one feedback data block to be sent to at least one of the set of source nodes is generated. Feedback relay node selection data selecting at least one of a plurality of nodes in the database system for facilitating routing of the at least one feedback data block from the node to the at least one of the set of source nodes based on a prioritized feedback communication mechanism and power utilization. The at least one feedback data block is sent to the at least one of the plurality of nodes selected in the feedback relay node selection data. The at least one feedback data block is routed from the node to the at least one of the set of source nodes via the prioritized feedback communication mechanism.

Abstract: A first set of processing core resources of a database system is operable to receive data of a dataset that includes rows of columnar data. The first set is further operable to, as sets of rows are received, store the sets of rows as pages of data in accordance with a page storage protocol. A second set of processing core resources of the database system is operable to, when a trigger condition regarding the storage of pages, obtain cluster key for the data of the dataset. The second set is further operable to organize the f pages based on the cluster key to produce groups of pages. The second set is further operable to generate raw segments from the groups of pages. The second set is further operable to generate, in accordance with an LTS protocol, LTS segments from the raw segments. The second set is further operable to store the LTS segments.

Abstract: A database system is operable to execute a first to generate the machine learning model from a training set of rows based on accessing and processing the training set of rows via a plurality of operators. system configuration data tracking a plurality of objects of a database system is updated to further track the machine learning model as a corresponding first object tracked via the system configuration data. A query output for a second query indicating applying of the machine learning model via execution of the second query based on applying the machine learning model to a set of rows in accordance with at least one property of the corresponding first object based on accessing the system configuration data.

Abstract: A set of computing devices of a database system is operable to receive a dataset that includes a plurality of data cells organized by rows and columns. The set is further operable to process the dataset to produce LTS data units. The set is further operable to store, in accordance with a storage model, the LTS data units in the distributed memory, where a first cell correlates to a first LTS data unit, and where the first LTS data unit is stored at one or more data block size addressable memory spaces of the data block size addressable memory spaces. The set is further operable to generate a file that records storing of the LTS data units within the distributed storage and that records correlation of the data cells to the LTS data units. The set is further operable to store the file for subsequent retrieval of data of the dataset.

Abstract: A computing device cluster of a database system generates a query operations for a query regarding data of a dataset. The query operations include a first set of query operations that cause hierarchical computing nodes to execute, in a wide parallelism mode, a first operation to produce a plurality of first outputs. Then, execute, in a narrow parallelism mode, another operation to produce a first output result. The query instructions further include a scatter operation that causes a computing node to divide the first output result into first output scattered data. The query instructions further includes a second set of query operations that cause the hierarchical computing nodes to receive, in the wide parallelism mode, the first scattered data. Then, execute, in the wide parallelism mode, a first operation of the second set to produce second outputs. Then, execute, in the narrow parallelism mode, another operation of the second set to produce a second output result.

Abstract: A database system is operable to generating a query operator execution flow that includes an optimized join tree implementing a plurality of join operations applied to a plurality of input row sets based on: identifying a set of filter predicates indicated by the query expression and generating, based on the plurality of input row sets and the set of filter predicates, cardinality data for each of the plurality of input row sets. The optimized join tree is generated based on generating the optimized join tree based on selecting an ordering for applying the plurality of join operations to the plurality of input row sets based on the cardinality data for the each of the plurality of input row sets. The query operator execution flow is executed in conjunction with executing the query expression based on executing the plurality of join operators of the optimized join tree.

Abstract: A database system operates by: receiving a set of records for storage; applying a replication factor dictated by the required fault tolerance level to generate a set of replicas of the set of records; temporarily storing the set of replicas in a temporary ingress storage system during a first temporal period in accordance with the required fault tolerance level; generating a plurality of redundancy-coded segments that encodes the set of record utilizing a redundancy storage coding scheme dictated by the required fault tolerance level; and migrating the set of records from the temporary ingress storage system to a long-term storage system by storing the set of segments in the long-term storage system and by removing the set of replicas from the temporary ingress storage system.

Abstract: A database system includes multiple computing device clusters having computing devices with computing nodes and processing core resources. A set of the processing core resources receives a query regarding a machine learning model stored as an object in memory of the database system, the query includes an object name that identifies the machine learning model as a non-standard data object and specifies a requested machine learning purpose. The system identifies data for use by the model to achieve the requested purpose, obtains a query plan that allocates computing resources for executing the model, and retrieves the model from memory based on the object name. The system executes the machine learning model using the identified data in accordance with the query plan to produce a machine learning result corresponding to the requested purpose.

Abstract: A database system includes a lead computing device of a computing device cluster of a plurality of computing device clusters operably coupled to determine that the computing device cluster is due for maintenance. The lead computing device further operable to determine a maintenance schedule of the computing device cluster based on a threshold number of computing nodes of the computing device cluster being available during maintenance, identify a computing device of the computing device cluster to take offline for the maintenance. For an upcoming offline maintenance period, reallocate tasks to be executed by computing nodes of the computing device to computing nodes of one or more other computing devices of the computing device cluster, transition the computing device to an offline maintenance mode after completion of current tasks in accordance with the maintenance schedule, and after completion of the offline maintenance period, transition the computing device to an available mode.

Abstract: A store and compute sub-system of a database system includes a computing cluster that is operable to receive a plurality of tasks. The computing cluster is further operable to execute, in a concurrent manner, the plurality of tasks. For a first task, a first lead computing device of the computing cluster is operable to: generate a plurality of first partial tasks based on the first task and allocate the plurality of first partial tasks to the plurality of computing devices, wherein the plurality of computing devices executes the plurality of first partial tasks. For a second task, a second lead computing device of the computing cluster is operable to: generate a plurality of second partial tasks based on the second task; and allocate the plurality of second partial tasks to the plurality of computing devices, wherein the plurality of computing device executes the plurality of second partial tasks.

Abstract: A database system includes a data ingest subsystem, a store and compute subsystem, and a query and response subsystem interconnected through a system communication network. Each subsystem includes a hierarchy of computing resources defined by an “a” number of computing clusters, a “b” number of computing entities per cluster providing an “a”*“b” number of computing entities, a “c” number of computing devices per entity providing an “a”*“b”*“c” number of computing devices, a “d” number of computing nodes per device providing an “a”*“b”*“c”*“d” number of computing nodes, and an “e” number of processing core resources per node providing an “a”*“b”*“c”*“d”*“e” number of processing core resources, wherein an asterisk (*) denotes multiplication. The network operably couples the subsystems to enable distributed data ingestion, storage, and query execution across scalable processing resources.

Abstract: A query and response sub-system of a database system, wherein a set of computing nodes of a set of computing devices of a set of computing device clusters is operable to: identify a memory intensive operation of a query regarding data of a dataset. The query and response sub-system is further operable to, when the memory intensive operation is a reorder operation, modify the reorder operation to enable reorder of a set of columnar data of the plurality of columnar data, wherein the modified reorder operation includes: an instruction to create new metadata regarding the sub-set of the packed column streams based on underlying memory layout of storage of the sub-set of the packed column streams.

Abstract: A query and response sub-system is operable to obtain a query and to generate an execution plan for a left inner join function of the query. The execution plan includes: identify left data of a column of the left table based on a reference column that is common to both tables; forward the rows of left data to memory allocated for a result table; identify right data of a column of the right table based on the reference column; temporarily store rows of the right data in memory allocated for the right table; identify rows of matching data from the rows of right data based on matching data of the reference column; add the rows of matching data to the result table; identify rows of non-matching data from the rows of right data based on data of the reference column; and add NULL to the rows of non-matching data to the result table.

Abstract: A method includes generating array field distribution data for an array field of a dataset. A query expression for execution that includes a query predicate indicating the array field of the dataset is received. The array field distribution data for the array field is utilized to generate query plan data for the query expression based on the query expression including the query predicate indicating the array field. Execution of the query expression is facilitated in accordance with the query plan data.

Abstract: A parallelized data input sub-system of a database system includes a first set of loader nodes of pluralities of computing nodes of a plurality of computing device clusters. The first set of loader nodes includes a plurality of memory devices and a plurality of processing modules. The first set of loader node is operable to ingest at least a portion of a dataset (“data”), a set of the memory devices stores the data, and a set of the processing modules determines whether the data is regarding a query. When it is regarding the query, the set of processing modules provides the data to a query and response sub-system. When it is not regarding the query, the set of processing modules determines long term storage parameters, processes the data in accordance with the parameters to produce formatted data, and provides the formatted data to a store and compute sub-system.

Abstract: A database system includes an analytics sub-system of an administrative sub-system and a parallelized query and results sub-system. The analytics sub-system includes a data management module operable to obtain and store user profile data related to end users of the database system, data provider profile data related to data providers of the database system, database usage data related to one or more current or past queries on the database system, and an analytics processing module operable to obtain query and results information from the parallelized query and results sub-system based on an analysis indication of a query, obtain analysis information from the data management module related to the query and results information, and compare the query and results information and the analysis information in light of the analysis indication to produce an analysis result.

Abstract: An object storage system is operable to store a plurality of records of a plurality of datasets via a plurality of objects. Configuration data mapping storage of the plurality of records of the plurality of datasets is stored via the plurality of objects. A request is received from a data processing system indicating filtering parameter data in accordance with object storage communication protocol data. A record identification pipeline is generated for execution based on the filtering parameter data and the configuration data. A filtered row set identifying a proper subset of the plurality of records meeting the filtering parameter data is generated based on executing the record identification pipeline by accessing at least one object of the plurality of objects. A response is sent to the data processing system that indicates the filtered row set in accordance with the object storage communication protocol data.

Abstract: A data input sub-system of a parallelized database system includes processing core resources. Data blocks of a first memory device of a first processing core resource correspond to a first set of logical data block addresses. The processing core resources are operable to obtain divisions of data slabs, compress the divisions of data slabs, and store a respective division of compressed data slabs. A first data slab of a first division of data slabs is mapped to at least a portion of the first set of logical data block addresses that includes at least a portion of a first set of fixed size data fields. The first data slab is compressed to produce a first compressed data slab and the first compressed data slab is mapped to a reduced amount of fixed size data fields of the 10 at least the portion of the first set of fixed size data fields.