Abstract: A transfer segment group task processing module of a database system is operable to send a first begin segment group transfer instruction to a first storage cluster based on a determination to transfer a set of segments stored via the first storage cluster to a second storage cluster. A target first ownership sequence number is received from the first storage cluster. A second begin segment group transfer instruction to the second storage cluster. A transfer complete notification is received from the second storage cluster. A complete storage instruction is sent to the second storage cluster. An updated second ownership sequence number is received from the second storage cluster indicating updating of second data ownership information of the second storage cluster. A commit transfer instruction is sent to the first storage cluster based on updated second ownership sequence number received from the second storage cluster.

Abstract: A record processing and storage system is operable to generating a set of pages that includes corresponding records included in a plurality of row data via a plurality of processing resources based on each processing resource in the plurality of processing resources generating a corresponding subset of the set of pages. A single storage transaction is performed to store the set of pages based on writing the corresponding records included in the ones of the plurality of row data to a page storage system. Based on completing the single storage transaction, row durability data indicating a least favorably ordered row number of a plurality of row numbers corresponding to the plurality of row data is generated and transmitted to a computing device.

Abstract: A database system is operable to generate an IO pipeline that includes a primary cluster key pipeline element serially after a secondary index pipeline element. The IO pipeline is executed based on applying the secondary index pipeline element of the IO pipeline to generate row list output, and applying the primary cluster key pipeline element to determine a first set of row ranges, determine a second set of row ranges of row list output generated by the secondary index pipeline element, and generate, from the first set of row ranges and the second set of row ranges, a result set having a plurality of outputs, each indicating a cluster key and a row range for the cluster key.

Abstract: A record processing and storage system is operable to generate data source assignment data by segregating a plurality of data sources into a plurality of data source sets mapped to a corresponding plurality of storage clusters of the record processing and storage system. For each of the plurality of storage clusters, a corresponding subset of a plurality of pages is generated to include only ones of a plurality of records received from data sources included in a corresponding one of the plurality of data source sets, and the corresponding subset of the plurality of pages via storage resources of the each of the plurality of storage clusters. For each of the plurality of storage clusters, deduplication of each page included in the corresponding subset of the plurality of pages is facilitated based on metadata comparisons with other pages included in the corresponding subset of the plurality of pages.

Abstract: A database system is operable to assign a first node as a task monitoring node for a task and assign a second node as a task execution node for the task. The task is executed via the first node and the second node based on the first node sending a plurality of polls to the second node, the second node initiating execution of the task based on one of the plurality of polls, the second node sending a plurality of task status data to the first node, the first node maintaining current task status data for the task a shared metadata state based on the plurality of task status data, the second node completing execution of the task and caching task results in memory resources, the first node receiving the task results from the second node; and the second node removing the task results from the memory resources.

Abstract: A database system is operable to generate an addendum part indicating deletion of a set of rows of a plurality of rows of a dataset based on generating a row list based on identifying the set of rows for deletion from the dataset and compressing the row list in a set of compressed blocks written to disk memory resources. Based on determining to process the addendum part, the addendum part is read based on decompressing the row list based on access in the set of compressed block in the disk memory resources and emitting the row list for processing.

Abstract: A database system operates by: executing a query against a relational database table indicating a predicate applied to geospatial data of a geospatial data column, based on: applying an inner predicate to internal level nodes when traversing a set of internal levels of the each tree-based index structure to identify a first subset of leaf nodes in a plurality of leaf nodes of the each tree-based index structure based on identifying internal nodes having internal node bounding boxes meeting the inner predicate; and applying a leaf predicate to only leaf nodes included in the first subset of leaf nodes of the tree-based index structure to identify a second subset of leaf nodes of the first subset of leaf nodes corresponding to only leaf nodes of the first subset of leaf nodes having corresponding leaf node bounding boxes meeting the leaf predicate; wherein a query resultant of the query is generated based on geospatial objects of the geospatial data column for ones of a plurality of rows of the relational datab

Abstract: A transfer segment group task processing module of a database system is operable to send a first begin segment group transfer instruction to a first storage cluster based on a determination to transfer a set of segments stored via the first storage cluster to a second storage cluster. A target first ownership sequence number is received from the first storage cluster. A second begin segment group transfer instruction to the second storage cluster. A transfer complete notification is received from the second storage cluster. A complete storage instruction is sent to the second storage cluster. An updated second ownership sequence number is received from the second storage cluster indicating updating of second data ownership information of the second storage cluster. A commit transfer instruction is sent to the first storage cluster based on updated second ownership sequence number received from the second storage cluster.

Abstract: A database system operates by storing a plurality of array field values for an array field of a plurality of rows and generating index data for the array field based on: indexing non-null element values of the plurality of array fields for the plurality of rows, indexing null-valued ones of the plurality of array fields for the plurality of rows, indexing ones of the plurality of array fields for the plurality of rows having an empty set of elements, and indexing ones of the plurality of fields for the plurality of rows having at least one null element value. Wherein the database system further operates by: determining a query including a query predicate indicating an array operation for the array field; and applying an IO pipeline in conjunction with execution of the query.

Abstract: A record processing and storage system operates by: generating a set of pages from a plurality of row data via a plurality of processing core resources, wherein each processing core resource in the plurality of processing core resources generate a corresponding subset of the set of pages, independently from and in parallel with processing of other subsets of the set of pages via other ones of the plurality of processing core resources; facilitating performance of a single storage transaction to store the set of pages; identifying a page set interval based on a plurality of row number intervals of the set of pages; generating, based on completing the single storage transaction, row durability data indicating a least favorably ordered row number of a plurality of row numbers corresponding to the plurality of row data; and transmitting the row durability data to a computing device associated with the plurality of row data.

Abstract: A node of a database system is operable to determine, at a first time, to prepare for a shutdown. The shutdown of the node is performed at a second time that is a period of time after the first time based on the node delaying the shutdown until a plurality of currently running processes being run by the node that initiated prior to the first time are determined to be complete. During the period of time after the first time and prior to performing the shutdown, a set of new processing requests are rejected by the node.

Abstract: A method includes performing a search of an inverted index structure indexing values of a column to generate an in-range indexed value set by identifying all indexed values of the inverted index structure falling within a range corresponding to a range-based filter. A set of characteristics of the in-range indexed value set are identified based on performing the search of an inverted index structure. When the set of characteristics compare favorably to the set of index-usage requirements, output is generated based on performing a plurality of searches to the inverted index structure based on the in-range indexed value set. When the set of characteristics compare unfavorably to the set of index-usage requirements, the output is generated without performing any searches to the inverted index structure.

Abstract: A method includes generating a plurality of pages from a plurality of records received from a plurality of data sources. Deduplication of the plurality of pages is facilitated based on a plurality of page metadata of the plurality of pages based on, for the each page of the plurality of pages. A filtered set of potentially-intersecting pages is identified for each given page as a proper subset of the plurality of pages stored in the page storage system based on first comparison parameters, and an intersecting set of pages that include a row number intersection with the given page is identified as a proper subset of the filtered set of potentially-intersecting pages based on second comparison parameters. Records with records with row numbers included in row number intersections with other pages in the intersecting set of pages are removed from the each page.

Abstract: A database system operates by: generating a plurality of pages that include a plurality of records in accordance with a row-based format; storing the plurality of pages via a page storage system; generating segment generation determination data based on storage utilization data of the page storage system; generating a plurality of segments from the plurality of pages that include the plurality of records in a column-based format based on the segment generation determination data indicating segments be generated; storing the plurality of segments via a segment storage system; receiving a query; generating an operator execution flow for execution of the query that includes an IO operator, generating a modified operator execution flow for execution of the query by replacing the IO operator of the operator execution flow with a segment IO operator in parallel with a page IO operator, and facilitating execution of the query.

Abstract: A database system is operable to generate an initial query operator execution that includes an IO operator for execution serially before an aggregation operator indicating performance of an aggregation. The initial query operator execution flow is converted into an updated query operator execution flow semantically equivalent to the initial query operator execution flow based on updating the IO operator to include the performance of the aggregation and adding a re-aggregation operator serially after the IO operator. A corresponding query is executed by applying the updated query operator execution flow based on executing, via each of a plurality of parallelized resources, the IO operator generate corresponding sub-aggregation output via performance of the aggregation, and executing the re-aggregation operator upon data blocks generated based on execution of the IO operator across the plurality of parallelized resources.

Abstract: A node of a database system is operable to determine, at a first time, to prepare for a shutdown. The shutdown of the node is performed at a second time that is a period of time after the first time based on the node delaying the shutdown until a plurality of currently running processes being run by the node that initiated prior to the first time are determined to be complete. During the period of time after the first time and prior to performing the shutdown, a set of new processing requests are rejected by the node.

Abstract: A database system is operable to generate an initial query operator execution that includes an IO operator for execution serially before an aggregation operator indicating performance of an aggregation. The initial query operator execution flow is converted into an updated query operator execution flow semantically equivalent to the initial query operator execution flow based on updating the IO operator to include the performance of the aggregation and adding a re-aggregation operator serially after the IO operator. A corresponding query is executed by applying the updated query operator execution flow based on executing, via each of a plurality of parallelized resources, the IO operator generate corresponding sub-aggregation output via performance of the aggregation, and executing the re-aggregation operator upon data blocks generated based on execution of the IO operator across the plurality of parallelized resources.

Abstract: A database system is operable to store a plurality of column values for a first column of a plurality of rows. A query having a query predicate indicating a negation of a condition for the first column based on a first value is executed. A first index element is applied to identify a first proper subset of the plurality of rows having values for the first column meeting the condition based on accessing index data. At least one second index element is applied to identify a second proper subset of the plurality of rows having values for the first column meeting at least one missing data-based condition based on accessing the index data. A set difference between the plurality of rows and a union of the first proper subset and the second proper subset is identified. A query resultant for the query is generated based on the set difference.

Abstract: A database system is operable to write to a file buffer corresponding to geospatial index data for a plurality of rows based on processing each given row of the plurality of rows. A new leaf node of a set of leaf nodes in a temporary leaf node buffer when the given row includes a geospatial object. When the temporary leaf node buffer is determined to have a number of leaf nodes meeting a predetermined threshold number of leaf nodes, a new tree-based index structure of a set of tree-based index structures of the geospatial index data is built via processing the temporary leaf node buffer. The geospatial index data is stored based on writing the file buffer to disk memory resources. A query is executed against a relational database table based on accessing the geospatial index data in the disk memory resources.

Abstract: A database system operates to: receive, via a communication network, a record stream from a data source that includes a plurality of row data, wherein each of the plurality of row data includes a set of records for storage as relational database rows of at least one database table and further indicates a batch number corresponding to the set of records; add each of the plurality of row data to a pending row data pool; and perform parallelized generation of a plurality of pages from the plurality of row data to each include multiple records of the plurality of row data via a plurality of processing core resources based on each processing core resource in the plurality of processing core resources including power, processing a corresponding subset of the plurality of row data, independently from and in parallel with processing of other subsets of the plurality of row data via other ones of the plurality of processing core resources.