Abstract: The present application provides a data processing method, a system, a computing device and a computer readable storage medium, wherein the data processing method is applied to a data processing node of a distributed data processing system and includes: receiving a plurality of data processing requests sent by a client, determining a target processing amount based on the number of the plurality of data processing requests, and limiting a flow of the plurality of data processing requests in accordance with the target processing amount to obtain a target data processing request; forwarding the target data processing request to a global transaction manager, receiving a processing result from processing the target data processing request by the global transaction manager, and returning the processing result to the client corresponding to each data processing request.

Abstract: Methods and computer systems for storing a query plan in a cache are provided. The method can include: determining a use frequency of the query plan; determining a target storage layer for storing the query plan based on the determined use frequency, wherein the cache includes at least two storage layers and each of the storage layers is associated with a compression level of query plans; processing the query plan according to the compression level of the target storage layer; and storing the processed query plan in the target storage layer.

Abstract: A method for dynamic filter pushdown for massive parallel processing databases on the cloud, including acquiring one or more filters corresponding to a query, acquiring statistics information of one or more database tables, determining a selectivity of the one or more database tables based on the statistics information, determining whether the selectivity satisfies a threshold condition, and pushing down the one or more filters to the one or more database tables based on the determination of whether the selectivity satisfies a threshold condition.

Abstract: Embodiments of the present disclosure provide methods and devices for formatting a database. The method can include clustering, into each block, a set of sequence of elements from each column of a database; generating a file storing one or more blocks having the clustered set of sequences of elements; providing a first table storing file schema information that associates the elements with the file; and providing a second table for storing block schema information that associates the columns with the one or more blocks.

Abstract: The present disclosure provides systems and methods for prefetching data in databases. One method for prefetching data in a database comprises receiving a database query on the database, determining one or more sets of adjacent columns access by the database query, and for each set of adjacent columns of the one or more determined sets, prefetching data in the adjacent columns.

Abstract: The present disclosure provides a method for generating a classification model in a database system. The method comprises generating a plurality of database tables by performing data clustering on a database, acquiring a plurality of queries associated with information from the database, generating training data by executing each of the plurality of queries on the plurality of database tables, and training a classification model using the training data, wherein the classification model is used to select a database table for a query based on corresponding performance information associated with the query on the database during executing of the query. The present disclosure also provides a method for executing a query on a database using a classification model. The method comprises obtaining the classification model, obtaining a database query, generating a target table by inputting the database query into the classification model, and executing the database query on the target table.

Abstract: The methods and computer systems for storing a query plan in a cache. The method can include: determining a use frequency of the query plan; determining a target storage layer for storing the query plan based on the determined use frequency, wherein the cache includes at least two storage layers and each of the storage layers is associated with a compression level of query plans; processing the query plan according to the compression level of the target storage layer; and storing the processed query plan in the target storage layer.

Abstract: Embodiments of the present disclosure provide methods and devices for inserting data in a database. The method can include caching one or more rows of data in a delta table in response to a request to insert the one or more rows in a destination table, wherein the destination table is one of a group of destination tables of a first database, and wherein the delta table contains requested changes of rows of the destination table, the changes being caused by a data operation; determining a priority of the destination table based on runtime information of the destination table relative to the group of destination tables; and scheduling the merging of the inserted rows from the delta table to the destination table based on the determined priority of the destination table. The destination table is in a format of a file storing one or more blocks, each block having a set of sequence of elements clustered from each column of the destination table.

Abstract: The present disclosure provides systems and methods for an indexing system in databases. One method for locating a row in a row-group columnar storage database comprises: receiving a query to locate a target row in the row-group columnar storage database that comprises rows of data divided into one or more blocks of data having columns of data that correspond to the rows of data, and one or more block information tables having column information corresponding to the columns of data in the one or more blocks of data, each block information table of the one or more block information tables being associated with a corresponding block of data of the one or more blocks of data; and locating the target row using the column information from the one or more block information tables.

Abstract: Embodiments of the present disclosure provide methods and devices for inserting data in a database. The method can include caching one or more rows of data in a delta table in response to a request to insert the one or more rows in a destination table, wherein the destination table is one of a group of destination tables of a first database, and wherein the delta table contains requested changes of rows of the destination table, the changes being caused by a data operation; determining a priority of the destination table based on runtime information of the destination table relative to the group of destination tables; and scheduling the merging of the inserted rows from the delta table to the destination table based on the determined priority of the destination table. The destination table is in a format of a file storing one or more blocks, each block having a set of sequence of elements clustered from each column of the destination table.

Abstract: The present disclosure provides systems and methods for prefetching data in databases. One method for prefetching data in a database comprises receiving a database query on the database, determining one or more sets of adjacent columns access by the database query, and for each set of adjacent columns of the one or more determined sets, prefetching data in the adjacent columns.

Abstract: Embodiments of the present disclosure provide methods and devices for formatting a database. The method can include clustering, into each block, a set of sequence of elements from each column of a database; generating a file storing one or more blocks having the clustered set of sequences of elements; providing a first table storing file schema information that associates the elements with the file; and providing a second table for storing block schema information that associates the columns with the one or more blocks.

Abstract: The present disclosure provides a method for compressing numeric data. The method comprises receiving a data set having a plurality of numeric values; for each numeric value of the plurality of numeric values of the data set, dividing a numeric value into a plurality of arrays arranged according to a specific location of the numeric value, wherein the plurality of arrays include a first array and a second array; grouping, across the plurality of numeric values, first arrays; grouping, across the plurality of numeric values, second arrays; and compressing the group of first arrays and the group of second arrays. The present disclosure also provides a method for decompressing numeric data. The method comprises receiving a data buffer comprising compressed numeric values; decompressing the compressed numeric values into groups of arrays; aligning the groups of arrays according to their relative positions from decimal points; and reconstructing numeric values according to the aligned groups of arrays.

Abstract: A User-Defined Function (UDF) provided by a user and accessed through a user query is identified. A wrapper for the UDF is generated. The wrapper represents a customized instance of the UDF. The wrapper, during execution, selectively calls the UDF provided by the user. Any query that references the UDF is rewritten to process the wrapper instead of the UDF.

Abstract: The present disclosure provides a method for generating a classification model in a database system. The method comprises generating a plurality of database tables by performing data clustering on a database, acquiring a plurality of queries associated with information from the database, generating training data by executing each of the plurality of queries on the plurality of database tables, and training a classification model using the training data, wherein the classification model is used to select a database table for a query based on corresponding performance information associated with the query on the database during executing of the query. The present disclosure also provides a method for executing a query on a database using a classification model. The method comprises obtaining the classification model, obtaining a database query, generating a target table by inputting the database query into the classification model, and executing the database query on the target table.

Abstract: Database values and their associated indicators can be arranged in multiple “buckets.” Adjacent buckets can be combined into a single bucket successively based one or more criteria associated with the indicators to effectively reduce the number of buckets until a desired number is reached.

Abstract: A method for dynamic filter pushdown for massive parallel processing databases on the cloud, including acquiring one or more filters corresponding to a query, acquiring statistics information of one or more database tables, determining a selectivity of the one or more database tables based on the statistics information, determining whether the selectivity satisfies a threshold condition, and pushing down the one or more filters to the one or more database tables based on the determination of whether the selectivity satisfies a threshold condition.

Abstract: Database values and their associated indicators can be arranged in multiple “buckets.” Adjacent buckets can be combined into a single bucket successively based one or more criteria associated with the indicators to effectively reduce the number of buckets until a desired number is reached.

Abstract: A User-Defined Function (UDF) provided by a user and accessed through a user query is identified. A wrapper for the UDF is generated. The wrapper represents a customized instance of the UDF. The wrapper, during execution, selectively calls the UDF provided by the user. Any query that references the UDF is rewritten to process the wrapper instead of the UDF.

Abstract: A data store system may include a persistent storage device that may store a plurality of data tables. The data store system may further include a processor array in communication with the persistent storage device. The processor array may receive a query comprising a call to a user-defined function (UDF) that may determine a plurality of aggregates for at least one data table from the plurality of data tables. The processor array may determine a number of the plurality of aggregates in the UDF. The processor array may perform a single read of rows associated with the plurality aggregates from the at least one data table. At least one row of the rows may be commonly associated with more than one of the plurality of aggregates. The processor array may generate each aggregate of the plurality of aggregates using the rows according to the UDF. A method and computer-readable medium may also be implemented.