Abstract: A method for data redistribution of a job data in a first datanode (DN) to at least one additional DN in a Massively Parallel Processing (MPP) Database (DB) is provided. The method includes recording a snapshot of the job data, creating a first data portion in the first DN and a redistribution data portion in the first DN, collecting changes to a job data copy stored in a temporary table, and initiating transfer of the redistribution data portion to the at least one additional DN.

Abstract: A method for data redistribution of a job data in a first datanode (DN) to at least one additional DN in a Massively Parallel Processing (MPP) Database (DB) is provided. The method includes recording a snapshot of the job data, creating a first data portion in the first DN and a redistribution data portion in the first DN, collecting changes to a job data copy stored in a temporary table, and initiating transfer of the redistribution data portion to the at least one additional DN.

Abstract: Data messages having different priorities may be stored in different communication buffers of a network node. The data messages may then be forwarded from the communication buffers to working buffers as space becomes available in the working buffers. After being forwarded to the working buffers, the data messages may be available to be processed by upper-layer operations of the network node. Priorities may be assigned to the data messages based on a priority level of a query associated with the data messages, a priority level of an upper-layer operation assigned to process the data messages, or combinations thereof.

Abstract: Architecture that includes an index creation algorithm that utilizes available resources and dynamically adjusts to successfully scale with increased resources and be able to do so for any data distribution. The resources can be processing resources, memory, and/or input/output, for example. A finer level of granularity, called a segment, is utilized to process tuples in a partition while creating an index. The segment also aligns with compression techniques for the index. By choosing an appropriate size for a segment and using load balancing the overall time for index creation can be reduced. Each segment can then be processed by a single thread thereby limiting segment skew. Skew is further limited by breaking down the work done by a thread into parallelizable stages.

Abstract: A computer-implemented method of relocating data in a distributed database comprises: creating, by one or more processors, a second table in the distributed database, the second table including all columns from a first table; copying, by the one or more processors, a first set of tuples from the first table to the second table; modifying, by the one or more processors, during the copying of the first set of tuples, data of the first table according to a modification; after the copying of the first set of tuples, modifying, by the one or more processors, data of the second table according to the modification; and switching, by the one or more processors, the second table for the first table in a catalog of the distributed database.

Abstract: System and method for hybrid distribution mode in massively parallel processing (MPP) database preventing storage imbalance issues caused by data skew. Key values of the database are identified as outliers if records of those keys cause database skew. In hybrid mode, records having the outlier key values are distributed using a random distribution scheme. Other records are distributed using a hash distribution scheme. A threshold skew amount is configurable for the system. Record lookups, insertions, deletions, and updates are processed according to a query plan optimized for the distribution mode of the records referenced in a database query.

Abstract: A method for processing query fault, where a database server receives a query statement and generates a corresponding query plan tree including multiple layers of operators in a pipeline relationship, and each layer includes operation symbols having logical relationship with each other. The server executes the query statement according to the query plan tree, extracts intermediate status information of a faulty operator when a fault occurs in a process of executing the query statement, updates operation symbols of the faulty operator and a logical relationship among the operation symbols according to the query plan tree and the intermediate status information to obtain a reconstructed query plan tree, and continues to execute the query statement according to the reconstructed query plan tree after the fault is recovered.

Abstract: Architecture that includes an index creation algorithm that utilizes available resources and dynamically adjusts to successfully scale with increased resources and be able to do so for any data distribution. The resources can be processing resources, memory, and/or input/output, for example. A finer level of granularity, called a segment, is utilized to process tuples in a partition while creating an index. The segment also aligns with compression techniques for the index. By choosing an appropriate size for a segment and using load balancing the overall time for index creation can be reduced. Each segment can then be processed by a single thread thereby limiting segment skew. Skew is further limited by breaking down the work done by a thread into parallelizable stages.

Abstract: The disclosure relates to technology for estimating a number of samples satisfying a database query. One or more subsets from a sample dataset of a collection of all data are randomly drawn. The one or more subsets are queried to determine a number of cardinalities as training data. A prediction model based on the training data is then trained using machine learning or statistical methods, and a sample size satisfying the database query of the collection of all data is estimated using the trained prediction model.

Abstract: A method for cloning data samples in a data set based on statistic information of the data samples. The method does not use any of the data samples to perform the cloning. The statistic information includes a first set of statistic parameters obtained from a data matrix formed by data entries of the data samples based on Eckart-Young theorem, and a second set of statistic parameters indicating statistical properties of the data entries of the data samples. The data samples are reconstructed using the first and the second sets of statistic parameters based on Eckart-Young theorem.

Abstract: A computer-implemented method and system are provided, including executing an application programming interface (API) in a network switch to define at least one of one or more database functions, performing, using one or more processors, the one or more database functions on at least a portion of data contained in a data message received at the switch, to generate result data, and routing the result data to one or more destination nodes. A database function-defined network switch includes a network switch and one or more processors to perform a pre-defined database function on query data contained in data messages received at the switch, to produce result data, wherein the pre-defined database function is performed on the query data in a first mode of operation to a state of full completion, generating complete result data and no skipped query data, or to a state of partial completion, generating partially completed result data and skipped query data.

Abstract: A computer-implemented method and system at a network switch provides using one or more processors to perform a pre-defined database function on query data contained in data messages received at the network switch, with the performing producing result data, and wherein the pre-defined database function is performed on the query data in a first mode of operation to a state of full completion, generating complete result data and no skipped query data, and in a second mode of operation to a state of partial completion, generating partially complete result data and skipped query data. Further, the method and system performing one or more network switch functions to route the complete result data, and/or route the partially complete result data and skipped query data, to one or more destination nodes. In addition, an application programming interface (API) is used to define the database function.

Abstract: A data processing method and device are provided, where the method includes encoding data by using an arithmetic coding algorithm, so as to obtain a code value interval. When a code value corresponding to the data exists in the code value interval, obtaining the code value according to the code value interval. The method further includes comparing a quantity of bits of the code value with a quantity of bits of the data, so as to obtain a comparison result, and performing a storage operation according to the comparison result.

Abstract: In one embodiment, a method includes determining a number of initial servers in a massively parallel processing (MPP) database cluster and determining an initial bucket configuration of the MPP database cluster, where the initial bucket configuration has a number of initial buckets. The method also includes adding a number of additional servers to the MPP database cluster to produce a number of updated servers, where the updated servers include the initial servers and the additional servers and creating an updated bucket configuration in accordance with the number of initial servers, the initial bucket configuration, and the number of additional servers, where the updated bucket configuration has a number of updated buckets. Additionally, the method includes redistributing data of the MPP cluster in accordance with the updated bucket configuration.

Abstract: A method for data redistribution of a job data in a first datanode (DN) to at least one additional DN in a Massively Parallel Processing (MPP) Database (DB) is provided. The method includes recording a snapshot of the job data, creating a first data portion in the first DN and a redistribution data portion in the first DN, collecting changes to a job data copy stored in a temporary table, and initiating transfer of the redistribution data portion to the at least one additional DN.

Abstract: The present technology relates to managing data caching in processing nodes of a massively parallel processing (MPP) database system. A directory is maintained that includes a list and a storage location of the data pages in the MPP database system. Memory usage is monitored in processing nodes by exchanging memory usage information with each other. Each of the processing nodes manages a list and a corresponding amount of available memory in each of the processing nodes based on the memory usage information. Data pages are read from a memory of the processing nodes in response to receiving a request to fetch the data pages, and a remote memory manager is queried for available memory in each of the processing nodes in response to receiving the request. The data pages are distributed to the memory of the processing nodes having sufficient space available for storage during data processing.

Abstract: A data processing method and device are provided, where the method includes encoding data by using an arithmetic coding algorithm, so as to obtain a code value interval. When a code value corresponding to the data exists in the code value interval, obtaining the code value according to the code value interval. The method further includes comparing a quantity of bits of the code value with a quantity of bits of the data, so as to obtain a comparison result, and performing a storage operation according to the comparison result.

Abstract: Architecture that includes an index creation algorithm that utilizes available resources and dynamically adjusts to successfully scale with increased resources and be able to do so for any data distribution. The resources can be processing resources, memory, and/or input/output, for example. A finer level of granularity, called a segment, is utilized to process tuples in a partition while creating an index. The segment also aligns with compression techniques for the index. By choosing an appropriate size for a segment and using load balancing the overall time for index creation can be reduced. Each segment can then be processed by a single thread thereby limiting segment skew. Skew is further limited by breaking down the work done by a thread into parallelizable stages.

Abstract: System and method embodiments are provided for using different storage formats for a primary database and its replicas in a database managed replication (DMR) system. As such, the advantages of both formats can be combined with suitable design complexity and implementation. In an embodiment, data is arranged in a sequence of rows and stored in a first storage format at the primary database. The data arranged in the sequence of rows is also stored in a second storage format at the replica database. The sequence of rows is determined according to the first storage format or the second storage format. The first storage format is a row store (RS) and the second storage format is a column store (CS), or vice versa. In an embodiment, the sequence of rows is determined to improve compression efficiency at the CS.

Abstract: This disclosure provides a clustering storage method and apparatus. The method includes: storing to-be-stored first data row by row into a local memory in a database system; determining a first sorting column, where the first sorting column is used to sort data that has been cached in the local memory; sorting second data according to the first sorting column if the second data that has been cached in the local memory meets a preset condition, where the second data is data, which has been cached into the local memory, in the first data; and storing the sorted second data in a clustering manner into a storage medium in the database system.