Abstract: A system for transferring data from a Relational Database Management System (“RDBMS”) to a big data platform and methods for making and using the same. The system can acquire a partitioning execution scheme of a selected table from the RDBMS and submitting partitioned queries from the big data platform to each mapper of partitions. The partitioned queries are generated based on the partitioning execution scheme. The partitioning execution scheme can be acquired by submitting a query explain request to an optimizer of the RDBMS to generating a parallel query plan. The partitioning execution scheme can also be acquired by querying statistics from a statistics catalog of the RDBMS or by user inputs. The system can use RDBMS capabilities and statistics for parallel data fetching. Thereby, the system can increase efficiency of the fetching and can avoid straggling when target data is not evenly distributed and can avoid table query-in-serial.

Abstract: An integrated data processing system with two-tier data caching system and techniques for use thereof in a hybrid RDBMS and BDS computing environment are provided. In one aspect, the system is RDBMS-centric and uses two caches, one on the RDBMS side (1st tier) and the other on the BDS side (2nd tier). In another aspect, a DRDA wrapper on the BDS side enables the RDBMS to communicate with the BDS as if the BDS is another RDBMS. This is advantageous because the RDBMS already supports the DRDA protocol standard. In yet another aspect, the DRDA wrapper performs the data transformation needed when transferring cached objects between the RDBMS cache and BDS cache because RDBMS and BDS save data objects in different formats. This is advantageous because it offloads the computation from RDBMS to BDS therefore reducing the performance impact on RDBMS for its normal query and transaction processing.

Abstract: A system for transferring data from a Relational Database Management System (“RDBMS”) to a big data platform and methods for making and using the same. The system can acquire a partitioning execution scheme of a selected table from the RDBMS and submitting partitioned queries from the big data platform to each mapper of partitions. The partitioned queries are generated based on the partitioning execution scheme. The partitioning execution scheme can be acquired by submitting a query explain request to an optimizer of the RDBMS to generating a parallel query plan. The partitioning execution scheme can also be acquired by querying statistics from a statistics catalog of the RDBMS or by user inputs. The system can use RDBMS capabilities and statistics for parallel data fetching. Thereby, the system can increase efficiency of the fetching and can avoid straggling when target data is not evenly distributed and can avoid table query-in-serial.