Abstract: The invention is generally directed to systems and methods of automatically tuning big data workloads across various cloud platforms, the system being in communication with a cloud platform and a user, the cloud platform including data storage and a data engine. The system may include: a system information module in communication with the cloud platform; a static tuner in communication with the system information module; a cloud tuner in communication with the static tuner and the user; and an automation module in communication with the cloud tuner. Methods may include extracting information impacting or associated with the performance of the big data workload from the cloud platform; determining recommendations based at least in part on the information extracted; iterating through different hardware configurations to determine optimal hardware and data engine configuration; and applying the determined configuration to the data engine.

Abstract: The invention is directed to systems and methods for scheduling interactive database queries from multiple tenants onto distributed query processing clusters with service level agreements (SLAs). SLAs may be provided through a combination of estimation of resources per query followed by scheduling of that query onto a cluster if enough resources are available or triggering proactive autoscaling to spawn new clusters if they are not. In some embodiments systems may include a workflow manager; a resource estimator cluster; one or more execution clusters; and one or more metastores. A workflow manager may include an active node and a passive node configured to send a query to the resource estimator cluster and receive a resource estimate. A resource estimator cluster may be in communication with the workflow manager. One or more execution clusters may be scaled by the workflow manager as part of a schedule or autoscale based on workload.

Abstract: The present invention is generally directed to a caching framework that provides a common abstraction across one or more big data engines, comprising a cache filesystem including a cache filesystem interface used by applications to access cloud storage through a cache subsystem, the cache filesystem interface in communication with a big data engine extension and a cache manager; the big data engine extension, providing cluster information to the cache filesystem and working with the cache filesystem interface to determine which nodes cache which part of a file; and a cache manager for maintaining metadata about the cache, the metadata comprising the status of blocks for each file. The invention may provide common abstraction across big data engines that does not require changes to the setup of infrastructure or user workloads, allows sharing of cached data and caching only the parts of files that are required, can process columnar format.

Abstract: The invention is directed to systems and methods for scheduling interactive database queries from multiple tenants onto distributed query processing clusters with service level agreements (SLAs). SLAs may be provided through a combination of estimation of resources per query followed by scheduling of that query onto a cluster if enough resources are available or triggering proactive autoscaling to spawn new clusters if they are not. In some embodiments systems may include a workflow manager; a resource estimator cluster; one or more execution clusters; and one or more metastores. A workflow manager may include an active node and a passive node configured to send a query to the resource estimator cluster and receive a resource estimate. A resource estimator cluster may be in communication with the workflow manager. One or more execution clusters may be scaled by the workflow manager as part of a schedule or autoscale based on workload.

Abstract: As part of managing the loading of data from a source onto a database, according to an example, an interface through which a user is to define logic related to the loading of the data onto the database is provided. The user-defined logic pertains to at least one of a user-defined location identification of the source, a user-defined filter to be applied on the data, and a user-defined parsing operation to be performed on the data to convert the data into an appropriate format for the database. In addition, the user-defined logic is received and the user-defined logic is implemented to load the data onto the database.

Abstract: The invention is generally directed to systems and methods of automatically tuning big data workloads across various cloud platforms, the system being in communication with a cloud platform and a user, the cloud platform including data storage and a data engine. The system may include: a system information module in communication with the cloud platform; a static tuner in communication with the system information module; a cloud tuner in communication with the static tuner and the user; and an automation module in communication with the cloud tuner. Methods may include extracting information impacting or associated with the performance of the big data workload from the cloud platform; determining recommendations based at least in part on the information extracted; iterating through different hardware configurations to determine optimal hardware and data engine configuration; and applying the determined configuration to the data engine.

Abstract: The present invention is generally directed to a caching framework that provides a common abstraction across one or more big data engines, comprising a cache filesystem including a cache filesystem interface used by applications to access cloud storage through a cache subsystem, the cache filesystem interface in communication with a big data engine extension and a cache manager; the big data engine extension, providing cluster information to the cache filesystem and working with the cache filesystem interface to determine which nodes cache which part of a file; and a cache manager for maintaining metadata about the cache, the metadata comprising the status of blocks for each file. The invention may provide common abstraction across big data engines that does not require changes to the setup of infrastructure or user workloads, allows sharing of cached data and caching only the parts of files that are required, can process columnar format.

Abstract: In general, the present invention is directed to systems and corresponding methods for providing metadata aware background caching amongst various tables in data processing systems, the system configured to process either an original copy of data stored or data stored in derived tables in one or more data stores, the system including: a query optimization module, a catalog module, and a dataset manager. Each of the query optimization module, catalog module, and dataset manager may be communicatively connected to the original copy of data and the derived tables in one or more data stores. The query optimization module configured to conduct queries against data stored in the original copy of data or in the derived tables; the catalog module configured to register tables of data across various types and formats of data stores; and the dataset manager configured to maintain the freshness of the data in the derived tables.

Abstract: In a method for managing transfer of data from a source machine cluster to a destination machine cluster, information relevant to the transfer of data from the source machine cluster to the destination machine cluster is accessed. In addition, a data transfer operation that substantially optimizes the transfer of the data based upon the accessed information is determined. Furthermore, the determined data transfer operation is implemented to transfer the data from the source machine cluster to the destination machine cluster.

Abstract: A database management system (DBMS) client calls an interface of a DBMS server to execute a statement that has been prepared to be executed in batch form in relation to a database managed by the DBMS server. The DBMS client provides a different batch of data to the DBMS server each time the DBMS client calls the interface. The DBMS server processes the different batch of data in accordance with the statement that has been prepared, to effectuate execution of the statement at the DBMS server on a batch basis in relation to the different batch of data. The DBMS server returns results of processing the different batch of data to the DBMS client, such that the DBMS client receives intermediate feedback as to status of processing the statement prior to the statement being completely processed.

Abstract: As part of managing the loading of data from a source onto a database, according to an example, an interface through which a user is to define logic related to the loading of the data onto the database is provided. The user-defined logic pertains to at least one of a user-defined location identification of the source, a user-defined filter to be applied on the data, and a user-defined parsing operation to be performed on the data to convert the data into an appropriate format for the database. In addition, the user-defined logic is received and the user-defined logic is implemented to load the data onto the database.

Abstract: Methods, systems and program products for database storage. In one implementation, data of a projection of a database is stored at least partly in grouped ROS format and partly in column format based on patterns of updating the projection data. The projection data is updated so that the updated projection is stored partly in grouped ROS format and partly in column format.

Abstract: In a method for managing transfer of data from a source machine cluster to a destination machine cluster, information relevant to the transfer of data from the source machine cluster to the destination machine cluster is accessed. In addition, a data transfer operation that substantially optimizes the transfer of the data based upon the accessed information is determined. Furthermore, the determined data transfer operation is implemented to transfer the data from the source machine cluster to the destination machine cluster.

Abstract: A database management system (DBMS) client calls an interface of a DBMS server to execute a statement that has been prepared to be executed in batch form in relation to a database managed by the DBMS server. The DBMS client provides a different batch of data to the DBMS server each time the DBMS client calls the interface. The DBMS server processes the different batch of data in accordance with the statement that has been prepared, to effectuate execution of the statement at the DBMS server on a batch basis in relation to the different batch of data. The DBMS server returns results of processing the different batch of data to the DBMS client, such that the DBMS client receives intermediate feedback as to status of processing the statement prior to the statement being completely processed.

Abstract: Methods, systems and program products for database storage. In one implementation, data of a projection of a database is stored at least partly in grouped ROS format and partly in column format based on patterns of updating the projection data. The projection data is updated so that the updated projection is stored partly in grouped ROS format and partly in column format.