Abstract: A query optimization system is described that, at runtime, optimizes the execution pipeline generated for a query. Based upon communications between nodes in the execution pipeline, the execution pipeline generated for a query is optimized by modifying the execution pipeline to create a modified execution pipeline. The modified execution pipeline is then executed to execute the query and results obtained for the query. The changes or modifications made to an execution pipeline may include changing the capabilities (e.g., changes to inputs and/or outputs of a node, changing the task(s) or function(s) performed by the node) of one or more nodes within the execution pipeline. The changes may include changing the position of one or more nodes within a directed acyclic graph representing the execution pipeline.

Abstract: Optimized techniques are disclosed for sorting variable-length records using an optimized amount of memory while maintaining good locality of references. The amount of memory required for sorting the variable length records is optimized by reusing some of the memory used for storing the variable length records being sorted. Pairs of input runs storing variable length records may be merged into a merged run that contains the records in a sorted order by incrementally scanning, sorting, and copying the records from the two input runs being merged into memory pages of the merged run. When all the records of a memory page of an input run have been processed or copied to the merged run, that memory page can be emptied and released to a cache of empty memory pages. Memory pages available from the cache of empty memory pages can then be used for generating the merged run.

Abstract: Innovative techniques are disclosed for performing a combined sort and aggregation operation involving eagerly performing aggregation while sorting. The techniques described herein enable an aggregation and a group-by operation to be performed using an amount of system memory that is far less than the system memory required to store all the data records being processed while minimizing the need to spill data to disk. This combined sort and aggregation operation has better performance than conventional techniques since system memory is used more efficiently. In certain embodiments, a combined sort and aggregation operation is disclosed that enables the efficient sorting and aggregation of data records when the desired aggregation function is composable (such as SUM, COUNT, MIN and MAX aggregate operations).

Abstract: A query optimization system is described that, at runtime, optimizes the execution pipeline generated for a query. Based upon communications between nodes in the execution pipeline, the execution pipeline generated for a query is optimized by modifying the execution pipeline to create a modified execution pipeline. The modified execution pipeline is then executed to execute the query and results obtained for the query. The changes or modifications made to an execution pipeline may include changing the capabilities (e.g., changes to inputs and/or outputs of a node, changing the task(s) or function(s) performed by the node) of one or more nodes within the execution pipeline. The changes may include changing the position of one or more nodes within a directed acyclic graph representing the execution pipeline.

Abstract: A query optimization system is described that, at runtime, optimizes the execution pipeline generated for a query. Based upon communications between nodes in the execution pipeline, the execution pipeline generated for a query is optimized by modifying the execution pipeline to create a modified execution pipeline. The modified execution pipeline is then executed to execute the query and results obtained for the query. The changes or modifications made to an execution pipeline may include changing the capabilities (e.g., changes to inputs and/or outputs of a node, changing the task(s) or function(s) performed by the node) of one or more nodes within the execution pipeline. The changes may include changing the position of one or more nodes within a directed acyclic graph representing the execution pipeline.

Abstract: Optimized techniques are disclosed for sorting variable-length records using an optimized amount of memory while maintaining good locality of references. The amount of memory required for sorting the variable length records is optimized by reusing some of the memory used for storing the variable length records being sorted. Pairs of input runs storing variable length records may be merged into a merged run that contains the records in a sorted order by incrementally scanning, sorting, and copying the records from the two input runs being merged into memory pages of the merged run. When all the records of a memory page of an input run have been processed or copied to the merged run, that memory page can be emptied and released to a cache of empty memory pages. Memory pages available from the cache of empty memory pages can then be used for generating the merged run.

Abstract: Optimized techniques are disclosed for sorting variable-length records using an optimized amount of memory while maintaining good locality of references. The amount of memory required for sorting the variable length records is optimized by reusing some of the memory used for storing the variable length records being sorted. Pairs of input runs storing variable length records may be merged into a merged run that contains the records in a sorted order by incrementally scanning, sorting, and copying the records from the two input runs being merged into memory pages of the merged run. When all the records of a memory page of an input run have been processed or copied to the merged run, that memory page can be emptied and released to a cache of empty memory pages. Memory pages available from the cache of empty memory pages can then be used for generating the merged run.

Abstract: Innovative techniques are disclosed for performing a combined sort and aggregation operation involving eagerly performing aggregation while sorting. The techniques described herein enable an aggregation and a group-by operation to be performed using an amount of system memory that is far less than the system memory required to store all the data records being processed while minimizing the need to spill data to disk. This combined sort and aggregation operation has better performance than conventional techniques since system memory is used more efficiently. In certain embodiments, a combined sort and aggregation operation is disclosed that enables the efficient sorting and aggregation of data records when the desired aggregation function is composable (such as SUM, COUNT, MIN and MAX aggregate operations).

Abstract: Techniques for dynamically changing the amount of memory that is allocated to and used by processes (e.g., sorter processes) executed concurrently by a data processing system in a manner that is adaptive to the overall memory usage of the system. The ability to dynamically change the amount of memory that can be allocated to and used by individual processes increases and optimizes the overall memory utilization of the data processing system, which in turn results in the operations (e.g., sorting) performed by the processes to be performed in a more efficient manner. A memory manager is provided that is responsible for periodically collecting memory related information for the data processing system and for the processes (e.g., sorter processes) executed by the data processing system, and based on the collected information, for periodically and dynamically causing the memory used by the processes to be changed during the lifetime of the processes.

Abstract: Innovative techniques are disclosed for performing a combined sort and aggregation operation involving eagerly performing aggregation while sorting. The techniques described herein enable an aggregation and a group-by operation to be performed using an amount of system memory that is far less than the system memory required to store all the data records being processed while minimizing the need to spill data to disk. This combined sort and aggregation operation has better performance than conventional techniques since system memory is used more efficiently. In certain embodiments, a combined sort and aggregation operation is disclosed that enables the efficient sorting and aggregation of data records when the desired aggregation function is composable (such as SUM, COUNT, MIN and MAX aggregate operations).

Abstract: A query optimization system is described that, at runtime, optimizes the execution pipeline generated for a query. Based upon communications between nodes in the execution pipeline, the execution pipeline generated for a query is optimized by modifying the execution pipeline to create a modified execution pipeline. The modified execution pipeline is then executed to execute the query and results obtained for the query. The changes or modifications made to an execution pipeline may include changing the capabilities (e.g., changes to inputs and/or outputs of a node, changing the task(s) or function(s) performed by the node) of one or more nodes within the execution pipeline. The changes may include changing the position of one or more nodes within a directed acyclic graph representing the execution pipeline.

Abstract: Optimized techniques are disclosed for sorting variable-length records using an optimized amount of memory while maintaining good locality of references. The amount of memory required for sorting the variable length records is optimized by reusing some of the memory used for storing the variable length records being sorted. Pairs of input runs storing variable length records may be merged into a merged run that contains the records in a sorted order by incrementally scanning, sorting, and copying the records from the two input runs being merged into memory pages of the merged run. When all the records of a memory page of an input run have been processed or copied to the merged run, that memory page can be emptied and released to a cache of empty memory pages. Memory pages available from the cache of empty memory pages can then be used for generating the merged run.

Abstract: Techniques for dynamically changing the amount of memory that is allocated to and used by processes (e.g., sorter processes) executed concurrently by a data processing system in a manner that is adaptive to the overall memory usage of the system. The ability to dynamically change the amount of memory that can be allocated to and used by individual processes increases and optimizes the overall memory utilization of the data processing system, which in turn results in the operations (e.g., sorting) performed by the processes to be performed in a more efficient manner. A memory manager is provided that is responsible for periodically collecting memory related information for the data processing system and for the processes (e.g., sorter processes) executed by the data processing system, and based on the collected information, for periodically and dynamically causing the memory used by the processes to be changed during the lifetime of the processes.

Abstract: Innovative techniques are disclosed for performing a combined sort and aggregation operation involving eagerly performing aggregation while sorting. The techniques described herein enable an aggregation and a group-by operation to be performed using an amount of system memory that is far less than the system memory required to store all the data records being processed while minimizing the need to spill data to disk. This combined sort and aggregation operation has better performance than conventional techniques since system memory is used more efficiently. In certain embodiments, a combined sort and aggregation operation is disclosed that enables the efficient sorting and aggregation of data records when the desired aggregation function is composable (such as SUM, COUNT, MIN and MAX aggregate operations).

Abstract: An aspect of the present invention simplifies generating a star schema from a snowflake schema. In an embodiment, a user first specifies fact tables to be included in a star schema, and a synchronization tool inspects the snowflake schema to determine the dimension tables linked to the specified fact tables. The determined dimension tables are included in the star schema sought to be generated.

Abstract: An aspect of the present invention simplifies generating a star schema from a snowflake schema. In an embodiment, a user first specifies fact tables to be included in a star schema, and a synchronization tool inspects the snowflake schema to determine the dimension tables linked to the specified fact tables. The determined dimension tables are included in the star schema sought to be generated.