Abstract: Systems and methods for optimizing memory in an interest-driven business intelligence system in accordance with embodiments of the invention are illustrated. A dictionary for storing values of a dataset may be partitioned in accordance with some embodiments. The partitions of the dictionary may be generated by mapping and reducer processes. The mapping processes receive a value, determine the dimension of the data to which the value belongs, and provides the value to a reducer process that handles the determined dimension. Each reducer process generates partitions of the dictionary for each dimension. The number of values in each partition is determined and compared to a threshold value. Partitions that have a number of values greater than the threshold are stored in a common memory. Partitions smaller than the threshold value can be combined with other partitions such that the cardinality of the combined partition exceeds the threshold value.

Abstract: Systems and methods for optimizing memory in an interest-driven business intelligence system in accordance with embodiments of the invention are illustrated. A dictionary for storing values of a dataset may be partitioned in accordance with some embodiments. The partitions of the dictionary may be generated by mapping and reducer processes. The mapping processes receive a value, determine the dimension of the data to which the value belongs, and provides the value to a reducer process that handles the determined dimension. Each reducer process generates partitions of the dictionary for each dimension. The number of values in each partition is determined and compared to a threshold value. Partitions that have a number of values greater than the threshold are stored in a common memory. Partitions smaller than the threshold value can be combined with other partitions such that the cardinality of the combined partition exceeds the threshold value.

Abstract: Systems and methods for optimizing memory in an interest-driven business intelligence system in accordance with embodiments of the invention are illustrated. A dictionary for storing values of a dataset may be partitioned in accordance with some embodiments. The partitions of the dictionary may be generated by mapping and reducer processes. The mapping processes receive a value, determine the dimension of the data to which the value belongs, and provides the value to a reducer process that handles the determined dimension. Each reducer process generates partitions of the dictionary for each dimension. The number of values in each partition is determined and compared to a threshold value. Partitions that have a number of values greater than the threshold are stored in a common memory. Partitions smaller than the threshold value can be combined with other partitions such that the cardinality of the combined partition exceeds the threshold value.

Abstract: Embodiments are included for methods, systems, and computer program products in which evaluating operations using an electronic computing device are presented including: receiving the operation on a first database partition in a shared nothing parallelism system, where the operation is non-collocated, reference based operation; generating a correlation sequence, where the correlation sequence includes a sequence of references pointing to input data required for the operation; receiving one of the references on a first table queue operator on a second database partition, where the second database partition includes input data corresponding with the received reference, and where the table queue operator is configured for providing communication between the first database partition and the second database partition; and processing the input data on the second database partition corresponding with the operation.

Abstract: Systems and methods for optimizing memory in an interest-driven business intelligence system in accordance with embodiments of the invention are illustrated. A dictionary for storing values of a dataset may be partitioned in accordance with some embodiments. The partitions of the dictionary may be generated by mapping and reducer processes. The mapping processes receive a value, determine the dimension of the data to which the value belongs, and provides the value to a reducer process that handles the determined dimension. Each reducer process generates partitions of the dictionary for each dimension. The number of values in each partition is determined and compared to a threshold value. Partitions that have a number of values greater than the threshold are stored in a common memory. Partitions smaller than the threshold value can be combined with other partitions such that the cardinality of the combined partition exceeds the threshold value.

Abstract: Systems and methods for data ingest in interest-driven business intelligence systems in accordance with embodiments of the invention are illustrated. The interest-driven business intelligence system may maintain a set of registered data ingest instruction data that includes at least one registered data ingest instruction data. Each of the at least one registered data ingest instruction data includes an identifier and data ingest instruction data associated with the identifier. The system may receive a request to generate data using registered data instruction data. The request may include the identifier of the registered data instruction data. Data is generated using the data ingest instruction data associated with the requested identifier and at least one of raw data, source data, and aggregate data, and provided for use.

Abstract: A database system performs hash join process for processing queries that join an inner and an outer database table. The hash join processes builds a hash table in memory for the inner table. The database system receives a limit on the memory for storing the hash table. The database system maximizes the number of partitions stored in memory for the hash table. If the hash table exceeds the limit of the memory while adding rows from the inner table, the database system selects a partition for spilling to a persistent storage. The partition selected for spilling to may be the largest partition or a partition larger than most of the partitions. The database system initializes the hash table to a number of partitions that is substantially equal to half of the total number of blocks that can be stored within the specified limit of memory for the hash table.

Abstract: Embodiments are included for methods, systems, and computer program products in which evaluating operations using an electronic computing device are presented including: receiving the operation on a first database partition in a shared nothing parallelism system, where the operation is non-collocated, reference based operation; generating a correlation sequence, where the correlation sequence includes a sequence of references pointing to input data required for the operation; receiving one of the references on a first table queue operator on a second database partition, where the second database partition includes input data corresponding with the received reference, and where the table queue operator is configured for providing communication between the first database partition and the second database partition; and processing the input data on the second database partition corresponding with the operation.

Abstract: Embodiments are included for methods, systems, and computer program products for evaluating operations in a shared nothing parallelism system using an electronic computing device are presented, the methods and computer program products including: causing the electronic computing device to receive the operation, where the operation is non-collocated, reference based operation, and where the operation is configured for execution on a first partition in the shared nothing parallelism system; causing the electronic computing device to insert a table queue operator on a second partition, where the second partition includes a reference to input data, causing the electronic computing device to place the reference to the input data on the first partition by the table queue operator; and causing the electronic computing device to execute the operation on the reference to the input data on the first partition.

Abstract: A database system performs hash join process for processing queries that join an inner and an outer database table. The hash join processes builds a hash table in memory for the inner table. The database system receives a limit on the memory for storing the hash table. The database system maximizes the number of partitions stored in memory for the hash table. If the hash table exceeds the limit of the memory while adding rows from the inner table, the database system selects a partition for spilling to a persistent storage. The partition selected for spilling to may be the largest partition or a partition larger than most of the partitions. The database system initializes the hash table to a number of partitions that is substantially equal to half of the total number of blocks that can be stored within the specified limit of memory for the hash table.

Abstract: A method and system for distributed garbage collection in a pipelined workflow environment comprising a plurality of processing nodes that pass item references to each other. Each node maintains a reference list of local item references and a reference dictionary of remote item references, and periodically synchronizes the reference list with the reference dictionaries of other nodes in the workflow, so that item references are not marked for garbage collection while other processing nodes may still have outstanding references to them.

Abstract: Embodiments are included for methods, systems, and computer program products for evaluating operations in a shared nothing parallelism system using an electronic computing device are presented, the methods and computer program products including: causing the electronic computing device to receive the operation, where the operation is non-collocated, reference based operation, and where the operation is configured for execution on a first partition in the shared nothing parallelism system; causing the electronic computing device to insert a table queue operator on a second partition, where the second partition includes a reference to input data, causing the electronic computing device to place the reference to the input data on the first partition by the table queue operator; and causing the electronic computing device to execute the operation on the reference to the input data on the first partition.

Abstract: A method and computer program for processing structured documents follows a processing framework that enables generation of events corresponding to instance document elements and events corresponding to definition components in a single serial process. The process comprises creating a graph data structure in which nodes of the graph represent components of a document definition. The process further involves reading an instance document conforming to the document definition, identifying elements of the document that correspond to nodes of the graph, identifying a path between nodes of the graph that correspond to elements of the document, and traversing the path to generate a start event when moving from a parent node to a child node and an end event when moving from a child node to a parent node.

Abstract: A method and system for distributed garbage collection in a pipelined workflow environment comprising a plurality of processing nodes that pass item references to each other. Each node maintains a reference list of local item references and a reference dictionary of remote item references, and periodically synchronizes the reference list with the reference dictionaries of other nodes in the workflow, so that item references are not marked for garbage collection while other processing nodes may still have outstanding references to them.

Abstract: A system and computer-based method is provided for context-sensitive decomposition of a markup based document into a relational database, based on schemas with reusable item declarations. The user creates a mapping document from a schema of a markup based document with at least one reusable item declaration defining an item being mapped to different tables/columns pairs, depending on the item's ancestry. Mapping document is then annotated with the item ancestry mapping annotation having a location path of the item. Each item is decomposed into a corresponding table column. Preferably, the mapping document is an annotated XML Schema.

Abstract: Methods for pre-processing mapping information for efficient decomposition of an XML document for storage in a database are provided. The method includes receiving a mapping document that describes how all of (or a portion of) an XML document is to be decomposed, transforming the mapping document into a data structure for decomposing an XML document, and making the data persistent for use with a subsequent decomposition operation that decomposes an XML document.

Abstract: A system for determining relationships between hierarchically structured schema components and their effects on and content of tuples, includes: analyzing the hierarchically structured schema with user-supplied mappings and finding elements or attributes mapped to a same relational table; determining relationships between the elements or attributes to be either a one-to-one relationship or a one-to-many relationship based on an information set in the hierarchically structured schema; recording the relationships; and processing a hierarchically structured document against the recorded relationships and generating tuples accordingly. The constructs of a hierarchically structured schema that may affect the cardinality between the attributes of a relation, and thus the contents of the tuples, are considered. A relationship between the hierarchically structured schema model and a relational model is established.

Abstract: Pre-processing mapping information for efficient decomposition of an XML document for storage in a database. A mapping document is received that describes how all of (or a portion of) an XML document is to decomposed, the mapping document is transformed into a data structure for decomposing an XML document, and the data is made persistent for use with a subsequent decomposition operation that decomposes an XML document.

Abstract: A method and computer program for processing structured documents follows a processing framework that enables generation of events corresponding to instance document elements and events corresponding to definition components in a single serial process. The process comprises creating a graph data structure in which nodes of the graph represent components of a document definition. The process further involves reading an instance document conforming to the document definition, identifying elements of the document that correspond to nodes of the graph, identifying a path between nodes of the graph that correspond to elements of the document, and traversing the path to generate a start event when moving from a parent node to a child node and an end event when moving from a child node to a parent node.

Abstract: A system and computer-based method is provided for decomposing and storing a markup based document into a relational database. For a schema of a markup based document a user identifies multiple items mapping into a same database table-column pair, creates a logical table and associates the logical table to each item and a corresponding database table. Next, a user creates a mapping document of the markup based document with mapping annotations defining mapping of the items into columns of the logical tables. Decomposition of each item into a corresponding logical table column is accomplished by collecting the item content from the markup based document and storing it in the corresponding row column.