Abstract: The disclosure generally describes methods, software, and systems, including a method for defining and using dependencies and equivalences for data elements in the data model in order to improve query efficiency. At design time of a data model, dependencies and equivalences for data elements in the data model are received over a computer network. The dependencies and equivalences are stored in persistent storage with the data model. A request is received over the computer network, the request to be executed to obtain results from a database associated with the data model. An optimized query is generated using the received request and the dependencies and equivalences. The optimized query is optimized relative to the received request.

Abstract: Methods and apparatus, including computer program products, are provided for split query optimization. In some example embodiments, a method may include: determining, for a query comprising a plurality of operations, a first workload distribution between a first data store and a second data store, the first workload distribution indicating that a first portion of the query is to be performed at the first data store and a second portion of the query is to be performed at the second data store; and determining, based at least on the first workload distribution, a second workload distribution, the determining of the second workload distribution includes pushing down, to the first portion of the query, a first operation from the second portion of the query such that the first operation is performed at the first data store instead of at the second data store.

Abstract: As disclosed herein a method, executed by a computer, for enabling a SPARQL explain tool that provides query execution statistics via an explain plan includes initiating a database query comprising a SPARQL query pattern, optimizing the database query according to query semantics and database metadata to produce an optimized query, and generating an explain plan for the optimized query. The explain plan may include a sequence of operations and an execution time, an input, and an output for each operation of the sequence of operations. A computer system, and a computer program product corresponding to the method are also disclosed herein.

Abstract: A method and system for synchronization of Relational Database Management System (RDBMS) to non-Structured Query Language (noSQL) database receives a user request and associated predefined business logic from a user device for retrieving one or more transactional data and synchronizes the RDBMS to the noSQL database based on the predefined business logic. The database synchronization system segregates the predefined business logic into one or more processing divisions and assigning the one or more processing divisions to the RDBMS and the noSQL database. Further, the noSQL database retrieves data related to one or more elements of one or more tables related to the one or more transactional data from the RDBMS. Thereafter, the noSQL database transforms the retrieved data and provide to the database synchronization system. Finally, database synchronization system provides transformed data received from the noSQL database to the user device.

Abstract: A computer-implemented method for optimizing a queue of queries for database efficiency is implemented by a controller computing device coupled to a memory device. The method includes receiving a plurality of database queries at the computing device from at least one host, evaluating the plurality of database queries to determine a resource impact associated with each database query of the plurality of database queries, prioritizing the plurality of database queries based upon a set of prioritization factors and the resource impact associated with each database query, and submitting the prioritized plurality of database queries to a database system for execution. The database system executes the plurality of database queries in order of priority.

Abstract: Methods and systems that allow of mapping data stored in a first format to a second format is provided. According to the method a data object to be mapped is identified. The object may have a particular type associated with it and one or more associated properties. A unique identifier can be generated for the based on the type of the first object. According to some embodiments, the unique identifier may contain information that can be used to retrieve the object when it is stored in the first format. One or more of the properties associated with the first can be mapped to a resource associated with the unique identifier and also with a second format. The first object can then be queried using a query language associated with the second format.

Abstract: A system performs database management. The system receives a request for a query of a plurality of tables under a join condition on an attribute and determines a uniform mapping that maps a domain of the attribute into a set of numbers, where the domain of the attribute includes all attribute values taken by the attribute in the tables. Then, for a row in a table, the system includes the row into a synopsis of the table if the row includes an attribute value that is mapped by the uniform mapping to a number less than an inclusion probability associated with the table. The system obtains a plurality of synopsis for the tables by repeating the including for all rows in the table and for all tables, and determines, based on the plurality of synopsis, an estimate join size of the tables under the join condition.

Abstract: There is described a method and system for obtaining results from one or more data sources without the need to extract, transform and load the data into a database prior to performing a query. The data may retain its original format and remain in its initial location, and selective access to the data is provided using dynamically generated query statements. Multiple data sources may be accessed concurrently, each data source having its own format for stored data. Query statements are generated to obtain data (query results) from the various data sources by defining data sources, target fields, and parameters to be used for a given query. The statements comprise predefined operations that are applied recursively in order to transform (and sometimes aggregate) the data from the data sources into query results. Query statements and query results may be saved for later use.

Abstract: The present subject matter relates to entity resolution, and in particular, relates to providing an entity resolution from documents. The method comprises obtaining a plurality of documents corresponding to a plurality of entities, from at least one data source. Upon receiving the plurality of documents, the plurality of documents is blocked into at least one bucket based on textual similarity. Further, a graph including a plurality of record vertices and at least one bucket vertex is created. The plurality of record vertices and the at least one bucket vertex are indicative of the plurality of documents and the at least one bucket, respectively. Subsequently, a notification is provided to a user for selecting one of a Bucket-Centric Parallelization (BCP) technique and a Record-Centric Parallelization (RCP) technique for resolving entities from the plurality of documents. Based on the selection, a resolved entity-document for each entity is created.

Abstract: A method, system, and computer program product for amorphous data preparation for efficient query formulation are provided. A normalized form of representing a set of data cubes is selected. The set of data cubes includes a first data cube conforming to first data representation and a second data cube conforming to a second data representation. The normalized form selects a third data representation to represent the data of each data cube in the set. A transformation is applied to the first data cube to represent the first data cube using the third data representation to create a normalized first data cube. A set of metadata elements corresponding to the normalized first data cube is computed. The normalized first data cube and the metadata of the normalized first data cube are saved in a data store.

Abstract: A system and method of caching and parameterizing intermediate representation code includes receiving, by a database, a query, parsing, by the database, the query to obtain a plan tree comprising a plurality of plan nodes arranged in hierarchical order descending from a top plan node, generating, by the database, node intermediate representations (IRs) for the plan nodes, executing, by the database, a first query using the node IRs, and reusing, by the database, the node IRs to execute subsequent queries.

Abstract: A tool for combining common processes shared by at least two or more sub-queries within a query is provided. The tool determines the query with the at least two or more sub-queries. The tool determines whether one or more sub set relationships are shared between the at least two or more sub-queries. Responsive to a determination that one or more sub set relationships are shared between the at least two or more sub-queries, the tool determines an order class for the at least two or more sub-queries based on the one or more sub set relationships. The tool determines an access path for the query. The tool executes the access path during run-time for data accessing.

Abstract: Methods and apparatus, including computer program products, are provided for union node pruning. In one aspect, there is provided a method, which may include receiving, by a calculation engine, a query; processing a calculation scenario including a union node; accessing a pruning table associated with the union node, wherein the pruning table includes semantic information describing the first input from the first data source node and the second input from the second data source node; determining whether the first data source node and the second data source node can be pruned by at least comparing the semantic information to at least one filter of the query; and pruning, based on a result of the determining, at least one the first data source node or the second data source node. Related apparatus, systems, methods, and articles are also described.

Abstract: A database system comprising: a client that executes a query to a database server; and the database server that, according to a query from the client, transmits a table of an execution result of the query, the table representing an answer to each query as a record and including the record; wherein the database server comprises a data transmission part which, in a case of transmission of the table of the execution result having a plurality of columns, performs the transmission for each column of the table of the execution result by using any of a plurality of data structures being capable of representing the column.

Abstract: A network analytics controller in a computer network receives specification of a network analysis task. The controller creates maplets. Each maplet corresponds to an agent executing on a network node in the computer network. Each maplet specifies the network data collection configuration, collected data processing, and processed data reporting to be performed to support the network analysis task at the node executing the corresponding agent. The controller communicates the maplet corresponding to the agent over the computer network to each agent. Each agent receiving a maplet configures its corresponding network node for network data collection, collected data processing, and processed data reporting in accordance with the received maplet. Each agent receiving a maplet collects and processes network data, and reports the processed data to the controller, in accordance with the configuration. The controller aggregates, as the result of the network analysis task, the data reported from each agent.

Abstract: A knowledge graph virtualization system (KGVS) implements data virtualization techniques using semantic knowledge when parsing a knowledge graph to return results to a query. The data virtualization using semantic knowledge enable end users and applications to describe desired output without detailed knowledge of underlying data storage, schema, and relationships.

Abstract: In an approach to determining an access method for a partition in a partition table, a computer receives a query and determines if there is a partition table utilized by the query. When there is a partition table utilized by the query, then the computer determines that a partition in the partition table meets the plurality of conditions of the query. The computer collects a plurality of partition level statistics for the partition that meets the plurality of conditions of the query. Additionally, the computer determines, based, at least in part, on the plurality of partition level statistics, a cost for one or more access methods for the partition that meets the plurality of conditions of the query. Furthermore, the computer determines, based, at least in part, on the cost for each access method, an access method for the partition that meets the plurality of conditions of the query.

Abstract: Seed patterns are derived from a sequence database. Execution costs for types of seed patterns are computed. Each seed pattern is iteratively distributed to distributed nodes along with that seed pattern's assigned execution cost. The distributed nodes processing in parallel to mine the sequence database for super patterns found in the sequence database. When a distributed node exhausts its execution budget, any remaining mining needed for the seed pattern being mined is reallocated to another distributed node having remaining execution budget.

Abstract: Identifying objects in a datastore with specified object properties, where each object is characterized by a Bloom filter, a Hamming value of the Bloom filter, and a binary logarithm of the Bloom filter. A set of object properties is received. A search Bloom filter is created from the object properties. A Hamming value of the search Bloom filter is determined. A binary logarithm of the search Bloom filter is calculated. Objects in the datastore which have respective Hamming values greater than or equal to the Hamming value of the search Bloom filter and respective binary logarithms that are greater than or equal to the binary logarithm of the search Bloom filter are identified.

Abstract: A method is described for monitoring data store performance and recommending a data store structure for a data store. The method may include analyzing a data store structure to determine whether the data store structure corresponds to a set of criteria, analyzing a query executed against the data store structure to determine performance of the query with respect to the data store structure and providing a proposed data store structure to a customer based at least on part on the analysis of the data store structure and analysis of the query.

Abstract: A data store system may include a persistent storage device that may store a plurality of data tables. The data store system may further include a processor array in communication with the persistent storage device. The processor array may receive a query comprising a call to a user-defined function (UDF) that may determine a plurality of aggregates for at least one data table from the plurality of data tables. The processor array may determine a number of the plurality of aggregates in the UDF. The processor array may perform a single read of rows associated with the plurality aggregates from the at least one data table. At least one row of the rows may be commonly associated with more than one of the plurality of aggregates. The processor array may generate each aggregate of the plurality of aggregates using the rows according to the UDF. A method and computer-readable medium may also be implemented.

Abstract: A compute device to generate deterministic compressed streams receives a current string to be matched to one or more prior instances of the current string, the current string being located within an input buffer and the one or more prior instances located within a history buffer. The compute device identifies a limited subset of index memory designated for storing pointers to the prior instances, identifying a reserved slop region in the index memory, and compares the current string to a prior instance, locating the at least one prior instance using at least one pointer to the at least one prior instance. The at least one pointer is stored within the limited subset of the index memory, and the compute device also prohibits use of any pointers stored in the reserved slop region of the index memory. Other embodiments are described and claimed.

Abstract: A method for performing dataset operations within a cognitive information processing system comprising: receiving data from a plurality of data sources; and, processing the data from the plurality of data sources, the processing the data establishing and maintaining a dynamic data ingestion and enrichment pipeline.

Abstract: A computer-implemented method for executing a query specification can include evaluating the query specification by a first computing device to identify a candidate expression in the query specification to be executed by a second computing device hosting a source of data specified in the candidate expression. A cost estimate for executing the candidate expression at the first computing device is determined. Whether the cost estimate of executing the candidate expression at the first computing device is greater than a threshold is determined. The candidate expression is replaced with a native query function when the cost estimate of executing the candidate expression at the first computing device is greater than the threshold. The native query function can be executed at the second computing device. It can be determined whether a runtime context is necessary, and if necessary, a context parameter is injected into the native query function.

Abstract: Various embodiments of systems and methods for optimizing in-memory column based database table compression. A system may receive a parsed in-memory columnar database table as an input and reads an aggregated compression score from memory corresponding to the parsed database table. A total compression score may be computed iteratively for the database table in order to determine whether the total compression score is greater than the aggregated compression score. When the total compression score determined to be greater than the aggregated compression score, the aggregated compression score is assigned to a best compression score. To optimize in-memory database table compression, the given database table may be compressed using the best compression score.

Abstract: Approaches, techniques, and mechanisms are disclosed for ensuring that a database command is executed according to a query plan that has been verified to be actually optimal. Except in rare circumstances, a database server does not execute a query plan unless it is first verified by the database server. The database server receives a request to execute a database command. The database server determines an unverified plan is the best plan for satisfying the request. Rather than risk the unknown behavior of an unverified plan, the database server instead satisfies the request according to a verified plan. Subsequently—for example as part of a scheduled job—the database server executes the unverified plan to determine performance statistics. Based at least on the performance statistics, the database server determines whether or not to verify the unverified plan. Techniques for concurrent and optimistic verifications are also disclosed.

Abstract: A method for optimizing query compilation by tuning memory across a database cluster is provided. The method may include receiving, by a global memory tuner, memory configuration information from a plurality of nodes within the database cluster. The method may also include determining a node within the plurality of nodes having a least available memory value relative to a remainder of nodes within the plurality of nodes based on the received memory configuration information. The method may then include calculating a globally tuned memory value based on a memory value associated with the determined node. The method may further include determining a memory distribution based on the calculated globally tuned memory value. The method may also include sending the determined memory distribution to the plurality of nodes.

Abstract: In an approach to determining an access method for a partition in a partition table, a computer receives a query and determines if there is a partition table utilized by the query. When there is a partition table utilized by the query, then the computer determines that a partition in the partition table meets the plurality of conditions of the query. The computer collects a plurality of partition level statistics for the partition that meets the plurality of conditions of the query. Additionally, the computer determines, based, at least in part, on the plurality of partition level statistics, a cost for one or more access methods for the partition that meets the plurality of conditions of the query. Furthermore, the computer determines, based, at least in part, on the cost for each access method, an access method for the partition that meets the plurality of conditions of the query.

Abstract: A method for processing a database query includes receiving a database query to be executed according to a first processing profile and determining whether the database query is likely to be an ad hoc query. The determining of whether the database query is likely to be an ad hoc query is based on a factor. The factor can include a query originator factor, a job type factor, a run history factor, or a query quality factor. A second processing profile is generated before executing the database query in response to determining that the database query is likely to be an ad hoc query. The second processing profile is configured to limit performance impacts, such as by providing a lower execution priority, prohibiting parallel database operations, limiting memory allocation, or delaying or blocking the query.

Abstract: A respective volatility attribute associated with each of one or more tables of a computerized database is used in any of various aspects to (a) determine how table data is stored in a physical storage device; (b) regulate the use of a materialized query table using database table data; and/or (c) influence circumstances under which indexes are created or advised by database analytic software. Various optional additional uses of a volatility attribute to manage a database are disclosed. Preferably, database parameters are automatically monitored over time and database table volatility state is automatically determined and periodically adjusted.

Abstract: In one embodiment, a method determines one or more records that are added, updated, or deleted for an entity. One or more effective from dates for the one or more records and a last effective from date for the entity in a database table for the entity is analyzed to classify the one or more records in one or more scenarios in a plurality of scenarios. The method selects a scenario in the plurality of scenarios based on the one or more effective from dates for the one or more scenarios and the last effective from date for the entity. Then, an update process associated with the scenario is performed to update a portion of records in the database table using the one or more records.

Abstract: Load Plan Generator (LPG) is a BIAPPS utility for generating ODI load plans based on desired subset of fact tables for loading BIAPPS Data Warehouse. The tool simplifies the configurations process by minimizing the manual steps and configurations and provides a guided list of configurations steps and checklists. The load plan components are basically different sets of load plans that will be stitched together by the load plan generator to create one load plan for loading chosen fact groups in the warehouse sourcing from different transaction systems.

Abstract: Techniques for a system capable of performing low-latency database query processing are disclosed herein. The system includes a gateway server and a plurality of worker nodes. The gateway server is configured to divide a database query, for a database containing data stored in a distributed storage cluster having a plurality of data nodes, into a plurality of partial queries and construct a query result based on a plurality of intermediate results. Each worker node of the plurality of worker nodes is configured to process a respective partial query of the plurality of partial queries by scanning data related to the respective partial query that stored on at least one data node of the distributed storage cluster and generate an intermediate result of the plurality of intermediate results that is stored in a memory of that worker node.

Abstract: Indexing objects in a datastore, where the objects are characterized by a set of object properties. For objects from a datastore: a Bloom filter for the object properties, based on a predetermined Bloom filter configuration, is created; the Hamming value of the Bloom filter is determined; a binary logarithm of the Bloom filter is calculated; and the Bloom filter, the Hamming value, the binary logarithm, and a representation of the object is stored in the datastore.

Abstract: In one embodiment, a method of processing a structured query language (SQL) statement is provided, comprising: determining whether a first query and a second query are equivalent, the first and second queries being respectively the left side and the right side operands of the OUTER JOIN operation; determining whether a SELECT output of the SQL statement does not refer to database columns from the left side of the OUTER JOIN operation which are not also partitioning columns of the OUTER JOIN operation; and responsive to determining that the first query and the second query are equivalent and that the SELECT output of the SQL statement does not refer to database columns from the left side of the OUTER JOIN operation which are not also partitioning columns of the OUTER JOIN operation, transforming the SQL statement into an optimized query SQL statement by removing the OUTER JOIN operation.

Abstract: Chunk de-duplication performance is improved. A de-duplication system has a cut-out processing unit which inputs a content from a client terminal thereinto, determines a calculation range from a predetermined maximum chunk size and a predetermined minimum chunk size, divides the calculation range into at least two small calculation ranges, sets the positions of windows for rolling hash calculation so that the rolling hash calculation is continuous between the two small calculation ranges, and subjects the at least two small calculation ranges to the rolling hash calculation with shifting of the windows based on parallel processing to cut out a chunk from the content, and a de-duplication processing unit which does not store the cut-out chunk into a storage device when the chunk having the same contents as those of the cut-out chunk is already stored in the storage device.

Abstract: The invention notably relates to a computer-implemented method of clustering reference queries in a database for prediction of the runtime of a target query in the database based on similarity of the target query with the reference queries. The method comprises providing a number of numerical values that represent the runtimes of the reference queries; computing the optimal K-means clustering of the numerical values for a predetermined number of clusters, wherein the computing includes iterating, a number of times corresponding to the predetermined number of clusters, a linear-time Row Minima Searching algorithm applied to a square matrix of order equal to the number of numerical values; and clustering the reference queries according to the computed clustering of the numerical values. Such a method improves the field of database query runtime prediction.

Abstract: The present invention discloses a method for optimizing the throughput of hardware accelerators (HWAs) in a computerized abstraction system, by utilizing the maximal data input bandwidth to the said HWAs. The method is comprised of the following steps: dynamically obtaining the quantities and properties of HWAs and storage units within the computerized abstraction system dynamically allocating cache memory space per each of the HWAs, according to the said obtained quantities and properties, to minimize the time required for reading data from storage instances to the said HWA dynamically allocating spoolers per each of the HWAs, according to the said obtained quantities and properties, to buffer the input data and ensure a continuous flow of input data, in the target HWA's maximal input bandwidth.

Abstract: A method for stratified-sampling-based query execution includes: receiving a query; collecting a first data sample from the first dataset using a non-stratified sampling technique; performing statistical analysis on the first data sample; identifying a stratum classifier from the statistical analysis; generating a stratum classification by calculating strata boundaries for the stratum classifier; and calculating a result to the query based on analysis of the second data sample.

Abstract: Techniques for remotely managed data extraction are disclosed whereby a client data extractor (CDE) is installed on a customer computer system to extract data from the customer computer system for transmission to a destination. Soft configuration techniques can be employed where the configuration data for governing the data extraction are provided to the customer computer system from a service provider computer system on an as-needed basis. Moreover, during extraction, data translation from a source format to a destination format can be performed at the query level to improve efficiency.

Abstract: A compute device to generate deterministic compressed streams receives a current string to be matched to one or more prior instances of the current string, the current string being located within an input buffer and the one or more prior instances located within a history buffer. The compute device identifies a limited subset of index memory designated for storing pointers to the prior instances, identifying a reserved slop region in the index memory, and compares the current string to a prior instance, locating the at least one prior instance using at least one pointer to the at least one prior instance. The at least one pointer is stored within the limited subset of the index memory, and the compute device also prohibits use of any pointers stored in the reserved slop region of the index memory. Other embodiments are described and claimed.

Abstract: A first database management system (DBMS) running on first computer having a first set of computing resources determines that with respect to the first set of computing resources a first query access plan for executing a query satisfies a set of query optimization criteria. A determination is then made that with respect to a second set of computing resources the first query access plan fails a set of preliminary criteria. In response to this failure determination, a second query access plan for executing the query is generated. Prior to having a second DBMS running on a second computer having the second set of computing resources receive a request to execute the query, a determination is made that with respect to the second set of computing resources the second query access plan satisfies the set query optimization criteria.

Abstract: A first database management system (DBMS) running on first computer having a first set of computing resources determines that with respect to the first set of computing resources a first query access plan for executing a query satisfies a set of query optimization criteria. A determination is then made that with respect to a second set of computing resources the first query access plan fails a set of preliminary criteria. In response to this failure determination, a second query access plan for executing the query is generated. Prior to having a second DBMS running on a second computer having the second set of computing resources receive a request to execute the query, a determination is made that with respect to the second set of computing resources the second query access plan satisfies the set query optimization criteria.

Abstract: A system for efficiently responding to proximity queries may include a memory storing an index for searching a graph-based data store, the index including posting lists for one or more proximity ranges compatible with a space. A posting list can include one or more entities of a type compatible with the space, each entity having a location within the space, the location being a basic unit in a location hierarchy for the space and, for each entity, at least one node in the location hierarchy that falls within the proximity range of the posting list with reference to the location of the entity. The system may also include a memory storing instructions that cause the system to use the index to respond to a query that includes a query proximity range for the space. The space can be a geographic space or a time space.

Abstract: In one embodiment, a method for improving cardinality estimation of a join predicate between a fact table and an overloaded dimension table is provided. The method includes receiving a dimension table and a fact table in a join predicate of one or more SQL statements. The method further includes identifying a majority of records in the fact table that refer to a subset of records in the dimension table. The method further includes computing a filter factor of the join predicate between the dimension table and the fact table. The method further includes creating a statistical view using one or more relevant portions of the dimension table that are referred to by the fact table.

Abstract: In one embodiment, a method for improving cardinality estimation of a join predicate between a fact table and an overloaded dimension table is provided. The method includes receiving a dimension table and a fact table in a join predicate of one or more SQL statements. The method further includes identifying a majority of records in the fact table that refer to a subset of records in the dimension table. The method further includes computing a filter factor of the join predicate between the dimension table and the fact table. The method further includes creating a statistical view using one or more relevant portions of the dimension table that are referred to by the fact table.

Abstract: Techniques for delivering video content are provided. A video distribution service maintains associations of viewer devices that have portions of specific items of video content. As new viewer devices, which lack the specific video content, make requests for the specific items of video content from the video distribution system, the video distribution system identifies a viewer device from which a portion of the item of video content may be acquired. Viewer devices having the requested content are identified by comparing distances between the locations of the requestor device and the other viewer devices of the plurality of viewers to a distance threshold. The first portion of a first item of video content is obtained from a source viewer device and a second portion of the first item of video content is obtained from the video distribution system. Other embodiments are disclosed.

Abstract: A method for use in a database cluster is provided. A transaction is received at a first node of the database cluster. The transaction is executed at the first node, wherein the transaction produces a plurality of replication events. The plurality of replication events of the transaction are divided into a plurality of fragments, each of which comprises one or more replication events. Subsequently, a set of fragment-specific parameters is assigned to each of the plurality of fragments. The plurality of fragments are then communicated to a second node of the database cluster, one by one, as soon as they are prepared in the first node. A transaction is established in the second node to apply the arriving fragments. With this streaming replication method, the replication of the transaction in the first node, can begin early, before the transaction commits in the first node. This will even out the delay of large transaction replication.

Abstract: A calculation engine is described that executes calculation scenarios comprising a plurality of calculation nodes that specify operations to be performed to execute the query. One of the nodes can be a semantic node that is used to modify the query for operations requiring special handling including handling of hierarchy views. Related apparatus, systems, methods, and articles are also described.

Abstract: A system and method for database storage may be dynamically optimized for entity-attribute-value (EAV) data. In one embodiment, this dynamic EAV database may include one column for the entity identification (“ID”) and a second column for the serialized data for that ID (each attribute and the values for those attributes). The database may be dynamically updated to move or copy certain attributes to a separate column. The dynamic updating of one or more attributes may be based on certain factors (seasonal, temporal, usage, etc.), such as frequent requests for a particular attribute. This dynamic updating and indexing of particular attributes may improve certain queries and data requests.