Abstract: Some embodiments include a plurality of virtual data warehouses having table link capabilities that are built on top of a data center (e.g., running Apache Hive). Each virtual data warehouse can be modeled as a database and manage data in forms of database tables. The virtual data warehouse can include links which import tables from other virtual data warehouses by reference. Each link may contain partition metadata for the table partitions by dates of the source table and retention metadata to declare the needed retention time period for the partitions of the source table. The links can be dynamic and update when the corresponding source table receives new partitions or drops partitions. When a virtual data warehouse is migrated to another data center, the system can retain necessary table partitions on the current data center based on the partition and retention metadata of the links.

Abstract: Techniques for detecting patterns in one or more data or event streams. A pattern to be detected may be specified using a regular expression. Events received in a data stream are processed during runtime to detect occurrences of the specified pattern in the data stream. In one embodiment, a pattern type or class is determined for the specified pattern and pattern matching is performed using a technique selected based upon the type or class determined for the specified pattern.

Abstract: A mechanism is used to improve the efficiency of performing resource-to-path resolution. A hierarchical link structure contains entries and/or records (“link entries”) that each defines a parent-child link within an information hierarchy. A certain linking entry may have data that refers to another link entry representing a link to the parent—a link in which the parent is a child. This data allows resource-to-path resolutions to be performed more efficiently.

Abstract: Techniques for reducing the memory used for processing events received in a data stream are provided. This may be achieved by reducing the memory required for storing tuples. A method for processing a data stream includes receiving a tuple and determining a tuple specification that defines a layout of the tuple. The layout identifies one or more data types that are included in the tuple. A tuple class corresponding to the tuple specification may be determined. A tuple object based on the tuple class is instantiated, and during runtime of the processing system. The tuple object is stored in a memory.

Abstract: A method system and program for performing a query in which context and information including opcodes accompanies an execution plan for the query. The opcodes determine operations to retrieve a set of bind variables. The execution plan and context information shared by a plurality of application programs.

Abstract: System and method for virtual data warehouses having table link capabilities are provided. In particular, some embodiments include a plurality of virtual data warehouses built on top of a data center running Apache Hive. Each virtual data warehouse can be modeled as a database and manage data in forms of database tables. The virtual data warehouse can include links which import tables from other virtual data warehouses by reference. Each link may contain partition metadata for the table partitions by dates of the source table and retention metadata to declare the needed retention time period for the partitions of the source table. The links can be dynamic and update when the corresponding source table receives new partitions or drops partitions. When a virtual data warehouse is migrated to another data center, the system can retain necessary table partitions to remain on the current data center based on the partition metadata and retention metadata of the links.

Abstract: Techniques for detecting patterns in one or more data streams. A pattern to be detected may be specified using a regular expression. Events received in a data stream are processed during runtime to detect occurrences of the specified pattern in the data stream.

Abstract: A data stream management system (DSMS) is designed to support a new user-defined function, by creating and using at least two structures as follows. A first structure (“metadata entry”) is created in response to a command for creation of the new function, and maps a single instance of a class to the function's name. A second structure is created with creation of an operator on receipt of each new continuous query that uses the new function. The second structure (“operator specific data structure”) contains a path to the newly-created instance, which path is obtained by looking up the first structure. Additional second structures are created on receipt of additional continuous queries which use the new function, but all second structures contain the same path. All continuous queries use the same instance. Repeated use of a single instance to compile and execute multiple queries eliminates repeated instantiation of the same function.

Abstract: Techniques for logging data pertaining to the operation of a data stream processing server. In one set of embodiments, logging configuration information can be received specifying a functional area of a data stream processing server to be logged. Based on the logging configuration information, logging can be dynamically enabled for objects associated with the functional area that are instantiated by the data stream processing server, and logging can be dynamically disabled for objects associated with the functional area that are discarded (or no longer used) by the data stream processing server. In another set of embodiments, a tool can be provided for visualizing the data logged by the data stream processing server.

Abstract: A computer is programmed to accept a command for creation of a new aggregation defined by a user to process data incrementally, one tuple at a time. One or more incremental function(s) in a set of instructions written by the user to implement the new aggregation maintain(s) locally any information that is to be passed between successive invocations, to support computing the aggregation for a given set of tuples as a whole. The user writes a set of instructions to perform the aggregation incrementally, including a plus function which is repeatedly invoked, only once, for each addition to a window of a message. The user also writes a minus function to be invoked with the message, to return the value of incremental aggregation over the window after removal of the message. In such embodiments, the computer does not maintain copies of messages in the window for use by aggregation function(s).

Abstract: Techniques for detecting patterns in data streams. A pattern to be detected may be specified using a regular expression. Events received in data streams are then processed during runtime to detect occurrences of the pattern specified by the regular expression in the data stream. If the pattern to be matched belongs to a certain predetermined class of patterns, then a pattern matching technique that is customized for that class of patterns is used during the runtime events processing.

Abstract: Techniques for logging data pertaining to the operation of a data stream processing server. In one set of embodiments, logging configuration information can be received specifying a functional area of a data stream processing server to be logged. Based on the logging configuration information, logging can be dynamically enabled for objects associated with the functional area that are instantiated by the data stream processing server, and logging can be dynamically disabled for objects associated with the functional area that are discarded (or no longer used) by the data stream processing server. In another set of embodiments, a tool can be provided for visualizing the data logged by the data stream processing server.

Abstract: Techniques for facilitating communication between one or more client applications and an event processing server. In one set of embodiments, a framework is provided that enables a client application to communicate with an event processing server via a standard database connectivity API, such as Java Database Connectivity (JDBC).

Abstract: Techniques for managing memory usage in a processing system are provided. This may be achieved by receiving a data stream including multiple tuples and determining a query plan that was generated for a continuous query applied to the multiple tuples in the data stream. The query plan may include one or more operators. Before scheduling an operator in the query plan, it is determined when an eviction is to be performed based a level of free memory of the processing system. An eviction candidate is determined and a page associated with the eviction candidate is evicted from the memory to a persistent storage.

Abstract: In one embodiment, a system for distributed event processing includes a first event processor and a second event processor. The first event processor receives event data associated with an event stream. The second event processor receives a query associated with the event stream. A request to the first event processor is generated by the second event processor based on the query for a first portion of event data associated with the event stream. The second event processor receives the first portion of event data from the first event processor. The second event processor then generates an event based on the first portion of event data.

Abstract: Techniques for facilitating communication between one or more client applications and an event processing server. In one set of embodiments, a framework is provided that enables a client application to communicate with an event processing server via a standard database connectivity API, such as Java Database Connectivity (JDBC).

Abstract: A new continuous query to a data stream management system (DSMS) may use several operators that are equivalent to operators currently being used by continuous queries that have been previously registered in the DSMS. To determine equivalence of operators, the DSMS checks at least the function and the data input to the operators. On finding equivalence, the DSMS modifies a global plan being executed, to use at least an existing subtree of operators during execution of the new continuous query, to generate a modified plan. The DSMS is also programmed to cause each relation source operator (which outputs a relation) to propagate a current state of the relation to each operator newly coupled to the relational operator. After propagation of current state to newly-coupled operators, each operator in the modified plan processes any new data and supplies the result to all operators coupled thereto, including newly-coupled operators and existing operators.

Abstract: Techniques are provided for accessing and storing files within a virtual folder. A virtual folder is a folder that is logically a part of a hierarchical folder structure of a file system repository, but the contents of the virtual folder are not physically stored in the file system repository. A client may issue, to an access mechanism, a request to perform a file operation on a file in a virtual folder. The access mechanism, which may be a database server or a component therein, may perform the file operation on the file in the virtual folder as if the file was stored in a particular folder within the hierarchical folder structure implemented by the file system repository. Advantageously, files may be physically stored outside of the file system repository, but still be a part of the hierarchical folder structure implemented by the file system repository.

Abstract: A computer is programmed to accept a command to create a new aggregation defined by a user during execution of continuous queries on streams of data. The computer is further programmed to thereafter accept and process new continuous queries using the new aggregation, in a manner similar to built-in aggregations. The user typically writes a set of instructions to perform the new aggregation, and identifies in the command, a location of the set of instructions. In response to such a command, the computer creates metadata identifying the new aggregation. The metadata is used to instantiate one aggregation for each group of data in a current window, grouped by an attribute identified in a new query.

Abstract: A system and method for enabling a second database instance to more quickly process a request to execute a database statement that has previously been executed by a first database instance is described. In one embodiment, the method involves sending the database statement from the first database instance to the second database instance, and generating by the second database instance one or more structures needed to prepare the statement for execution, such as a parse tree and an execution plan for the statement. If at some point in the future, the second database instance receives a request to execute the same statement, the above structures can be used for execution, thereby eliminating the need for one or more potentially time-consuming operations, such as generation of a parse tree or execution plan for the statement.