Abstract: Techniques for processing event streams are provided that retain attributes or fields from a current event or group during stream aggregation. An exemplary technique includes receiving a stream of events having an input shape having a plurality of attributes and executing a query on the stream of events. The executing includes: (i) summarizing data of a first attribute of the plurality of attributes based on an aggregate function to generate values of data for a result-set, (ii) collecting data of events with values that are identical for a second attribute of the plurality of attributes based on a group by function to group the result-set by the second attribute, and (iii) preserving data of a third attribute of the plurality of attributes based on a current function. The technique further includes outputting events having an output shape having the first attribute, the second attribute, and the third attribute.

Abstract: An event processing system is disclosed that processes events of an event stream, performs the recovery of events during system failure and preserves the state of the system reliably and accurately while achieving desired system performance. In an embodiment, the event processing system processes a first batch of events of a continuous input stream of events using a continuous query and generates an output stream of events related to an application. The event processing system identifies one or more operators of the continuous query and determines that an operator is a journaled operator. The event processing system generates a journaled snapshot of a current state of the system based on execution of the journaled operator on at least the first batch of events and stores the journaled snapshot of the current state of the system.

Abstract: Some event ordering requirements can be determined based on continuous event processing queries. Other event ordering requirements can be determined based on distribution flow types being used to distribute events from event streams to node executing the queries. Events from event streams can be ordered according to ordering semantics that are based on a combination of all of these event ordering requirements. Additionally, virtual computing nodes can be associated with constraints, and computing processors can be associated with capabilities. Virtual computing nodes for processing event streams can be assigned to execute on various computing processors based on both these constraints and capabilities. Additionally, for each of several events in an event stream, a ratio between a total latency and a communication latency can be for determined. Based on an average of these ratios, a quantity of reducing nodes that will be involved in a map-reduce operation can be selected.

Abstract: Techniques for managing continuous queries with archived relations are provided. In some examples, a query that includes at least a data stream may be identified. Additionally, the query may be initialized with at least a portion of historical data. Further, in some instances, the query may be evaluated based at least in part on the data stream and the portion of the historical data.

Abstract: Techniques for managing configurable data windows for streams and/or archived relations are provided. In some examples, a query that identifies at least a data stream or an archived relation may be identified. Additionally, a configurable window size may be identified. In some instances, the query may be initialized with at least a portion of the historical data based at least in part on the window size. Further, the query may be evaluated based at least in part on the data stream and the portion of the historical data.

Abstract: A method for detecting patterns across multiple input data streams related to one or more applications is disclosed. The method includes receiving multiple input data streams and generating one or more dynamic data types for one or more attributes of the input data streams. In some embodiments, the method may include combining the input data streams to generate a combined input data stream based on the dynamic data types and processing a continuous query over the combined data stream to detect a pattern.

Abstract: An event processing system for processing events in an event stream is disclosed. The system can execute instructions to receive a continuous data stream related to an application, identify an input geometry associated with the continuous data stream, generate a cluster of geometries based at least in part on the input geometry, generate an output geometry based at least in part on the cluster of geometries and a number of geometries in each cluster of the cluster of geometries, determining one or more partitions for the output geometry based on the cluster of geometries and the number of geometries in each cluster of the cluster of geometries, dynamically change a size of the one or more partitions for the output geometry, and transmitting the output geometry associated with the continuous data stream.

Abstract: An event processing system for processing events in an event stream is disclosed. The system can execute instructions to receive a continuous data stream related to an application, convert geometry of the continuous data stream into a relation, track a plurality of moving objects in the continuous data stream, determine a relationship between at least a first object and a second object (e.g., a first moving object and a second moving object in the plurality of moving objects) based at least on the geometry of a continuous data stream and the relation. and perform an action based at least one the determined relationship.

Abstract: An event processing system for processing events in an event stream is disclosed. The system can launch a first CQL engine in a cluster of CQL engines using a CQL engine tracking engine. The system can schedule, using the CQL engine tracking engine, the first CQL engine to process a batch of a continuous stream of input events related to an application. The system can track, using the CQL engine tracking engine, the first CQL engine to be scheduled for execution. The system can then execute, using the CQL engine tracking engine, the first CQL engine to process the batch of the continuous stream of input events to generate a set of output events related to the application.

Abstract: An event processing system for processing events in an event stream is disclosed. The system can execute instructions to receive a micro-batch stream of input events, process the input events using the CQL engine to generate a set of output events, generate, using a snapshot management algorithm implemented by the CQL engine, a snapshot of a current state of a system based at least in part on the set of output events related to the application, generate a first directory structure to access snapshot information associated with the snapshot of the current state of the system, generate a second directory structure to generate a list of snapshots associated with the current state of the system, and determine based at least in part on the snapshot management algorithm, a process to get, add, or clean the list of snapshots associated with the current state of the system.

Abstract: An event processing system for processing events in an event stream is disclosed. The system can execute instructions to receive a continuous query, apply a transformation to the continuous query to generate a query plan for the continuous query, monitor the continuous query using a monitoring transformation process, receive a micro-batch stream of input events related to an application, and process the input events of the micro-batch stream based at least in part on the monitoring transformation process to generate a set of output events related to the application.

Abstract: An event processing system for processing events in an event stream is disclosed. The system can execute instructions to receive a continuous query, apply a transformation to the continuous query to generate a query plan for the continuous query, transform the query plan using a transformation algorithm to generate a transformed query plan, receive a micro-batch stream of input events related to an application, process the input events of the micro-batch stream based at least in part on the transformed query plan to generate a set of output events related to the application, and store the set of output events related to the application in an output queue.

Abstract: A distributed event processing system is disclosed that receives a batch of events via a continuous data stream and performs the serialization of data in the batch of events. In certain embodiments, the system identifies a first data type of a first attribute for each event in a batch of events and determines a first type of data compression to be performed on data values represented by the first attribute. The system determines a first type of data compression to be performed on data values represented by the first attribute based on the first data type of the first attribute. The system then generates a first set of serialized data values for the first attribute. The system processes the first set of serialized data values against a set of one or more continuous queries to generate a first set of output events.

Abstract: An event processing system for processing events in an event stream is disclosed. The system is configured for configuring a stream processor to micro-batch incoming events from a stream source. The system is also configured for generating a single timestamp for a micro-batch of the incoming events and/or receiving the micro-batch of the incoming events from the stream source. The system can also be configured for assigning the single timestamp to each event of the micro-batch and/or generating separate timestamp values for each respective event of the micro-batch. In some examples, the system can also be configured for assigning, for each respective event of the micro-batch, an individual one of the separate timestamp values.

Abstract: An event processing system for processing events in an event stream is disclosed. The system is configured for determining a stage for a continuous query language (CQL) query being processed by an event processing system and/or determining a stage type associated with the stage. The system is also configured for determining a transformation to be computed for the stage based at least in part on the stage type and/or determining a classification for the CQL query based at least in part on a plurality of rules. The system can also be configured for generating a transformation in a Directly Acyclic Graph (DAG) of a data transformation pipeline for the stage based at least in part on the partitioning criteria for the stage. In some examples, the system can also be configured for determining a partitioning of the stage based at least in part on the transformation.

Abstract: Systems and methods for guaranteeing the event order for multi-stage processing in distributed systems are disclosed. In some examples, a warm-up period is used to accurately determine a starting point for ordered events of an event stream. Skip-beats may be utilized as dummy events so that the event processor does not wait too long for events that were filtered out at earlier stages.

Abstract: Systems and methods for query decomposition for scalability of continuous query processing are disclosed. In some examples, it is determined whether a query is fully stateful. If it is, the query may be divided into two functions, an aggregate function and a global function. The aggregate functions may be distributed across multiple distributed nodes and executed in parallel. The global function may then receive the results from the distributed nodes and join the results to achieve the fully stateful result.

Abstract: Systems and methods for guaranteeing the event order for multi-stage processing in distributed systems are disclosed. In some examples, a warm-up period is used to accurately determine a starting point for ordered events of an event stream. Skip-beats may be utilized as dummy events so that the event processor does not wait too long for events that were filtered out at earlier stages.

Abstract: An event processing system is disclosed that processes events of an event stream and performs the recovery of events during system failure. The recovery of events is performed by generating output sequence numbers for each deterministic output event generated by the event processing system. In an embodiment, the event processing system determines a current output sequence number of a most recently transmitted output event in a output stream of events, a last output sequence number of an output event corresponding to a most recently processed batch of input events and determines a set of one or more output events of the output stream to be transmitted based on the current sequence number and the last output sequence number. The event processing system then transmits the output events related to the application.

Abstract: An event processing system is disclosed that processes events of an event stream, performs the recovery of events during system failure and preserves the state of the system reliably and accurately while achieving desired system performance. In an embodiment, the event processing system processes a first batch of events of a continuous input stream of events using a continuous query and generates an output stream of events related to an application. The event processing system identifies one or more operators of the continuous query and determines that an operator is a journaled operator. The event processing system generates a journaled snapshot of a current state of the system based on execution of the journaled operator on at least the first batch of events and stores the journaled snapshot of the current state of the system.