Abstract: Two streams of data items are received. A first estimated processing time for the first stream of data items and a second estimated processing time for the second stream of data items are determined. Data items of the first stream and data items of the second stream are dynamically maintained in a first buffer and a second buffer respectively. The data items of the second stream maintained in the second buffer have associated event times that are within a first join window based on the first estimated processing time for the first stream. A selected data item of the first stream maintained in the first buffer is joined with one or more data items of the second stream maintained in the second buffer that have associated event times that are within a second join window based on an event time associated with the selected data item of the first stream.

Abstract: Two streams of data items are received. A first estimated processing time for the first stream of data items and a second estimated processing time for the second stream of data items are determined. Data items of the first stream and data items of the second stream are dynamically maintained in a first buffer and a second buffer respectively. The data items of the second stream maintained in the second buffer have associated event times that are within a first join window based on the first estimated processing time for the first stream. A selected data item of the first stream maintained in the first buffer is joined with one or more data items of the second stream maintained in the second buffer that have associated event times that are within a second join window based on an event time associated with the selected data item of the first stream.

Abstract: Two streams of data items are received. A first estimated processing time for the first stream of data items and a second estimated processing time for the second stream of data items are determined. Data items of the first stream and data items of the second stream are dynamically maintained in a first buffer and a second buffer respectively. The data items of the second stream maintained in the second buffer have associated event times that are within a first join window based on the first estimated processing time for the first stream. A selected data item of the first stream maintained in the first buffer is joined with one or more data items of the second stream maintained in the second buffer that have associated event times that are within a second join window based on an event time associated with the selected data item of the first stream.

Abstract: A system, method and computer program product for adaptive incremental checkpointing an operator state in a streaming application. The system and method enable reduced costs of checkpointing an operator state in a streaming application, by i) logging updates to operator state and checkpointing operator state in either base (full state) or delta (logged updates) form, ii) dynamically and adaptively adjusting checkpointing options, and iii) maintaining dependencies between checkpoints to allow automatic restoration and checkpoint recycling.

Abstract: A system to manage clustering vertices of a streaming graph includes a memory storing a computer program, a vertex structure storing vertices of the graph, and a cluster structure storing clusters of the graph, and a processor. Each cluster structure comprises a plurality of edge structures represent edges of the graph and each edge structure comprises two of the vertices. The processor is configured to execute the program to cluster the vertices using the structures and a maximum cluster size that is constant.

Abstract: A streams manager intelligently restores multiple consistent regions of streaming applications in a streaming environment to increase performance and reduce overloading of other operators and resources. The streams manager detects a failure that affects multiple consistent regions and determines a preferred schedule to restore the multiple consistent regions by analyzing the relationships of the operators and resources associated with the multiple consistent regions. The streams manager may create a dependency graph that maps the interrelationships of the operators and resources associated with the consistent regions of the streaming application and then consult the dependency graph to determine the schedule to restore the multiple consistent regions affected by the failure.

Abstract: A system, method and computer program product for adaptive incremental checkpointing an operator state in a streaming application. The system and method enable reduced costs of checkpointing an operator state in a streaming application, by i) logging updates to operator state and checkpointing operator state in either base (full state) or delta (logged updates) form, ii) dynamically and adaptively adjusting checkpointing options, and iii) maintaining dependencies between checkpoints to allow automatic restoration and checkpoint recycling.

Abstract: A streams manager intelligently restores multiple consistent regions of streaming applications in a streaming environment to increase performance and reduce overloading of other operators and resources. The streams manager detects a failure that affects multiple consistent regions and determines a preferred schedule to restore the multiple consistent regions by analyzing the relationships of the operators and resources associated with the multiple consistent regions. The streams manager may create a dependency graph that maps the interrelationships of the operators and resources associated with the consistent regions of the streaming application and then consult the dependency graph to determine the schedule to restore the multiple consistent regions affected by the failure.

Abstract: A streams manager intelligently restores multiple consistent regions of streaming applications in a streaming environment to increase performance and reduce overloading of other operators and resources. The streams manager detects a failure that affects multiple consistent regions and determines a preferred schedule to restore the multiple consistent regions by analyzing the relationships of the operators and resources associated with the multiple consistent regions. The streams manager may create a dependency graph that maps the interrelationships of the operators and resources associated with the consistent regions of the streaming application and then consult the dependency graph to determine the schedule to restore the multiple consistent regions affected by the failure.

Abstract: A system, method and computer program product for adaptive incremental checkpointing an operator state in a streaming application. The system and method enable reduced costs of checkpointing an operator state in a streaming application, by i) logging updates to operator state and checkpointing operator state in either base (full state) or delta (logged updates) form, ii) dynamically and adaptively adjusting checkpointing options, and iii) maintaining dependencies between checkpoints to allow automatic restoration and checkpoint recycling.

Abstract: A system to manage clustering vertices of a streaming graph includes a memory storing a computer program, a vertex structure storing vertices of the graph, and a cluster structure storing clusters of the graph, and a processor. Each cluster structure comprises a plurality of edge structures represent edges of the graph and each edge structure comprises two of the vertices. The processor is configured to execute the program to cluster the vertices using the structures and a maximum cluster size that is constant.

Abstract: Embodiments of the disclosure include a method for providing stream processing with runtime adaptation includes registering one or more events, wherein each of the events is associated with a stream processing application. The method also includes monitoring, by a processor, for an occurrence of the one or more events associated with the stream processing application, wherein each of the one or more events is associated with one or more runtime metrics. The method further includes receiving an event notification, wherein the event notification includes event identification and an event context and executing an adaptation of the stream processing application.

Abstract: Embodiments of the disclosure include a system for providing stream processing with runtime adaptation, having a stream processing application that receives an incoming data stream and a runtime infrastructure configured to execute the stream processing application. The system also includes an orchestrator configured to communicate with the runtime infrastructure and the stream processing application, the orchestrator configured to perform a method. The method includes registering one or more events, wherein each of the events is associated with a stream processing application. The method also includes monitoring, by a processor, for an occurrence of the one or more events associated with the stream processing application, wherein each of the one or more events is associated with one or more runtime metrics.

Abstract: A method for managing distributed computer data stream acceleration devices is provided that utilizes distributed acceleration devices on nodes within the computing system to process inquiries by programs executing on the computing system. The available nodes and available acceleration devices in the computing system are identified. In addition, a plurality of virtual acceleration device definitions is created. Each virtual acceleration device definition includes attributes used to configure at least one of the plurality of identified acceleration devices. When an inquiry containing an identification of computing system resources to be used in processing the inquiry is received, at least one virtual acceleration device definition that is capable of configuring an acceleration device in accordance with the computing system resources identified by the inquiry is identified.

Abstract: In one embodiment, the invention comprises partial fault tolerant stream processing applications. One embodiment of a method for implementing partial fault tolerance in a stream processing application comprising a plurality of stream operators includes: defining a quality score function that expresses how well the application is performing quantitatively, injecting a fault into at least one of the plurality of operators, assessing an impact of the fault on the quality score function, and selecting at least one partial fault-tolerant technique for implementation in the application based on the quantitative metric-driven assessment.

Abstract: In one embodiment, the invention is a method and apparatus for failure recovery for stream processing applications. One embodiment of a method for providing a failure recovery mechanism for a stream processing application includes receiving source code for the stream processing application, wherein the source code defines a fault tolerance policy for each of the components of the stream processing application, and wherein respective fault tolerance policies defined for at least two of the plurality of components are different, generating a sequence of instructions for converting the state(s) of the component(s) into a checkpoint file comprising a sequence of storable bits on a periodic basis, according to a frequency defined in the fault tolerance policy, initiating execution of the stream processing application, and storing the checkpoint file, during execution of the stream processing application, at a location that is accessible after failure recovery.

Abstract: Embodiments of the disclosure include a system for providing stream processing with runtime adaptation, having a stream processing application that receives an incoming data stream and a runtime infrastructure configured to execute the stream processing application. The system also includes an orchestrator configured to communicate with the runtime infrastructure and the stream processing application, the orchestrator configured to perform a method. The method includes registering one or more events, wherein each of the events is associated with a stream processing application. The method also includes monitoring, by a processor, for an occurrence of the one or more events associated with the stream processing application, wherein each of the one or more events is associated with one or more runtime metrics.

Abstract: Embodiments of the disclosure include a method for providing stream processing with runtime adaptation includes registering one or more events, wherein each of the events is associated with a stream processing application. The method also includes monitoring, by a processor, for an occurrence of the one or more events associated with the stream processing application, wherein each of the one or more events is associated with one or more runtime metrics. The method further includes receiving an event notification, wherein the event notification includes event identification and an event context and executing an adaptation of the stream processing application.

Abstract: In one embodiment, the invention comprises partial fault tolerant stream processing applications. One embodiment of a method for implementing partial fault tolerance in a stream processing application comprising a plurality of stream operators includes: defining a quality score function that expresses how well the application is performing quantitatively, injecting a fault into at least one of the plurality of operators, assessing an impact of the fault on the quality score function, and selecting at least one partial fault-tolerant technique for implementation in the application based on the quantitative metric-driven assessment.

Abstract: In one embodiment, the invention comprises partial fault tolerant stream processing applications. One embodiment of a method for implementing partial fault tolerance in a stream processing application comprising a plurality of stream operators includes: defining a quality score function that expresses how well the application is performing quantitatively, injecting a fault into at least one of the plurality of operators, assessing an impact of the fault on the quality score function, and selecting at least one partial fault-tolerant technique for implementation in the application based on the quantitative metric-driven assessment.