Abstract: Methods and apparatus for integrating relational and hierarchical data, schema definitions, and queries in a data processing system are provided. One or more schema definitions are converted into an intermediate schema language component of an intermediate data language when one or more schema definitions are provided. One or more query expressions are converted into an intermediate query language component of the intermediate data language when one or more query expressions are provided. The intermediate schema language component or the intermediate query language component is compiled in an intermediate data language processing engine into a run-time representation in accordance with a relational-hierarchical analysis.

Abstract: A distributed messaging system supporting stateful subscriptions is provided. A stateful publish-subscribe system extends the functionality of the content-based approach to include more general state-valued expressions. Stateful subscriptions may refer to one or more message histories and may include more complex expressions. Therefore, subscribers may receive different information than that provided in the published messages. A plurality of broker machines is provided to deliver messages sent by publishing clients toward subscribing clients based upon the contents of the messages and stateful transformations requested by the subscribing clients. Subscription specifications are analyzed by a compiler and converted into a collection of transform objects and view objects. The messaging system builds a structure containing all transform objects and view objects needed for all intermediate and subscribed views of all subscriptions.

Abstract: Controlling live migration of a running application between a source machine and a target machine is provided. At least one application characteristic and at least one network characteristic are obtained. An objective function that defines a cost to migrate from the source machine to the target machine as a function such as a weighted combination of at least one cost property is obtained. The objective function is optimized using the at least one application characteristic and the at least one network characteristic to determine a policy specifying when to switch execution of the running application from the source machine to the target machine. Live migration of the application from the source machine is controlled to send pages from the source machine to the target machine in accordance with the policy, and to switch the execution of the application from the source machine to the target machine at a state specified by the policy.

Abstract: A computer implemented method, system and computer program product for delivering published messages to at least one subscriber in a publish-subscribe messaging system. A computer implemented method for delivering published messages to a subscriber includes receiving a plurality of message streams, and analyzing at least one message from each of the plurality of message streams in accordance with criteria established by the subscriber specifying a message content and a message content quality. At least one message that satisfies the criteria is delivered to the subscriber.

Abstract: The present invention solves the disadvantages of the prior art and provides a distributed messaging system supporting stateful subscriptions. A stateful publish-subscribe system extends the functionality of the content-based approach to include more general state-valued expressions. A plurality of broker machines is provided to deliver messages sent by publishing clients toward subscribing clients based upon the contents of the messages and stateful transformations requested by the subscribing clients. Subscription specifications are analyzed by a compiler and converted into a collection of message transforms and views. A deployment service component deploys tasks to optimize system performance. A monitoring services component detects a possible need to reconfigure. A performance optimization service component computes new assignment of transforms.

Abstract: Method, system and computer usable program code for dynamic component placement in an event processing system having producers, consumers, a plurality of nodes between the producers and the consumers, and a flow graph representing computational components to be executed between the producers and the consumers. A description of a change to the system is received. At each node, next-hop neighbor nodes for each consumer are identified. A routing value is assigned to each next-hop neighbor node for each consumer. Using the routing values in a context of the change, a performance cost of the system is estimated based on hypothetical changed placements of the computational components at nodes along paths from a producer to a consumer through the next-hop neighbor nodes for each consumer, and a changed placement of the computational components that minimizes performance cost of the system relative to the hypothetical changed placements is selected.

Abstract: Method, apparatus and computer program product for fault recovery in a distributed stateful publish-subscribe system. The system includes the capability of recovering from failures that may occur when a stateful publish-subscribe service is implemented on an overlay network. Such failures may include, for example, temporary crashes of broker machines, and network errors causing messages to possibly be lost, duplicated or delivered out of order. The system requires stable storage logging only when a published event enters the system, and requires that logged messages be retrieved from stable storage only in the event all brokers between a failed link or broker and the publishing sites have failed. The publish-subscribe system of the present invention does not require that broker-to-broker connections use reliable FIFO protocols, such as TCP/IP, but may advantageously use faster, less reliable protocols.

Abstract: Systems, methods and computer program products for optimizing an objective function (e.g. the overall end-to-end runtime latency) of a flow graph of message transformations which are placed onto an overlay network of broker machines by replicating stateless transformations in a guided manner. Exemplary embodiments include a guided replication method, including given an initial transformation graph and a network topology, running a placement algorithm to obtain top N best task-to-broker assignments, identifying the R-task which forms the performance bottleneck, checking if the objective function improvement drops below a designated threshold, if yes, then selecting the optimal task-to-broker assignment and the program terminates; otherwise, replicating the R-task, optimally distributing the data load among R-task's replicas, deriving a set of starting points from the above obtained top N task-to-broker assignments, and re-executing the placement algorithm with these starting points in an iterative fashion.

Abstract: A computer implemented method, apparatus, and computer usable program code for controlling when to send messages in a stream processing system. A policy is determined by utilizing probability statistics and a cost function prior to stream processing. The policy specifies under which conditions a message is sent eagerly and under which other conditions the message is delayed. A filter is operated during stream processing that selects which of the messages to send from a sending transform based upon a threshold. A controller is operated during stream processing that observes a current state of a receiving transform and that applies the policy based on the current state to determine whether to change the threshold.

Abstract: A mechanism for controlling live migration of a running application between a source machine and a target machine. At least one application characteristic and at least one network characteristic are obtained. An objective function that defines a cost to migrate from the source machine to the target machine as a function such as a weighted combination of at least one cost property is obtained. The objective function is optimized using the at least one application characteristic and the at least one network characteristic to determine a policy specifying when to switch execution of the running application from the source machine to the target machine. Live migration of the application from the source machine is controlled to send pages from the source machine to the target machine in accordance with the policy, and to switch the execution of the application from the source machine to the target machine at a state specified by the policy.

Abstract: A method and system for controlling message flow in distributed stream processing. State transition probabilities in a Markov model having one state per staleness value of data are determined for sending or withholding updates of data to subscribers using expected message rates from an information provider. A cost function annotates each state transition in the model with a state transition cost for each decision to “send” or “withhold”. A propagation policy specifying whether to send or withhold the message is determined for each state. The propagation policy is then deployed. If a new message comprising an update of data is received during a lapsed time unit, a staleness value of the data held by subscribers is increased. The propagation policy is used to determine whether to send or withhold the message. If the message should be sent, the message is propagated and the staleness value of the data is reset.

Abstract: Computer implemented method, system and computer usable program code for achieving transactional quality of service in a transactional object store system. A transaction is received from a client and is executed, wherein the transaction comprises reading a read-only derived object, or reading or writing another object, and ends with a decision to request committing the transaction or a decision to request aborting the transaction. Responsive to a decision to request committing the transaction, wherein the transaction comprises writing a publishing object, events are delivered to event stream processing queries, and are executed in parallel with executing of the transaction.

Abstract: Methods and apparatus for integrating relational and hierarchical data, schema definitions, and queries in a data processing system are provided. One or more schema definitions are converted into an intermediate schema language component of an intermediate data language when one or more schema definitions are provided. One or more query expressions are converted into an intermediate query language component of the intermediate data language when one or more query expressions are provided. The intermediate schema language component or the intermediate query language component is compiled in an intermediate data language processing engine into a run-time representation in accordance with a relational-hierarchical analysis.

Abstract: Method, system and computer usable program code for dynamic component placement in an event processing system having producers, consumers, a plurality of nodes between the producers and the consumers, and a flow graph representing computational components to be executed between the producers and the consumers. A description of a change to the system is received. At each node, next-hop neighbor nodes for each consumer are identified. A routing value is assigned to each next-hop neighbor node for each consumer. Using the routing values in a context of the change, a performance cost of the system is estimated based on hypothetical changed placements of the computational components at nodes along paths from a producer to a consumer through the next-hop neighbor nodes for each consumer, and a changed placement of the computational components that minimizes performance cost of the system relative to the hypothetical changed placements is selected.

Abstract: Systems, methods and computer program products for optimizing an objective function (e.g. the overall end-to-end runtime latency) of a flow graph of message transformations which are placed onto an overlay network of broker machines by replicating stateless transformations in a guided manner. Exemplary embodiments include a guided replication method, including given an initial transformation graph and a network topology, running a placement algorithm to obtain top N best task-to-broker assignments, identifying the R-task which forms the performance bottleneck, checking if the objective function improvement drops below a designated threshold, if yes, then selecting the optimal task-to-broker assignment and the program terminates; otherwise, replicating the R-task, optimally distributing the data load among R-task's replicas, deriving a set of starting points from the above obtained top N task-to-broker assignments, and re-executing the placement algorithm with these starting points in an iterative fashion.

Abstract: A computer implemented method for delivering published messages to at least one subscriber in a publish-subscribe messaging system. The method for delivering published messages to a subscriber includes receiving a plurality of message streams and analyzing at least one message from each of the plurality of message streams in accordance with criteria established by the subscriber specifying a message content and a message content quality. At least one message that satisfies the criteria is delivered to the subscriber.

Abstract: A method and system for controlling message flow in distributed stream processing. State transition probabilities in a Markov model having one state per staleness value of data are determined for sending or withholding updates of data to subscribers using expected message rates from an information provider. A cost function annotates each state transition in the model with a state transition cost for each decision to “send” or “withhold”. A propagation policy specifying whether to send or withhold the message is determined for each state. The propagation policy is then deployed. If a new message comprising an update of data is received during a lapsed time unit, a staleness value of the data held by subscribers is increased. The propagation policy is used to determine whether to send or withhold the message. If the message should be sent, the message is propagated and the staleness value of the data is reset.

Abstract: A system and method for maintaining checkpoints of a keyed data structure using a sequential log are provided. The system and method are built upon the idea of writing all updates to a keyed data structure in a physically sequential location. The system and method make use of a two-stage operation. In a first stage, various values of the same key are combined such that only the latest value in a given checkpoint interval is maintained for writing to persistent storage. In a second stage of the operation, a periodic write operation is performed to actually store the latest values for the key-value pairs to a persistent storage. All such updates to key-value pairs are written to the end of a sequential log. This minimizes the physical storage input/output (I/O) overhead for the write operations. Data structures are provided for identifying the most current entries in the sequential log for each key-value pair.

Abstract: A computer implemented method, apparatus, and computer usable program code for controlling when to send messages in a stream processing system. A policy is determined by utilizing probability statistics and a cost function prior to stream processing. The policy specifies under which conditions a message is sent eagerly and under which other conditions the message is delayed. A filter is operated during stream processing that selects which of the messages to send from a sending transform based upon a threshold. A controller is operated during stream processing that observes a current state of a receiving transform and that applies the policy based on the current state to determine whether to change the threshold.

Abstract: The present invention solves the disadvantages of the prior art and provides a distributed messaging system supporting stateful subscriptions. A stateful publish-subscribe system extends the functionality of the content-based approach to include more general state-valued expressions. A plurality of broker machines is provided to deliver messages sent by publishing clients toward subscribing clients based upon the contents of the messages and stateful transformations requested by the subscribing clients. Subscription specifications are analyzed by a compiler and converted into a collection of message transforms and views. A deployment service component deploys tasks to optimize system performance. A monitoring services component detects a possible need to reconfigure. A performance optimization service component computes new assignment of transforms.