Abstract: Techniques for performing capacity planning for applications running on a computational infrastructure are provided. The techniques include instrumenting an application under development to receive one or more performance metrics under a physical deployment plan, receiving the one or more performance metrics from the computational infrastructure hosting one or more applications that are currently running, using a predictive inference engine to determine how the application under development can be deployed, and using the determination to perform capacity planning for the applications on the computational infrastructure.

Abstract: Architecture-dependent assets are automatically built and retargeted. An asset originally built for one architecture is downloaded and automatically retargeted on another architecture. This automatically retargeting may be performed on demand, at runtime.

Abstract: An application provisioning model is automatically created. The model is created from a high-level application and specifies dependencies of the application. It is used to provision the application on one or more nodes or other actions.

Abstract: A publisher-subscriber system includes a broker configured to receive and distribute at least one data stream from publishers to subscribers in accordance with subscriptions. An annotator/classifier is configured to annotate or classify the data stream by employing one or more additional stream-level attributes to create an annotated data stream wherein the annotated data stream applies the stream-level attribute to all messages therein. Subscribers can subscribe to the annotated data stream created by the annotator/classifier.

Abstract: A method is provided for generating a resource function estimate of resource usage by an instance of a processing element configured to consume zero or more input data streams in a stream processing system having a set of available resources that comprises receiving at least one specified performance metric for the zero or more input data streams and a processing power of the set of available resources, wherein one specified performance metric is stream rate; generating a multi-part signature of executable-specific information for the processing element and a multi-part signature of context-specific information for the instance; accessing a database of resource functions to identify a static resource function corresponding to the executable-specific information and a context-dependent resource function corresponding to the context-specific information; combining the static resource function and the context-dependent resource function to form a composite resource function for the instance; and applying the res

Abstract: One embodiment of a method for providing failure recovery for an application that processes stream data includes providing a plurality of operators, each of the operators comprising a software element that performs an operation on the stream data, creating one or more groups, each more groups including a subset of the operators, assigning a policy to each of the groups, the policy comprising a definition of how the subset of the operators will function in the event of a failure, and enforcing the policy through one or more control elements that are interconnected with the operators.

Abstract: Techniques for determining feasibility of a set of one or more operator partitioning constraints are provided. The techniques include receiving one or more sets of operator partitioning constraints, wherein each set of one or more constraints define one or more desired conditions for grouping together of operators into partitions and placing partitions on hosts, wherein each operator is embodied as software that performs a particular function, processing each set of one or more operator partitioning constraints to determine feasibility of each set of one or more operator partitioning constraints, creating and outputting one or more candidate partitions and one or more host placements for each set of feasible partitioning constraints, and creating and outputting a certificate of infeasibility for each set of infeasible partitioning constraints, wherein the certificate of infeasibility outlines one or more reasons for infeasibility.

Abstract: A keyed delay is used in the processing of streaming data to decrease the processing performed and the output provided. A first event, within a particular window, having a particular key starts a delay condition. Arriving events with the same key replace the previous arrival for that key until the delay condition is satisfied. In response thereto, the latest event with that key is output.

Abstract: Keyed aggregation is used in the processing of streaming data to streamline processing to provide higher throughput and decreased use of resources. The most recent event for each unique replacement key value(s) is maintained. In response to an incoming event having a same key as a previous event, the effect on an aggregation of the previous event is removed. The aggregation is then updated with one or more values from the arriving event and the updated aggregation is output.

Abstract: A method to optimize performance of an operator on a computer system includes determining whether the system is busy, decreasing a software thread level within the operator if the system is busy, and increasing the software thread level within the operator if the system is not busy and a performance measure of the system at a current software thread level of the operator is greater than a performance measure of the system when the operator has a lower software thread level.

Abstract: A keyed join is used in the processing of streaming data to streamline processing to provide higher throughput and decreased use of resources. The most recent event for each unique replacement key value(s) is maintained substituting older events with the same key. An incoming event is joined with the data received from one or more other data sources, and the correlations are output.

Abstract: Techniques for partitioning an operator flow graph are provided. The techniques include receiving source code for a steam processing application, wherein the source code comprises an operator flow graph, wherein the operator flow graph comprises a plurality of operators, receiving profiling data associated with the plurality of operators and one or more processing requirements of the operators, defining a candidate partition as a coalescing of one or more of the operators into one or more sets of processing elements (PEs), using the profiling data to create one or more candidate partitions of the processing elements, using the one or more candidate partitions to choose a desired partitioning of the operator flow graph, and compiling the source code into an executable code based on the desired partitioning.

Abstract: Techniques for composing an application resource model in a data stream processing system are disclosed. The application resource model may be used to understand what resources will be consumed by an application when executed by the data stream processing system. For example, a method for composing an application resource model for a data stream processing system comprises the following steps. One or more operator-level metrics are obtained from an execution of a data stream processing application in accordance with a first configuration. The application is executed by one or more nodes of the data stream processing system, and the application is comprised of one or more processing elements that are comprised of one or more operators. One or more operator-level resource functions are generated based on the obtained one or more operator-level metrics. A processing element-level resource function is generated based on the one or more generated operator-level resource functions.

Abstract: Techniques for compiling a data stream processing application are provided. The techniques include receiving, by a compiler executing on a computer system, source code for a data stream processing application, wherein the source code comprises source code for a plurality of operators, each of which performs a data processing function, determining, by the compiler, one or more characteristics of operators within the data stream processing application, grouping, by the compiler, the operators into one or more execution containers based on the one or more characteristics, and compiling, by the compiler, the source code for the data stream processing application into executable code, wherein the executable code comprises a plurality of execution units, wherein each execution unit contains one or more of the operators, wherein each operator is assigned to an execution unit based on the grouping, and wherein each execution unit is to be executed in a partition.

Abstract: Techniques for dynamically modifying inter-connections between components in an application are provided. The techniques include receiving a data producer profile for each output port within a software application to be executed on one or more processors, receiving a data subscription profile for each input port of each component of the application, establishing connections between the output ports and the input ports of the components in the application based on a comparison of each data producer profile and each data subscription profile, executing the application on one or more processors to process streams of data, receiving either or both of a new data producer profile or a new data subscription profile during the execution of the application, and establishing at least one new connection between an output port and an input port based upon a revised comparison of the received data profiles that include the new data profile.

Abstract: One embodiment of a method for on-demand de-marshalling of a network message includes receiving a first network message at a receiver device, wherein the first network message was sent from a sender device, and wherein the first network message comprises one or more attributes that characterize an object at the sender device, and further wherein the first network message is in an encoded format according to a network protocol, storing the first network message in a buffer of the receiver device in the encoded format, identifying, at the receiver device and prior to de-marshalling of any of the attributes, that at least one of the attributes is to be manipulated by the receiver device, de-marshalling, in response to the identifying, the identified attribute(s), and manipulating the identified attribute(s) at the receiver device.

Abstract: In one embodiment, the invention is a method and apparatus for use of vectorization instruction sets. One embodiment of a method for generating vector instructions includes receiving source code written in a high-level programming language, wherein the source code includes at least one high-level instruction that performs multiple operations on a plurality of vector operands, and compiling the high-level instruction(s) into one or more low-level instructions, wherein the low-level instructions are in an instruction set of a specific computer architecture.

Abstract: In one embodiment, the invention is a method and apparatus for failure recovery for stream processing applications. One embodiment of a method for providing a failure recovery mechanism for a stream processing application includes receiving source code for the stream processing application, wherein the source code defines a fault tolerance policy for each of the components of the stream processing application, and wherein respective fault tolerance policies defined for at least two of the plurality of components are different, generating a sequence of instructions for converting the state(s) of the component(s) into a checkpoint file comprising a sequence of storable bits on a periodic basis, according to a frequency defined in the fault tolerance policy, initiating execution of the stream processing application, and storing the checkpoint file, during execution of the stream processing application, at a location that is accessible after failure recovery.

Abstract: One embodiment of a method for constructing executable code for a component-based application includes receiving a request to compile source code for the component-based application, wherein the request identifies the source code, and wherein the source code comprises a plurality of source code components, each of the source code components implementing a different component of the application, and performing a series of steps for each source code component where the series of steps includes: deriving a signature for the source code component, retrieving a stored signature corresponding to a currently available instance of executable code for the source code component, comparing the derived signature with the stored signature, compiling the source code component into the executable code when the derived signature does not match the stored signature, and obtaining the executable code for the source code component from a repository when the derived signature matches the stored signature.

Abstract: A method for cross-triggering transitions in independent finite state machines is provided. For a given plurality of finite state machine definitions having a plurality of states and a plurality of transitions among the states, two or more independent instances of the plurality of finite state machine definitions are created. In addition, associations between two or more of the independent finite state machine instances are identified. The method uses cross-triggering rules that identify a condition in a first one of the associated independent finite state machine instances that triggers a transition action in a second one of the associated independent finite state machine instances. Each cross-triggering rule is triggered upon an occurrence of the cross-triggering rule condition, and the transition action in the second associated independent finite state machine instance is performed in response to the triggering of the cross-triggering rule.