Abstract: In one embodiment, the invention comprises partial fault tolerant stream processing applications. One embodiment of a method for implementing partial fault tolerance in a stream processing application comprising a plurality of stream operators includes: defining a quality score function that expresses how well the application is performing quantitatively, injecting a fault into at least one of the plurality of operators, assessing an impact of the fault on the quality score function, and selecting at least one partial fault-tolerant technique for implementation in the application based on the quantitative metric-driven assessment.

Abstract: Data analysis applications include model building components and stream processing components. To increase utility of the data analysis application, in one embodiment, the model building component of the data analysis application is managed. Management includes resource allocation and/or configuration adaptation of the model building component, as examples.

Abstract: A virtual execution environment (VEE) for a streaming Intermediate Language (IL), wherein the streaming IL represents a streaming program, communicates streaming data in queues, stores data-at-rest in variables, and determines data by functions, where inputs are read from the queues and the variables, and outputs are written to the queues and the variables.

Abstract: Techniques for performing capacity planning for applications running on a computational infrastructure are provided. The techniques include instrumenting an application under development to receive one or more performance metrics under a physical deployment plan, receiving the one or more performance metrics from the computational infrastructure hosting one or more applications that are currently running, using a predictive inference engine to determine how the application under development can be deployed, and using the determination to perform capacity planning for the applications on the computational infrastructure.

Abstract: Architecture-dependent assets are automatically built and retargeted. An asset originally built for one architecture is downloaded and automatically retargeted on another architecture. This automatically retargeting may be performed on demand, at runtime.

Abstract: An application provisioning model is automatically created. The model is created from a high-level application and specifies dependencies of the application. It is used to provision the application on one or more nodes or other actions.

Abstract: A method for processing a first data stream specifying locations of a user at different times and a second data stream specifying values of a monitored attribute at a location of interest at different times includes: receiving a location-centric trigger specifying at least one spatial predicate condition relative to the location of interest and at least one non-spatial predicate condition relevant to the location of interest, calculating a safe region that includes locations whose probability of satisfying the spatial predicate condition falls below a first threshold, calculating a safe value container that includes values whose probability of satisfying the non-spatial predicate condition falls below a second threshold, and processing the first data stream and the second data stream against the location-centric trigger, by considering only those locations that are not contained within the safe region and only those values that are not contained within the safe value container.

Abstract: One embodiment of a method for providing failure recovery for an application that processes stream data includes providing a plurality of operators, each of the operators comprising a software element that performs an operation on the stream data, creating one or more groups, each more groups including a subset of the operators, assigning a policy to each of the groups, the policy comprising a definition of how the subset of the operators will function in the event of a failure, and enforcing the policy through one or more control elements that are interconnected with the operators.

Abstract: Techniques for searching multi-dimensional data are provided. The techniques include providing a parallelization framework for a search algorithm, wherein the search algorithm exposes one or more architecture-sensitive tunable optimization parameters, and using the one or more architecture-sensitive tunable optimization parameters to tune the search algorithm to search multi-dimensional data in any underlying architecture.

Abstract: Techniques for determining feasibility of a set of one or more operator partitioning constraints are provided. The techniques include receiving one or more sets of operator partitioning constraints, wherein each set of one or more constraints define one or more desired conditions for grouping together of operators into partitions and placing partitions on hosts, wherein each operator is embodied as software that performs a particular function, processing each set of one or more operator partitioning constraints to determine feasibility of each set of one or more operator partitioning constraints, creating and outputting one or more candidate partitions and one or more host placements for each set of feasible partitioning constraints, and creating and outputting a certificate of infeasibility for each set of infeasible partitioning constraints, wherein the certificate of infeasibility outlines one or more reasons for infeasibility.

Abstract: Keyed aggregation is used in the processing of streaming data to streamline processing to provide higher throughput and decreased use of resources. The most recent event for each unique replacement key value(s) is maintained. In response to an incoming event having a same key as a previous event, the effect on an aggregation of the previous event is removed. The aggregation is then updated with one or more values from the arriving event and the updated aggregation is output.

Abstract: A keyed delay is used in the processing of streaming data to decrease the processing performed and the output provided. A first event, within a particular window, having a particular key starts a delay condition. Arriving events with the same key replace the previous arrival for that key until the delay condition is satisfied. In response thereto, the latest event with that key is output.

Abstract: A method to optimize performance of an operator on a computer system includes determining whether the system is busy, decreasing a software thread level within the operator if the system is busy, and increasing the software thread level within the operator if the system is not busy and a performance measure of the system at a current software thread level of the operator is greater than a performance measure of the system when the operator has a lower software thread level.

Abstract: A keyed join is used in the processing of streaming data to streamline processing to provide higher throughput and decreased use of resources. The most recent event for each unique replacement key value(s) is maintained substituting older events with the same key. An incoming event is joined with the data received from one or more other data sources, and the correlations are output.

Abstract: Techniques for partitioning an operator flow graph are provided. The techniques include receiving source code for a steam processing application, wherein the source code comprises an operator flow graph, wherein the operator flow graph comprises a plurality of operators, receiving profiling data associated with the plurality of operators and one or more processing requirements of the operators, defining a candidate partition as a coalescing of one or more of the operators into one or more sets of processing elements (PEs), using the profiling data to create one or more candidate partitions of the processing elements, using the one or more candidate partitions to choose a desired partitioning of the operator flow graph, and compiling the source code into an executable code based on the desired partitioning.

Abstract: Techniques for dynamically modifying inter-connections between components in an application are provided. The techniques include receiving a data producer profile for each output port within a software application to be executed on one or more processors, receiving a data subscription profile for each input port of each component of the application, establishing connections between the output ports and the input ports of the components in the application based on a comparison of each data producer profile and each data subscription profile, executing the application on one or more processors to process streams of data, receiving either or both of a new data producer profile or a new data subscription profile during the execution of the application, and establishing at least one new connection between an output port and an input port based upon a revised comparison of the received data profiles that include the new data profile.

Abstract: Techniques for composing an application resource model in a data stream processing system are disclosed. The application resource model may be used to understand what resources will be consumed by an application when executed by the data stream processing system. For example, a method for composing an application resource model for a data stream processing system comprises the following steps. One or more operator-level metrics are obtained from an execution of a data stream processing application in accordance with a first configuration. The application is executed by one or more nodes of the data stream processing system, and the application is comprised of one or more processing elements that are comprised of one or more operators. One or more operator-level resource functions are generated based on the obtained one or more operator-level metrics. A processing element-level resource function is generated based on the one or more generated operator-level resource functions.

Abstract: Techniques for compiling a data stream processing application are provided. The techniques include receiving, by a compiler executing on a computer system, source code for a data stream processing application, wherein the source code comprises source code for a plurality of operators, each of which performs a data processing function, determining, by the compiler, one or more characteristics of operators within the data stream processing application, grouping, by the compiler, the operators into one or more execution containers based on the one or more characteristics, and compiling, by the compiler, the source code for the data stream processing application into executable code, wherein the executable code comprises a plurality of execution units, wherein each execution unit contains one or more of the operators, wherein each operator is assigned to an execution unit based on the grouping, and wherein each execution unit is to be executed in a partition.

Abstract: One embodiment of a method for on-demand de-marshalling of a network message includes receiving a first network message at a receiver device, wherein the first network message was sent from a sender device, and wherein the first network message comprises one or more attributes that characterize an object at the sender device, and further wherein the first network message is in an encoded format according to a network protocol, storing the first network message in a buffer of the receiver device in the encoded format, identifying, at the receiver device and prior to de-marshalling of any of the attributes, that at least one of the attributes is to be manipulated by the receiver device, de-marshalling, in response to the identifying, the identified attribute(s), and manipulating the identified attribute(s) at the receiver device.

Abstract: In one embodiment, the invention is a method and apparatus for use of vectorization instruction sets. One embodiment of a method for generating vector instructions includes receiving source code written in a high-level programming language, wherein the source code includes at least one high-level instruction that performs multiple operations on a plurality of vector operands, and compiling the high-level instruction(s) into one or more low-level instructions, wherein the low-level instructions are in an instruction set of a specific computer architecture.