Abstract: An embodiment of the invention provides a method of displaying a data flow, wherein a description of a data flow application to be displayed is received. The data flow application includes nodes and edges connecting the nodes, wherein the nodes represent operators and the edges represent data connections for data flowing between the operations. A reason that a user is to view the data flow and/or a user constraint on a complexity of the data flow application to be displayed is determined with a processor; and, the time required to render a display of the data flow application is estimated. A transformed representation of the data flow application is created with the processor. The transformed representation is created based upon the user reason, the user constraint, the estimated time of rendering, and/or a layout strategy. The transformed representation is displayed on a graphical user interface.

Abstract: Techniques for compiling a data stream processing application are provided. The techniques include receiving, by a compiler executing on a computer system, source code for a data stream processing application, wherein the source code comprises source code for a plurality of operators, each of which performs a data processing function, determining, by the compiler, one or more characteristics of operators within the data stream processing application, grouping, by the compiler, the operators into one or more execution containers based on the one or more characteristics, and compiling, by the compiler, the source code for the data stream processing application into executable code, wherein the executable code comprises a plurality of execution units, wherein each execution unit contains one or more of the operators, wherein each operator is assigned to an execution unit based on the grouping, and wherein each execution unit is to be executed in a partition.

Abstract: An application provisioning model is automatically created. The model is created from a high-level application and specifies dependencies of the application. It is used to provision the application on one or more nodes or other actions.

Abstract: Data sharing is facilitated in stream processing environments, including distributed stream processing environments. A processor of the stream processing environment obtains at least one of usage information for shared data of the stream processing environment, one or more pre-declared characteristics of the shared data, or performance information relating to the stream processing environment. Based on at least one of the usage information, the one or more pre-declared characteristics or the performance information, code is generated for managing the shared data.

Abstract: One embodiment of the present method and apparatus adaptive in-operator load shedding includes receiving at least two data streams (each comprising a plurality of tuples, or data items) into respective sliding windows of memory. A throttling fraction is then calculated based on input rates associated with the data streams and on currently available processing resources. Tuples are then selected for processing from the data streams in accordance with the throttling fraction, where the selected tuples represent a subset of all tuples contained within the sliding window.

Abstract: Techniques for partitioning an operator flow graph are provided. The techniques include receiving source code for a stream processing application, wherein the source code comprises an operator flow graph, wherein the operator flow graph comprises a plurality of operators, receiving profiling data associated with the plurality of operators and one or more processing requirements of the operators, defining a candidate partition as a coalescing of one or more of the operators into one or more sets of processing elements (PEs), using the profiling data to create one or more candidate partitions of the processing elements, using the one or more candidate partitions to choose a desired partitioning of the operator flow graph, and compiling the source code into an executable code based on the desired partitioning.

Abstract: In one embodiment, the invention comprises partial fault tolerant stream processing applications. One embodiment of a method for implementing partial fault tolerance in a stream processing application comprising a plurality of stream operators includes: defining a quality score function that expresses how well the application is performing quantitatively, injecting a fault into at least one of the plurality of operators, assessing an impact of the fault on the quality score function, and selecting at least one partial fault-tolerant technique for implementation in the application based on the quantitative metric-driven assessment.

Abstract: A method for processing a first data stream specifying locations of a user at different times and a second data stream specifying values of a monitored attribute at a location of interest at different times includes: receiving a location-centric trigger specifying at least one spatial predicate condition relative to the location of interest and at least one non-spatial predicate condition relevant to the location of interest, calculating a safe region that includes locations whose probability of satisfying the spatial predicate condition falls below a first threshold, calculating a safe value container that includes values whose probability of satisfying the non-spatial predicate condition falls below a second threshold, and processing the first data stream and the second data stream against the location-centric trigger, by considering only those locations that are not contained within the safe region and only those values that are not contained within the safe value container.

Abstract: A virtual execution environment (VEE) for a streaming Intermediate Language (IL), wherein the streaming IL represents a streaming program, communicates streaming data in queues, stores data-at-rest in variables, and determines data by functions, where inputs are read from the queues and the variables, and outputs are written to the queues and the variables.

Abstract: A method and computer program product for selecting an expression evaluation technique for domain-specific language (DSL) compilation. An application written in DSL for a programming task is provided, the application including a plurality of components configured by expressions. A technique that most quickly implements the programming task is selected from a plurality of techniques for evaluating the expressions. The DSL application is compiled in accordance with the selected expression evaluation technique to generate general-purpose programming language (GPL) code.

Abstract: Techniques for partitioning an operator flow graph are provided. The techniques include receiving source code for a stream processing application, wherein the source code comprises an operator flow graph, wherein the operator flow graph comprises a plurality of operators, receiving profiling data associated with the plurality of operators and one or more processing requirements of the operators, defining a candidate partition as a coalescing of one or more of the operators into one or more sets of processing elements (PEs), using the profiling data to create one or more candidate partitions of the processing elements, using the one or more candidate partitions to choose a desired partitioning of the operator flow graph, and compiling the source code into an executable code based on the desired partitioning.

Abstract: Techniques for performing capacity planning for applications running on a computational infrastructure are provided. The techniques include instrumenting an application under development to receive one or more performance metrics under a physical deployment plan, receiving the one or more performance metrics from the computational infrastructure hosting one or more applications that are currently running, using a predictive inference engine to determine how the application under development can be deployed, and using the determination to perform capacity planning for the applications on the computational infrastructure.

Abstract: One embodiment of the present method and apparatus adaptive in-operator load shedding includes receiving at least two data streams (each comprising a plurality of tuples, or data items) into respective sliding windows of memory. A throttling fraction is then calculated based on input rates associated with the data streams and on currently available processing resources. Tuples are then selected for processing from the data streams in accordance with the throttling fraction, where the selected tuples represent a subset of all tuples contained within the sliding window.

Abstract: In one embodiment, the invention comprises partial fault tolerant stream processing applications. One embodiment of a method for implementing partial fault tolerance in a stream processing application comprising a plurality of stream operators includes: defining a quality score function that expresses how well the application is performing quantitatively, injecting a fault into at least one of the plurality of operators, assessing an impact of the fault on the quality score function, and selecting at least one partial fault-tolerant technique for implementation in the application based on the quantitative metric-driven assessment.

Abstract: A method for processing a first data stream specifying locations of a user at different times and a second data stream specifying values of a monitored attribute at a location of interest at different times includes: receiving a location-centric trigger specifying at least one spatial predicate condition relative to the location of interest and at least one non-spatial predicate condition relevant to the location of interest, calculating a safe region that includes locations whose probability of satisfying the spatial predicate condition falls below a first threshold, calculating a safe value container that includes values whose probability of satisfying the non-spatial predicate condition falls below a second threshold, and processing the first data stream and the second data stream against the location-centric trigger, by considering only those locations that are not contained within the safe region and only those values that are not contained within the safe value container.

Abstract: Techniques for generating a distributed stream processing application are provided. The techniques include obtaining a declarative description of one or more data stream processing tasks from a graph of operators, wherein the declarative description expresses at least one stream processing task, generating one or more containers that encompass a combination of one or more stream processing operators, and generating one or more execution units from the declarative description of one or more data stream processing tasks, wherein the one or more execution units are deployable across one or more distributed computing nodes, and comprise a distributed data stream processing application binary.

Abstract: Techniques for searching multi-dimensional data are provided. The techniques include providing a parallelization framework for a search algorithm, wherein the search algorithm exposes one or more architecture-sensitive tunable optimization parameters, and using the one or more architecture-sensitive tunable optimization parameters to tune the search algorithm to search multi-dimensional data in any underlying architecture.

Abstract: State sharing is facilitated in stream processing environments, including distributed stream processing environments. A customized shared state implementation representing the state to be shared is automatically created based on at least one of user preferences, hints of usage, and system performance.

Abstract: A method and apparatus are provided for hardware-assisted local triangle counting in a graph. The method includes converting vertex relationships of the graph into rule patterns. The method also includes compiling the rule patterns into a binary file, wherein the rule patterns are organized into a finite state machine. The method further includes loading at least a part of the binary file and a search string to be compared there against into a hardware pattern matching accelerator. The method additionally includes receiving a number of matching outputs from the pattern matching accelerator.

Abstract: A method and apparatus are provided for hardware-assisted local triangle counting in a graph. The method includes converting vertex relationships of the graph into rule patterns. The method also includes compiling the rule patterns into a binary file, wherein the rule patterns are organized into a finite state machine. The method further includes loading at least a part of the binary file and a search string to be compared there against into a hardware pattern matching accelerator. The method additionally includes receiving a number of matching outputs from the pattern matching accelerator.