Abstract: A method for adjusting a data parallel region of a stream processing application includes measuring congestion of each parallel channel of the data parallel region, measuring a total throughput of all the parallel channels, and adjusting the number of parallel channels based on the current measured congestion and throughput.

Abstract: In one embodiment, the invention is a method and apparatus for use of vectorization instruction sets. One embodiment of a method for generating vector instructions includes receiving source code written in a high-level programming language, wherein the source code includes at least one high-level instruction that performs multiple operations on a plurality of vector operands, and compiling the high-level instruction(s) into one or more low-level instructions, wherein the low-level instructions are in an instruction set of a specific computer architecture.

Abstract: Keyed aggregation is used in the processing of streaming data to streamline processing to provide higher throughput and decreased use of resources. The most recent event for each unique replacement key value(s) is maintained. In response to an incoming event having a same key as a previous event, the effect on an aggregation of the previous event is removed. The aggregation is then updated with one or more values from the arriving event and the updated aggregation is output.

Abstract: An embodiment of the invention provides a method of displaying a data flow, wherein a description of a data flow application to be displayed is received. The data flow application includes nodes and edges connecting the nodes, wherein the nodes represent operators and the edges represent data connections for data flowing between the operations. A reason that a user is to view the data flow and/or a user constraint on a complexity of the data flow application to be displayed is determined with a processor; and, the time required to render a display of the data flow application is estimated. A transformed representation of the data flow application is created with the processor. The transformed representation is created based upon the user reason, the user constraint, the estimated time of rendering, and/or a layout strategy. The transformed representation is displayed on a graphical user interface.

Abstract: A method of integrating an existing source code into a target source code, the method including mapping a plurality of candidate mappings between a change point of the existing source code and a plurality of change points of the target source code, selecting a candidate mapping of the plurality of candidate mappings, automatically updating the existing source code based on the candidate mapping having been selected, and transplanting the existing source code having been updated into the target source code.

Abstract: A computer system for integrating existing source code into target source code, the computer system including a change point mapper that outputs a plurality of suggested candidate mappings between a change point of the existing source code and a plurality of change points of the target source code, a change point transformer that receives the plurality of suggested candidate mappings and outputs an updated existing source code, the change point transformer being configured to automatically update the existing source code based on a selected candidate mapping of the plurality of suggested candidate mappings, and a change point transplanter being configured to insert the updated existing source code into the target source code.

Abstract: Architecture-dependent assets are automatically built and retargeted. An asset originally built for one architecture is downloaded and automatically retargeted on another architecture. This automatically retargeting may be performed on demand, at runtime.

Abstract: A computer system for integrating existing source code into target source code, the computer system including a change point mapper that outputs a plurality of suggested candidate mappings between a change point of the existing source code and a plurality of change points of the target source code, a change point transformer that receives the plurality of suggested candidate mappings and outputs an updated existing source code, the change point transformer being configured to automatically update the existing source code based on a selected candidate mapping of the plurality of suggested candidate mappings, and a change point transplanter being configured to insert the updated existing source code into the target source code.

Abstract: A method of integrating an existing source code into a target source code, the method including mapping a plurality of candidate mappings between a change point of the existing source code and a plurality of change points of the target source code, selecting a candidate mapping of the plurality of candidate mappings, automatically updating the existing source code based on the candidate mapping having been selected, and transplanting the existing source code having been updated into the target source code.

Abstract: Techniques for determining feasibility of a set of one or more operator partitioning constraints are provided. The techniques include receiving one or more sets of operator partitioning constraints, wherein each set of one or more constraints define one or more desired conditions for grouping together of operators into partitions and placing partitions on hosts, wherein each operator is embodied as software that performs a particular function, processing each set of one or more operator partitioning constraints to determine feasibility of each set of one or more operator partitioning constraints, creating and outputting one or more candidate partitions and one or more host placements for each set of feasible partitioning constraints, and creating and outputting a certificate of infeasibility for each set of infeasible partitioning constraints, wherein the certificate of infeasibility outlines one or more reasons for infeasibility.

Abstract: Data analysis applications include model building components and stream processing components. To increase utility of the data analysis application, in one embodiment, the model building component of the data analysis application is managed. Management includes resource allocation and/or configuration adaptation of the model building component, as examples.

Abstract: Techniques for partitioning an operator flow graph are provided. The techniques include receiving source code for a stream processing application, wherein the source code comprises an operator flow graph, wherein the operator flow graph comprises a plurality of operators, receiving profiling data associated with the plurality of operators and one or more processing requirements of the operators, defining a candidate partition as a coalescing of one or more of the operators into one or more sets of processing elements (PEs), using the profiling data to create one or more candidate partitions of the processing elements, using the one or more candidate partitions to choose a desired partitioning of the operator flow graph, and compiling the source code into an executable code based on the desired partitioning.

Abstract: An embodiment of the invention provides a method for adaptive auto-pipelining of a stream processing application, wherein the stream processing application includes one or more threads. Runtime of the stream processing application is initiated with a stream processing application manager. The stream processing application is monitored with a monitoring module during the runtime, wherein the monitoring of the stream processing application includes identifying threads in the stream processing application that execute operators in a data flow graph, and determining an amount of work that each of the threads are performing on operators of the logical data flow graph. A processor identifies one or more operators in the data flow graph to add one or more additional threads based on the monitoring of the stream processing application during the runtime.

Abstract: An embodiment of the invention provides a method for exploiting stateless and stateful data parallelism in a streaming application, wherein a compiler determines whether an operator of the streaming application is safe to parallelize based on a definition of the operator and an instance of the definition. The operator is not safe to parallelize when the operator has selectivity greater than 1, wherein the selectivity is the number of output tuples generated for each input tuple. Parallel regions are formed within the streaming application with the compiler when the operator is safe to parallelize. Synchronization strategies for the parallel regions are determined with the compiler, wherein the synchronization strategies are determined based on the definition of the operator and the instance of the definition. The synchronization strategies of the parallel regions are enforced with a runtime system.

Abstract: An embodiment of the invention provides a method for exploiting stateless and stateful data parallelism in a streaming application, wherein a compiler determines whether an operator of the streaming application is safe to parallelize based on a definition of the operator and an instance of the definition. The operator is not safe to parallelize when the operator has selectivity greater than 1, wherein the selectivity is the number of output tuples generated for each input tuple. Parallel regions are formed within the streaming application with the compiler when the operator is safe to parallelize. Synchronization strategies for the parallel regions are determined with the compiler, wherein the synchronization strategies are determined based on the definition of the operator and the instance of the definition. The synchronization strategies of the parallel regions are enforced with a runtime system.

Abstract: Embodiments of the disclosure include a method for providing stream processing with runtime adaptation includes registering one or more events, wherein each of the events is associated with a stream processing application. The method also includes monitoring, by a processor, for an occurrence of the one or more events associated with the stream processing application, wherein each of the one or more events is associated with one or more runtime metrics. The method further includes receiving an event notification, wherein the event notification includes event identification and an event context and executing an adaptation of the stream processing application.

Abstract: Embodiments of the disclosure include a system for providing stream processing with runtime adaptation, having a stream processing application that receives an incoming data stream and a runtime infrastructure configured to execute the stream processing application. The system also includes an orchestrator configured to communicate with the runtime infrastructure and the stream processing application, the orchestrator configured to perform a method. The method includes registering one or more events, wherein each of the events is associated with a stream processing application. The method also includes monitoring, by a processor, for an occurrence of the one or more events associated with the stream processing application, wherein each of the one or more events is associated with one or more runtime metrics.

Abstract: A method to optimize performance of an operator on a computer system includes determining whether the system is busy, decreasing a software thread level within the operator if the system is busy, and increasing the software thread level within the operator if the system is not busy and a performance measure of the system at a current software thread level of the operator is greater than a performance measure of the system when the operator has a lower software thread level.

Abstract: A method is provided for tracing dataflow in a distributed computing application. For example, the method includes incrementally advancing a dataflow in a dataflow path of one or more dataflow paths according to two or more directives encoded in two or more data messages. The method further includes performing the two or more directives. The dataflow path includes one or more operators including at least one merge operator operative to merge the two or more data messages and merge the two or more directives. One or more of the incrementally advancing of the dataflow and the performing of the two or more directives are implemented as instruction code performed on a processor device.

Abstract: Data analysis applications include model building components and stream processing components. To increase utility of the data analysis application, in one embodiment, the model building component of the data analysis application is managed. Management includes resource allocation and/or configuration adaptation of the model building component, as examples.