Abstract: A method comprises establishing a packet size limit. The packet size limit may govern the maximum permitted size of packets that are transmitted between a first node and a second node of a network. The method also comprises monitoring a buffer of the second node in the network. The method also comprises determining, based on the monitoring, that the buffer of the second node is filled above an upper capacity threshold. The method also comprises increasing, based on the determining, the first packet size limit.

Abstract: Techniques for optimizing updates to a distributed data processing application are provided. A runtime engine compares a first version of a distributed application to a second, new version of the distributed application. The runtime engine identifies change(s) to the first version. The runtime engine generates compiler hint(s) for the change(s). The second version of the distributed application and compiler hint(s) are sent to a compiler. The runtime engine receives, from the compiler, a compiled second version of the distributed application and compiler-scheduler hint(s) describing respective optimizations the compiler made to the compiled second version of the distributed application based on the change(s) made to the first version of the distributed application and the compiler hint(s).

Abstract: An XML fragmenting mechanism uses an XML schema for the XML file to split up the XML file in a hierarchal structure of data blocks for storage in a storage system with a limited block size such as a cluster coordination service. The XML fragmenting mechanism creates an XML file map to document the structure of the XML file in the storage system. The XML fragmenting mechanism stores the data blocks in the storage system according to the XML file map and supports retrieval of all or part of the data in a format that supports XML validation.

Abstract: A computer-implemented method, in an example, may include calculating a time-to-live value for at least one packet based on a hop count between each of a plurality of processing element containers within an application streaming network; updating at least one processing element container output connection based on the calculated time-to-live value; and monitoring a streams resource metrics service for a change in a packet delivery rate.

Abstract: A connection can be made to a processing element of a remotely deployed and live streaming application executed by a first data processing system, the processing element containing at least one operator that processes at least one tuple. As the live streaming application is executed, without slowing or modifying data flow of the live streaming application execution to client devices, a copy of the tuple and a memory dump of state data for a state of the operator can be received, and the tuple can be tracked through a call graph. The state data can be loaded into a local instance of the operator loaded into a debugger. At least a portion of the call graph can be presented to a user, and a flow of the tuple through the call graph based on the state data for the operator can be indicated.

Abstract: A method, computer program product, and computer system for identifying, in a dynamically formed cloud, a plurality of voluntary compute nodes, wherein the plurality of voluntary compute nodes may include personal computing devices of a plurality of users, and wherein at least a portion of the plurality of voluntary compute nodes may be configured to communicate with each other to receive and send streaming data. A first compute node of at least the portion of the plurality of voluntary compute nodes may be identified to receive and distribute a streams operator of a streams graph requested by a client computing device. A second compute node of at least the portion of the plurality of voluntary compute nodes may be identified for verifying computational correctness of a tuple stream computation for the stream operator by replicating the tuple stream computations for the stream operator on the second compute node. A final result tuples of the graph may be sent to the client computing device.

Abstract: In using a virtual extensible local area network (VXLAN) for stream processing, a management system allocates resources for a streaming application based on an operator graph of the streaming application. The management system assigns the resources to a group of VXLAN segments based on the operator graph of the streaming application. A first processing element of the streaming application multicasts data on the group of VXLAN segments. A second processing element on a given VXLAN segment of the group of VXLAN segments receives the data. If the second processing element is an intended recipient of the data, then the second processing element processes the data. If the second processing element is not the intended recipient of the data, then the second processing element ignores the data.

Abstract: System resource usage by a streaming application processing workloads can be monitored, the streaming application comprising at least one parallelizable operator, at least a first instance of the parallelizable operator being initialized to execute. Based on the monitoring, at least one performance metric for the streaming application in processing the workloads can be determined. Based on the at least one performance metric, whether hosting of the streaming application is, or is near to, violating at least one criteria can be determined. If so, at least one additional instance of at least one parallelizable operator of the streaming application that is currently executing can be initialized. Responsive to initializing the at least one additional instance of at least one of the parallelizable operators, a portion of the workloads can be directed to the at least one additional instance of at least one of the parallelizable operators.

Abstract: Provided are techniques for operator isolation based on data security requirements. At a cloud node, a graph is received that includes ingest portions of data and operators. For each of the operators, it is determined whether the operator processes protected data. In response to determining that the operator is tagged with an indication that the operator processes protected data, the operator is forwarded to a tenant secure node for processing. In response to determining that the operator is not tagged with an indication that the operator processes protected data, the operator is forwarded to a tenant general node for processing. Then, while the tenant general node is processing the operator, in response to determining that the operator is processing protected data a tag is associated with the operator to indicate that the operator processes protected data and the operator is forwarded to the tenant secure node for processing.

Abstract: Disclosed aspects relate to cache management in a stream computing environment that uses a set of many-core hardware processors to process a stream of tuples by a plurality of processing elements which operate on the set of many-core hardware processors. The stream of tuples to be processed by the plurality of processing elements which operate on the set of many-core hardware processors may be received. A tuple-processing hardware-route on the set of many-core hardware processors may be determined based on a cache factor associated with the set of many-core hardware processors. The stream of tuples may be routed based on the tuple-processing hardware-route on the set of many-core hardware processors. The stream of tuples may be processed by the plurality of processing elements which operate on the set of many-core hardware processors.

Abstract: A computer system configures processing elements within a distributed computing system. A processing element within a distributed computing environment is determined to be affected by a software update, wherein each processing element of the distributed computing system includes a plurality of components and the software update modifies the components of the processing elements. The determined processing element is split into a plurality of processing elements based on a set of factors. The plurality of components of the processing element are assigned among the plurality of processing elements based on components affected by the software update. Embodiments of the present invention further include a method and program product for configuring processing elements within a distributed computing system in substantially the same manner described above.

Abstract: A computer system configures processing elements within a distributed computing system. A processing element within a distributed computing environment is determined to be affected by a software update, wherein each processing element of the distributed computing system includes a plurality of components and the software update modifies the components of the processing elements. The determined processing element is split into a plurality of processing elements based on a set of factors. The plurality of components of the processing element are assigned among the plurality of processing elements based on components affected by the software update. Embodiments of the present invention further include a method and program product for configuring processing elements within a distributed computing system in substantially the same manner described above.

Abstract: A method, computer program product, and computer system for consolidating, at a computing device, a number of parallel streams, entering a hardware accelerator in a cluster, into fewer streams. The fewer streams, exiting the hardware accelerator in the cluster, may be returned back into the number of parallel streams.

Abstract: A deployment manager deploys processing elements of a streaming application in a non-uniform memory access (NUMA) aware manner to reduce memory coherency overhead in a streaming application. The deployment manager is able to utilize information about an application's operators and the architecture of the NUMA nodes to place whole processing elements on a single NUMA node. Where the operators of a processing element would cross NUMA node boundaries, the deployment manager may consolidate the threads of a processing element to place an application's operators on a single NUMA node to increase efficiency of the system.

Abstract: A method, computer program product, and computer system for defining, on at least one processor, one or more potential overflow paths for processing at least a portion of a data load through a plurality of stream operators deployed on a plurality of computing devices. Embodiments may include determining, on the at least one processor, a conditional availability for each computing device of the plurality of computing devices. Embodiments may also include identifying, on the at least one processor, a change in one or more of a configuration of the plurality of computing devices and the at least a portion of the data load. Embodiments may further include dynamically activating, on the at least one processor, one or more potential overflow paths in response to the identified change based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices.

Abstract: Embodiments generally relate to processing element restart precedence in a job overlay environment. In some embodiments, a method includes determining a job overlay, wherein the job overlay involves updates to a subset of processing elements of a plurality of processing elements of a job. The method further includes determining processing requirements of the plurality of processing elements. The method further includes determining computation capabilities of computational resources associated with the plurality of processing elements. The method further includes determining a processing element restart order based at least in part on processing requirements and computation capabilities. The method further includes updating the subset of processing elements. The method further includes restarting the subset of processing elements based at least in part on the processing element restart order.

Abstract: Techniques for optimizing updates to a distributed data processing application are provided. A runtime engine compares a first version of a distributed application to a second, new version of the distributed application. The runtime engine identifies change(s) to the first version. The runtime engine generates compiler hint(s) for the change(s). The second version of the distributed application and compiler hint(s) are sent to a compiler. The runtime engine receives, from the compiler, a compiled second version of the distributed application and compiler-scheduler hint(s) describing respective optimizations the compiler made to the compiled second version of the distributed application based on the change(s) made to the first version of the distributed application and the compiler hint(s).

Abstract: Methods, systems, and computer program products are provided for eliminating run-time errors in a stream processing environment. Processing devices receive tuples in a distributed stream processing environment. Each respective processing device checks the received tuples to determine whether a pattern of the received tuples matches one or more pattern profiles that cause run-time errors. When the pattern of the received tuples matches the one or more pattern profiles that cause run-time errors, the respective processing device performs an action to adjust a processing order of the received tuple to reduce an impact of the run-time errors. When the pattern of the received tuples does not match the one or more pattern profiles that cause run-time errors, the respective processing device performs a function on the received tuples. The functions are performed by an operator residing in the respective processing device.

Abstract: A method, system and computer program product for providing malicious application detection and prevention for stream computing applications deployed in cloud environments. Static and run-time analyses are performed of the stream computing application and any files associated therewith to create an expected profile of the stream computing application's behavior. The stream computing application is invoked in an execution environment to compare the stream computing application's behavior to the expected profile as the stream computing application runs in the execution environment. The stream computing application is isolated within the cloud computing environment when the stream computing application's behavior deviates significantly from the expected profile. The isolated stream computing application is investigated, wherein the output data that is stored is used for forensic investigation or replayed later if the stream computing application is not harmful.

Abstract: A computer system dynamically balances a workload of a distributed computing application at runtime. A distributed computing application is monitored at runtime for the occurrence of one or more conditions, wherein the distributed computing application comprises one or more nodes that include one or more processing elements, and wherein each processing element includes one or more operators. One or more adjustments to the distributed computing application are determined. The one or more adjustments to the distributed computing application are performed at runtime. Embodiments of the present invention further include a method and program product for configuring processing elements within a distributed computing system in substantially the same manner described above.