Abstract: Methods and apparatus, including computer program products, implementing and using techniques for data stream processing in a runtime data processing environment. A stream processing graph that includes several connected operators is received. Source code of the operators is analyzed to identify hints describing whether an operator contains data structures, method parameters or other data that can be applied in a parallelization data processing environment. Performance metrics of the data processing environment within parallel regions is evaluated to determine whether data processing resources can be dynamically scaled up or down. In response to determining that the data processing resources can be dynamically scaled up, one or more operators are split to be processed on two or more parallel processing resources. In response to determining that the data processing resources can be dynamically scaled down, one or more operators are combined to be processed on a single parallel processing resource.

Abstract: Embodiments generally relate to processing element restart precedence in a job overlay environment. In some embodiments, a method includes determining a job overlay, wherein the job overlay involves updates to a subset of processing elements of a plurality of processing elements of a job. The method further includes determining processing requirements of the plurality of processing elements. The method further includes determining computation capabilities of computational resources associated with the plurality of processing elements. The method further includes determining a processing element restart order based at least in part on processing requirements and computation capabilities. The method further includes updating the subset of processing elements. The method further includes restarting the subset of processing elements based at least in part on the processing element restart order.

Abstract: In preventing malicious operator placement in a streaming application, a stream computing management system receives a submission of an operator graph for the streaming application. A scheduler of the stream computing management system places the operators across a plurality of computing nodes. A threat detector of the stream computing management system monitors placements of the operators during the submission of the operator graph and a runtime of the streaming application. The threat detector further monitors runtime activities of the operators of the streaming application. The threat detector compares the runtime activities of the operators of the streaming application with stored activities patterns. The threat detector determines whether any anomalies identified based on the comparing exceeds a risk tolerance threshold. In response to an anomaly exceeding the risk tolerance threshold, the scheduler changes a placement of at least one of the operators.

Abstract: A method, system and computer program product for providing malicious application detection and prevention for stream computing applications deployed in cloud environments. Static and run-time analyses are performed of the stream computing application and any files associated therewith to create an expected profile of the stream computing application's behavior. The stream computing application is invoked in an execution environment to compare the stream computing application's behavior to the expected profile as the stream computing application runs in the execution environment. The stream computing application is isolated within the cloud computing environment when the stream computing application's behavior deviates significantly from the expected profile. The isolated stream computing application is investigated, wherein the output data that is stored is used for forensic investigation or replayed later if the stream computing application is not harmful.

Abstract: Methods for data stream processing in a runtime data processing environment. A stream processing graph that includes several connected operators is received. Source code of the operators is analyzed to identify hints describing whether an operator contains data structures, method parameters or other data that can be applied in a parallelization data processing environment. Performance metrics of the data processing environment within parallel regions is evaluated to determine whether data processing resources can be dynamically scaled up or down. In response to determining that the data processing resources can be dynamically scaled up, one or more operators are split to be processed on two or more parallel processing resources. In response to determining that the data processing resources can be dynamically scaled down, one or more operators are combined to be processed on a single parallel processing resource.

Abstract: Methods, systems, and computer program products are provided for eliminating run-time errors in a stream processing environment. Processing devices receive tuples in a distributed stream processing environment. Each respective processing device checks the received tuples to determine whether a pattern of the received tuples matches one or more pattern profiles that cause run-time errors. When the pattern of the received tuples matches the one or more pattern profiles that cause run-time errors, the respective processing device performs an action to adjust a processing order of the received tuple to reduce an impact of the run-time errors. When the pattern of the received tuples does not match the one or more pattern profiles that cause run-time errors, the respective processing device performs a function on the received tuples. The functions are performed by an operator residing in the respective processing device.

Abstract: A method, system and computer program product for optimizing memory management. An analytics engine receives from a computing device an execution sequence that led to the detected execution of an operation to create a new process, which involves copying of the parent process. The analytics engine searches profiles stored in a repository for patterns of execution sequences that match the received execution sequence. Upon identifying a stored pattern with an execution sequence within a threshold degree of similarity as the received execution sequence, the analytics engine instructs the computing device to handle the operation to create the new process in a certain manner based on the contents of the profile containing the matching stored pattern. For example, the computing device may be instructed to preemptively fail the operation in response to the profile indicating that the execution of the operation resulted in an out of memory error.

Abstract: Disclosed aspects relate to cache management in a stream computing environment that uses a set of many-core hardware processors to process a stream of tuples by a plurality of processing elements which operate on the set of many-core hardware processors. The stream of tuples to be processed by the plurality of processing elements which operate on the set of many-core hardware processors may be received. A tuple-processing hardware-route on the set of many-core hardware processors may be determined based on a cache factor associated with the set of many-core hardware processors. The stream of tuples may be routed based on the tuple-processing hardware-route on the set of many-core hardware processors. The stream of tuples may be processed by the plurality of processing elements which operate on the set of many-core hardware processors.

Abstract: A computer system configures processing elements within a distributed computing system. A processing element within a distributed computing environment is determined to be affected by a software update, wherein each processing element of the distributed computing system includes a plurality of components and the software update modifies the components of the processing elements. The determined processing element is split into a plurality of processing elements based on a set of factors. The plurality of components of the processing element are assigned among the plurality of processing elements based on components affected by the software update. Embodiments of the present invention further include a method and program product for configuring processing elements within a distributed computing system in substantially the same manner described above.

Abstract: A connection can be made to a processing element of a remotely deployed and live streaming application executed by a first data processing system, the processing element containing at least one operator that processes at least one tuple. As the live streaming application is executed, without slowing or modifying data flow of the live streaming application execution to client devices, a copy of the tuple and a memory dump of state data for a state of the operator can be received, and the tuple can be tracked through a call graph. The state data can be loaded into a local instance of the operator loaded into a debugger. At least a portion of the call graph can be presented to a user, and a flow of the tuple through the call graph based on the state data for the operator can be indicated.

Abstract: A computer system configures processing elements within a distributed computing system. A processing element within a distributed computing environment is determined to be affected by a software update, wherein each processing element of the distributed computing system includes a plurality of components and the software update modifies the components of the processing elements. The determined processing element is split into a plurality of processing elements based on a set of factors. The plurality of components of the processing element are assigned among the plurality of processing elements based on components affected by the software update. Embodiments of the present invention further include a method and program product for configuring processing elements within a distributed computing system in substantially the same manner described above.

Abstract: Disclosed aspects relate to tuple traffic management in a stream computing environment to process a stream of tuples using a set of control group data with respect to a set of operation-system-level virtualization and resource isolation containers. The set of control group data may be collected with respect to a set of containers in the stream computing environment. A tuple flow model may be determined with respect to the set of containers in the stream computing environment based on the set of control group data. The stream of tuples may be processed using the tuple flow model with respect to the set of containers in the stream computing environment.

Abstract: A method for data analysis in streaming data includes receiving a stream of data, the stream of data including ordered compressed files. The method may also include partitioning the stream of data into portions of the ordered compressed files. The method may also include concurrently filtering each of the portions of ordered compressed files with a filter. The method may further include forward matching portions of the ordered compressed files downstream of the received stream of data.

Abstract: A connection can be made to a processing element of a remotely deployed and live streaming application executed by a first data processing system, the processing element containing at least one operator that processes at least one tuple. As the live streaming application is executed, without slowing or modifying data flow of the live streaming application execution to client devices, a copy of the tuple and a memory dump of state data for a state of the operator can be received, and the tuple can be tracked through a call graph. The state data can be loaded into a local instance of the operator loaded into a debugger. At least a portion of the call graph can be presented to a user, and a flow of the tuple through the call graph based on the state data for the operator can be indicated.

Abstract: A computer-implemented method, in an example, may include calculating a time-to-live value for at least one packet based on a hop count between each of a plurality of processing element containers within an application streaming network; updating at least one processing element container output connection based on the calculated time-to-live value; and monitoring a streams resource metrics service for a change in a packet delivery rate.

Abstract: An XML fragmenting mechanism uses an XML schema for the XML file to split up the XML file in a hierarchal structure of data blocks for storage in a storage system with a limited block size such as a cluster coordination service. The XML fragmenting mechanism creates an XML file map to document the structure of the XML file in the storage system. The XML fragmenting mechanism stores the data blocks in the storage system according to the XML file map and supports retrieval of all or part of the data in a format that supports XML validation.

Abstract: A streams manager assesses the security risk of streaming communication and dynamically migrates operators of a streaming application based on the security risk. The streams manager may also dynamically change encryption of intranode communication between the operators to increase performance of the computer node. The streams manager includes a stream security module (SSM) with a monitor that monitors risk in the system and changes placement of operators and encryption between operators on a node depending on the risk. The stream security module may use security data and node profile data collected by the monitor or the system to determine the risk. The stream security module may provide recommendations to a customer for the customer to override changes in placement and encryption.

Abstract: Disclosed aspects relate to cache management in a stream computing environment that uses a set of many-core hardware processors to process a stream of tuples by a plurality of processing elements which operate on the set of many-core hardware processors. The stream of tuples to be processed by the plurality of processing elements which operate on the set of many-core hardware processors may be received. A tuple-processing hardware-route on the set of many-core hardware processors may be determined based on a cache factor associated with the set of many-core hardware processors. The stream of tuples may be routed based on the tuple-processing hardware-route on the set of many-core hardware processors. The stream of tuples may be processed by the plurality of processing elements which operate on the set of many-core hardware processors.

Abstract: A streams manager assesses the security risk of streams communication and when possible turns off encryption of intranode communication between operators of a streaming application on a computer node to increase performance of the computer node. The streams manager includes a stream security module (SSM) with a monitor that monitors risk in the system and changes encryption between operators on a node depending on the risk. The stream security module may use security data and node profile data collected by the monitor or the system to determine the risk. The stream security module may provide recommendations to a customer for the customer to override changes in encryption.

Abstract: Systems, methods, and computer program products to perform an operation comprising receiving a first data frame and a second data frame by a device driver of a first virtual network interface controller (vNIC) of a plurality of vNICs sharing a physical network interface controller (NIC) of a computing system, storing, by the device driver of the first vNIC, the first and second data frames in a first queue for the first vNIC, generating, by the device driver of the first vNIC, an optimized data frame comprising the first and second data frames, wherein the optimized data frame specifies a modified Ethernet preamble, and sending the optimized data frame to a hypervisor of the computing system.