Abstract: A total number of messages in a set of messages that are processed by a computing node in a virtual computer cluster during a time interval is determined. The virtual computer cluster is deployed with a cloud computing service and includes the computing node and other computing nodes at an end time of the time interval. It is determined whether the total number of messages in the set of messages processed by the computing node is no less than a maximum per-interval message number threshold. In response to determining that the total number of messages is no less than a maximum per-interval message number threshold, the cloud computing service is caused to start an additional computing node in the virtual computer cluster after the time interval, wherein the additional computing node is used to process one or more subsequent messages to be processed after the time interval.

Abstract: Embodiments regard transfer of data streaming services to provide continuous data flow. An embodiment of an apparatus includes a processor to process data for streaming to one or more organizations; and a memory to store data for streaming to the one or more organizations, wherein the apparatus is to provide a centralized work distribution service to track status of each of a plurality of data streams to the one or more organizations, and a plurality of nodes, each node being a virtual machine to stream one or more data streams to the one or more organizations, each node including a first daemon service to monitor connectivity of the node to dependency services for the node and, upon detecting a loss of connection to one or more of the dependency services, the node to discontinue ownership of the one or more data streams of the node and a second daemon service to poll the centralized work distribution service for data streams that are not assigned.

Abstract: A cloud computing service is used to deploy a virtual computer cluster. The virtual computer cluster is initialized with a set of one or more streaming nodes for processing first messages of one or more streaming jobs. It is determined whether the virtual computer cluster is to process second messages of a non-streaming job. In response to determining that the virtual computer cluster is to process the second messages of the non-streaming job, for example using control messages, the cloud computing service is caused to start a non-streaming node in the virtual computer cluster. The non-streaming node is tasked to process the second messages of the non-streaming job.

Abstract: Embodiments regard transfer of data streaming services to provide continuous data flow. An embodiment of an apparatus includes a processor to process data for streaming to one or more organizations; and a memory to store data for streaming to the one or more organizations, wherein the apparatus is to provide a centralized work distribution service to track status of each of a plurality of data streams to the one or more organizations, and a plurality of nodes, each node being a virtual machine to stream one or more data streams to the one or more organizations, each node including a first daemon service to monitor connectivity of the node to dependency services for the node and, upon detecting a loss of connection to one or more of the dependency services, the node to discontinue ownership of the one or more data streams of the node and a second daemon service to poll the centralized work distribution service for data streams that are not assigned.

Abstract: A total number of messages in a set of messages that are processed by a computing node in a virtual computer cluster during a time interval is determined. The virtual computer cluster is deployed with a cloud computing service and includes the computing node and other computing nodes at an end time of the time interval. It is determined whether the total number of messages in the set of messages processed by the computing node is no less than a maximum per-interval message number threshold. In response to determining that the total number of messages is no less than a maximum per-interval message number threshold, the cloud computing service is caused to start an additional computing node in the virtual computer cluster after the time interval, wherein the additional computing node is used to process one or more subsequent messages to be processed after the time interval.

Abstract: A cloud computing service is used to deploy a virtual computer cluster. The virtual computer cluster is initialized with a set of one or more streaming nodes for processing first messages of one or more streaming jobs. It is determined whether the virtual computer cluster is to process second messages of a non-streaming job. In response to determining that the virtual computer cluster is to process the second messages of the non-streaming job, for example using control messages, the cloud computing service is caused to start a non-streaming node in the virtual computer cluster. The non-streaming node is tasked to process the second messages of the non-streaming job.

Abstract: Techniques are disclosed relating to bulk data extraction systems. In some embodiments, a streaming server system may receive a first request, from a data storage system, that is sent prior to initiation of a bulk data extraction for a first group of users. In response to the first request, the streaming server system may receive, from the data storage system, a first notification message that includes a particular event identifier for a most recent data event generated at the data storage system. The streaming server system may receive, from the data storage system, those messages associated with the bulk data extraction for the first group. Subsequent to completion of the bulk data extraction, the streaming server system may send, to the data storage system, a request to subscribe to notification messages for data events associated with the first group.

Abstract: Techniques are disclosed relating to verifying the sequence of messages sent via data streaming systems. In some embodiments, a streaming server system may receive a plurality of messages associated with events at a data storage system that is accessible to various groups of users. In some embodiments, each of the plurality of messages may include an event identifier for an associated event and a group-specific sequence number. The streaming server system may detect, based on group-specific sequence numbers included in a set of messages, that a particular message for a particular group has not been received. The streaming server system may send, to the data storage system, a request for the particular message and, in response to the request, may receive the particular message from the data storage system.

Abstract: Techniques are disclosed relating to bulk data extraction systems. In some embodiments, a streaming server system may receive a first request, from a data storage system, that is sent prior to initiation of a bulk data extraction for a first group of users. In response to the first request, the streaming server system may receive, from the data storage system, a first notification message that includes a particular event identifier for a most recent data event generated at the data storage system. The streaming server system may receive, from the data storage system, those messages associated with the bulk data extraction for the first group. Subsequent to completion of the bulk data extraction, the streaming server system may send, to the data storage system, a request to subscribe to notification messages for data events associated with the first group.

Abstract: Embodiments regard transfer of data streaming services to provide continuous data flow. An embodiment of an apparatus includes a processor to process data for streaming to one or more organizations; and a memory to store data for streaming to the one or more organizations, wherein the apparatus is to provide a centralized work distribution service to track status of each of a plurality of data streams to the one or more organizations, and a plurality of nodes, each node being a virtual machine to stream one or more data streams to the one or more organizations, each node including a first daemon service to monitor connectivity of the node to dependency services for the node and, upon detecting a loss of connection to one or more of the dependency services, the node to discontinue ownership of the one or more data streams of the node and a second daemon service to poll the centralized work distribution service for data streams that are not assigned.