Abstract: Various embodiments of systems and methods for providing virtualized (e.g., serverless) broker clusters for a data streaming service are disclosed. A data streaming service uses a front-end proxy layer and a back-end broker layer to provide virtualized broker clusters, for example in a Kafka-based streaming service. Resources included in a virtualized broker cluster are monitored and automatically scaled-up, scaled-down, or re-balanced in a way that is transparent to data producing and/or data consuming clients of the data streaming service.

Abstract: Configuration information indicating that one or more stream consumers are granted read-only access to contents of a shared-access data stream is stored at a stream management service. A virtual stream associated with the shared-access stream may be established. In response to a read request directed to the virtual stream, contents of a particular record of the shared-access data stream are provided.

Abstract: In response to a first programmatic request, metadata indicating that a first isolated read channel of a real-time category has been associated with a first target stream is stored at a stream management service. In response to another request, metadata indicating that a second isolated read channel of a non-real-time category has been associated with a second target stream is stored. In response to a read request indicating the first channel or the second channel, one or more data records of the corresponding target streams are provided.

Abstract: At a data stream management service, a first set of metadata indicating that a first isolated read channel has been associated with a first data stream is stored. The first isolated read channel has an associated read performance limit setting. A second set of metadata indicating that a second isolated read channel, with its own performance limit setting, has been associated with a data stream is also stored. Based on determining that the difference between a metric of read operations associated with the first channel and the read performance limit setting of the first channel meets a first criterion, the service initiates a throttling operation for reads associated with the first channel. The throttling decision is made independently of read metrics of the second channel.

Abstract: In response to a first programmatic request, metadata indicating that a first isolated read channel of a real-time category has been associated with a first target stream is stored at a stream management service. In response to another request, metadata indicating that a second isolated read channel of a non-real-time category has been associated with a second target stream is stored. In response to a read request indicating the first channel or the second channel, one or more data records of the corresponding target streams are provided.

Abstract: From a stream management service, indications of transmissions of data records of a first data stream via a persistent network connection associated with a subscription are provided to a stream processing application. In response to detecting that a transfer throttling condition associated with the subscription has been met, a delay is introduced before additional data records are transmitted via the same persistent network connection. After a subscription expiration period has elapsed, an indication of the expiration is provided to the application via the persistent connection.

Abstract: At a platform at which stream data retrieval requests directed to a stream management service are processed, respective metrics of reads being performed using a first set of one or more persistent network connections established between the platform and one or more stream processing applications are collected. A request to read additional stream data requests is directed to the platform based at least in part on a decision made by a load balancer. The request is rejected based on a determination that the collected metrics satisfy a threshold.

Abstract: At a stream management service, one or more storage servers of a first repository are assigned to store at least a first partition of a first data stream. At the first repository, records of different partitions are interleaved. In response to determining that a set of data records of the first partition meet a first criterion, a rearranged version of the set in which the data records of the first partition are contiguous is stored at a second repository. Using the rearranged version, one or more records of the first partition are provided to an application which meets a lag criterion.

Abstract: Disclosed are various embodiments for consuming fragments of time-associated data streams. A streaming gateway receives a request from a consumer via a network to obtain a stream of time-associated data. The gateway determines that a first fragment of the stream of time-associated data is stored in a data store. The first fragment is obtained from the data store and sent to the consumer via the network. The gateway determines that the stream of time-associated data is currently being received by an endpoint from a producer. A second segment of the stream is obtained from the endpoint and sent to the consumer via the network.

Abstract: In response to a first programmatic request, metadata indicating that a first isolated read channel associated with a data stream has been established is stored at a stream management service. A read request that indicates the first isolated read channel and the stream is obtained. In response to the read request, contents of data records of the stream are provided.

Abstract: In response to a first programmatic request, metadata indicating that a first isolated read channel of a real-time category has been associated with a first target stream is stored at a stream management service. In response to another request, metadata indicating that a second isolated read channel of a non-real-time category has been associated with a second target stream is stored. In response to a read request indicating the first channel or the second channel, one or more data records of the corresponding target streams are provided.

Abstract: Disclosed are various embodiments for fragmentation of time-associated streaming data. A producer application receives a stream of time-associated data. The producer application then generates a first fragment of the stream of time-associated data and assigns a first unique fragment identifier and a first timestamp to the first fragment. The producer application sends the first fragment via a network using an application-layer protocol to an endpoint. The producer application receives at least one acknowledgement for the first fragment from the endpoint via the network using the application-layer protocol.

Abstract: Configuration information indicating that one or more stream consumers are granted read-only access to contents of a shared-access data stream is stored at a stream management service. A virtual stream associated with the shared-access stream may be established. In response to a read request directed to the virtual stream, contents of a particular record of the shared-access data stream are provided.

Abstract: From a stream management service, indications of transmissions of data records of a first data stream via a persistent network connection associated with a subscription are provided to a stream processing application. In response to detecting that a transfer throttling condition associated with the subscription has been met, a delay is introduced before additional data records are transmitted via the same persistent network connection. After a subscription expiration period has elapsed, an indication of the expiration is provided to the application via the persistent connection.

Abstract: Various embodiments replicate and index fragments of a stream of time-associated data. In one embodiment, a stream endpoint application receives a first fragment of a stream of time-associated data from a producer via a network using an application-layer protocol. The stream endpoint application determines a producer timestamp from the first fragment. The stream endpoint application sends at least one acknowledgment to the producer via the network using the application-layer protocol. The stream endpoint application indexes the first fragment and sends the first fragment to a destination.

Abstract: At a data stream management service, a first set of metadata indicating that a first isolated read channel has been associated with a first data stream is stored. The first isolated read channel has an associated read performance limit setting. A second set of metadata indicating that a second isolated read channel, with its own performance limit setting, has been associated with a data stream is also stored. Based on determining that the difference between a metric of read operations associated with the first channel and the read performance limit setting of the first channel meets a first criterion, the service initiates a throttling operation for reads associated with the first channel. The throttling decision is made independently of read metrics of the second channel.

Abstract: In response to a first programmatic request, metadata indicating that a first isolated read channel of a real-time category has been associated with a first target stream is stored at a stream management service. In response to another request, metadata indicating that a second isolated read channel of a non-real-time category has been associated with a second target stream is stored. In response to a read request indicating the first channel or the second channel, one or more data records of the corresponding target streams are provided.

Abstract: Configuration information indicating that one or more stream consumers are granted read-only access to contents of a shared-access data stream is stored at a stream management service. A virtual stream associated with the shared-access stream may be established. In response to a read request directed to the virtual stream, contents of a particular record of the shared-access data stream are provided.

Abstract: A journal service maintains a journal that holds a number of journal entries. Access to the journal entries is controlled by providing a lead resource with access to a named role associated with the journal. A pool of alternate resources containing potential replacements for the lead resource is maintained. A succession order may be established for the pool of alternate resources, and individual resources in the pool of alternate resources may be managed based at least in part on their position within the succession order. The succession order may be established using various criteria that are indicative of a likelihood that a particular alternate resource will be available if the lead resource fails, such as the presence of shared points of failure between the current lead resource and an alternate resource. The lead resource may be replaced with an alternate resource as a result of various impeachment events.

Abstract: From a stream management service, indications of transmissions of data records of a first data stream via a persistent network connection associated with a subscription are provided to a stream processing application. In response to detecting that a transfer throttling condition associated with the subscription has been met, a delay is introduced before additional data records are transmitted via the same persistent network connection. After a subscription expiration period has elapsed, an indication of the expiration is provided to the application via the persistent connection.