Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating tracer events. One of the methods includes generating, by a first system in an event processing pipeline, tracer event data for a data event; receiving, by a downstream system in the event processing pipeline, data for the data event and the tracer event data for the data event; updating, by the downstream system, the tracer event data for the data event using metric data generated while the downstream system processed the data event; after updating the tracer event data, receiving the data event that has been processed at each of multiple different layers in the event processing pipeline, and the updated tracer event data for the data event; and causing a change to the event processing pipeline using a result of an analysis of the updated tracer event data.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for improved delivery likelihood of batch events. One of the methods includes receiving, from an upstream system in an event processing pipeline, a request to store a first batch of events in persistent memory, the request including the first batch of events and a first identifier for the first batch of events; providing, to a downstream system in the event processing pipeline, a data storage request that includes the first batch of events and the first identifier; receiving, from the downstream system, the first confirmation that includes the first identifier and indicates that the first batch of events was successfully committed; and in response to receiving the first confirmation: sending, to the upstream system, a second confirmation message that includes the first identifier and indicates that the first batch of events was successfully committed.