Abstract: Computer-readable media, methods, and systems are disclosed for performing rollback recovery with data lineage capture for data pipelines. A middle operator receives ingested input events from a source operator reading data from an external input data source. The middle operator then logs information regarding middle input events to a middle operator input log, designating the logged middle input event information as incomplete. The middle operator then processes data associated with the middle input events and updates the middle input log entries setting them to a completed logging status designation for middle input events that were consumed to produce the one or more middle output events. The middle operator then transmits the middle output events to subsequent operators. Garbage collection is performed to remove completed entries from the middle operator output log. Finally, based on receiving a recovering message from a subsequent operator, corresponding middle output events are re-sent.

Abstract: Computer-readable media, methods, and systems are disclosed for performing rollback recovery with data lineage capture for data pipelines. A middle operator receives ingested input events from a source operator reading data from an external input data source. The middle operator then logs information regarding middle input events to a middle operator input log, designating the logged middle input event information as incomplete. The middle operator then processes data associated with the middle input events and updates the middle input log entries setting them to a completed logging status designation for middle input events that were consumed to produce the one or more middle output events. The middle operator then transmits the middle output events to subsequent operators. Garbage collection is performed to remove completed entries from the middle operator output log. Finally, based on receiving a recovering message from a subsequent operator, corresponding middle output events are re-sent.

Abstract: Disclosed herein are system, method, and computer program product embodiments for determining clusters of operations in a data processing pipeline. By clustering the operations according to the operations' programming languages using the algorithm disclosed herein, performance efficiency gains can be achieved. The algorithm is iterative, traversing all operations in a data processing pipeline with subsequent iterations addressing the clustering regime determined by prior iterations.

Abstract: Disclosed herein are system, method, and computer program product embodiments for determining clusters of operations in a data processing pipeline. By clustering the operations according to the operations' programming languages using the algorithm disclosed herein, performance efficiency gains can be achieved. The algorithm is iterative, traversing all operations in a data processing pipeline with subsequent iterations addressing the clustering regime determined by prior iterations.