Abstract: Systems and methods are described for updating text files for a processing pipeline without restarting the processing pipeline. A processing pipeline may include a frontend thread and a backend thread. The frontend thread of the processing pipeline may generate transformed data using the text file. A backend thread of the processing pipeline may periodically determine whether an updated text file has been uploaded. The backend thread can determine that an updated text file has been uploaded and cause the frontend thread to pause generating transformed data. The backend thread can validate the updated text file by comparing the text file and the updated text file. Based on validating the updated text file, the backend thread can cause the frontend thread to resume transforming data using the updated text file.

Abstract: Systems and methods are described for graphical user interfaces that enable users to graphically design nodes and interconnections of pipelines for preprocessing data later ingested into an indexing system. The preprocessing can include receiving messages published to a first publish-subscribe messaging system, the messages containing raw machine data generated by one or more components in an information technology environment, performing one or more processing operations on at least some of the messages to generate preprocessed messages, republishing the preprocessed messages to a second publish-subscribe messaging system, and providing to the indexing system, a subset of the messages from the second publish-subscribe messaging system. Nodes in the pipeline can specify the preprocessing operations, and interconnections in the pipeline can represent data flow through the nodes of the pipeline.

Abstract: Systems and methods are described for performing dynamic data enrichment in accordance with data streaming. In this regard, dynamic data enrichment can be performed to process data later ingested into an indexing system. In performing dynamic data enrichment, a message can be obtained via a data stream. The message is analyzed to identify a data portion within the message to add to a lookup table. Thereafter, the data portion is written to the lookup table to enhance the lookup table with data from the message. The enhanced lookup table can then be used to lookup a data value to append to the message to enrich the message, and the enriched message can be output (e.g., to an indexing system).

Abstract: Systems and methods are described for graphical user interfaces that enable users to graphically design nodes and interconnections of pipelines for preprocessing data later ingested into an indexing system. The preprocessing can include receiving messages published to a first publish-subscribe messaging system, the messages containing raw machine data generated by one or more components in an information technology environment, performing one or more processing operations on at least some of the messages to generate preprocessed messages, republishing the preprocessed messages to a second publish-subscribe messaging system, and providing to the indexing system, a subset of the messages from the second publish-subscribe messaging system. Nodes in the pipeline can specify the preprocessing operations, and interconnections in the pipeline can represent data flow through the nodes of the pipeline.