Abstract: A method for receiving text files, each having data defined in various format types. The text files are received without a reference to a schema defining the text files. A description file for each of the text files describing one of the format types is received and the description file includes a separate file from a corresponding text file. Before processing the text file and absent receiving the schema, the method identifies a data structure of the description file, the data structure comprising one or more parameters and definitions of the various format types. Each of the text files is processed and during processing of each of the text files, dynamically generating, at run-time, one or more reader engines to read the data in the text files and dynamically extracting, at run-time, data based on the identified data structure. The method generates an output file with the extracted data.

Abstract: A method for receiving text files, each having data defined in various format types. The text files are received without a reference to a schema defining the text files. A description file for each of the text files describing one of the format types is received and the description file includes a separate file from a corresponding text file. Before processing the text file and absent receiving the schema, the method identifies a data structure of the description file, the data structure comprising one or more parameters and definitions of the various format types. Each of the text files is processed and during processing of each of the text files, dynamically generating, at run-time, one or more reader engines to read the data in the text files and dynamically extracting, at run-time, data based on the identified data structure. The method generates an output file with the extracted data.

Abstract: A method and a system are provided for constructing a data pipeline having scalability and low latency. In one example, the system provides a primary data distributor cluster. The system provides one or more non-primary data distributor clusters. The system arranges a cascade configuration that includes the primary data distributor cluster and the one or more non-primary data distributor clusters.

Abstract: A method and a system are provided for processing an event stream for a data pipeline. In one example, the system receives one or more plug-in components from one or more consumer devices. The system generates a layout for the one or more plug-in components. The system receives one or more events from an input source. The system then provides filtered events to at least one consumer device by using the layout for the one or more plug-in components.

Abstract: A method and a system are provided for constructing a data pipeline having scalability and low latency. In one example, the system provides a primary data distributor cluster. The system provides one or more non-primary data distributor clusters. The system arranges a cascade configuration that includes the primary data distributor cluster and the one or more non-primary data distributor clusters.

Abstract: Systems and methods for tracking event loss are set forth in this disclosure. More specifically, systems and methods for tracking event loss within a first time period and second time period are set forth in this disclosure.