Abstract: In an embodiment of the present invention, a record is read a single time, and it is evaluated against all conditions to identify the ones that it satisfies. By organizing the conditions into appropriate data structures, the process may be further optimized by ensuring that the record is not scanned multiple times.

Abstract: In an embodiment of the present invention, a record is read a single time, and it is evaluated against all conditions to identify the ones that it satisfies. By organizing the conditions into appropriate data structures, the process may be further optimized by ensuring that the record is not scanned multiple times.

Abstract: A solution is provided wherein path information is stored for efficient retrieval. Raw path information may be stored in a path file. A node path index file may then be created containing entries for each of one or more corresponding nodes in the path information. Each node path entry corresponds to a unique appearance of the corresponding node in the path file, and wherein each node path entry contains a path file offset and a position of the corresponding node in the path file in the path indicated by the path file offset. A node index file may then be created containing, for one or more nodes in the path information, a single node entry containing an indication of the number of times the corresponding node in the node path index file appears in the node path index file and also containing a node path index file offset.

Abstract: A solution is provided wherein a master process and two or more drone processes may be utilized to identify path information containing a pattern. The master process may send the pattern to the two or more drone processes, which may identify the pattern in path data. Each drone process may then send the paths that satisfy the pattern back to the master process, which may aggregate the path data so that two or more identical paths appearing in the path data are reduced to a single occurrence of a path.

Abstract: Data quality measurement is provided for use in a data processing stream, which comprises at least one upstream data processing system and at least one downstream data processing system. An input alert component can be used to provide a measurement of data prior to its input to a data processing system (e.g., a downstream data processing system or an upstream data processing system). An output alert component can be used to provide a measurement on data output by a data processing system. A self-consistency component can be used to measure consistency between items of input, or output data. An end-to-end component can be used to measure data quality using data items from both input data and output data. These components can be used in some combination, or independent of the other, and in any order. In addition, the data quality measurements can be performed separate from that processing performed by either the upstream or downstream processing system.