Abstract: A data processing method is provided. The method comprises collecting a stream of data records received from one or more data sources connected in a communications network; dividing the stream of data records into sets of data records for parallel processing by a plurality of concurrently running tasks, wherein a first task loads a persistent index associated with a first set of data records into memory to generate an in-memory version of the first persistent index for the first set of data records; and identifying duplicate and non-duplicate data records in the first set of data records, based on searching the in-memory version of the first persistent index.

Abstract: A method of scheduling highly parallel jobs with global interdependencies is provided herein. The method includes the following steps: grouping input elements, each group being associated with an interdependency tag reflecting a level of interdependency between data associated with different input elements within a group; clustering the groups into collections of groups, wherein the clustered groups are associated with an interdependency tag reflecting a level of interdependency between groups, above a specified value; applying a conflict check to the collections of groups and to active jobs of a working set, to yield a conflict level between each collection of groups and each active job, by analyzing the interdependency tags of the collections of groups vis à vis interdependency tags associated with the active jobs; and adding collections of groups into the working set, wherein added collections of groups are associated with a conflict level below an acceptable conflict level.

Abstract: A data processing method is provided. The method comprises collecting a stream of data records received from one or more data sources connected in a communications network; dividing the stream of data records into sets of data records for parallel processing by a plurality of concurrently running tasks, wherein a first task loads a persistent index associated with a first set of data records into memory to generate an in-memory version of the first persistent index for the first set of data records; and identifying duplicate and non-duplicate data records in the first set of data records, based on searching the in-memory version of the first persistent index.

Abstract: A data processing method is provided. The method comprises collecting a stream of data records from one or more devices in a network; loading one or more persistent indexes associated with the stream of data records into memory; identifying duplicate data records in the stream of data records using the in-memory indexes; and updating a repository such that the duplicate data records are not stored in the repository or managed differently than non-duplicate data records.

Abstract: A data processing method is provided. The method comprises collecting a stream of data records from one or more devices in a network; loading one or more persistent indexes associated with the stream of data records into memory; identifying duplicate data records in the stream of data records using the in-memory indexes; and updating a repository such that the duplicate data records are not stored in the repository or managed differently than non-duplicate data records.

Abstract: A method of scheduling highly parallel jobs with global interdependencies is provided herein. The method includes the following steps: grouping input elements, each group being associated with an interdependency tag reflecting a level of interdependency between data associated with different input elements within a group; clustering the groups into collections of groups, wherein the clustered groups are associated with an interdependency tag reflecting a level of interdependency between groups, above a specified value; applying a conflict check to the collections of groups and to active jobs of a working set, to yield a conflict level between each collection of groups and each active job, by analyzing the interdependency tags of the collections of groups vis à vis interdependency tags associated with the active jobs; and adding collections of groups into the working set, wherein added collections of groups are associated with a conflict level below an acceptable conflict level.

Abstract: A method for processing a document that includes a plurality of fields having respective contents. The method includes providing labels to be assigned respectively to the fields and one or more rules applicable to the contents of the fields according to the labels assigned thereto. A computer reads the contents of the fields, and assigns the labels to the fields responsive to application of the rules to the contents.

Abstract: A method for processing an input image includes applying one or more transformations to the input image, whereby different shifts are applied to different pixels in the input image, so as to generate an output image. A region containing content of interest is selected in the output image, and the region corresponding to the selected region in the output image is located in the input image. The content of the located region in the input image is substituted for the content of the selected region in the output image.

Abstract: A method for processing a document that includes a plurality of fields having respective contents. The method includes providing labels to be assigned respectively to the fields and one or more rules applicable to the contents of the fields according to the labels assigned thereto. A computer reads the contents of the fields, and assigns the labels to the fields responsive to application of the rules to the contents.