Abstract: A data processing method is provided. The method comprises collecting a stream of data records received from one or more data sources connected in a communications network; dividing the stream of data records into sets of data records for parallel processing by a plurality of concurrently running tasks, wherein a first task loads a persistent index associated with a first set of data records into memory to generate an in-memory version of the first persistent index for the first set of data records; and identifying duplicate and non-duplicate data records in the first set of data records, based on searching the in-memory version of the first persistent index.

Abstract: A data processing method is provided. The method comprises collecting a stream of data records received from one or more data sources connected in a communications network; dividing the stream of data records into sets of data records for parallel processing by a plurality of concurrently running tasks, wherein a first task loads a persistent index associated with a first set of data records into memory to generate an in-memory version of the first persistent index for the first set of data records; and identifying duplicate and non-duplicate data records in the first set of data records, based on searching the in-memory version of the first persistent index.

Abstract: A data processing method is provided. The method comprises collecting a stream of data records from one or more devices in a network; loading one or more persistent indexes associated with the stream of data records into memory; identifying duplicate data records in the stream of data records using the in-memory indexes; and updating a repository such that the duplicate data records are not stored in the repository or managed differently than non-duplicate data records.

Abstract: A data processing method is provided. The method comprises collecting a stream of data records from one or more devices in a network; loading one or more persistent indexes associated with the stream of data records into memory; identifying duplicate data records in the stream of data records using the in-memory indexes; and updating a repository such that the duplicate data records are not stored in the repository or managed differently than non-duplicate data records.

Abstract: A database architecture system includes a database including a plurality of tables for storing data records having a time of creation and a time of arrival. A plurality of current tables include current data records and have their time of creation meeting a first specified creation time period, and their time of arrival meets a specified arrival time period. The current data records which meet a specified extraction time period after the first specified creation time period are extracted to files. Data records in current tables, having their time of creation overlapping the specified extraction time period are not updated during extraction, but are accessible for querying. The current data records having a time of creation not overlapping the specified extraction time period are accessible for querying and updating during extraction. The data records in the files are deleted when their time of creation meets a specified deletion time period.

Abstract: A relational database architecture with Dynamic Load capability, including a relational database for storing records, a storage unit, a record dispatcher for storing a plurality of objects, each containing at least one record, in the storage unit, and a mapping of the objects to the records which they contain, where the record dispatcher is configured to receive a Dynamic Load Request including a key value of desired records, search the mapping to determine which of the objects in the storage unit contain records that match the key value, and load any of the matching records into the database.

Abstract: A relational database architecture with Dynamic Load capability, including a relational database for storing records, a storage unit, a record dispatcher for storing a plurality of objects, each containing at least one record, in the storage unit, and a mapping of the objects to the records which they contain, where the record dispatcher is configured to receive a Dynamic Load Request including a key value of desired records, search the mapping to determine which of the objects in the storage unit contain records that match the key value, and load any of the matching records into the database.