Abstract: A distributed data processing system executes database operations, and includes a first group of host processors and a second group of job processing units (JPUs). A software application for executing database operations executes in a distributed fashion with portions of the database application executing on at least one central database processor and other portions executing on the data storage processors. At least a portion of the database application is implemented within and/or coordinated by a communication process that is executing the communication protocol. The communication process implements a load balancing function in front of the host processors, the load balancing function directing individual transactions to selected processors so as to evenly distribute workload.

Abstract: A field oriented pipeline processor for a data engine that can be programmed to recognize record and field structures of data received from a source such as a mass storage device. The processor has an interface for receiving field-delineated data from a field parser. The field parser parses non-field delineated data from a streaming data source into field delineated data under instruction from an external processing unit. The pipeline processor receives a field delineated data stream and employs logical arithmetic methods to compare fields with one another, or with values otherwise supplied by general purpose processors to precisely determine which records are worth transferring to memory of the more general purpose distributed processors. The architecture of the data engine allows for the use of substitution tables, temporary registers, and a data string register to assist in the efficiency and accuracy of the data engine processing.

Abstract: A field oriented pipeline processor for a data engine that can be programmed to recognize record and field structures of data received from a source such as a mass storage device. The processor has an interface for receiving field-delineated data from a field parser. The field parser parses non-field delineated data from a streaming data source into field delineated data under instruction from an external processing unit. The pipeline processor receives a field delineated data stream and employs logical arithmetic methods to compare fields with one another, or with values otherwise supplied by general purpose processors to precisely determine which records are worth transferring to memory of the more general purpose distributed processors. The architecture of the data engine allows for the use of substitution tables, temporary registers, and a data string register to assist in the efficiency and accuracy of the data engine processing.

Abstract: A field oriented pipeline processor for a data engine that can be programmed to recognize record and field structures of data received from a source such as a mass storage device. The processor has an interface for receiving field-delineated data from a field parser. The field parser parses non-field delineated data from a streaming data source into field delineated data under instruction from an external processing unit. The pipeline processor receives a field delineated data stream and employs logical arithmetic methods to compare fields with one another, or with values otherwise supplied by general purpose processors to precisely determine which records are worth transferring to memory of the more general purpose distributed processors. The architecture of the data engine allows for the use of substitution tables, temporary registers, and a data string register to assist in the efficiency and accuracy of the data engine processing.

Abstract: A programmable streaming data processor that can be programmed to recognize record and field structures of data received from a streaming data source such as a mass storage device. Being programmed with, for example, field information, the unit can locate record and field boundaries and employ logical arithmetic methods to compare fields with one another or with values otherwise supplied by general purpose processors to precisely determine which records are worth transferring to memory of the more general purpose distributed processors. The remaining records arrive and are discarded by the streaming data processor or are tagged with status bits to indicate to the more general purpose processor that they are to be ignored. In a preferred embodiment, the streaming data processor may analyze and discard records for several reasons. The first reason may be an analysis of contents of the field.