Abstract: A system and method from processing database queries allows for cost and locale based distribution for execution of database queries. The database queries are executed on execution engines that provide flexible configuration and overlapping functionality. The system reduces various costs, including elapsed time, required to perform database queries. The method provides processing of a database query using a database catalog comprising database table locality information, record locality information and execution engine information. A query optimizer receives the query and accesses the catalog to create a query execution plan comprising locality-based database operations. A central database operation processor providing a first execution engine executes the query execution plan by performing at least a portion of the locality-based database operations and distributing at least a portion of the locality-based database operations as a subplan.

Abstract: A technique for read error failover processing in a mirrored disk system such as a Redundant Array of Inexpensive Disks (RAID) system, where individual disk units perform Logical Block Address (LBA) remapping. A read error causes a disk controller to report an “unrecoverable” read error to a RAID controller. After receiving this report of an unrecoverable read error, rather than simply switching over to a mirror, the RAID controller first checks to see whether the disk that caused the error can successfully reassign an LBA. In particular, the RAID controller can retrieve the data that was associated with the failed LBA from the mirror, and then write that data to the offending disk. Only if this process does not succeed is the offending disk then treated by the RAID controller as having failed sufficiently to require failover to the mirror.

Abstract: A field oriented pipeline processor for a data engine that can be programmed to recognize record and field structures of data received from a source such as a mass storage device. The processor has an interface for receiving field-delineated data from a field parser. The field parser parses non-field delineated data from a streaming data source into field delineated data under instruction from an external processing unit. The pipeline processor receives a field delineated data stream and employs logical arithmetic methods to compare fields with one another, or with values otherwise supplied by general purpose processors to precisely determine which records are worth transferring to memory of the more general purpose distributed processors. The architecture of the data engine allows for the use of substitution tables, temporary registers, and a data string register to assist in the efficiency and accuracy of the data engine processing.

Abstract: A programmable streaming data processor that can be programmed to recognize record and field structures of data received from a streaming data source such as a mass storage device. Being programmed with, for example, field information, the unit can locate record and field boundaries and employ logical arithmetic methods to compare fields with one another or with values otherwise supplied by general purpose processors to precisely determine which records are worth transferring to memory of the more general purpose distributed processors. The remaining records arrive and are discarded by the streaming data processor or are tagged with status bits to indicate to the more general purpose processor that they are to be ignored. In a preferred embodiment, the streaming data processor may analyze and discard records for several reasons. The first reason may be an analysis of contents of the field.

Abstract: A usage model and the underlying technology used to provide sequence analysis as part of a relational database system. Included components include the semantic and syntactic integration of the sequence analysis with an existing query language, the storage methods for the sequence data, and the design of a multipart execution scheme that runs the sequence analysis as part of a potentially larger database query, especially using parallel execution techniques.

Abstract: A field oriented pipeline processor for a data engine that can be programmed to recognize record and field structures of data received from a source such as a mass storage device. The processor has an interface for receiving field-delineated data from a field parser. The field parser parses non-field delineated data from a streaming data source into field delineated data under instruction from an external processing unit. The pipeline processor receives a field delineated data stream and employs logical arithmetic methods to compare fields with one another, or with values otherwise supplied by general purpose processors to precisely determine which records are worth transferring to memory of the more general purpose distributed processors. The architecture of the data engine allows for the use of substitution tables, temporary registers, and a data string register to assist in the efficiency and accuracy of the data engine processing.

Abstract: In a preferred embodiment, the present invention is an asymmetric data processing system having two or more groups of processors that have attributes that are optimized for their assigned functions. A first processor group, which may be SMP machines, are responsible for interfacing with applications and/or end users to obtain queries, and for planning query execution. A second processor group consists of many streaming record-oriented processors called Job Processing Units (JPUs), typically arranged as an MPP structure. The JPUs carry out the bulk of the data processing required to implement the logic of a query, running autonomously and asynchronously from other processors in the system. The JPUs preferably use a multi-tasking operating system that permits multiple tasks to run at a given instant in time, in either an absolute-priority-based or a weighted-priority-based demand scheduling environment.

Abstract: A data processing system having two or more groups of data processors that have attributes that are optimized for their assigned functions. A first group consists of one or more host computers responsible for interfacing with applications and/or end users to obtain queries and for planning query execution. A second processor group consists of many streaming record-oriented processors called Job Processing Units (JPUs), preferably arranged as an MPP structure. The JPUs typically carry out the bulk of the data processing required to implement the logic of a query. Each of the JPUs typically include a general purpose microcomputer, local memory, one or more mass storage devices, and one or more network connections. Each JPU also has a special purpose programmable processor, referred to herein as a Programmable Streaming Data Processor (PSDP). The PSDP serves as an interface between the CPU of a JPU and the mass storage device, to offload functions from the CPU of the JPU.

Abstract: An asymmetric data record processor and method includes host computers and Job processing units (JPU's) coupled together on a network. Each host computer and JPU forms a node on the network. A plurality of software operators allow each node to process streams of records. For each operator in a given sequence within nodes and across nodes, output of the operator is input to a respective succeeding operator. Data processing follows a logical data flow based on readiness of a record. As soon as a record is ready it is passed for processing from one part to a next part in the logical data flow. The flow of records during data processing is substantially continuous and of a streaming fashion.

Abstract: This invention relates generally to a system for processing database queries, and more particularly to a method for generating high level language or machine code to implement query execution plans. In one preferred embodiment, the method begins by receiving a subject query, and then forming an execution plan corresponding to the subject query. The execution plan will typically have a sequence of component snippets or pieces and corresponding processes for implementing the pieces. For at least one piece in the plan, the process then (a) generates source code using different code generation techniques as a function of expected runtime processing machine types; and then (b) compiles the generated source code to form machine executable code for implementing the subject query. As a result, the query executed directly as machine executable code thereby avoiding runtime interpretation of the pieces in the execution plan.

Abstract: A system from processing database queries allows for cost and locale based distribution for execution of database queries. The database queries are executed on execution engines that provide flexible configuration and overlapping functionality. The system reduces various costs, including elapsed time, required to perform database queries. The system provides processing of a database query using a database catalog comprising database table locality information, record locality information and execution engine information. A query optimizer receives the query and accesses the database catalog to create a query execution plan comprising locality-based database operations. A central database operation processor providing a first execution engine executes the query execution plan by performing at least a portion of the locality-based database operations and distributing at least a portion of the locality-based database operations as a subplan.

Abstract: This invention relates generally to a system for processing database queries, and more particularly to a method for generating high level language or machine code to implement query execution plans. in one preferred embodiment, the method begins by receiving a subject query, and then forming an execution plan corresponding to the subject query. The execution plan will typically have a sequence of component snippets or pieces and corresponding processes for implementing the pieces. For at least one piece in the plan, the process then (a) generates source code using different code generation techniques as a function of expected runtime processing machine types; and then (b) compiles the generated source code to form machine executable code for implementing the subject query. As a result, the query executed directly as machine executable code thereby avoiding runtime interpretation of the pieces in the execution plan.

Abstract: A disk is segmented into a first data segment and a secondary data segment. The secondary data segment stores a logical mirror of the first data segment of another disk. Upon detecting a failure, the logical mirror of data stored in the first data segment of the failed disk is accessible from the secondary data segment of a non-failed disk. The first data segment can be rebuilt quickly on another disk from the logical mirror stored in the secondary data segment. During regenerating, accesses to the first data segment on the disk containing the logical mirror are handled by its own logical mirror, which is not involved in the regenerating process.

Abstract: A multi-version database system controls visibility of data during transaction processing. A transaction includes a transaction identifier that identifies the transaction and an invisibility list of transactions whose effects are invisible to the transaction. Changes made by other transactions are visible to the transaction based on the isolation level of the transaction. Records are visible to a transaction based on a creator identifier stored in the record that identifies the transaction that created the record.

Abstract: A technique for read error failover processing in a mirrored disk system such as a Redundant Array of Inexpensive Disks (RAID) system, where individual disk units perform Logical Block Address (LBA) remapping. A read error causes a disk controller to report an “unrecoverable” read error to a RAID controller. After receiving this report of an unrecoverable read error, rather than simply switching over to a mirror, the RAID controller first checks to see whether the disk that caused the error can successfully reassign an LBA. In particular, the RAID controller can retrieve the data that was associated with the failed LBA from the mirror, and then write that data to the offending disk. The disk controller for that disk will then perform its standard LBA remapping, and write the data to a new, good sector. Only if this process does not succeed is the offending disk then treated by the RAID controller as having failed sufficiently to require failover to the mirror.

Abstract: A database appliance having multiple internetworked data storage units and central database processing units. The central database processors and storage units communicate as network nodes. Execution of a distributed database application is coordinated by a communication process such that data blocks are passed by using data block reference information. Operations executing within the same context as the communication process may include join, sort, aggregate, restrict, reject, expression evaluation, statistical analysis or other database operations.

Abstract: A mechanism controls concurrency among database transactions through the use of serial ordering relations. The ordering relations are computed dynamically in response to patterns of use. An embodiment of the present invention serializes a transaction that accesses a resource before a transaction that modifies the resource, even if the accessor starts after the modifier starts or commits after the modifier commits. A method of concurrency control for a database transaction in a distributed database system stores an intended use of a database system resource by the database transaction in a serialization graph. A serialization ordering is asserted between the database transaction and other database transactions based on the intended use of the database system resource by the database transaction. The serialization ordering is then communicated to a node in the distributed database system that needs to know the serialization ordering to perform concurrency control.

Abstract: Described herein is a system for creating and maintaining materialized views. The maintenance of the materialized views is performed in a distributed manner that improves efficiency and performance in response to the queries for materialized views or views on top of the materialized views.

Abstract: A disk is segmented into a first data segment and a secondary data segment. The secondary data segment stores a logical mirror of the first data segment of another disk. Fast access to data stored on the disk is provided by partitioning the disk such that the first data segment includes the fast tracks of the disk and the secondary data segment includes the slow tracks of the disk and forwarding all data requests to the first data segment. Upon detecting a failure, the logical mirror of data stored in the first data segment of the failed disk is accessible from the secondary data segment of a non-failed disk. The first data segment can be rebuilt quickly on another disk from the logical mirror stored in the secondary data segment.

Abstract: A mechanism controls concurrency among database transactions through the use of serial ordering relations. The ordering relations are computed dynamically in response to patterns of use. An embodiment of the present invention serializes a transaction that accesses a resource before a transaction that modifies the resource, even if the accessor starts after the modifier starts or commits after the modifier commits. A method of concurrency control for a database transaction in a distributed database system stores an intended use of a database system resource by the database transaction in a serialization graph. A serialization ordering is asserted between the database transaction and other database transactions based on the intended use of the database system resource by the database transaction. The serialization ordering is then communicated to a node in the distributed database system that needs to know the serialization ordering to perform concurrency control.