Abstract: A method includes determining a length of a file and storing the length of the file in a first memory location. An endpoint of a last complete record within the file is determined and the endpoint is stored in a second memory location. The length of the file stored in the first memory location is compared to a current length of the file, and a data structure associated with the file is updated beginning at the endpoint if the current length of the file exceeds the length of the file stored in the first memory location.

Abstract: Processing transactions using graph-based computations includes determining that at least one of a plurality of graph elements of a computation graph of a set of one or more computation graphs includes a computation to be performed for a given transaction, associating the given transaction with an instance of the computation graph that includes reusable computation elements associated with respective graph elements, and executing the graph to perform the computation.

Abstract: An approach to performing graph-based computation uses one or both of an efficient startup approach and efficient control using process pools. Efficient startup of a graph-based computation involves precomputing data representing a runtime structure of a computation graph such that an instance of the computation graph is formed using the precomputed data for the required type of graph to form the runtime data structure for the instance of the computation graph. Pools of processes that are each suitable for performing computations associated with one or more vertices of the computation graphs are formed such that at runtime, members of these pools of processes are dynamically assigned to particular vertices of instances of computation graphs when inputs are available for processing at those vertices.

Abstract: A method, system, and program for providing runtime graph parameters and conditional components for data flow graphs. Runtime parameters allow an application builder to defer the value of a parameter setting to runtime. The values of runtime parameters may be supplied by the end user or be derived from a combination of other runtime parameters or objects stored in an object repository. A developer may specify a prompt for each parameter and the conditions for displaying the prompt, which are used to determine a graphical user interface control for receiving the parameter value. A conditional components mechanism permits changes to a graph structure based on parameter values and computed metadata. Each component of a graph has a condition which controls whether or not that component will appear in the graph at runtime. The condition can be computed directly or indirectly through runtime parameters.

Abstract: A system and method for parallelizing applications of script-driven software tools. Scripts in the software tool scripting language are automatically analyzed in order to produce a specification for a parallel computation plus a set of “script fragments”, the combination of which is functionally equivalent to the original script. The computational specification plus the script fragments are then executed by a parallel runtime system, which causes multiple instances of the original software tool and/or supplemental programs to be run as parallel processes. The resulting processes will read input data and produce output data, performing the same computation as was specified by the original script.

Abstract: A method for analyzing the performance and capacity of an application and a parallel processing system. Based upon a graph representation of the application and system and upon supplied performance values, a data file is created which describes the graph and values. Using that data file, performance equations are generated which model the performance of the application and the system. The model is displayed and modified by the user to allow analysis, evaluation, and extrapolation.

Abstract: A data processing system and method that provides two processes, checkpointing and compute point propagation, and permits a continuous flow of data processing by allowing each process to (1) return to normal operation after checkpointing or (2) respond to receipt of a compute point indicator, independently of the time required by other processes for similar responsive actions. Checkpointing makes use of a command message from a checkpoint processor that sequentially propagates through a process stage from data sources through processes to data sinks, triggering each process to checkpoint its state and then pass on a checkpointing message to connected “downstream” processes. A compute point indicator marks blocks of records that should be processed as a group within each process. A compute point indicator is triggered and sequentially propagates through a process stage from data sources through processes to data sinks without external control.

Abstract: A data processing system and method that provides checkpointing and permits a continuous flow of data processing by allowing each process to return to operation after checkpointing, independently of the time required by other processes to checkpoint their state. Checkpointing in accordance with the invention makes use of a command message from a checkpoint processor that sequentially propagates through a process stage from data sources through processes to data sinks, triggering each process to checkpoint its state and then pass on a checkpointing message to connected “downstream” processes. This approach provides checkpointing and permits a continuous flow of data processing by allowing each process to return to normal operation after checkpointing, independently of the time required by other processes to checkpoint their state.

Abstract: A method for analyzing the performance and capacity of an application and a parallel processing system. Based upon a graph representation of the application and system and upon supplied performance values, a data file is created which describes the graph and values. Using that data file, performance equations are generated which model the performance of the application and the system. The model is displayed and modified by the user to allow analysis, evaluation, and extrapolation.

Abstract: A method for analyzing the performance and capacity of an application and a parallel processing system. Based upon a graph representation of the application and system and upon supplied performance values, a data file is created which describes the graph and values. Using that data file, performance equations are generated which model the performance of the application and the system. The model is displayed and modified by the user to allow analysis, evaluation, and extrapolation.

Abstract: A buffer deadlock prevention technique for dataflow computations. The invention that implements the following algorithm: (1) providing a supplemental buffer for each input of a "downstream" program; (2) partitioning the inputs of each downstream program into disjoint input sets, such that two inputs are in the same input set if and only if such inputs come, either directly or indirectly, from a common upstream program; (3) attempting to read data into a downstream program from an "upstream" program via an input I in an input set; (4) if no data is currently available from input I, and if any other input J in the same input set has data available, then reading into the downstream program available data from each such input J and storing such that data in the supplemental buffer corresponding to such input J until such time as available data is exhausted on all such inputs J or data becomes available on input I.

Abstract: A method and apparatus by which a graph can be used to invoke computations directly. Methods get information into and out of individual processes represented on a graph, move information between the processes, and define a running order for the processes. An application writer informs a system incorporating the invention how processes should access necessary data. The invention adds "adaptor processes", if necessary, to assist in getting information into and out of processes.

Abstract: A parallel virtual file system for parallel processing systems including single-processor systems having multiple storage devices.

Abstract: A method and system that applies transaction techniques to file system operations in non-database applications executing on parallel processing systems. For each of a set of file operations, methods embodied in program routines are defined for performing, finalizing, and undoing the operations, so that the operations may be used in a non-database application to create a transaction processing environment. In general, in one aspect, the invention provides a computer program library for adding the semantics of transactions to a set of native operations of a native file system.

Abstract: Two methods for partitioning the work to be done by a computer program into smaller pieces so that checkpoints may be done more frequently. Initially, a parallel task starts with one or more input data sets having q initial partitions, divides the input data sets into p partitions by some combination of partitioning elements (i.e., partitioners/gatherers), runs an instance of a component program on each of the p partitions of the data, and produces one or more sets of output files, with each set being considered a partitioned data set. The invention is applied to such a task to create a new, "overpartitioned" task as follows: (1) the partitioner is replaced with an "overpartitioner" which divides its q inputs into n*p partitions, for some integer factor n; (2) the component program is run in a series of n execution phases, with p instances of the component program being run at any time.

Abstract: Methods and systems for running and checkpointing parallel and distributed applications which does not require modification to the programs used in the system nor changes to the underlying operating system. One embodiment of the invention includes the following general steps: (1) starting an application on a parallel processing system; (2) controlling processes for the application, including recording of commands and responses; (3) controlling a commit protocol; (4) detecting failures of the application; (5) continuing execution of the application from the most recently committed transaction after "replaying" the recorded commands and responses.