Abstract: An approach to performing graph-based computation uses one or both of an efficient startup approach and efficient control using process pools. Efficient startup of a graph-based computation involves precomputing data representing a runtime structure of a computation graph such that an instance of the computation graph is formed using the precomputed data for the required type of graph to form the runtime data structure for the instance of the computation graph. Pools of processes that are each suitable for performing computations associated with one or more vertices of the computation graphs are formed such that at runtime, members of these pools of processes are dynamically assigned to particular vertices of instances of computation graphs when inputs are available for processing at those vertices.

Abstract: An approach to performing graph-based computation uses one or both of an efficient startup approach and efficient control using process pools. Efficient startup of a graph-based computation involves precomputing data representing a runtime structure of a computation graph such that an instance of the computation graph is formed using the precomputed data for the required type of graph to form the runtime data structure for the instance of the computation graph. Pools of processes that are each suitable for performing computations associated with one or more vertices of the computation graphs are formed such that at runtime, members of these pools of processes are dynamically assigned to particular vertices of instances of computation graphs when inputs are available for processing at those vertices.

Abstract: Determining metadata associated with a graph-based computation includes functionally transforming metadata associated with a first portion of a graph to generate transformed metadata associated with a second portion of the graph; determining a third portion of the graph related to the second portion of the graph; and propagating the transformed metadata from the second portion of the graph to the third portion of the graph.

Abstract: A method, and corresponding system and software, is described for writing data to a plurality of queues, each portion of the data being written to a corresponding one of the queues. The method includes, without requiring concurrent locking of more than one queue, determining if a space is available in each queue for writing a corresponding portion of the data, and if available, reserving the spaces in the queues. The method includes writing each portion of the data to a corresponding one of the queues.

Abstract: A method, and corresponding software and system, for translating an expression are described. The method includes accepting a first expression in a first syntax, the expression including a first character sequence representing a string value using the first syntax; identifying a second syntax for representing the expression; and generating a second expression in the second syntax representing the first expression, including generating a second character sequence using the first character sequence such that the second character sequence represents the string value using the second syntax.

Abstract: A method, and corresponding software and system, is described for paging memory used for one or more sequentially-accessed data structure. The method includes providing a data structure representing an order in which memory pages are to be reused; and maintaining the data structure according to a history of access to a memory page associated with one of the sequentially-accessed data structures. A position of the memory page in the order depends on a transition of sequential access off of the memory page.

Abstract: Aggregating data includes accepting a first data set that includes records, each record holding a value for each of a plurality of fields. A second data set is generated from the first data set. The second data set includes one or more aggregated records each corresponding to one or more records from the first data set that match values in a subset of fields. A third data set is generated from the second data set. The third data set includes one or more aggregated records each corresponding to one or more aggregated records of the first data set that match values in a subset of fields. An aggregate value associated with an aggregated record in the third data set represents a result of performing a non-cascadable operation on values associated with a plurality of records from the first data set.

Abstract: A number of tasks are defined according to a dependency graph. Multiple parameter contexts are maintained, each associated with a different scope of the tasks. A parameter used in a first of the tasks is bound to a value. This binding includes identifying a first of the contexts according to the dependency graph and retrieving the value for the parameter from the identified context.

Abstract: A specification of a graphical representation of task dependency has a plurality of task elements each associated with a different task, a resource element having a plurality of attachment locations, and linking elements coupling the task elements to the resource element at the plurality of associated attachment locations. Associations of task elements to attachment locations on the resource element specify an ordering constraint on the tasks associated with the task elements. The tasks are executed according to the graphical representation of task dependency.

Abstract: An approach to automatically specifying, or assisting with the specification of, a parallel computation graph involves determining data processing characteristics of the linking elements that couple data processing elements of the graph. The characteristics of the linking elements are determined according to the characteristics of the upstream and/or downstream data processing elements associated with the linking element, for example, to enable computation by the parallel computation graph that is equivalent to computation of an associated serial graph.

Abstract: A parallel virtual file system for parallel processing systems including single-processor systems having multiple storage devices.

Abstract: Two methods for partitioning the work to be done by a computer program into smaller pieces so that checkpoints may be done more frequently. Initially, a parallel task starts with one or more input data sets having q initial partitions, divides the input data sets into p partitions by some combination of partitioning elements (i.e., partitioners/gatherers), runs an instance of a component program on each of the p partitions of the data, and produces one or more sets of output files, with each set being considered a partitioned data set. The invention is applied to such a task to create a new, "overpartitioned" task as follows: (1) the partitioner is replaced with an "overpartitioner" which divides its q inputs into n*p partitions, for some integer factor n; (2) the component program is run in a series of n execution phases, with p instances of the component program being run at any time.

Abstract: Methods and systems for running and checkpointing parallel and distributed applications which does not require modification to the programs used in the system nor changes to the underlying operating system. One embodiment of the invention includes the following general steps: (1) starting an application on a parallel processing system; (2) controlling processes for the application, including recording of commands and responses; (3) controlling a commit protocol; (4) detecting failures of the application; (5) continuing execution of the application from the most recently committed transaction after "replaying" the recorded commands and responses.