Abstract: A system for processing a batch job comprises a processor and a memory. The processor is configured to receive a batch job comprising a sequential or parallel flow of operations, wherein each operation has a defined input type and a defined output type. The processor is further configured to verify that the batch job can run successfully, wherein verifying includes checking that a first operation output defined type is compatible with a second operation input defined type when a first operation output is connected to a second operation input, and wherein verifying includes checking that a parameter used by a calculation in an operation is input to the operation. The memory is coupled to the processor and configured to provide the processor with instructions.

Abstract: A system for processing a batch job comprises a processor and a memory. The processor is configured to receive a job name for a job submitted to execute, to receive one or more job parameters, and to determine one or more nodes to run the job. The processor is configured to determine one or steps, where for each step: a step is executed on a node using a state of data associated with a start state of the step; and upon completion of executing the step, a result is stored to a durable storage. The durable storage stores the state of data associated with the start state of the step and the completion state of the step and are accessible by other execution processes as associated with either the start state of the step or the completion state of the step. The memory of the system is coupled to the processor and configured to provide processor with instructions.