Abstract: An apparatus includes a processor to: receive a request to perform a job flow; within a performance container, based on the data dependencies among a set of tasks of the job flow, derive an order of performance of the set of tasks that includes a subset able to be performed in parallel, and derive a quantity of task containers to enable the parallel performance of the subset; based on the derived quantity of task containers, derive a quantity of virtual machines (VMs) to enable the parallel performance of the subset; provide, to a VM allocation routine, an indication of a need for provision of the quantity of VMs; and store, within a task queue, multiple task routine execution request messages to enable parallel execution of task routines within the quantity of task containers to cause the parallel performance of the subset.

Abstract: An apparatus includes a processor to receive a request to provide a view of an object associated with a job flow, and in response to determining that the object is associated with a task type requiring access to a particular resource not accessible to a first interpretation routine: store, within a job queue, a job flow generation request message to cause generation of a job flow definition the defines another job flow for generating the requested view; within a task container in which a second interpretation routine that does have access to the particular resource is executed, generate the job flow definition; store, within a task queue, a job flow generation completion message that includes a copy of the job flow definition; use the job flow definition to perform the other job flow to generate the requested view; and transmit the requested view to the requesting device.

Abstract: An apparatus including a processor to: within a kill container, in response to a set of error messages indicative of errors in executing multiple instances of a task routine to perform a task of a job flow with multiple data object blocks of a data object, and in response to the quantity of error messages reaching a threshold, output a kill tasks request message that identifies the job flow; within a task container, in response to the kill tasks request message, cease execution of the task routine and output a task cancelation message that identifies the task and the job flow; and within a performance container, in response to he task cancelation message, output a job cancelation message to cause the transmission of an indication of cancelation of the job flow, via a network, and to a requesting device that requested the performance of the job flow.

Abstract: An apparatus includes a processor to: output a request message to cause a first task to be performed; within a task container, in response to the request message and a data object not being divided, divide the data object into a set of data object blocks based on at least the sizes of the data object and the atomic unit of organization of data therein, as well as the storage resources allocated to task containers, and output a task completion message indicating that the first task has been performed, and including a set of data block identifiers indicating the location of the set of data object blocks within at least one federated area; and in response to the task completion message, output a set of request messages to cause a second task to be performed by executing multiple instances of a task routine within multiple task containers.

Abstract: An apparatus including a processor to: output a first request message onto a group sub-queue shared by multiple task containers to request execution of a first task routine; within a task container, respond to the first request message, by outputting a first task in-progress message onto an individual sub-queue not shared with other task containers to accede to executing the first task routine, followed by a task completion message; and respond to the task completion message by allowing the task completion message to remain on the individual sub-queue to keep the task container from executing another task routine from another request message on the group sub-queue, outputting a second request message onto the individual sub-queue to cause execution of a second task routine within the same task container to perform a second task, and responding to the second task in-progress message by de-queuing the task completion message.

Abstract: An apparatus includes a processor to receive a request to provide a view of an object associated with a job flow, and in response to determining that the object is associated with a task type requiring access to a particular resource not accessible to a first interpretation routine: store, within a job queue, a job flow generation request message to cause generation of a job flow definition the defines another job flow for generating the requested view; within a task container in which a second interpretation routine that does have access to the particular resource is executed, generate the job flow definition; store, within a task queue, a job flow generation completion message that includes a copy of the job flow definition; use the job flow definition to perform the other job flow to generate the requested view; and transmit the requested view to the requesting device.

Abstract: An apparatus includes a processor to: receive a request to perform a job flow; within a performance container, based on the data dependencies among a set of tasks of the job flow, derive an order of performance of the set of tasks that includes a subset able to be performed in parallel, and derive a quantity of task containers to enable the parallel performance of the subset; based on the derived quantity of task containers, derive a quantity of virtual machines (VMs) to enable the parallel performance of the subset; provide, to a VM allocation routine, an indication of a need for provision of the quantity of VMs; and store, within a task queue, multiple task routine execution request messages to enable parallel execution of task routines within the quantity of task containers to cause the parallel performance of the subset.

Abstract: An apparatus includes a processor to: derive an order of performance of a set of tasks of a job flow; based on the order of performance, store, within a task queue, a first task routine execution request message to cause a first task to be performed; within a first task container, and in response to storage of the first task routine execution request message, execute instructions of a first task routine of a set of task routines, store a mid-flow data set output of the first task within a federated area, and store a first task completion message within the task queue after completion of storage of the mid-flow data set; and in response to the storage of the first task completion message, and based on the order of performance, store, within the task queue, a second task routine execution request message to cause a second task to be performed.

Abstract: An apparatus includes processor(s) to: within a kill container, monitor a task kill queue for execution failure message(s) indicating failure of a task routine to perform a task of a job flow, and in response to a quantity of execution failure messages reaching a threshold, store, within the task kill queue, a kill tasks request message; within a task container, in response to the kill tasks request message, cease execution of the task routine to cease performance of the task, and store, within the task queue, a task cancelation message indicating cancelation of the task routine; within a performance container, in response to the task canceled message, store, within a job queue, a job cancelation message indicating cancelation of the job flow; and in response to the job cancelation message, transmit an indication of cancelation of the job flow to a requesting device.

Abstract: An apparatus includes processor(s) to: within a kill container, monitor a task kill queue for execution failure message(s) indicating failure of a task routine to perform a task of a job flow, and in response to a quantity of execution failure messages reaching a threshold, store, within the task kill queue, a kill tasks request message; within a task container, in response to the kill tasks request message, cease execution of the task routine to cease performance of the task, and store, within the task queue, a task cancelation message indicating cancelation of the task routine; within a performance container, in response to the task canceled message, store, within a job queue, a job cancelation message indicating cancelation of the job flow; and in response to the job cancelation message, transmit an indication of cancelation of the job flow to a requesting device.

Abstract: An apparatus includes processor(s) to: receive, a request to cancel a performance of a job flow; store, within a task kill queue, a kill tasks request message; within a task container, in response to the storage of the kill tasks request message, cease execution of instructions of a task routine to cease performance of a task of a set of tasks of the job flow, and store, within a task queue, a task cancelation message indicating cancelation of execution of the task routine; within a performance container, execute instructions of a performance routine to cause the at least one processor to, in response to the storage of the task canceled message, store, within a job queue, a job cancelation message indicating cancelation of the job flow; and in response to the storage of the job cancelation message, transmit an indication of cancelation of the job flow to the requesting device.

Abstract: An apparatus includes processor(s) to: receive a request to perform a job flow; retrieve a job flow definition from federated area(s); store, within a job queue, a job performance request message comprising the job flow definition; within a first performance container, execute a performance routine causing the processor(s) to, in response to the storage of the job performance request message within the job queue, derive an order of performance of tasks specifying a first task based on dependencies among the tasks, and store, within a task queue, a first task routine execution request message comprising an identifier associated with the first task; and within a first task container, in response to the storage of the first task routine execution request message within the task queue, use the identifier to retrieve a first task routine from the federated area(s), and execute the first task routine to perform the first task.

Abstract: An apparatus includes a processor to: receive a request to perform a set of tasks; analyze input data objects used as inputs and output data objects generated as outputs by the set of tasks to identify at least one dependency between tasks and at least one pair of tasks performable in parallel; determine an order of performance of the set of tasks; retrieve executable instructions of a corresponding set of task routines that define input interfaces and output interfaces for the input data objects and output data objects, respectively; generate a job flow definition defining the order of performance, each input interface and each output interface; use the job flow definition to perform the set of tasks; and transmit one of the output data objects to the requesting device as a result report serving as the output of the set of tasks.

Abstract: An apparatus includes a processor to: receive a request from a remote device to perform a job flow; retrieve a job flow definition defining the job flow and each of a set of task routines to perform tasks of the job flow from a set of storage devices where each is stored as an undivided object within one storage device; and in response to determining that a data set is stored as multiple data object blocks, generate a container containing the job flow definition and set of task routines to enable each storage device to perform the job flow using a locally stored data object block of the data set as input to generate a corresponding data object block of a result report, provide a copy of the container to each storage device, and transmit the result report assembled from the data object blocks thereof to the remote device.

Abstract: An apparatus includes a processor to: receive a request to perform a job flow defined in a job flow definition; retrieve the most recent versions of a set of task routines to perform a set of tasks of the job flow; analyze each interface of each task routine by which a data set is input or output during execution to identify dependencies where two task routines exchange a data set, and are written in different programming languages; execute the set of task routines to perform the job flow; in response to identifying such a dependency, convert the data set from a form supported by the programming language of one of the two task routines and into a form supported by the programming language of the other; store one of the forms within a federated area; and transmit a result report output during the performance to a remote device.

Abstract: An apparatus including a processor to: parse comments of multiple task routines to identify I/O parameters; for each task routine, generate a macro including its I/O parameters; transmit the macros to a requesting device to enable it to generate a visualization of a DAG to include visual representations of the task routines; wherein each representation includes a task graph object, an input data graph object representing an input and indicating a characteristic of the input, and an output data graph object representing an output and indicating a characteristic of the output; and wherein the requesting device is to: identify, in the I/O parameters, each dependency between an output and an input of a pair of task routines; augment the visualization, for each identified dependency, with a dependency marker that visually links the visual representations of the pair of task routines; and visually output the visualization.

Abstract: An apparatus including a processor to: parse comments of multiple task routines to identify I/O parameters; generate a visualization of a DAG to include a visual representation of each task routine, wherein each representation includes a task graph object of the task routine, at least one input data graph object that represents an input to the task routine and that includes a visual indication of at least one characteristic of the input; and at least one output data graph object that represents an output of the task routine and that includes a visual indication of at least one characteristic of the output; in the I/O parameters, identify each dependency between an output of one task routine and an input of another; for each identified dependency, augment the visualization with a dependency marker that visually links the visual representations of each associated pair of task routines; and visually output the visualization.