Abstract: Implementing static loaders and savers for the transfer of local and distributed data containers to and from storage systems can be difficult because there are so many different configurations of output formats, data containers and storage systems. Described herein is an extensible componentized data transfer framework for performant and scalable authoring of data loaders and data savers. Abstracted local and distributed workflows drive selection of plug-ins that can be composed by the framework into particular local or distributed scenario loaders and savers. Reusability and code sparsity are maximized.

Abstract: Implementing static loaders and savers for the transfer of local and distributed data containers to and from storage systems can be difficult because there are so many different configurations of output formats, data containers and storage systems. Described herein is an extensible componentized data transfer framework for performant and scalable authoring of data loaders and data savers. Abstracted local and distributed workflows drive selection of plug-ins that can be composed by the framework into particular local or distributed scenario loaders and savers. Reusability and code sparsity are maximized.

Abstract: Implementing static loaders and savers for the transfer of local and distributed data containers to and from storage systems can be difficult because there are so many different configurations of output formats, data containers and storage systems. Described herein is an extensible componentized data transfer framework for performant and scalable authoring of data loaders and data savers. Abstracted local and distributed workflows drive selection of plug-ins that can be composed by the framework into particular local or distributed scenario loaders and savers. Reusability and code sparsity are maximized.

Abstract: A platform that provides a way to automatically compose and execute even complex workflows without writing code is described. A set of pre-built functional building blocks can be provided. The building blocks perform data transformation and machine learning functions. The functional blocks have well known plug types. The building blocks can be composed build complex compositions. Input and output files are converted to a standard data type so that modules are pluggable.

Abstract: Implementing static loaders and savers for the transfer of local and distributed data containers to and from storage systems can be difficult because there are so many different configurations of output formats, data containers and storage systems. Described herein is an extensible componentized data transfer framework for performant and scalable authoring of data loaders and data savers. Abstracted local and distributed workflows drive selection of plug-ins that can be composed by the framework into particular local or distributed scenario loaders and savers. Reusability and code sparsity are maximized.

Abstract: Implementing static loaders and savers for the transfer of local and distributed data containers to and from storage systems can be difficult because there are so many different configurations of output formats, data containers and storage systems. Described herein is an extensible componentized data transfer framework for performant and scalable authoring of data loaders and data savers. Abstracted local and distributed workflows drive selection of plug-ins that can be composed by the framework into particular local or distributed scenario loaders and savers. Reusability and code sparsity are maximized.

Abstract: A platform that provides a way to automatically compose and execute even complex workflows without writing code is described. A set of pre-built functional building blocks can be provided. The building blocks perform data transformation and machine learning functions. The functional blocks have well known plug types. The building blocks can be composed build complex compositions. Input and output files are converted to a standard data type so that modules are pluggable.

Abstract: A platform that provides a way to automatically compose and execute even complex workflows without writing code is described. A set of pre-built functional building blocks can be provided. The building blocks perform data transformation and machine learning functions. The functional blocks have well known plug types. The building blocks can be composed build complex compositions. Input and output files are converted to a standard data type so that modules are pluggable.

Abstract: A platform that provides a way to automatically compose and execute even complex workflows without writing code is described. A set of pre-built functional building blocks can be provided. The building blocks perform data transformation and machine learning functions. The functional blocks have well known plug types. The building blocks can be composed build complex compositions. Input and output files are converted to a standard data type so that modules are pluggable.

Abstract: Implementing static loaders and savers for the transfer of local and distributed data containers to and from storage systems can be difficult because there are so many different configurations of output formats, data containers and storage systems. Described herein is an extensible componentized data transfer framework for performant and scalable authoring of data loaders and data savers. Abstracted local and distributed workflows drive selection of plug-ins that can be composed by the framework into particular local or distributed scenario loaders and savers. Reusability and code sparsity are maximized.

Abstract: In one embodiment, a process may be performed in parallel on a parallel server by defining a data type that may be used to reference data stored on the parallel server and overloading a previously-defined operation, such that when the overloaded operation is called, a command is sent to the parallel server to manipulate the data stored on the parallel server. In some embodiments, the previously-defined operation that is overloaded may be an operation of an operating system. Further, in some embodiments, when the data stored on the parallel server is no longer needed, a command may be sent to the parallel server to reallocate the memory used to store the data.

Abstract: One embodiment of the invention is directed to executing a computer program that manipulates data stored in a distributed manner on a parallel server. At least one portion of the computer program treats the data from a global perspective, wherein commands to the parallel server are not specific to the manner in which the data is distributed on the parallel server. In addition, at least one portion of the computer program treats the data from a local perspective, wherein commands to the parallel server are specific to the manner in which the data is distributed on the parallel server.