Abstract: A virtual file system is arranged between a physical file system and an analytics platform comprising a plurality of compute nodes, and is configured to control movement of data between the analytics platform and the physical file system. For example, the virtual file system may be configured to present to the analytics platform a software application programming interface (API) through which the analytics platform accesses data stored in the physical file system. The analytics platform illustratively comprises a unified analytics platform that is configurable to support both database analytics and map-reduce analytics. In some embodiments, the physical file system comprises one or more of a Lustre file system, a Hadoop Distributed File System (HDFS) and a General Parallel File System (GPFS), and the virtual file system comprises a parallel log-structured file system (PLFS).

Abstract: Techniques are provided for storing files in a parallel computing system based on a user-specification. A plurality of files generated by a distributed application in a parallel computing system are stored by obtaining a specification from the distributed application indicating how the plurality of files should be stored; and storing one or more of the plurality of files in one or more storage nodes of a multi-tier storage system based on the specification. The plurality of files comprise a plurality of complete files and/or a plurality of sub-files. The specification can optionally be processed by a daemon executing on one or more nodes in a multi-tier storage system. The specification indicates how the plurality of files should be stored, for example, identifying one or more storage nodes where the plurality of files should be stored.

Abstract: A cluster file system comprises a plurality of clients and a plurality of servers configured to communicate over a network. A given server of the cluster file system interfaces with an associated storage array, and comprises a block cache and a request translator. The request translator is configured to translate input-output requests in a protocol of the cluster file system to a format suitable for use with the block cache. The cluster file system may more particularly comprise a Lustre file system, and the given server may comprise a metadata server or an object storage server of the Lustre file system. In such an embodiment, the request translator is more particularly configured to translate Lustre client input-output requests directed to the storage array into block input-output requests that are processed by the block cache. The block cache is illustratively implemented using a VFCache or other type of server flash cache.

Abstract: Improved sorting techniques are provided that perform a parallel sort using a ranged, partitioned key-value store in a high performance computing (HPC) environment. A plurality of input data files comprising unsorted key-value data in a partitioned key-value store are sorted. The partitioned key-value store comprises a range server for each of a plurality of ranges. Each input data file has an associated reader thread. Each reader thread reads the unsorted key-value data in the corresponding input data file and performs a local sort of the unsorted key-value data to generate sorted key-value data. A plurality of sorted, ranged subsets of each of the sorted key-value data are generated based on the plurality of ranges. Each sorted, ranged subset corresponds to a given one of the ranges and is provided to one of the range servers corresponding to the range of the sorted, ranged subset. Each range server sorts the received sorted, ranged subsets and provides a sorted range.

Abstract: Improved techniques are provided for asynchronous checkpointing in parallel computing environments. A burst buffer appliance is configured to communicate with a plurality of compute nodes of a parallel computing system over a network and also to store message logs for a plurality of processes executing on the compute nodes, wherein the plurality of processes employ asynchronous checkpointing. The processes executing on the compute nodes can exchange messages and/or perform other compute operations during an asynchronous checkpointing operation. The burst buffer appliance can optionally store checkpoint data that results from the asynchronous checkpointing operations. The burst buffer appliance can optionally store the messages using a partitioned data store, such as Multidimensional Data Hashing Indexing Middleware.

Abstract: Parallel de-duplication of data chunks being written to a shared object is provided. A client executing on a compute node or a burst buffer node in a parallel computing system stores a data chunk to a shared data object on a storage node by processing the data chunk to obtain a de-duplication fingerprint; comparing the de-duplication fingerprint to de-duplication fingerprints of other data chunks; and providing original data chunks to the storage node that stores the shared object. A reference to an original data chunk can be stored when the de-duplication fingerprint matches another data chunk. The client and storage node may employ Log-Structured File techniques. A storage node stores a data chunk in the shared object by receiving only an original version of the data chunk from a compute node; and storing the original version of the data chunk to the shared data object on the storage node as a shared object.

Abstract: A cluster file system includes a physical file system and a virtual file system arranged between the physical file system and multiple clients. The virtual file system comprises a data object processing module configured to receive multiple data files from each of one or more of the clients and to transform the data files into data objects for object-based storage in the physical file system. The virtual file system further comprises a metadata object processing module configured to receive metadata files associated with the data files and to transform the metadata files into metadata objects for object-based storage in the physical file system. The virtual file system thereby implements multiple object stores utilizing the physical file system for independent storage of data objects and metadata objects but presents a file-based interface to the clients. In some embodiments, the virtual file system illustratively comprises a parallel log-structured file system (PLFS).

Abstract: Improved techniques are provided for parallel writing of data to a shared object in a parallel computing system. A method is provided for storing data generated by a plurality of parallel processes to a shared object in a parallel computing system. The method is performed by at least one of the processes and comprises: dynamically determining a block size for storing the data; exchanging a determined amount of the data with at least one additional process to achieve a block of the data having the dynamically determined block size; and writing the block of the data having the dynamically determined block size to a file system. The determined block size comprises, e.g., a total amount of the data to be stored divided by the number of parallel processes. The file system comprises, for example, a log structured virtual parallel file system, such as a Parallel Log-Structured File System (PLFS).

Abstract: A cluster file system comprises storage server units each configured for communication with a plurality of clients over a network. At least one of the storage server units comprises an object storage server, an object storage target associated with the object storage server, a metadata server, a metadata target associated with the metadata server, and a scale-out network attached storage cluster. The scale-out network attached storage cluster comprises storage directories corresponding to respective ones of the object storage and metadata targets. The object storage server and its associated object storage target may form part of a first storage tier of the storage server unit, and a plurality of nodes of the scale-out network attached storage cluster may form part of a second storage tier of the storage server unit. Parallel log-structured file system (PLFS) middleware may be used to control movement of data between the first and second storage tiers.

Abstract: Techniques are provided for storing files in a parallel computing system using different resolutions. A method is provided for storing at least one file generated by a distributed application in a parallel computing system. The file comprises one or more of a complete file and a sub-file. The method comprises the steps of obtaining semantic information related to the file; generating a plurality of replicas of the file with different resolutions based on the semantic information; and storing the file and the plurality of replicas of the file in one or more storage nodes of the parallel computing system. The different resolutions comprise, for example, a variable number of bits and/or a different sub-set of data elements from the file. A plurality of the sub-files can be merged to reproduce the file.

Abstract: Interactive requests are processed from users of log-in nodes. A metadata server node is provided for use in a file system shared by one or more interactive nodes and one or more batch nodes. The interactive nodes comprise interactive clients to execute interactive tasks and the batch nodes execute batch jobs for one or more batch clients. The metadata server node comprises a virtual machine monitor; an interactive client proxy to store metadata requests from the interactive clients in an interactive client queue; a batch client proxy to store metadata requests from the batch clients in a batch client queue; and a metadata server to store the metadata requests from the interactive client queue and the batch client queue in a metadata queue based on an allocation of resources by the virtual machine monitor. The metadata requests can be prioritized, for example, based on one or more of a predefined policy and predefined rules.

Abstract: A computer-executable method, system, and computer program product for managing I/O requests from a compute node in communication with a data storage system, including a first burst buffer node and a second burst buffer node, the computer-executable method, system, and computer program product comprising striping data on the first burst buffer node and the second burst buffer node, wherein a first portion of the data is communicated to the first burst buffer node and a second portion of the data is communicated to the second burst buffer node, processing the first portion of the data at the first burst buffer node, and processing the second portion of the data at the second burst buffer node.

Abstract: Improved techniques are provided for storing files in a parallel computing system using a list-based index to identify file replicas. A file and at least one replica of the file are stored in one or more storage nodes of the parallel computing system. An index for the file comprises at least one list comprising a pointer to a storage location of the file and a storage location of the at least one replica of the file. The file comprises one or more of a complete file and one or more sub-files. The index may also comprise a checksum value for one or more of the file and the replica(s) of the file. The checksum value can be evaluated to validate the file and/or the file replica(s). A query can be processed using the list.

Abstract: A burst buffer appliance is adapted for coupling between a computer system and a file system. The burst buffer appliance comprises a flash memory or other high-speed memory having a substantially lower access time than the file system, and is configured to include a plurality of virtual machines for processing respective different types of input-output operations that involve utilization of the high-speed memory, with each of the virtual machines providing a different performance level for its associated type of input-output operations. The performance levels provided by the plurality of virtual machines may comprise respective different quality of service (QoS) levels for the respective different types of input-output operations, specified in terms of parameters such as latency and throughput rate. A highest QoS level may be provided by a particular virtual machine for operations involving writing checkpoints from the computer system to the high-speed memory.

Abstract: Cloud object storage is enabled for archived data, such as checkpoints and results, of high performance computing applications using a middleware process. A plurality of archived files, such as checkpoint files and results, generated by a plurality of processes in a parallel computing system are stored by obtaining the plurality of archived files from the parallel computing system; converting the plurality of archived files to objects using a log structured file system middleware process; and providing the objects for storage in a cloud object storage system. The plurality of processes may run, for example, on a plurality of compute nodes. The log structured file system middleware process may be embodied, for example, as a Parallel Log-Structured File System (PLFS). The log structured file system middleware process optionally executes on a burst buffer node.

Abstract: A burst buffer appliance is adapted for coupling between a computer system and a file system. The burst buffer appliance comprises a flash memory or other high-speed memory having a substantially lower access time than the file system, and an analytics engine, and is configured to aggregate multiple data files having a size below a designated threshold into a container having at least one associated metadata file for processing by the analytics engine. The burst buffer appliance directs storage of the container in the file system. The analytics engine is configured to generate at least a content-based portion of the one or more metadata files of the container based at least in part on content of one or more of the multiple data files so as to permit one or more subsequent processing operations to be performed using the metadata file(s) rather than the corresponding multiple data files.

Abstract: Techniques are provided for storing files in a parallel computing system using sub-files with semantically meaningful boundaries. A method is provided for storing at least one file generated by a distributed application in a parallel computing system. The file comprises one or more of a complete file and a plurality of sub-files. The method comprises the steps of obtaining a user specification of semantic information related to the file; providing the semantic information as a data structure description to a data formatting library write function; and storing the semantic information related to the file with one or more of the sub-files in one or more storage nodes of the parallel computing system. The semantic information provides a description of data in the file. The sub-files can be replicated based on semantically meaningful boundaries.

Abstract: Techniques are provided for storing files in a parallel computing system based on a user-specified parser function. A plurality of files generated by a distributed application in a parallel computing system are stored by obtaining a parser from the distributed application for processing the plurality of files prior to storage; and storing one or more of the plurality of files in one or more storage nodes of the parallel computing system based on the processing by the parser. The plurality of files comprise one or more of a plurality of complete files and a plurality of sub-files. The parser can optionally store only those files that satisfy one or more semantic requirements of the parser. The parser can also extract metadata from one or more of the files and the extracted metadata can be stored with one or more of the plurality of files and used for searching for files.

Abstract: Techniques are provided for small file aggregation in a parallel computing system. An exemplary method for storing a plurality of files generated by a plurality of processes in a parallel computing system comprises aggregating the plurality of files into a single aggregated file; and generating metadata for the single aggregated file. The metadata comprises an offset and a length of each of the plurality of files in the single aggregated file. The metadata can be used to unpack one or more of the files from the single aggregated file.