Abstract: Message passing is provided among a plurality of interdependent parallel processes using a shared memory. Inter-process communication among a plurality of interdependent processes executing on a plurality of compute nodes is performed by obtaining a message from a first process for a second process; and storing the message in a memory location of a Peripheral Component Interconnect Express (PCIE)-linked storage device, wherein the second process reads the memory location to obtain the message. The message is optionally persistently stored in the PCIE-linked storage device for an asynchronous checkpoint until the message is no longer required for an asynchronous restart.

Abstract: An apparatus in one embodiment comprises a multi-tier storage system and at least one dispatcher node. The multi-tier storage system comprises at least a front-end storage tier and a back-end storage tier. The at least one dispatcher node is configured to load portions of graph data from the front-end storage tier and the back-end storage tier for graph analytics computations performed at respective ones of a plurality of compute nodes configured for communication with the storage system over at least one network. The multi-tier storage system configured to switch logical connections of respective ones of the compute nodes to the at least one dispatcher node based on progress of graph analytics computations at the compute nodes.

Abstract: An apparatus in one embodiment comprises a multi-tier storage system having at least a front-end storage tier and a back-end storage tier. The storage system comprises a software-defined storage controller configured to implement the front-end storage tier as a unified memory cluster accessible to respective processors of a plurality of compute nodes. The software-defined storage controller is implemented using at least one processing device comprising a processor coupled to a memory. The unified memory cluster in some embodiments comprises portions of respective memories of at least a subset of the compute nodes. The portions of respective memories of at least a subset of the compute nodes may comprise, for example, respective flash memories of those compute nodes, or respective dynamic random access memories of those compute nodes. Other illustrative embodiments include systems, methods and processor-readable storage media.

Abstract: An apparatus in one embodiment comprises a plurality of compute nodes each having a processor coupled to a memory. The apparatus further comprises a storage system configured for communication with the compute nodes over at least one network, with the storage system comprising a software-defined storage controller configured to implement an in-memory database accessible as a unified memory cluster to respective processors of the compute nodes. The in-memory database comprises portions of respective memories of the compute nodes. The in-memory database may be expanded, for example, using an additional portion of the memory of one of the compute nodes, responsive to an overflow event associated with the memory of another one of the compute nodes. The compute nodes may illustratively comprise respective analysis nodes that run respective applications configured to utilize analytics functionality of the in-memory database.

Abstract: A multi-tier storage system is provided that employs a burst buffer middleware appliance for batch messaging. An exemplary multi-tier storage system comprises at least first and second storage tiers comprising storage for storing data; and at least one burst buffer appliance, wherein the at least one burst buffer appliance comprises an abstract storage interface comprising a software interface module that communicates with at least one persistent storage device in at least one of the first and second storage tiers, wherein the at least one persistent storage device provides a list application programming interface, wherein the software interface module provides a batch operation comprised of a plurality of individual operations to the at least one persistent storage device using an aggregated message. The batch operation optionally comprises a batch operation for processing metadata. A burst buffer appliance is also provided for implementing the batch messaging techniques described herein.

Abstract: Techniques are provided for content-aware storage tiering techniques within a job scheduling system. A data node in a job scheduling environment receives at least one job to be executed for a portion of a larger logical file, and notifies one or more sibling data nodes to prefetch one or more related portions of the larger logical file. A write operation performed by an application client to store data on the data node is optionally intercepted at the data node and the associated data is stored in a performance storage tier of the data node. The data is optionally moved from the performance storage tier to a capacity storage tier if one or more predefined eviction criteria are satisfied. Data objects to be evicted are selected based on an anticipated future access of the data objects. The eviction selection is optionally based on grey box knowledge of the application client and/or information obtained from the application client.

Abstract: Improved techniques are provided for storing bursty data, such as checkpoints, in parallel computing environments. A parallel file system is provided comprising at least first and second storage tiers comprising respective disjoint subsets of storage; and at least one processing device configured to store burst data from a plurality of distributed processes for a given burst operation during the given burst operation on both of the at least first and second storage tiers. The given burst operation comprises a multi-phase input/output (IO) task, such as a checkpoint, having alternating periods of idle time and bursts of write and/or storage activity. A method is also provided for provisioning a parallel file system by obtaining a specification of burst data requirements for the parallel file system and determining an amount of storage required for at least first and second storage tiers comprising respective disjoint subsets of storage to satisfy the burst data requirements.

Abstract: An information processing system comprises a plurality of front-end storage tiers, a plurality of back-end storage tiers, a plurality of data mover modules arranged between the front-end and back-end file storage tiers, and a metadata warehouse associated with the data mover modules and the front-end and back-end storage tiers. The data mover modules are configured to control movement of data between the storage tiers. The metadata warehouse is configured to store for each of a plurality of data items corresponding metadata comprising movement information characterizing movement of the data item between the storage tiers. The movement information for a given data item illustratively comprises locations, timestamps and checksums for different stages of movement of the given data item. Other types of metadata for the given data item illustratively include lineage information, access history information and compliance information.

Abstract: A key-value store is provided having one or more key-value storage interfaces. A key-value store on at least one compute node comprises a memory for storing a plurality of key-value pairs; and an abstract storage interface comprising a software interface module that communicates with at least one persistent storage device providing a key-value interface for persistent storage of one or more of the plurality of key-value pairs, wherein the software interface module provides the one or more key-value pairs to the at least one persistent storage device in a key-value format. The abstract storage interface optionally processes one or more batch operations on the plurality of key-value pairs. A distributed embodiment for a partitioned key-value store is also provided.

Abstract: Improved multi-tier data storage is provided using inclusive/exclusive burst buffer caching techniques based on reference counts. An exemplary multi-tier storage system comprises at least first and second storage tiers for storing data, wherein at least one of the first and second storage tiers comprises at least one cache, and wherein the data is retained in the at least one cache as a given cached data item based on a reference count indicating a number of expected requests for the given cached data item. The number of expected requests for the given cached data item in a given cache is based, for example, on a number of nodes serviced by the given cache. A burst buffer appliance is also provided for implementing the cache retention policies described herein.

Abstract: Cluster file systems are provided having a FUSE interface. A cluster file system comprises a FUSE object storage device that communicates with at least one object storage server of the cluster file system; and provides an interface between the cluster file system and at least one additional file system having a FUSE implementation. The additional file system optionally includes a burst buffer appliance. The burst buffer appliance can be configured, for example, to control the movement of data between first and second storage tiers. The burst buffer appliance can optionally communicate with a plurality of clients over the network and process requests from the clients. The FUSE object storage device provides an interface to, for example, a parallel log-structured file system and/or a Lustre clustered file system.

Abstract: A cluster file system is provided having a plurality of distributed metadata servers with shared access to one or more shared low latency persistent key-value metadata stores. A metadata server comprises an abstract storage interface comprising a software interface module that communicates with at least one shared persistent key-value metadata store providing a key-value interface for persistent storage of key-value metadata. The software interface module provides the key-value metadata to the at least one shared persistent key-value metadata store in a key-value format. The shared persistent key-value metadata store is accessed by a plurality of metadata servers. A metadata request can be processed by a given metadata server independently of other metadata servers in the cluster file system. A distributed metadata storage environment is also disclosed that comprises a plurality of metadata servers having an abstract storage interface to at least one shared persistent key-value metadata store.

Abstract: A sparse file is stored without holes by storing a data portion of the sparse file using a parallel log-structured file system; and generating an index entry for the data portion, the index entry comprising a logical offset, physical offset and length of the data portion. The holes can be restored to the sparse file upon a reading of the sparse file. The data portion can be stored at a logical end of the sparse file. Additional storage efficiency can optionally be achieved by (i) detecting a write pattern for a plurality of the data portions and generating a single patterned index entry for the plurality of the patterned data portions; and/or (ii) storing the patterned index entries for a plurality of the sparse files in a single directory, wherein each entry in the single directory comprises an identifier of a corresponding sparse file.

Abstract: Data from a group of distributed processes to a shared file is written using a parallel log-structured file system. A metadata server of a cluster file system is configured to communicate with a plurality of object storage servers of the cluster file system over a network. The metadata server further configured to implement a Parallel Log Structured File System (PLFS) library to coordinate storage on one or more of the plurality of object storage servers of a plurality of portions of a shared file generated by a plurality of applications executing on compute nodes of the cluster file system and to store metadata for the plurality of portions of the shared file. Concurrent writes to the shared file are decoupled by writing the plurality of portions of the shared file generated by each of the plurality of applications to independent write streams for each application.

Abstract: Techniques are provided for storing data and metadata on sharded storage arrays. In one embodiment, data is processed in a sharded distributed data storage system that stores data in a plurality of shards on one or more storage nodes by providing a plurality of addressable virtual shards within each of the shards, wherein at least a first one of the addressable virtual shards stores the data, and wherein at least a second one of the addressable virtual shards stores the metadata related to the data; obtaining the data from a compute node; and providing the data and the metadata related to the data stored to the sharded distributed data storage system for storage in the respective first and second addressable virtual shards. The metadata related to the data is stored together at a portion of a corresponding stripe for the data in the second one of the addressable virtual shards. A third one of the addressable virtual shards optionally stores a checksum value related to the data.

Abstract: A computer-executable method, system, and computer program product of managing I/O received within a Lustre file system, the computer-executable method, system, and computer program product comprising: receiving a data I/O request, wherein the data I/O request relates to data stored within the Lustre file system; processing the data I/O request in a journal stored on a fast data storage device within the Lustre file system; analyzing the journal to make a determination related to the data I/O request; and responding to the data I/O request.

Abstract: End-to-end data integrity is provided in parallel computing systems, such as High Performance Computing (HPC) environments. An exemplary method is provided for processing data in a distributed data storage system by obtaining the data and one or more corresponding checksum values from a compute node; and providing the data and the one or more corresponding checksum values to the distributed data storage system for storage. One or more checksum values corresponding to the data can be generated if the one or more checksum values are not received from a compute node. Exemplary processes are provided for copy; slice; merge: and slice and merge functions. The distributed data storage system comprises, for example, one or more Parallel Log-Structured File System (PLFS) storage elements and/or key-value storage elements storing one or more key-value pairs.

Abstract: Improved techniques are provided for storing metadata associated with a plurality of sub-files associated with a single shared file in a parallel file system. The shared file is generated by a plurality of applications executing on a plurality of compute nodes. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file generated by an application executing on the compute node and metadata for the at least one portion of the shared file on one or more object storage servers. The compute node is also configured to implement a partitioned data store for storing a partition of the metadata for the shared file, wherein the partitioned data store communicates with partitioned data stores on other compute nodes using a message passing interface. The partitioned data store can be implemented, for example, using Multidimensional Data Hashing Indexing Middleware (MDHIM).

Abstract: A computer-executable method, computer program product, and system for managing I/Os from an Application on a host, wherein the host is in communication with a data storage system including one or more burst buffer servers, the computer-executable method, computer program product, and system comprising providing a portion of shared memory using the one or more burst buffer servers of the data storage system, wherein the portion of shared memory is enabled to be visible from each of the one or more burst buffer servers, receiving, at a first burst buffer server of the one or more burst buffer servers, a data I/O request from the application on the host, wherein the data I/O request is associated with data on the portion of shared memory, determining whether data associated with the data I/O request is located on the first buffer server, and processing the data I/O request based on the determination.

Abstract: A computer-executable method, system, and computer program product for managing meta-data in a distributed storage system, wherein the distributed storage system includes one or more burst buffers enabled to operate with a distributed key-value store, the co computer-executable method, system, and computer program product comprising receiving a request for meta-data associated with a block of data stored in a first burst buffer of the one or more burst buffers in the distributed storage system, wherein the meta data is associated with a key-value, determining which of the one or more burst buffers stores the requested metadata, and upon determination that a first burst buffer of the one or more burst buffers stores the requested metadata, locating the key-value in a portion of the distributed key-value store accessible from the first burst buffer.