Abstract: A multi-tier storage system is provided with direct client access to an archive storage tier for input/output operations. An exemplary method comprises communicating over a network with (i) a cluster file system on a first storage tier, and (ii) a second archive storage tier comprising an object store; providing a client of the cluster file system with access to one or more files in the cluster file system on the first storage tier; and executing a translation shim to provide the client of the cluster file system with one or more of read and write access to one or more files on the second archive storage tier. The translation shim converts between the protocols of the cluster file system and the protocols of the second archive storage tier, to allow unmodified applications to optionally access the second archive storage tier using existing cluster file system protocols.

Abstract: Example embodiments of the present invention relate to a method, a system, and a computer program product for maintaining dynamically loadable namespaces. The method includes presenting, to an application, a file system having a global single namespace (SN) comprising a plurality of dynamically-loadable namespaces (DLNs) maintained in a capacity tier (CT) of object storage of a storage system. In response to a job submitted by the application, dynamically loading a first dynamically-loadable namespace, including the metadata only for the first dynamically-loadable namespace, associated with the job from the capacity tier of object storage into a fast tier (FT) of storage of the storage system.

Abstract: An apparatus in one embodiment comprises a storage system having at least first and second storage tiers each comprising a plurality of storage devices. The storage system implements a parallel file system comprising at least first and second sets of storage servers. The first set of storage servers is configured to access directories of the parallel file system that are stored in storage devices of the first storage tier, and the second set of storage servers is configured to access directories of the parallel file system that are stored in storage devices of the second storage tier. The storage system is further configured to move data between the first and second storage tiers by moving data files between different directories of the respective first and second storage tiers. One or more servers of the first set may be generated as respective new server instances on one or more compute nodes.

Abstract: A computer-executable method, system, and computer program product for managing a cloud computing resource in communication with a data storage system, the computer-executable method, system, and computer program product comprising receiving configuration information at the data storage system related to a host application; determining an optimized configuration for the host application; and creating an implementation of the optimized configuration based on one or more resources enabled to be accessed by the data storage system.

Abstract: A storage system comprises a shared storage environment that includes a storage array having at least one storage volume shared between first and second host devices. The storage system further comprises a server associated with the storage array, at least first and second clients associated with the respective first and second host devices, and a first block cache arranged between the first client and the storage array. The server is configured to coordinate operations of the first and second clients relating to the storage volume shared between the first and second host devices in a manner that ensures coherency of data stored in the first block cache. The server may comprise a storage block mapping protocol (SBMP) server and the first and second clients may comprise respective SBMP clients. The block cache is illustratively implemented using a VFCache or other type of server flash cache.

Abstract: Techniques are provided for scheduling computational tasks among multiple classes of storage resources based on a job classification. A job to be executed is classified into one of a plurality of predefined job classes. Each predefined job class is associated with a corresponding one of a plurality of predefined storage classes. The job is then assigned based on the classification to one of the storage resources of the predefined storage class associated with the classified predefined job class. Exemplary predefined storage classes include a performance class, a capacity class, a key-value storage class, and a shingled disk drive class. Exemplary predefined job classes include a CPU Intensive job class, an IO Intensive job class and a Small IO job class. Data required for a job is optionally prefetched before the job is assigned to a storage device. Data objects to be evicted from a storage device are optionally selected based on an anticipated future access.

Abstract: Techniques are provided for encryption and decryption of time series data using a digital filter array. A plurality of digital time series samples generated by a digital time series generation device are applied to a digital filter array (e.g., a finite impulse response filter or an infinite impulse response filter) that combines delayed versions of the digital time series samples using a plurality of coefficients to generate a plurality of encrypted digital time series samples. The plurality of coefficients comprise a portion of an encryption key associated with the digital time series generation device. The digital time series generation device has an associated device identifier, and the device identifier and the corresponding encryption key are registered with a server. The plurality of encrypted digital time series samples and the device identifier are provided to the server for decryption using an inverse digital filter array based on the encryption key.

Abstract: An apparatus in one embodiment comprises a storage system configured to implement at least one scalable video server. The storage system comprises a software-defined storage pool, and the scalable video server comprises a plurality of file system storage nodes each including a corresponding portion of the software-defined storage pool and an associated file system server. A streaming bandwidth of the scalable video server for a given video stream is controlled by adjusting the number of file system storage nodes utilized for the given video stream in the scalable video server. The file system servers of the respective file system storage nodes are configured to interact with a file system client associated with the given video stream. The streaming bandwidth of the scalable video server for the given video stream may be dynamically adjusted by adding or deleting file system storage nodes to or from the scalable video server.

Abstract: An apparatus in one embodiment comprises a reconfigurable multi-tier storage system having at least a front-end storage tier and a back-end storage tier. The multi-tier storage system is implemented at least in part utilizing a plurality of replicated storage units. One or more of the replicated storage units are each utilized in a first configuration of the reconfigurable multi-tier storage system to implement a portion of at least one of a burst buffer appliance and a software-defined storage pool of the front-end storage tier. At least one of the one or more replicated storage units is utilized in a second configuration of the reconfigurable multi-tier storage system to implement a portion of at least one of a scale-out network-attached storage cluster and an object store of the back-end storage tier. A given one of the replicated storage units may be reconfigured by rebooting it using different software modules.

Abstract: An apparatus in one embodiment comprises a multi-tier storage system having at least a front-end storage tier and a back-end storage tier. The multi-tier storage system is implemented at least in part utilizing a plurality of virtual machines of cloud infrastructure. The front-end storage tier comprises a plurality of storage nodes of a cluster file system, with the storage nodes being implemented on respective ones of the virtual machines. The front-end storage tier further comprises a software-defined storage pool accessible to the storage nodes and implemented utilizing local disk resources of respective ones of the virtual machines. The back-end storage tier of the multi-tier storage system comprises at least one object store. At least a subset of the virtual machines may further comprise respective compute nodes configured to access the multi-tier storage system. Other illustrative embodiments include systems, methods and processor-readable storage media.

Abstract: Metadata associated with a plurality of sub-files associated with a single shared file is stored in a parallel file system. A plurality of processes generate a shared file. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file and metadata for the at least one portion of the shared file on one or more of the plurality of object storage servers. The compute node is further configured to store the metadata by striping the metadata across a plurality of subdirectories of the shared file. The metadata is optionally striped across the plurality of subdirectories in a round-robin manner. The plurality of subdirectories are stored on one or more of the object storage servers. Write and read processes optionally communicate using a message passing interface. A given write process optionally writes metadata for a given portion of the shared file to an index file in a particular one of the subdirectories corresponding to the given portion.

Abstract: An apparatus in one embodiment comprises a storage system having at least first and second storage tiers each comprising a plurality of storage devices. The storage system implements at least one configurable data mover module adapted to interact with the first and second storage tiers and comprising at least one power management configuration file. The data mover module alters a manner in which data is moved between the first and second storage tiers based at least in part on a change in state of the power management configuration file. For example, the data mover module may be configurable via the power management configuration file in a mode of operation in which data movement from the first storage tier to the second storage tier occurs for data files utilized by a given application only after completion of a compute phase for that application.

Abstract: An apparatus comprises a multi-tier storage system having a front-end storage tier, a back-end storage tier and a data mover module coupled to the front-end and back-end storage tiers and configured to control movement of data between the storage tiers. A translator at least partially incorporated in or otherwise associated with the multi-tier storage system implements a plurality of translation services and one or more index tables. A given one of the translation services is configured to access at least one of the index tables in order to determine a dynamically loadable namespace identifier associated with an application running on an analysis node that communicates with the multi-tier storage system over at least one network, and to instantiate a name node corresponding to the dynamically loadable namespace identifier. Particular objects of an object store of the back-end storage tier are made accessible to the application via the name node.

Abstract: A multi-tier storage system is provided that employs a burst buffer middleware appliance for batch messaging. An exemplary multi-tier storage system comprises at least first and second storage tiers comprising storage for storing data; and at least one burst buffer appliance, wherein the at least one burst buffer appliance comprises an abstract storage interface comprising a software interface module that communicates with at least one persistent storage device in at least one of the first and second storage tiers, wherein the at least one persistent storage device provides a list application programming interface, wherein the software interface module provides a batch operation comprised of a plurality of individual operations to the at least one persistent storage device using an aggregated message. The batch operation optionally comprises a batch operation for processing metadata. A burst buffer appliance is also provided for implementing the batch messaging techniques described herein.

Abstract: Techniques are provided for content-aware storage tiering techniques within a job scheduling system. A data node in a job scheduling environment receives at least one job to be executed for a portion of a larger logical file, and notifies one or more sibling data nodes to prefetch one or more related portions of the larger logical file. A write operation performed by an application client to store data on the data node is optionally intercepted at the data node and the associated data is stored in a performance storage tier of the data node. The data is optionally moved from the performance storage tier to a capacity storage tier if one or more predefined eviction criteria are satisfied. Data objects to be evicted are selected based on an anticipated future access of the data objects. The eviction selection is optionally based on grey box knowledge of the application client and/or information obtained from the application client.

Abstract: Improved techniques are provided for storing bursty data, such as checkpoints, in parallel computing environments. A parallel file system is provided comprising at least first and second storage tiers comprising respective disjoint subsets of storage; and at least one processing device configured to store burst data from a plurality of distributed processes for a given burst operation during the given burst operation on both of the at least first and second storage tiers. The given burst operation comprises a multi-phase input/output (IO) task, such as a checkpoint, having alternating periods of idle time and bursts of write and/or storage activity. A method is also provided for provisioning a parallel file system by obtaining a specification of burst data requirements for the parallel file system and determining an amount of storage required for at least first and second storage tiers comprising respective disjoint subsets of storage to satisfy the burst data requirements.

Abstract: A key-value store is provided having one or more key-value storage interfaces. A key-value store on at least one compute node comprises a memory for storing a plurality of key-value pairs; and an abstract storage interface comprising a software interface module that communicates with at least one persistent storage device providing a key-value interface for persistent storage of one or more of the plurality of key-value pairs, wherein the software interface module provides the one or more key-value pairs to the at least one persistent storage device in a key-value format. The abstract storage interface optionally processes one or more batch operations on the plurality of key-value pairs. A distributed embodiment for a partitioned key-value store is also provided.

Abstract: A cluster file system is provided having a plurality of distributed metadata servers with shared access to one or more shared low latency persistent key-value metadata stores. A metadata server comprises an abstract storage interface comprising a software interface module that communicates with at least one shared persistent key-value metadata store providing a key-value interface for persistent storage of key-value metadata. The software interface module provides the key-value metadata to the at least one shared persistent key-value metadata store in a key-value format. The shared persistent key-value metadata store is accessed by a plurality of metadata servers. A metadata request can be processed by a given metadata server independently of other metadata servers in the cluster file system. A distributed metadata storage environment is also disclosed that comprises a plurality of metadata servers having an abstract storage interface to at least one shared persistent key-value metadata store.

Abstract: A storage system comprises a shared storage environment that includes a storage array having at least one storage volume shared between first and second host devices. The storage system further comprises a server associated with the storage array, at least first and second clients associated with the respective first and second host devices, and a first block cache arranged between the first client and the storage array. The server is configured to coordinate operations of the first and second clients relating to the storage volume shared between the first and second host devices in a manner that ensures coherency of data stored in the first block cache. The server may comprise a storage block mapping protocol (SBMP) server and the first and second clients may comprise respective SBMP clients. The block cache is illustratively implemented using a VFCache or other type of server flash cache.

Abstract: For simulating a parallel supercomputing cluster, a simulation program includes a model of the system of compute nodes and a model of an I/O subsystem that stores checkpoints from the compute nodes. When executed, instructions in the simulation program perform the steps of receiving input parameters defining the compute nodes and the I/O subsystem, computing a total number of computational flops for a time between checkpoints and an amount of disk storage required to store the checkpoint data for different configurations of the parallel supercomputing cluster, and presenting a summary to a user of the computed number of computational flops for a time between checkpoints and an amount of disk storage required to store the checkpoint data for the different configurations of the parallel supercomputing cluster.