Abstract: Cloud services intelligently provision new VMs for a VM scale set when the original label that included the OS or other software images used to provision existing VMs of the VM scale set is no longer available or has been changed. Metadata of the existing VMs are analyzed to identify an OS image or other software image used to provision the existing VMs. The metadata also reveals updates that are running on the new VMs. These updates include software that was not part of the original label used to provision the existing VMs and are used to find the second label in a label database. The second labels include the OS or software of the existing VMs—or a later version thereof—and some portion of the updates. A VM-provisioning service uses the second label to provision the new VM.

Abstract: A cluster file system comprises a burst buffer appliance coupled to a plurality of object storage servers via a network. The burst buffer appliance comprises a controller configured to implement storage tiering control functionality for at least first and second storage tiers comprising respective disjoint subsets of the plurality of object storage servers. The burst buffer appliance is thereby configured to control movement of data between the first and second storage tiers. The object storage servers in the first storage tier may be configured to interface with object storage targets of a first type and the object storage servers in the second storage tier may be configured to interface with object storage targets of a second type different than the first type. For example, the object storage targets of the first type may comprise non-volatile electronic storage devices such as flash storage devices, and the object storage targets of the second type may comprise disk storage devices.

Abstract: A set of data chunks stored in a first data storage system is accessed. The set of data chunks includes original data chunks and replicated data chunks respectively corresponding to the original data chunks. A given original data chunk and the corresponding replicated data chunk are stored in separate storage nodes of the first data storage system. For each of at least a subset of storage nodes of the first data storage system, unique ones of the original data chunks and the replicated data chunks stored on the storage node are aggregated to form a data object. The data objects thereby formed collectively represent a given data volume. Each of the data objects is stored in separate storage nodes of a second data storage system.

Abstract: Metadata associated with a plurality of sub-files associated with a single shared file is stored in a parallel file system. A plurality of processes generate a shared file. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file and metadata for the at least one portion of the shared file on one or more of the plurality of object storage servers. The compute node is further configured to store the metadata by striping the metadata across a plurality of subdirectories of the shared file. The metadata is optionally striped across the plurality of subdirectories in a round-robin manner. The plurality of subdirectories are stored on one or more of the object storage servers. Write and read processes optionally communicate using a message passing interface. A given write process optionally writes metadata for a given portion of the shared file to an index file in a particular one of the subdirectories corresponding to the given portion.

Abstract: An apparatus comprises a multi-tier storage system having a front-end storage tier, a back-end storage tier and a data mover module coupled to the front-end and back-end storage tiers and configured to control movement of data between the storage tiers. A translator at least partially incorporated in or otherwise associated with the multi-tier storage system implements a plurality of translation services and one or more index tables. A given one of the translation services is configured to access at least one of the index tables in order to determine a dynamically loadable namespace identifier associated with an application running on an analysis node that communicates with the multi-tier storage system over at least one network, and to instantiate a name node corresponding to the dynamically loadable namespace identifier. Particular objects of an object store of the back-end storage tier are made accessible to the application via the name node.

Abstract: An apparatus in one embodiment comprises a multi-tier storage system having at least a front-end storage tier and a back-end storage tier. The storage system comprises a software-defined storage controller configured to implement the front-end storage tier as a unified memory cluster accessible to respective processors of a plurality of compute nodes. The software-defined storage controller is implemented using at least one processing device comprising a processor coupled to a memory. The unified memory cluster in some embodiments comprises portions of respective memories of at least a subset of the compute nodes. The portions of respective memories of at least a subset of the compute nodes may comprise, for example, respective flash memories of those compute nodes, or respective dynamic random access memories of those compute nodes. Other illustrative embodiments include systems, methods and processor-readable storage media.

Abstract: A multi-tier storage system is provided that employs a burst buffer middleware appliance for batch messaging. An exemplary multi-tier storage system comprises at least first and second storage tiers comprising storage for storing data; and at least one burst buffer appliance, wherein the at least one burst buffer appliance comprises an abstract storage interface comprising a software interface module that communicates with at least one persistent storage device in at least one of the first and second storage tiers, wherein the at least one persistent storage device provides a list application programming interface, wherein the software interface module provides a batch operation comprised of a plurality of individual operations to the at least one persistent storage device using an aggregated message. The batch operation optionally comprises a batch operation for processing metadata. A burst buffer appliance is also provided for implementing the batch messaging techniques described herein.

Abstract: Techniques are provided for content-aware storage tiering techniques within a job scheduling system. A data node in a job scheduling environment receives at least one job to be executed for a portion of a larger logical file, and notifies one or more sibling data nodes to prefetch one or more related portions of the larger logical file. A write operation performed by an application client to store data on the data node is optionally intercepted at the data node and the associated data is stored in a performance storage tier of the data node. The data is optionally moved from the performance storage tier to a capacity storage tier if one or more predefined eviction criteria are satisfied. Data objects to be evicted are selected based on an anticipated future access of the data objects. The eviction selection is optionally based on grey box knowledge of the application client and/or information obtained from the application client.

Abstract: Improved techniques are provided for storing bursty data, such as checkpoints, in parallel computing environments. A parallel file system is provided comprising at least first and second storage tiers comprising respective disjoint subsets of storage; and at least one processing device configured to store burst data from a plurality of distributed processes for a given burst operation during the given burst operation on both of the at least first and second storage tiers. The given burst operation comprises a multi-phase input/output (IO) task, such as a checkpoint, having alternating periods of idle time and bursts of write and/or storage activity. A method is also provided for provisioning a parallel file system by obtaining a specification of burst data requirements for the parallel file system and determining an amount of storage required for at least first and second storage tiers comprising respective disjoint subsets of storage to satisfy the burst data requirements.

Abstract: A key-value store is provided having one or more key-value storage interfaces. A key-value store on at least one compute node comprises a memory for storing a plurality of key-value pairs; and an abstract storage interface comprising a software interface module that communicates with at least one persistent storage device providing a key-value interface for persistent storage of one or more of the plurality of key-value pairs, wherein the software interface module provides the one or more key-value pairs to the at least one persistent storage device in a key-value format. The abstract storage interface optionally processes one or more batch operations on the plurality of key-value pairs. A distributed embodiment for a partitioned key-value store is also provided.

Abstract: Cluster file systems are provided having a FUSE interface. A cluster file system comprises a FUSE object storage device that communicates with at least one object storage server of the cluster file system; and provides an interface between the cluster file system and at least one additional file system having a FUSE implementation. The additional file system optionally includes a burst buffer appliance. The burst buffer appliance can be configured, for example, to control the movement of data between first and second storage tiers. The burst buffer appliance can optionally communicate with a plurality of clients over the network and process requests from the clients. The FUSE object storage device provides an interface to, for example, a parallel log-structured file system and/or a Lustre clustered file system.

Abstract: Data from a group of distributed processes to a shared file is written using a parallel log-structured file system. A metadata server of a cluster file system is configured to communicate with a plurality of object storage servers of the cluster file system over a network. The metadata server further configured to implement a Parallel Log Structured File System (PLFS) library to coordinate storage on one or more of the plurality of object storage servers of a plurality of portions of a shared file generated by a plurality of applications executing on compute nodes of the cluster file system and to store metadata for the plurality of portions of the shared file. Concurrent writes to the shared file are decoupled by writing the plurality of portions of the shared file generated by each of the plurality of applications to independent write streams for each application.

Abstract: A cluster file system comprises a metadata server and a plurality of object storage servers each having one or more object storage targets. The object storage servers and their corresponding object storage targets are arranged in a first storage tier of the cluster file system. The cluster file system further comprises a second storage tier that includes a plurality of nodes of a scale-out network attached storage cluster. The cluster file system is configured to control movement of data between the first and second storage tiers. For example, in some embodiments, the object storage servers comprise one or more components of parallel log-structured file system (PLFS) middleware configured to control migration of data between the first and second storage tiers.

Abstract: A cluster file system comprises a burst buffer appliance coupled to a plurality of object storage servers via a network. The burst buffer appliance is configured to implement storage tiering control functionality based on a user specification for at least first and second storage tiers comprising respective disjoint subsets of the plurality of object storage servers. The user specification comprises, for example, a user request to move data from a first storage tier to a second storage tier; for an indication of where a given file is stored; for an indication of a given storage tier where a given file is stored and/or for an indication of an object storage server storing one or more given file portions.

Abstract: A gateway of a cluster file system comprises a plurality of object storage servers each having one or more object storage targets implemented using storage devices of a first type. The gateway is configured to communicate with a network file system cluster implemented using storage devices of a second type different than the first type. The network file system cluster comprises storage directories corresponding to respective ones of the object storage targets. A controller that may be implemented at least in part within the gateway is configured to control movement of data between the object storage targets and the storage directories of the network file system cluster. The movement of data between the object storage targets and the storage directories may be controlled at least in part based on one or more of policy information provided by a policy engine and information provided by an application running on a client.

Abstract: Data migration techniques are provided using parallel log-structured file system middleware to overcome file system limitations. A data file is migrated from a first storage tier to an archive storage tier, where the data file exceeds performance and/or capacity limitations of the archival storage tier, by writing the data file to a plurality of sub-files using a parallel data mover process, wherein each of the plurality of sub-files satisfy the performance and capacity limitations of the archival storage tier; and storing the plurality of sub-files to the archival storage tier using a log-structured file system. The log-structured file system optionally comprises a Parallel Log-Structured File System (PLFS) and the archival storage tier optionally comprises a multi disk storage tier. The parallel data mover process divides a size of the data file by a file size limit of the archival storage tier to determine of a number of the plurality of sub-files.

Abstract: An apparatus comprises a cluster file system, a burst buffer appliance coupled to the cluster file system, a high-speed memory pool configured for allocation between the cluster file system and the burst buffer appliance, and a controller associated with the high-speed memory pool and configured to dynamically adapt portions thereof between utilization as object storage servers of the cluster file system and utilization as burst buffers of the burst buffer appliance. By way of example, the high-speed memory pool comprises a plurality of flash servers that are reconfigurable between utilization as respective ones of the object storage servers of the cluster file system and utilization as respective ones of the burst buffers of the burst buffer appliance. The controller is configured to dynamically adapt the flash servers between utilization as respective object storage servers of the cluster file system and utilization as respective burst buffers of the burst buffer appliance.

Abstract: A cluster file system comprises a burst buffer appliance coupled to a plurality of object storage servers via a network. The burst buffer appliance is configured to implement storage tiering control functionality for at least first and second storage tiers comprising respective disjoint subsets of the plurality of object storage servers. The burst buffer appliance implements a coordinated movement of data between the first and second storage tiers to pre-fetch at least one additional portion of a single logical file that is stored across a plurality of said object storage devices from another of said plurality of object storage devices. The burst buffer appliance optionally implements the coordinated movement of data between the first and second storage tiers such that substantially all portions of a single logical file that are stored across a plurality of said object storage devices in said cluster file system are stored in only one of said storage tiers at a given time.

Abstract: A cluster file system comprises storage server units each configured for communication with a plurality of clients over a network. At least one of the storage server units comprises an object storage server, an object storage target associated with the object storage server, a metadata server, a metadata target associated with the metadata server, and a scale-out network attached storage cluster. The scale-out network attached storage cluster comprises storage directories corresponding to respective ones of the object storage and metadata targets. The object storage server and its associated object storage target may form part of a first storage tier of the storage server unit, and a plurality of nodes of the scale-out network attached storage cluster may form part of a second storage tier of the storage server unit. Parallel log-structured file system (PLFS) middleware may be used to control movement of data between the first and second storage tiers.

Abstract: Extensions to the Fragment Mapping Protocol are introduced which protect a disk array from malicious client access by exporting file system access information to the storage device. FMP requests received at the storage device can be authorized at a block granularity prior to completion, thereby limiting the exposure of the disk array to malicious clients. Client authorizations can be cached at the storage device to enable the permissions to be quickly extracted for subsequent client accesses to pre-authorized volumes.