Abstract: Improved multi-tier storage techniques are provided for storing data, such as checkpoints or other bursty data, in parallel computing environments. A burst buffer appliance is provided for use in a first storage tier of a multi-tier storage system comprising at least the first storage tier and a second storage tier. The exemplary burst buffer appliance comprises a memory for storing data; and at least one processing device to transform at least a portion of the data for storage on the second storage tier based on one or more performance characteristics of the second storage tier. In at least one embodiment, the at least one processing device is further configured to perform at least one function on the at least the portion of the data on behalf of the second storage tier. The performance characteristics of the second storage tier comprise, for example, a stripe size and/or network topology information.

Abstract: Partitioned key-value stores with one-sided communications are provided for secondary global key lookup by range-knowledgeable clients. A secondary global lookups is performed in a partitioned key-value store by receiving a request from an application for a secondary global key; accessing metadata to identify a first server storing the secondary global key; instructing the identified first server to store a first value corresponding to the requested secondary global key into a memory address of the client; and returning program control to the application, wherein the first server identifies a primary key corresponding to the requested secondary global key and instructs a second server storing the corresponding primary key to store a second value corresponding to the primary key into the memory address of the client. The second server stores the second value corresponding to the primary key into the first memory address of the client.

Abstract: A computer-executable method, computer program product, and system for managing I/Os from a legacy compliant Application on a host, wherein the host is in communication with a data storage system including a burst buffer node, the computer-executable method comprising receiving a POSIX compliant message from the Application, wherein the message is associated with data on the data storage system and processing the POSIX message on the data storage system.

Abstract: Techniques are provided for globally appending data from a group of distributed processes to a shared file using a log-structured file system. Data generated by a plurality of processes in a parallel computing system are appended to a shared file by storing the data to the shared file using a log-structured file system (such as a Parallel Log-Structured File System (PLFS)); and generating an index entry for the data, the index entry comprising a logical offset entry and a timestamp entry indicating a time of the storage, wherein the logical offset entry is resolved at read time. The logical offset entry can be populated with an append placeholder that is resolved when the shared file is read. At read time, a plurality of the index entries associated with the shared file can be sorted using the timestamp entry to deliver the requested shared file to a requesting application.

Abstract: Methods and apparatus are provided for filtering dynamically loadable namespaces (DLNs). An exemplary method comprises, in response to a job submitted by an application, obtaining a dynamically loadable namespace (DLN) portion of a global single namespace of a file system, wherein the DLN is associated with the job and is maintained in a capacity tier of a storage system; obtaining filtering directives from a user; reducing the DLN using a filtering mechanism on a directory tree associated with the DLN, based on the filtering directives, by removing files in the directory tree of the DLN that do not satisfy requirements of the filtering directives to generate a filtered DLN; and dynamically loading the filtered DLN, including reduced metadata for the filtered DLN relative to the DLN, from the capacity tier into a performance tier of the storage system for processing by the application.

Abstract: Improved techniques are provided for patent analytics using MapReduce clustering. An exemplary patent analytics method comprises obtaining one or more patent cluster keywords to query one or more public patent databases; obtaining one or more product cluster keywords to query one or more enterprise databases of an enterprise for patents related to a given product; mapping the one or more public patent databases for at least one patent cluster; reducing the map for the at least one patent cluster to obtain the at least one patent cluster; mapping the one or more enterprise databases for a plurality of patents in the one or more enterprise databases related to the given product; reducing the map for the one or more enterprise databases to identify a plurality of patents of the enterprise related to the given product; sorting the patent clusters according to one or more criteria; and processing the sorted patent clusters to obtain one or more patent analytic information.

Abstract: Metadata associated with a plurality of sub-files associated with a single shared file is stored in a parallel file system. A plurality of processes generate a shared file. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file and metadata for the at least one portion of the shared file on one or more of the plurality of object storage servers. The compute node is further configured to store the metadata by striping the metadata across a plurality of subdirectories of the shared file. The metadata is optionally striped across the plurality of subdirectories in a round-robin manner. The plurality of subdirectories are stored on one or more of the object storage servers. Write and read processes optionally communicate using a message passing interface. A given write process optionally writes metadata for a given portion of the shared file to an index file in a particular one of the subdirectories corresponding to the given portion.

Abstract: An apparatus in one embodiment comprises a storage system having a plurality of object storage servers. A software-defined storage controller of the storage system configures a unified memory cluster accessible to each of the plurality of object storage servers. The object storage servers collectively implement a distributed tiered parallel file system, such as a distributed tiered Lustre file system. The unified memory cluster may comprise portions of respective flash memories of a plurality of nodes of the storage system where each of the nodes comprises a corresponding one of the object storage servers. Objects stored in the flash memory of one of the nodes are mirrored into the flash memories of one or more other ones of the nodes under the control of the software-defined storage controller.

Abstract: Methods and apparatus are provided for fault resilient distributed computing using a continuous data protection feature of virtual machines. An exemplary method by a compute node for executing a process of a distributed application comprises providing a virtual machine having continuous data protection to store a copy of a state of the process in a performance storage tier; and providing a virtual machine to intercept messages of the process and to store a copy of the intercepted messages in a message log, wherein the process communicates with a plurality of other processes executing on other compute nodes, and wherein the plurality of processes employ asynchronous checkpointing. The process optionally communicates with the other processes in the distributed application using one or more virtual networks. The state is optionally moved from the performance storage tier to a capacity storage tier when a new state is stored.

Abstract: A file system is provided having a plurality of distributed metadata servers with separate metadata servers for file metadata and directory metadata. An exemplary distributed metadata storage system comprises a plurality of compute nodes, wherein at least one compute node comprises a file system client that generates a directory metadata request and/or a file metadata request; a plurality of directory metadata servers to serve directory metadata in response to a given directory metadata request; and a plurality of file metadata servers to serve file metadata in response to a given file metadata request. The plurality of directory metadata servers are distinct from the plurality of file metadata servers. The compute nodes optionally comprise an object client that requests object data from an object store system.

Abstract: Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.

Abstract: Methods and apparatus are provided for filtering dynamically loadable namespaces (DLNs). In one exemplary embodiment, a method comprises the steps of, in response to a job submitted by an application, obtaining a dynamically loadable namespace (DLN) portion of a global single namespace of a file system, wherein the DLN is associated with the job and is maintained in a capacity tier of object storage of a storage system; obtaining one or more filtering directives from a user; reducing the DLN based on the one or more filtering directives to generate a filtered DLN; and dynamically loading the filtered DLN, including the metadata only for the filtered DLN, from the capacity tier of object storage into a performance tier of storage of the storage system for processing by the application. The filtered DLN is merged into the original DLN following one or more modifications to the filtered DLN.

Abstract: An apparatus comprises a multi-tier storage system having a front-end storage tier, a back-end storage tier and a data mover module coupled to the front-end and back-end storage tiers and configured to control movement of data between the storage tiers. A translator at least partially incorporated in or otherwise associated with the multi-tier storage system implements a plurality of translation services and one or more index tables. A given one of the translation services is configured to access at least one of the index tables in order to determine a dynamically loadable namespace identifier associated with an application running on an analysis node that communicates with the multi-tier storage system over at least one network, and to instantiate a name node corresponding to the dynamically loadable namespace identifier. Particular objects of an object store of the back-end storage tier are made accessible to the application via the name node.

Abstract: A System, Computer program product, and computer-executable method of dynamically modifying data storage resilience within a two tiered data storage system, where the two tiered data storage system includes one or more fast data storage appliances and storage capacity tier wherein the fast data storage appliance is enabled to provide reliable data storage, the System, Computer program product, and computer-executable method including receiving configuration information from a node utilizing the two-tiered data storage system, wherein the configuration information provides characteristics of a job, analyzing the configuration information, updating use of the fast data storage appliances based on the analysis.

Abstract: A cluster file system comprises a front-end file system, a back-end file system, data mover modules arranged between the front-end and back-end file systems, and a quota manager associated with at least a given one of data mover modules. The data mover modules are configured to control archiving of data between the front-end file system and the back-end file system for respective users based at least in part on respective user quotas established by the quota manager and identifying corresponding portions of the back-end file system available to the users. The front-end file system may comprise archive directories configured for respective ones of the users, with the data mover modules being configured to archive a given file from the front-end file system in the back-end file system responsive to a link to that file being stored in a corresponding one of the archive directories of the front-end file system.

Abstract: A cluster file system comprises a front-end file system, a back-end file system, data mover modules arranged between the front-end and back-end file systems, and a quota manager associated with at least a given one of data mover modules. The data mover modules are configured to control archiving of data between the front-end file system and the back-end file system for respective users based at least in part on respective user quotas established by the quota manager and identifying corresponding portions of the back-end file system available to the users. The front-end file system may comprise archive directories configured for respective ones of the users, with the data mover modules being configured to archive a given file from the front-end file system in the back-end file system responsive to a link to that file being stored in a corresponding one of the archive directories of the front-end file system.

Abstract: An apparatus comprises an object store, back-end storage supporting the object store, and a file system associated with the back-end storage. Objects of the object store are mapped to files of the file system. The file system is configured to provide file system metadata extensions for respective ones of the files, with the file system metadata extension for a given one of the files comprising a plurality of sets of attributes characterizing respective ones of a plurality of accesses to the given file. The file system metadata extension for the given one of the files may be updated responsive to each of the plurality of accesses to that file to incorporate a corresponding one of the sets of attributes. Each of the sets of attributes may comprise, for example, at least one time-based attribute and at least one data integrity check attribute.

Abstract: A system, computer program product, and computer-executable method including presenting, to an application, data storage from a data storage system including a fast tier (FT) of data storage and a capacity tier (CT) of data storage, wherein the data storage system is enabled augment data stored within the data storage system to facilitate moving data between tiers of data storage within the data storage system, analyzing data stored on the FT of the data storage system to determine whether the application is sending write I/Os associated with the data, upon a negative determination, augmenting the data for the CT, and moving the data to the CT.

Abstract: A cluster file system comprises a parallel file system and a plurality of burst buffers, with the burst buffers collectively configured to provide an interface between the parallel file system and compute nodes of a computer system. Each of the burst buffers comprises a virtual file system component and a metadata server component. The virtual file system components of respective ones of the burst buffers collectively implement a virtual file system. Metadata relating portions of the virtual file system to portions of the parallel file system is stored in a distributed manner over the metadata server components of the respective burst buffers. Each burst buffer independently manages its own flash memory, but the burst buffers are configured such that data consistency is ensured across their flash memories. The virtual file system illustratively comprises a parallel log-structured file system (PLFS) and the metadata server components illustratively comprise respective TO forwarding servers.

Abstract: A cluster file system includes a front-end file system comprising a hierarchical storage management application programming interface (API), a back-end file system, and a data mover module arranged between the front-end and back-end file systems and adapted for communication with the hierarchical storage management API of the front-end file system. The data mover module is configured to control movement of data between the front-end file system and the back-end file system responsive to information received via the hierarchical storage management API. For example, in some embodiments, the data mover module splits a given file received from the front-end file system into multiple smaller files for storage in the back-end file system and associates with each of the smaller files a corresponding file extension from a designated namespace to permit reconstruction of the given file upon retrieval of the smaller files from the back-end file system.