Abstract: Data migration techniques are provided using parallel log-structured file system middleware to overcome file system limitations. A data file is migrated from a first storage tier to an archive storage tier, where the data file exceeds performance and/or capacity limitations of the archival storage tier, by writing the data file to a plurality of sub-files using a parallel data mover process, wherein each of the plurality of sub-files satisfy the performance and capacity limitations of the archival storage tier; and storing the plurality of sub-files to the archival storage tier using a log-structured file system. The log-structured file system optionally comprises a Parallel Log-Structured File System (PLFS) and the archival storage tier optionally comprises a multi disk storage tier. The parallel data mover process divides a size of the data file by a file size limit of the archival storage tier to determine of a number of the plurality of sub-files.

Abstract: An apparatus comprises a cluster file system, a burst buffer appliance coupled to the cluster file system, a high-speed memory pool configured for allocation between the cluster file system and the burst buffer appliance, and a controller associated with the high-speed memory pool and configured to dynamically adapt portions thereof between utilization as object storage servers of the cluster file system and utilization as burst buffers of the burst buffer appliance. By way of example, the high-speed memory pool comprises a plurality of flash servers that are reconfigurable between utilization as respective ones of the object storage servers of the cluster file system and utilization as respective ones of the burst buffers of the burst buffer appliance. The controller is configured to dynamically adapt the flash servers between utilization as respective object storage servers of the cluster file system and utilization as respective burst buffers of the burst buffer appliance.

Abstract: A parallel supercomputing cluster includes compute nodes interconnected in a mesh of data links for executing an MPI job, and solid-state storage nodes each linked to a respective group of the compute nodes for receiving checkpoint data from the respective compute nodes, and magnetic disk storage linked to each of the solid-state storage nodes for asynchronous migration of the checkpoint data from the solid-state storage nodes to the magnetic disk storage. Each solid-state storage node presents a file system interface to the MPI job, and multiple MPI processes of the MPI job write the checkpoint data to a shared file in the solid-state storage in a strided fashion, and the solid-state storage node asynchronously migrates the checkpoint data from the shared file in the solid-state storage to the magnetic disk storage and writes the checkpoint data to the magnetic disk storage in a sequential fashion.

Abstract: Techniques are provided for storing files in a parallel computing system using different resolutions. A method is provided for storing at least one file generated by a distributed application in a parallel computing system. The file comprises one or more of a complete file and a sub-file. The method comprises the steps of obtaining semantic information related to the file; generating a plurality of replicas of the file with different resolutions based on the semantic information; and storing the file and the plurality of replicas of the file in one or more storage nodes of the parallel computing system. The different resolutions comprise, for example, a variable number of bits and/or a different sub-set of data elements from the file. A plurality of the sub-files can be merged to reproduce the file.

Abstract: Solid-State Drive (SSD) burst buffer nodes are interposed into a parallel supercomputing cluster to enable fast burst checkpoint of cluster memory to or from nearby interconnected solid-state storage with asynchronous migration between the burst buffer nodes and slower more distant disk storage. The SSD nodes also perform tasks offloaded from the compute nodes or associated with the checkpoint data. For example, the data for the next job is preloaded in the SSD node and very fast uploaded to the respective compute node just before the next job starts. During a job, the SSD nodes perform fast visualization and statistical analysis upon the checkpoint data. The SSD nodes can also perform data reduction and encryption of the checkpoint data.

Abstract: A computer-executable method, system, and computer program product for managing I/O requests from a compute node in communication with a data storage system, including a first burst buffer node and a second burst buffer node, the computer-executable method, system, and computer program product comprising striping data on the first burst buffer node and the second burst buffer node, wherein a first portion of the data is communicated to the first burst buffer node and a second portion of the data is communicated to the second burst buffer node, processing the first portion of the data at the first burst buffer node, and processing the second portion of the data at the second burst buffer node.

Abstract: Improved techniques are provided for storing files in a parallel computing system using a list-based index to identify file replicas. A file and at least one replica of the file are stored in one or more storage nodes of the parallel computing system. An index for the file comprises at least one list comprising a pointer to a storage location of the file and a storage location of the at least one replica of the file. The file comprises one or more of a complete file and one or more sub-files. The index may also comprise a checksum value for one or more of the file and the replica(s) of the file. The checksum value can be evaluated to validate the file and/or the file replica(s). A query can be processed using the list.

Abstract: Techniques are provided for storing files in a parallel computing system using sub-files with semantically meaningful boundaries. A method is provided for storing at least one file generated by a distributed application in a parallel computing system. The file comprises one or more of a complete file and a plurality of sub-files. The method comprises the steps of obtaining a user specification of semantic information related to the file; providing the semantic information as a data structure description to a data formatting library write function; and storing the semantic information related to the file with one or more of the sub-files in one or more storage nodes of the parallel computing system. The semantic information provides a description of data in the file. The sub-files can be replicated based on semantically meaningful boundaries.

Abstract: Techniques are provided for storing files in a parallel computing system based on a user-specified parser function. A plurality of files generated by a distributed application in a parallel computing system are stored by obtaining a parser from the distributed application for processing the plurality of files prior to storage; and storing one or more of the plurality of files in one or more storage nodes of the parallel computing system based on the processing by the parser. The plurality of files comprise one or more of a plurality of complete files and a plurality of sub-files. The parser can optionally store only those files that satisfy one or more semantic requirements of the parser. The parser can also extract metadata from one or more of the files and the extracted metadata can be stored with one or more of the plurality of files and used for searching for files.

Abstract: Techniques are provided for small file aggregation in a parallel computing system. An exemplary method for storing a plurality of files generated by a plurality of processes in a parallel computing system comprises aggregating the plurality of files into a single aggregated file; and generating metadata for the single aggregated file. The metadata comprises an offset and a length of each of the plurality of files in the single aggregated file. The metadata can be used to unpack one or more of the files from the single aggregated file.

Abstract: Cloud infrastructure of an information processing system comprises one or more processing devices implementing a plurality of virtual machines. The cloud infrastructure is configured to receive a processing job from a tenant, to obtain a first key specific to the tenant, to determine a second key utilizing information supplied by the tenant, and to encrypt one or more results of the processing job utilizing a combination of the first key and the second key. At least a portion of the second key is determined by at least one application that is run on at least one virtual machine of the cloud infrastructure in conjunction with performance of the processing job. The encrypted results of the processing job may be stored in a virtual memory of the cloud infrastructure and transmitted to the tenant.

Abstract: A file server includes active storage containing frequently accessed files, and active/inactive disk drives for containing infrequently accessed files. Groups of the files having become inactive in the active storage are successively migrated to respective evacuated active/inactive disk drives so that each active/inactive disk drive is loaded with files having a similar probability of access when access to the active/inactive disk drive reaches an inactivity threshold for powering down the active/inactive disk drive. Storage of the active/inactive disk drives is reclaimed when an oldest group of the files is archived or when an active/inactive disk drive is evacuated by migrating files from the active/inactive disk drive to storage having been released in other disk drives by promotion of files for client access to the promoted files. Therefore, recovery of storage can be planned and scheduled in advance and performed efficiently in a background process.

Abstract: This invention is a system and method for managing data in a secure manner in a data storage environment that is in communication with a network including an internet-based network. The system includes logic for securely managing internet client's access to data volumes stored on a data storage system, and may also include logic operating with a file server for providing dynamic access of data available to such clients in a secure fashion.

Abstract: This invention is a system and for managing data in a secure manner in a data storage environment that is in communication with a network including an internet-based network. The system includes computer-executable logic or software for securely managing internet client's access to data volumes stored on a data storage system, and may also include logic operating with a file server for providing dynamic access of data available to such client's in a secure fashion.

Abstract: A network file server includes a first set of data processors for receiving requests from clients, and a second set of data processors for accessing read-write file systems. A respective data processor in the second set is assigned to each file system for exclusive management of read and write locks on the file system. Each data processor in the first set can authorize access to file systems directly accessed by more than one data processor in the second set. Processing of a request for access that is authorized is continued by the data processor that is assigned to manage the locks on the file system to be accessed. The exclusivity of lock management eliminates cache coherency problems, and dynamic load balancing can be used to prevent the lock management from becoming a bottleneck to performance. A preferred embodiment includes a cached disk storage subsystem linked to data mover computers.

Abstract: A client is permitted to send data access commands directly to network data storage of a network file server after obtaining a lock on at least a portion of the file and obtaining metadata indicating storage locations for the data in the data storage. For example, the client sends to the file server at least one request for access to a file. In response, the file server grants a lock to the client, and returns to the client metadata of the file including information specifying data storage locations in the network data storage for storing data of the file. The client receives the metadata, and uses the metadata to produce at least one data access command for accessing the data storage locations in the network storage. The client sends the data access command to the network data storage to read or write data to the file. For a write operation, the client may modify the metadata. When the client is finished writing to the file, the client returns any modified metadata to the file server.

Abstract: A client is permitted to send data access commands directly to network data storage of a network file server after obtaining a lock on at least a portion of the file and obtaining metadata indicating storage locations for the data in the data storage. For example, the client sends to the file server at least one request for access to a file. In response, the file server grants a lock to the client, and returns to the client metadata of the file including information specifying data storage locations in the network data storage for storing data of the file. The client receives the metadata, and uses the metadata to produce at least one data access command for accessing the data storage locations in the network storage. The client sends the data access command to the network data storage to read or write data to the file. For a write operation, the client may modify the metadata. When the client is finished writing to the file, the client returns any modified metadata to the file server.

Abstract: A client is permitted to send data access commands directly to network data storage of a network file server after obtaining a lock on at least a portion of the file and obtaining metadata indicating storage locations for the data in the data storage. For example, the client sends to the file server at least one request for access to a file. In response, the file server grants a lock to the client, and returns to the client metadata of the file including information specifying data storage locations in the network data storage for storing data of the file. The client receives the metadata, and uses the metadata to produce at least one data access command for accessing the data storage locations in the network storage. The client sends the data access command to the network data storage to read or write data to the file. For a write operation, the client may modify the metadata. When the client is finished writing to the file, the client returns any modified metadata to the file server.

Abstract: A client is permitted to send data access commands directly to network data storage of a network file server after obtaining a lock on at least a portion of the file and obtaining metadata indicating storage locations for the data in the data storage. For example, the client sends to the file server at least one request for access to a file. In response, the file server grants a lock to the client, and returns to the client metadata of the file including information specifying data storage locations in the network data storage for storing data of the file. The client receives the metadata, and uses the metadata to produce at least one data access command for accessing the data storage locations in the network storage. The client sends the data access command to the network data storage to read or write data to the file. For a write operation, the client may modify the metadata. When the client is finished writing to the file, the client returns any modified metadata to the file server.

Abstract: A network file server includes a first set of data processors for receiving requests from clients, and a second set of data processors for accessing read-write file systems. A respective data processor in the second set is assigned to each file system for exclusive management of read and write locks on the file system. Each data processor in the first set can authorize access to file systems directly accessed by more than one data processor in the second set. Processing of a request for access that is assigned to manage the locks on the file system to be accessed. The exclusivity of lock management eliminates cache coherency problems, and dynamic load balancing can be used to prevent the lock management from becoming a bottleneck to performance.