Abstract: A computer-implemented method according to one embodiment includes dividing data into a plurality of partitions, creating a plurality of groups of different data types within each of the plurality of partitions, independently compressing, within each of the plurality of partitions, each of the plurality of groups of different data types to create a plurality of independently compressed partitions, and validating each of the plurality of independently compressed partitions to create a plurality of validated independently compressed partitions.

Abstract: A computer-implemented method according to one embodiment includes dividing data into a plurality of partitions, creating a plurality of groups of different data types within each of the plurality of partitions, independently compressing, within each of the plurality of partitions, each of the plurality of groups of different data types to create a plurality of independently compressed partitions, and validating each of the plurality of independently compressed partitions to create a plurality of validated independently compressed partitions.

Abstract: One embodiment provides a method for re-balancing data and metadata across multiple sub-file systems of a file system. The method includes determining sub-file systems including an amount of data that exceeds a threshold. At least one cell in the sub-file systems is identified as a candidate for re-balancing. A re-balance process is performed on the at least one cell that includes performing a flush operation to flush dirty data from file system buffers, copying an inode table for an independent set of files in the at least one cell to a destination sub-file system, notifying an allocation manager for the destination sub-file system of a new storage pool to manage, and performing an un-quiesce operation on I/O operations to each element in the at least one cell.

Abstract: One embodiment provides a method for storage management in a hierarchical file system that includes creating a structure for the hierarchical file system including a top-file system and sub-file systems that each include files and directories. Directories for the sub-file systems of the hierarchical file system are tracked. A request for creation of a unique directory in the hierarchical file system is received. Upon a request by a particular sub-file system for more inodes without any inodes available, inode numbers are revoked from at least one other sub-file system having unused inodes.

Abstract: One embodiment provides a method for storage management in an aggregated file system. The method includes tracking inode numbers for sub-file systems of the aggregated file system. A request for a unique range of inode numbers in the aggregated file system is received. The unique range of inode numbers is provided to one or more sub-file systems of the aggregated file system while providing that each sub-file system of the aggregated file system has a required range of inode numbers and that no two sub-file systems have overlapping inode numbers.

Abstract: In one embodiment, a method includes receiving, at a centralized index operating in a cloud environment via a messaging bus, first events spawned by a first processor managing an object store accessible via the cloud environment. The method also includes receiving, at the centralized index, second events spawned by a second processor managing an archival store accessible via the cloud environment. Also, the method includes normalizing metadata in the first and second events. Moreover, the method includes indexing and storing the metadata in the centralized index. The object store and the archival store operate in different object subsystems of the cloud environment. Other systems, methods, and computer program products are described for managing import, export, and copy of data within a public tiered object storage environment in accordance with more embodiments.

Abstract: In one embodiment, a computer-implemented method for namespace and content manipulation within a storage system includes: defining policy rule(s), each policy rule dictating: how namespaces within the storage system may be transformed; and/or how content stored within the storage system may be transformed; receiving a request for access to a particular data set stored within the storage system; parsing the request to determine one or more transformations requested therein and/or required thereby; determining whether the particular data set is compliant with the one or more transformations; in response to determining the particular data set is compliant with the one or more transformations, providing access to the particular data set; and in response to determining the particular data set is not compliant with the one or more transformations, performing the one or more transformations to generate a transformed data set, and providing access to the transformed data set.

Abstract: One embodiment provides a method for optimizing data read-ahead for workflow and analytics applications including obtaining, by a processor, next file information from a workflow scheduler for next files for a next processing stage that are to be accessed by a process. Data for the next processing stage for at least one application and at least one system job is prefetched. The next files are prefetched as the prefetching data reaches an end of current inputs.

Abstract: Embodiments relate to enabling sequential access for data scans on files with metadata. More specifically, object files are stored in a data storage. The objects are classified. A metaobject file is created in the data storage. The metaobject file is created by extracting attributes from the file from objects in a first classification. An entry is created in the metaobject file and the extracted attributes are stored in that entry. A file attribute is extracted from a file in the second classification and associated with the created entry in the metaobject file. A link is created between the file in the second classification and the related entry in the metaobject file.

Abstract: One embodiment provides a method for storage management in a hierarchical file system. The method includes tracking directories for sub-file systems of the hierarchical set of file systems and maintaining a corresponding directory in a top-file system and a number of symbolic pointers in the corresponding directory which each point from the corresponding directory to a sub-file system where any portion of the tracked directory resides. If available space in a first sub-file system is below a threshold, the system identifies a second sub-file system with available space above the threshold, wherein the first sub-file system contains a first portion of the directory. If a second portion of the directory does not exist in the second sub-file system, the system creates the second portion of the directory in the second sub-file system and creates a link from the first portion of the directory to the second portion of the directory.

Abstract: One embodiment provides a method for storage management in a hierarchical file system. The method includes tracking directories for sub-file systems of the hierarchical file system. A request for a unique directory in the hierarchical file system is received. The sub-file system of the hierarchical file system containing the unique directory is provided while ensuring that each directory resides in only one sub-file system. The system also maintains, in a top-file system, a corresponding directory and a symbolic pointer which points from the corresponding directory to a sub-file system where a given directory resides. Creation of new directories in sub-file systems also includes creation of corresponding directories in the top-file system and symbolic pointers from corresponding directory to new directory.

Abstract: One embodiment provides a method for storage management in a hierarchical file system. The method includes tracking directories for sub-file systems of the hierarchical set of file systems and maintaining a directory while providing the directory is capable of residing in more than one sub-file system within the hierarchical file system. If available space in a first sub-file system is below a threshold, the system identifies a second sub-file system with available space above the threshold, wherein the first sub-file system contains a first portion of the directory. If a second portion of the directory exists in the second sub-file system, the system creates or expands a file in the second portion of the directory, otherwise the system first creates the second portion of the directory in the second sub-file system and then creates a link from the first portion of the directory to the second portion of the directory.

Abstract: One embodiment provides a method for storage management in a hierarchical file system. The method includes tracking directories for sub-file systems of the hierarchical file system. A request for creation of an additional directory in the hierarchical file system is received. In response to the request, the additional directory is created in a sub-file system of the hierarchical file system while providing that each directory resides in only one sub-file system. If the additional directory is a child directory, the additional directory is created in the sub-file system containing the parent directory of the child directory.

Abstract: A method for collecting digital provenance from heterogeneous sources is disclosed. In one embodiment, such a method includes detecting events generated by multiple heterogeneous sources. The method further filters the events to extract relevant events therefrom. The method further transforms the relevant events into a standard format and saves the relevant events in a queue. The relevant events are then consumed from the queue. Consuming relevant events in the queue includes determining actions to be performed for each relevant event in the queue. These actions may include collecting digital provenance associated with the relevant events. The method then executes the actions. A corresponding system and computer program product are also disclosed.

Abstract: A computer-implemented method according to one embodiment includes identifying genomic data within a system, dividing the genomic data into a plurality of partitions, creating a plurality of groups of different data types within each of the plurality of partitions, independently compressing, within each of the plurality of partitions, each of the plurality of groups of different data types to create a plurality of independently compressed partitions, validating each of the plurality of independently compressed partitions to create a plurality of validated independently compressed partitions, and saving the plurality of validated independently compressed partitions within the system.

Abstract: A method for scheduling data services in a heterogeneous storage environment is disclosed. In one embodiment, such a method includes instrumenting, in a heterogeneous storage environment, each of a plurality of storage systems to generate events when operations are performed thereon. The events are detected and indexed in a single namespace. These events may then be analyzed to generate a statistical model of I/O activity occurring in the heterogeneous storage environment over a period of time. From the statistical model, the method determines periods of reduced I/O workload across the heterogeneous storage environment. The method then schedules data services to occur during these periods of reduced I/O workload. A corresponding system and computer program product are also disclosed.

Abstract: A computer-implemented method according to one embodiment includes identifying an accessing of a file within an operating system, checking an in-memory cache for path information associated with the file, checking an external cache for the path information associated with the file, conditionally retrieving the path information associated with the file by performing a file system lookup and adding the path information associated with the file to the in-memory cache and the external cache, and returning the path information.

Abstract: Embodiments for a system using a migration data placement tool between storage systems based on data access by a processor. Data may be cataloged in a migration data placement tool by receiving events from a plurality of heterogeneous storage systems. The events may be migrated into the migration data placement tool using analytics to determine data placement of the data in a target storage system according to the data identified as hot data, warm data, or cold data. The data may be placed according to the migration template during migration to the target storage system.

Abstract: Embodiments for using a migration data placement tool between storage systems based on data access by a processor. Events from a plurality of heterogeneous storage systems may be indexed into a common repository. A migration data placement tool uses analytics to determine data placement of the data in a target storage system according to the data identified as hot data, warm data, or cold data based on the events indexed in the common repository. The data may be placed according to a migration template during migration to the target storage system.

Abstract: A computer-implemented method according to one embodiment includes identifying an accessing of a file within an operating system, checking an in-memory cache for path information associated with the file, checking an external cache for the path information associated with the file, conditionally retrieving the path information associated with the file by performing a file system lookup and adding the path information associated with the file to the in-memory cache and the external cache, returning the path information, and reorganizing the external cache to put file information that is frequently accessed together in a single block or adjacent blocks.