Abstract: A processor-based method for locating data and metadata closely together in a storage system is provided. The method includes writing a first range of a file and a first metadata relating to attributes of the file into at least one segment controlled by a first authority of the file. The method includes delegating, by the first authority, a second authority for a second range of the file, and writing the second range of the file and second metadata relating to the attributes of the file into at least one segment controlled by the second authority.

Abstract: Data transformation caching in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: identifying, in dependence upon one or more machine learning models to be executed on the GPU servers, one or more transformations to apply to a dataset; generating, in dependence upon the one or more transformations, a transformed dataset; storing, within one or more of the storage systems, the transformed dataset; receiving a plurality of requests to transmit the transformed dataset to one or more of the GPU servers; and responsive to each request, transmitting, from the one or more storage systems to the one or more GPU servers without re-performing the one or more transformations on the dataset, the transformed dataset.

Abstract: A method for snapshots with crash consistency is provided. The method includes indicating intent to create a snapshot of storage system contents associated with an existing snapshot copy identifier, responsive to confirmation that all of a plurality of distributed agents are in a snapshot creation mode. The method includes responding, with a new snapshot copy identifier that replaces the existing snapshot copy identifier, to one or more inquiries from the plurality of distributed agents regarding one or more I/O operations, after the indicating the intent to create the snapshot. The method includes creating the snapshot using the existing snapshot copy identifier and committing the snapshot, responsive to receiving responses from the plurality of distributed agents that service is complete for all I/O operations that started before the indicating the intent to create the snapshot or that are associated with the existing snapshot copy identifier.

Abstract: A method for distributed file deletion or truncation, performed by a storage system, is provided. The method includes determining, by an authority owning an inode of a file, which authorities own data portions to be deleted, responsive to a request for the file deletion or truncation. The method includes recording, by the authority owning the inode, the file deletion or truncation in a first memory, and deleting, in background by the authorities that own the data portions to be deleted, the data portions in one of a first memory or a second memory. A system and computer readable media are also provided.

Abstract: A system for cloud-based file services, comprising: a plurality of single-tenant file system nodes configured to provide file system access to an object store via a plurality of multitenant storage nodes; the plurality of multitenant storage nodes sharing access to the object store; and one or more management nodes configured to provision resources for the plurality of single-tenant file system nodes and the plurality of multitenant storage nodes, including modifying resources within the system.

Abstract: A method for snapshots with crash consistency is provided. The method includes indicating intent to create a snapshot of storage system contents associated with an existing snapshot copy identifier, responsive to confirmation that all of a plurality of distributed agents are in a snapshot creation mode. The method includes responding, with a new snapshot copy identifier that replaces the existing snapshot copy identifier, to one or more inquiries from the plurality of distributed agents regarding one or more I/O operations, after the indicating the intent to create the snapshot. The method includes creating the snapshot using the existing snapshot copy identifier and committing the snapshot, responsive to receiving responses from the plurality of distributed agents that service is complete for all I/O operations that started before the indicating the intent to create the snapshot or that are associated with the existing snapshot copy identifier.

Abstract: A method for distributed file deletion or truncation, performed by a storage system, is provided. The method includes determining, by an authority owning an inode of a file, which authorities own data portions to be deleted, responsive to a request for the file deletion or truncation. The method includes recording, by the authority owning the inode, the file deletion or truncation in a first memory, and deleting, in background by the authorities that own the data portions to be deleted, the data portions in one of a first memory or a second memory. A system and computer readable media are also provided.

Abstract: Improving machine learning models in an artificial intelligence infrastructure includes: storing, within one or more storage systems of an artificial intelligence infrastructure, information describing a dataset and one or more transformations applied to the dataset resulting in a transformed dataset; and storing, within the one or more storage systems, information describing only portions of previous versions of a machine learning model that differ from a current version of the machine learning model, wherein the previous versions used the transformed dataset as input during one or more prior executions by the artificial intelligence infrastructure.

Abstract: Ensuring reproducibility in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: identifying, by a unified management plane, one or more transformations applied to a dataset by the artificial intelligence infrastructure, wherein applying the one or more transformations to the dataset causes the artificial intelligence infrastructure to generate a transformed dataset; storing, within the one or more storage systems, information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset; identifying, by the unified management plane, one or more machine learning models executed by the artificial intelligence infrastructure using the transformed dataset as input; and storing, within the one or more storage systems, information describing one or more machine learning models executed using the transformed dataset as input.

Abstract: A system for cloud-based file services, comprising: a plurality of single-tenant file system nodes configured to provide file system access to an object store via a plurality of multitenant storage nodes; the plurality of multitenant storage nodes sharing access to the object store; and one or more management nodes configured to provision resources for the plurality of single-tenant file system nodes and the plurality of multitenant storage nodes.

Abstract: A storage system has a first memory, and a second memory that includes storage memory. The storage system has a processing device. The processing device is to select whether to write data to the first memory and write the data from the first memory to the second memory, or to write the data to the second memory bypassing the first memory. The processing device is to write portions of data for storage according to such selection.

Abstract: A storage system has a first memory, a second memory that include solid-state storage memory, and a processing device. The processing device is to select a mode for each portion of data to be written. Selection of the mode is based at least on size of the portion of data. Selection of the mode is from among modes that include a first mode of writing the portion of data in mirrored RAID form to the first memory for later transfer from the first memory to the second memory, a second mode of writing the portion of data in parity-based RAID form to the first memory for later transfer from the first memory to the second memory, and a third mode of writing the portion of data to the second memory, bypassing the first memory. The processing device is to handle portions of data to be written according to such selection.

Abstract: A method of replication in a distributed storage system, performed by the distributed storage system is provided. The method includes managing a first index for data or metadata in a first storage system, the first storage system having a first partitioning scheme. The method includes managing a second index for data or metadata in a second storage system, the second storage system having a second partitioning scheme. The method includes replicating the data or metadata from the first storage system to the second storage system, translating an identifier of the data or metadata from the first storage system, and mapping the replicated data or metadata into the second partitioning scheme, via the translating of the identifier of the data or metadata from the first storage system.

Abstract: A method for adaptive concurrency for write persistence in a storage system, performed by the storage system, is provided. The method includes selecting a write process from among a plurality of write processes, responsive to receiving a write request for writing data into the storage system, and writing the data into the storage system in accordance with the selected write process. One of the plurality of write processes includes transferring the data into the storage system, locking an inode associated with file information of the data in memory, updating the file information in the inode while the inode is locked, committing the data while the inode is locked, and unlocking the inode.

Abstract: A method for distribution of directories in a storage system is provided. The method includes distributing information, regarding location in the storage system of a plurality of files in a directory, to a plurality of owners in a plurality of storage nodes of the storage system, wherein ownership of differing subsets of the plurality of files of the directory is distributed among differing owners in differing storage nodes. The method includes receiving a request for location information in the storage system of a file of the plurality of files in the directory and determining, based on a file name of the file and an identity of the directory, which of the plurality of owners has ownership of the file and the location information for the file.

Abstract: A method of processing transactions associated with a command in a storage system is provided. The method includes receiving, at a first authority of the storage system, a command relating to user data. The method includes sending a transaction of the command, from the first authority to a second authority of the storage system, wherein a token accompanies the transaction and writing data in accordance with the transaction as permitted by the token into a partition that is allocated to the second authority in a storage device of the storage system.

Abstract: Executing a machine learning model in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: receiving, by a graphical processing unit (‘GPU’) server, a dataset transformed by a storage system that is external to the GPU server; and executing, by the GPU server, one or more machine learning algorithms using the transformed dataset as input.

Abstract: A method of replication in a distributed storage system, performed by the distributed storage system is provided. The method includes managing a first index for data or metadata in a first storage system, the first storage system having a first partitioning scheme. The method includes managing a second index for data or metadata in a second storage system, the second storage system having a second partitioning scheme. The method includes replicating the data or metadata from the first storage system to the second storage system, translating an identifier of the data or metadata from the first storage system, and mapping the replicated data or metadata into the second partitioning scheme, via the translating of the identifier of the data or metadata from the first storage system.

Abstract: A method for distribution of directories in a storage system is provided. The method includes distributing information, regarding location in the storage system of a plurality of files in a directory, to a plurality of owners in a plurality of storage nodes of the storage system, wherein ownership of differing subsets of the plurality of files of the directory is distributed among differing owners in differing storage nodes. The method includes receiving a request for location information in the storage system of a file of the plurality of files in the directory and determining, based on a file name of the file and an identity of the directory, which of the plurality of owners has ownership of the file and the location information for the file.

Abstract: A system with storage memory and a processing device has a logical deletion to physical erasure time bound. The system dereferences data, responsive to a direction to delete the data. The system monitors physical blocks in storage memory for live data and the dereferenced data. The system cooperates garbage collection with monitoring the physical blocks, so that at least a physical block having the dereferenced data is garbage collected and erased within a logical deletion to physical erasure time bound.