Abstract: Data transformation offloading in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: storing, within the storage system, a dataset; identifying, in dependence upon one or more machine learning models to be executed on the GPU servers, one or more transformations to apply to the dataset; and generating, by the storage system in dependence upon the one or more transformations, a transformed dataset.

Abstract: A method of processing transactions associated with a command in a storage system is provided. The method includes receiving, at a first authority of the storage system, a command relating to user data. The method includes sending a transaction of the command, from the first authority to a second authority of the storage system, wherein a token accompanies the transaction and writing data in accordance with the transaction as permitted by the token into a partition that is allocated to the second authority in a storage device of the storage system.

Abstract: Systems and methods of deduplication aware scalable content placement are described. A method may include receiving data to be stored on one or more nodes of a storage array and calculating a plurality of hashes corresponding to the data. The method further includes determining a first subset of the plurality of hashes, determining a second subset of the plurality of hashes of the first subset, and generating a node candidate placement list. The method may further include sending the first subset to one or more nodes represented on the node candidate placement list and receiving, from the nodes represented on the node candidate placement list, characteristics corresponding to the nodes represented on the candidate placement list. The method may further include identifying one of the one or more nodes represented on the candidate placement list in view of the characteristic and sending the data to the identified node.

Abstract: Dynamically allocated thread storage in a computing device is disclosed. The dynamically allocated thread storage is configured to work with a process including two or more threads. Each thread includes a statically allocated thread-local slot configured to store a table. Each table is configured to include a table slot corresponding with a dynamically allocated thread-local value. A dynamically allocated thread-local instance corresponds with the table slot.

Abstract: A plurality of storage nodes is provided. Each of the plurality of storage nodes includes nonvolatile solid-state memory for user data storage. The plurality of storage nodes is configured to distribute the user data and metadata associated with the user data throughout the plurality of storage nodes such that the plurality of storage nodes maintain the ability to read the user data, using erasure coding, despite a loss of two of the plurality of storage nodes. The plurality of storage nodes is configured to initiate an action based on the redundant copies of the metadata, responsive to achieving a level of redundancy for the redundant copies of the metadata. A method for accessing user data in a plurality of storage nodes having nonvolatile solid-state memory is also provided.

Abstract: A method for proactively rebuilding user data in a plurality of storage nodes of a storage cluster is provided. The method includes distributing user data and metadata throughout the plurality of storage nodes such that the plurality of storage nodes can read the user data, using erasure coding, despite loss of two of the storage nodes. The method includes determining that one of the storage nodes is unreachable and determining to rebuild the user data for the one of the storage nodes that is unreachable. The method includes reading the user data across a remainder of the plurality of storage nodes, using the erasure coding and writing the user data across the remainder of the plurality of storage nodes, using the erasure coding. A plurality of storage nodes within a single chassis that can proactively rebuild the user data stored within the storage nodes is also provided.

Abstract: Systems and methods of deduplication aware scalable content placement am described. A method may include receiving data to be stored on one or more nodes of a storage array and calculating a plurality of hashes corresponding to the data. The method further includes determining a first subset of the plurality of hashes, determining a second subset of the plurality of hashes of the first subset, and generating a node candidate placement list. The method may further include sending the first subset to one or more nodes represented on the node candidate placement list and receiving, from the nodes represented on the node candidate placement list, characteristics corresponding to the nodes represented on the candidate placement list. The method may further include identifying one of the one or more nodes represented on the candidate placement list m view of the characteristic and sending the data to the identified node.

Abstract: A method for file locks in a distributed file system, performed by a storage system is provided. The method includes receiving, at a storage system having a distributed file system, a request for access of a file and locking the file, through one of a plurality of persistent file locks in the storage system. The method includes accessing the file, through the distributed file system, and unlocking the file, through the one of the plurality of persistent file locks.

Abstract: A method for accessing a file in a storage system is provided. The method includes determining, for each file chunk of the file, an authority among differing storage nodes of the storage system and receiving from the authority having ownership of the file chunk, location information for the file chunk. The method includes accessing file chunks of the file as directed by each of the determined authorities.

Abstract: A method for proactively rebuilding user data in a plurality of storage nodes of a storage cluster is provided. The method includes distributing user data and metadata throughout the plurality of storage nodes such that the plurality of storage nodes can read the user data, using erasure coding, despite loss of two of the storage nodes. The method includes determining that one of the storage nodes is unreachable and determining to rebuild the user data for the one of the storage nodes that is unreachable. The method includes reading the user data across a remainder of the plurality of storage nodes, using the erasure coding and writing the user data across the remainder of the plurality of storage nodes, using the erasure coding. A plurality of storage nodes within a single chassis that can proactively rebuild the user data stored within the storage nodes is also provided.

Abstract: A method for file locks in a distributed file system, performed by a storage system is provided. The method includes receiving, at a storage system having a distributed file system, a request for access of a file and locking the file, through one of a plurality of persistent file locks in the storage system. The method includes accessing the file, through the distributed file system, and unlocking the file, through the one of the plurality of persistent file locks.

Abstract: A method for accessing a file in a storage system is provided. The method includes determining, for each file chunk of the file, an authority among differing storage nodes of the storage system and receiving from the authority having ownership of the file chunk, location information for the file chunk. The method includes accessing file chunks of the file as directed by each of the determined authorities.

Abstract: A method of encryption key management in a storage system having a plurality of nodes and more than one key manager, performed by the storage system, is provided. The method includes setting, in a first atomic operation to a distributed store of the plurality of nodes, a version identifier to a new value, and writing shards of a key encryption key, to node-specific memory of the plurality of nodes. The method includes committing the shards of the key encryption key by updating, in a second atomic operation, a set of version identifiers in the distributed store including a current version identifier, responsive to finding no change to the new value of the version identifier.

Abstract: A method of replication in a distributed storage system, performed by the distributed storage system is provided. The method includes managing a first index for data or metadata in a first storage system, the first storage system having a first partitioning scheme. The method includes managing a second index for data or metadata in a second storage system, the second storage system having a second partitioning scheme. The method includes replicating the data or metadata from the first storage system to the second storage system, translating an identifier of the data or metadata from the first storage system, and mapping the replicated data or metadata into the second partitioning scheme, via the translating of the identifier of the data or metadata from the first storage system.

Abstract: A processor-based method for locating data and metadata closely together in a storage system is provided. The method includes writing a first range of a file and a first metadata relating to attributes of the file into at least one segment controlled by a first authority of the file. The method includes delegating, by the first authority, a second authority for a second range of the file, and writing the second range of the file and second metadata relating to the attributes of the file into at least one segment controlled by the second authority.

Abstract: A system for cloud-based file services, comprising: a plurality of single-tenant file system nodes configured to provide file system access to an object store via a plurality of multitenant storage nodes; the plurality of multitenant storage nodes sharing access to the object store; and one or more management nodes configured to provision resources for the plurality of single-tenant file system nodes and the plurality of multitenant storage nodes.

Abstract: A method of distributed file deletion, performed by a storage system, is provided. The method includes receiving, at the storage system, a request to delete a directory and contents of the directory and adding the directory to a first set, listed in a memory in the storage system. The method includes operating on the first set, by examining each directory in the first set to identify subdirectories, adding each identified subdirectory to the first set as a directory, and adding each examined directory to a second set listed in the memory. The method includes deleting in a distributed manner across the storage system without concern for order, contents of directories, and the directories, listed in the second set.

Abstract: Data transformation caching in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: identifying, in dependence upon one or more machine learning models to be executed on the GPU servers, one or more transformations to apply to a dataset; generating, in dependence upon the one or more transformations, a transformed dataset; storing, within one or more of the storage systems, the transformed dataset; receiving a plurality of requests to transmit the transformed dataset to one or more of the GPU servers; and responsive to each request, transmitting, from the one or more storage systems to the one or more GPU servers without re-performing the one or more transformations on the dataset, the transformed dataset.

Abstract: A processor-based method for locating data and metadata closely together in a storage system is provided. The method includes writing a first range of a file and a first metadata relating to attributes of the file into at least one segment controlled by a first authority of the file. The method includes delegating, by the first authority, a second authority for a second range of the file, and writing the second range of the file and second metadata relating to the attributes of the file into at least one segment controlled by the second authority.

Abstract: A method of processing transactions associated with a command in a storage system is provided. The method includes receiving, at a first authority of the storage system, a command relating to user data. The method includes sending a transaction of the command, from the first authority to a second authority of the storage system, wherein a token accompanies the transaction and writing data in accordance with the transaction as permitted by the token into a partition that is allocated to the second authority in a storage device of the storage system.