Abstract: An artificial intelligence and machine learning infrastructure system, including: one or more storage systems comprising, respectively, one or more storage devices; and one or more graphical processing units, wherein the graphical processing units are configured to communicate with the one or more storage systems over a communication fabric; where the one or more storage systems, the one or more graphical processing units, and the communication fabric are implemented within a single chassis.

Abstract: An artificial intelligence and machine learning infrastructure system, including: one or more storage systems comprising, respectively, one or more storage devices; and one or more graphical processing units, wherein the graphical processing units are configured to communicate with the one or more storage systems over a communication fabric; where the one or more storage systems, the one or more graphical processing units, and the communication fabric are implemented within a single chassis.

Abstract: A non-volatile solid-state storage is provided. The non-volatile solid state storage includes a non-volatile random access memory (NVRAM) addressable by a processor external to the non-volatile solid state storage. The NVRAM is configured to store user data and metadata relating to the user data. The non-volatile solid state storage includes a flash memory addressable by the processor. The flash memory is configured to store the user data responsive to the processor directing transfer of the user data from the NVRAM to the flash memory.

Abstract: A plurality of storage nodes is provided. Each of the plurality of storage nodes includes nonvolatile solid-state memory for user data storage. The plurality of storage nodes is configured to distribute the user data and metadata associated with the user data throughout the plurality of storage nodes such that the plurality of storage nodes maintain the ability to read the user data, using erasure coding, despite a loss of two of the plurality of storage nodes. The plurality of storage nodes is configured to initiate an action based on the redundant copies of the metadata, responsive to achieving a level of redundancy for the redundant copies of the metadata. A method for accessing user data in a plurality of storage nodes having nonvolatile solid-state memory is also provided.

Abstract: Storage layer data obfuscation, including: determining a subset of a dataset to obfuscate in accordance with a security policy; generating, based at least in part on the security policy, an obfuscated snapshot of the dataset that is representative of the dataset with the subset of the dataset obfuscated; and sending, to a target computer system, the obfuscated snapshot from which a restored version of the dataset includes the subset of the dataset obfuscated.

Abstract: A method of applying an address space to data storage in a non-volatile solid-state storage is provided. The method includes receiving a plurality of portions of user data for storage in the non-volatile solid-state storage and assigning to each successive one of the plurality of portions of user data one of a plurality of sequential, nonrepeating addresses of an address space. The address range of the address space exceeds a maximum number of addresses expected to be applied during a lifespan of the non-volatile solid-state storage. The method includes writing each of the plurality of portions of user data to the non-volatile solid-state storage such that each of the plurality of portions of user data is identified and locatable for reading via the one of the plurality of sequential, nonrepeating addresses of the address space.

Abstract: A plurality of storage nodes is provided. Each of the plurality of storage nodes includes nonvolatile solid-state memory for user data storage. The plurality of storage nodes is configured to distribute the user data and metadata associated with the user data throughout the plurality of storage nodes such that the plurality of storage nodes maintain the ability to read the user data, using erasure coding, despite a loss of two of the plurality of storage nodes. The plurality of storage nodes is configured to initiate an action based on the redundant copies of the metadata, responsive to achieving a level of redundancy for the redundant copies of the metadata. A method for accessing user data in a plurality of storage nodes having nonvolatile solid-state memory is also provided.

Abstract: A method of applying an address space to data storage in a non-volatile solid-state storage is provided. The method includes receiving a plurality of portions of user data for storage in the non-volatile solid-state storage and assigning to each successive one of the plurality of portions of user data one of a plurality of sequential, nonrepeating addresses of an address space. The address range of the address space exceeds a maximum number of addresses expected to be applied during a lifespan of the non-volatile solid-state storage. The method includes writing each of the plurality of portions of user data to the non-volatile solid-state storage such that each of the plurality of portions of user data is identified and locatable for reading via the one of the plurality of sequential, nonrepeating addresses of the address space.

Abstract: Ensuring reproducibility in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: identifying, by a unified management plane, one or more transformations applied to a dataset by the artificial intelligence infrastructure, wherein applying the one or more transformations to the dataset causes the artificial intelligence infrastructure to generate a transformed dataset; storing, within the one or more storage systems, information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset; identifying, by the unified management plane, one or more machine learning models executed by the artificial intelligence infrastructure using the transformed dataset as input; and storing, within the one or more storage systems, information describing one or more machine learning models executed using the transformed dataset as input.

Abstract: A method of applying a data format in a direct memory access transfer is provided. The method includes distributing user data throughout a plurality of storage nodes through erasure coding, wherein the plurality of storage nodes are housed within a single chassis that couples the storage nodes as a cluster, each of the plurality of storage nodes having nonvolatile solid-state memory for user data storage. The method includes reading a self-describing data portion from a first memory of the nonvolatile solid-state memory and extracting a destination from the self-describing data portion. The method includes writing data, from the self-describing data portion, to a second memory of the nonvolatile solid-state memory according to the destination.

Abstract: Data transformation offloading in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: storing, within the storage system, a dataset; identifying, in dependence upon one or more machine learning models to be executed on the GPU servers, one or more transformations to apply to the dataset; and generating, by the storage system in dependence upon the one or more transformations, a transformed dataset.

Abstract: Data transformation caching in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: identifying, in dependence upon one or more machine learning models to be executed on the GPU servers, one or more transformations to apply to a dataset; generating, in dependence upon the one or more transformations, a transformed dataset; storing, within one or more of the storage systems, the transformed dataset; receiving a plurality of requests to transmit the transformed dataset to one or more of the GPU servers; and responsive to each request, transmitting, from the one or more storage systems to the one or more GPU servers without re-performing the one or more transformations on the dataset, the transformed dataset.

Abstract: Data transformation offloading in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: storing, within the storage system, a dataset; identifying, in dependence upon one or more machine learning models to be executed on the GPU servers, one or more transformations to apply to the dataset; and generating, by the storage system in dependence upon the one or more transformations, a transformed dataset.

Abstract: Data transformation caching in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: identifying, in dependence upon one or more machine learning models to be executed on the GPU servers, one or more transformations to apply to a dataset; generating, in dependence upon the one or more transformations, a transformed dataset; storing, within one or more of the storage systems, the transformed dataset; receiving a plurality of requests to transmit the transformed dataset to one or more of the GPU servers; and responsive to each request, transmitting, from the one or more storage systems to the one or more GPU servers without re-performing the one or more transformations on the dataset, the transformed dataset.

Abstract: Ensuring reproducibility in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: identifying, by a unified management plane, one or more transformations applied to a dataset by the artificial intelligence infrastructure, wherein applying the one or more transformations to the dataset causes the artificial intelligence infrastructure to generate a transformed dataset; storing, within the one or more storage systems, information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset; identifying, by the unified management plane, one or more machine learning models executed by the artificial intelligence infrastructure using the transformed dataset as input; and storing, within the one or more storage systems, information describing one or more machine learning models executed using the transformed dataset as input.

Abstract: A method for a transactional commit in a storage unit is provided. The method includes receiving a logical record from a storage node into a transaction engine of a storage unit of the storage node and writing the logical record into a data structure of the transaction engine. The method includes writing, to a command queue of the transaction engine, an indication to perform an atomic update using the logical record and transferring each portion of the logical record from the data structure of the transaction engine to non-persistent memory of the storage unit as a committed transaction. A storage unit for a storage system is also provided.

Abstract: A method of applying an address space to data storage in a non-volatile solid-state storage is provided. The method includes receiving a plurality of portions of user data for storage in the non-volatile solid-state storage and assigning to each successive one of the plurality of portions of user data one of a plurality of sequential, nonrepeating addresses of an address space. The address range of the address space exceeds a maximum number of addresses expected to be applied during a lifespan of the non-volatile solid-state storage. The method includes writing each of the plurality of portions of user data to the non-volatile solid-state storage such that each of the plurality of portions of user data is identified and locatable for reading via the one of the plurality of sequential, nonrepeating addresses of the address space.

Abstract: A method of applying a data format in a direct memory access transfer is provided. The method includes distributing user data throughout a plurality of storage nodes through erasure coding, wherein the plurality of storage nodes are housed within a single chassis that couples the storage nodes as a cluster, each of the plurality of storage nodes having nonvolatile solid-state memory for user data storage. The method includes reading a self-describing data portion from a first memory of the nonvolatile solid-state memory and extracting a destination from the self-describing data portion. The method includes writing data, from the self-describing data portion, to a second memory of the nonvolatile solid-state memory according to the destination.

Abstract: A method of applying scheduling policies is provided. The method includes distributing user data throughout a plurality of storage nodes through erasure coding, wherein the plurality of storage nodes are housed within a single chassis coupling the storage nodes as a cluster. The method includes receiving operations relating to a non-volatile memory of one of the plurality of storage nodes into a plurality of operation queues. The method includes evaluating each of the operations in the plurality of operation queues as to benefit to the non-volatile solid-state storage according to a plurality of policies. For each channel of a plurality of channels coupling the operation queues to the non-volatile memory, the method includes iterating a selection and an execution of a next operation from the plurality of operation queues, with each next operation having a greater benefit than at least a subset of operations remaining in the operation queues.

Abstract: A method for a transactional commit in a storage unit is provided. The method includes receiving a logical record from a storage node into a transaction engine of a storage unit of the storage node and writing the logical record into a data structure of the transaction engine. The method includes writing, to a command queue of the transaction engine, an indication to perform an atomic update using the logical record and transferring each portion of the logical record from the data structure of the transaction engine to non-persistent memory of the storage unit as a committed transaction. A storage unit for a storage system is also provided.