Abstract: Monitoring write amplification metrics for a flash-based storage system, including: generating, by a controller of a storage system, data describing write activity for each storage device of a plurality of storage devices of the storage system, wherein the data describing the write activity comprises data describing application write activity for each storage device of the plurality of storage devices; calculating, by the controller of the storage system, for each storage device of the plurality of storage devices, a corresponding write amplification metric; and providing, by the controller of the storage system, to each storage device of the plurality of storage devices, the corresponding write amplification metric.

Abstract: A method for a transactional commit in a storage unit is provided. The method includes receiving a logical record from a storage node into a transaction engine of a storage unit of the storage node and writing the logical record into a data structure of the transaction engine. The method includes writing, to a command queue of the transaction engine, an indication to perform an atomic update using the logical record and transferring each portion of the logical record from the data structure of the transaction engine to non-persistent memory of the storage unit as a committed transaction. A storage unit for a storage system is also provided.

Abstract: A direct data path between a storage system and memory of one or more processing devices is enabled for a machine learning workload to be executed on the processing devices. Remote direct memory access (RDMA) is configured to transfer data between the storage system and the memory of the one or more processing devices while bypassing host memory. Data transfers for the machine learning workload are executed over the direct data path.

Abstract: Log data generated by a computing system is received. An analytics operation is executed on the log data. Pattern recognition associated with the log data is applied to identify anomalous behaviors of the computing system.

Abstract: Executing a machine learning model in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: receiving, by a graphical processing unit (‘GPU’) server, a dataset transformed by a storage system that is external to the GPU server; and executing, by the GPU server, one or more machine learning algorithms using the transformed dataset as input.

Abstract: A method for updates in a storage system is provided. The method includes writing identifiers, associated with data to be stored, to storage units of the storage system and writing trim records indicative of identifiers that are allowed to not exist in the storage system to the storage units. The method includes determining whether stored data corresponding to records of identifiers is valid based on the records of the identifiers and the trim records.

Abstract: Buffering and persisting artificial intelligence (AI) training checkpoints, including: storing a plurality of artificial intelligence (AI) training checkpoints in a write buffer comprising a first storage memory type; determining that a condition for persisting an AI training checkpoint of the plurality of AI training checkpoints has been satisfied; and storing, in response to the condition being satisfied, the AI training checkpoint to persistent storage comprising a second storage memory type.

Abstract: Execution of a pipeline of a client system and access patterns to one or more storage resources used by the pipeline is monitored. A bottleneck in the pipeline is identified based at least in part on the access patterns. A reconfiguration of resources is initiated, including reallocating compute resources or storage resources, to resolve the bottleneck in the pipeline.

Abstract: A dataset is received from a data source. A real-time analytics operation is executed on at least a portion of the dataset prior to fully persisting the dataset. Results of the real-time analytics operation are stored.

Abstract: A method is disclosed for managing transformed datasets in a compute cluster environment. The method includes identifying, based on one or more machine learning models to be executed on a compute cluster comprising a plurality of GPU servers, one or more transformations to apply to a dataset. The method further includes generating a transformed dataset based on the one or more transformations, storing the transformed dataset, receiving a request to transmit the transformed dataset to at least one GPU server of the plurality of GPU servers, and, responsive to the request, transmitting the stored transformed dataset to the at least one GPU server without re-performing the one or more transformations on the dataset.

Abstract: Executing a machine learning model in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: receiving, by a graphical processing unit (‘GPU’) server, a dataset transformed by a storage system that is external to the GPU server; and executing, by the GPU server, one or more machine learning algorithms using the transformed dataset as input.

Abstract: Generating a transformed dataset for use by a machine learning model in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: storing, within one or more storage systems, a transformed dataset generated by applying one or more transformations to a dataset that are identified based on one or more expected input formats of data received as input data by one or more machine learning models to be executed on one or more servers; and transmitting, from the one or more storage systems to the one or more servers without reapplying the one or more transformations on the dataset, the transformed dataset including data in the one or more expected formats of data to be received as input data by the one or more machine learning models.

Abstract: Improving machine learning models in an artificial intelligence infrastructure includes: storing, within one or more storage systems of an artificial intelligence infrastructure, information describing a dataset and one or more transformations applied to the dataset resulting in a transformed dataset; and storing, within the one or more storage systems, information describing only portions of previous versions of a machine learning model that differ from a current version of the machine learning model, wherein the previous versions used the transformed dataset as input during one or more prior executions by the artificial intelligence infrastructure.

Abstract: Generating a transformed dataset for use by a machine learning model in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: storing, within one or more storage systems, a transformed dataset generated by applying one or more transformations to a dataset that are identified based on one or more expected input formats of data received as input data by one or more machine learning models to be executed on one or more servers; and transmitting, from the one or more storage systems to the one or more servers without reapplying the one or more transformations on the dataset, the transformed dataset including data in the one or more expected formats of data to be received as input data by the one or more machine learning models.

Abstract: Improving machine learning models in an artificial intelligence infrastructure includes: storing, within one or more storage systems of an artificial intelligence infrastructure, information describing a dataset and one or more transformations applied to the dataset resulting in a transformed dataset; and storing, within the one or more storage systems, information describing only portions of previous versions of a machine learning model that differ from a current version of the machine learning model, wherein the previous versions used the transformed dataset as input during one or more prior executions by the artificial intelligence infrastructure.

Abstract: A first command including data to be stored in non-volatile random access memory (NVRAM) of a storage device is received from a storage system controller by a storage device controller. In response to receiving the first command, the data is stored in the NVRAM. One or more second commands including a location in flash memory of the storage device for storing the data and an instruction to store the data in the location of the flash memory are received from the storage system controller. In response to receiving the one or more second commands, the data is stored in the location of the flash memory.

Abstract: A method of data transfer between storage systems includes determining to transfer a dataset stored at a first storage system to a second storage system and invoking an operation to retrieve, for portions of the dataset having a size less than a threshold size, an identifier, metadata, and data stored in the portions of the dataset. The method further includes transmitting the identifiers, the metadata, and the data for the portions of the dataset having a size less than the threshold size to the second storage system.

Abstract: Deleting objects based on key pattern matching, including: receiving, by an object storage system, a request to delete any objects matching a key pattern from a bucket of the object storage system; and deleting, in response to the request, a plurality of objects matching the key pattern from the bucket of the object storage system.

Abstract: Data for a first type of workload that is to be stored in a storage system is received. A first request to store the data for the first type of workload in flash memory is transmitted to a storage device of the storage system. The data for the first type of workload is to be mapped using a first indirection unit (IU) size. Data for a second type of workload that is to be stored in the storage system is received. A second request to store the data for the second type of workload in the flash memory is transmitted to the storage device. The data for the second type of workload is to be mapped using a second IU size that is different than the first IU size.