Abstract: Generating a transformed dataset for use by a machine learning model in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: storing, within one or more storage systems, a transformed dataset generated by applying one or more transformations to a dataset that are identified based on one or more expected input formats of data received as input data by one or more machine learning models to be executed on one or more servers; and transmitting, from the one or more storage systems to the one or more servers without reapplying the one or more transformations on the dataset, the transformed dataset including data in the one or more expected formats of data to be received as input data by the one or more machine learning models.

Abstract: Scaling storage resources in a storage volume, including: monitoring a usage of a volume in a storage pool that includes one or more cloud-based storage systems; determining that the usage of the volume exceeds a threshold usage; and based on the determination, expanding the resources that are included in the storage pool for servicing the volume, including: instantiating one or more new virtual drives that are included in the one or more cloud-based storage systems; and adding the one or more new virtual drives to the storage pool.

Abstract: Processing data through a storage system in a data pipeline including receiving, by the storage system, a dataset from a collector on a data producer, wherein the dataset is disaggregated from metadata for the dataset by the collector; storing the dataset on the storage system; receiving, by the storage system from a data indexer, a request for data from the dataset, wherein the request for the data comprises the metadata gathered by the collector on the data producer; servicing, by the storage system, the request for the data by locating the data using the metadata gathered by the collector on the data producer and received in the request for the data; and receiving, from the data indexer, indexed data indexed using the metadata gathered by the collector on the data producer.

Abstract: Application-managed fault detection for cross-region replicated object stores is disclosed. An embodiment includes determining, by a first storage system among a plurality of storage systems replicating an object store, a faulted state in response to identifying a fault that prevents replication of updates to the object store to at least a second storage system of the plurality of storage systems; providing, through an API, an indication that the first storage system has entered the faulted state; and receiving a request indicating how to proceed in the presence of the fault.

Abstract: Controlling the direction of replication between cross-region replicated object stores is disclosed.

Abstract: Providing application-side infrastructure to control cross-region replicated object stores is disclosed. An embodiment includes determining, for an object store replicated across a plurality of storage systems at a plurality of locations, a minimum time interval between requests to modify the object store that are received by differing storage systems of the plurality of storage systems that ensures a predictable result on all storage systems among which the requested modifications are replicated; and providing, through an application programming interface (API) to a client that utilizes the object store, one or more object store parameters including the minimum time interval.

Abstract: Creating a containerized data analytics pipeline, including: creating a data analytics pipeline having a component that is deployed within a container that executes in a container cluster that is external to shared storage accessible by components of the data analytics pipeline; and responsive to detecting that a component of the data analytics pipeline has failed, deploying another instance of the component in a failover container that is configured to support one or more components of the data analytics pipeline.

Abstract: Executing a machine learning model in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: receiving, by a graphical processing unit (‘GPU’) server, a dataset transformed by a storage system that is external to the GPU server; and executing, by the GPU server, one or more machine learning algorithms using the transformed dataset as input.

Abstract: Generating a transformed dataset for use by a machine learning model in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: storing, within one or more storage systems, a transformed dataset generated by applying one or more transformations to a dataset that are identified based on one or more expected input formats of data received as input data by one or more machine learning models to be executed on one or more servers; and transmitting, from the one or more storage systems to the one or more servers without reapplying the one or more transformations on the dataset, the transformed dataset including data in the one or more expected formats of data to be received as input data by the one or more machine learning models.

Abstract: Scaling storage resources in a storage volume, including: monitoring a usage of a volume in a storage pool that includes one or more cloud-based storage systems; determining that the usage of the volume exceeds a threshold usage; and based on the determination, expanding the resources that are included in the storage pool for servicing the volume, including: instantiating one or more new virtual drives that are included in the one or more cloud-based storage systems; and adding the one or more new virtual drives to the storage pool.

Abstract: Establishing a guarantee for maintaining a replication relationship between object stores during a communications outage, an embodiment including identifying, by a first storage system, respective local clock values of one or more second storage systems, wherein the first storage system and the one or more second storage systems are among a plurality of storage systems replicating an object store, wherein the plurality of storage systems are configured to receive requests directed to the replicated object store; determining, by the first storage system in dependence upon the respective local clock values, respective clock differences of the one or more second storage systems relative to the first storage system; and ordering, by the first storage system, one or more updates to the replicated object store in dependence upon the respective clock differences.

Abstract: Providing for high availability in a data analytics pipeline without replicas, including: creating a data analytics pipeline, wherein each component of the data analytics pipeline is deployed within a container; creating a failover container; detecting that a component within the data analytics pipeline has failed; and responsive to detecting that the component within the data analytics pipeline has failed, deploying the component within the data analytics pipeline that has failed in the failover container.

Abstract: Generating a transformed dataset for use by a machine learning model in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: storing, within one or more storage systems, a transformed dataset generated by applying one or more transformations to a dataset that are identified based on one or more expected input formats of data received as input data by one or more machine learning models to be executed on one or more servers; and transmitting, from the one or more storage systems to the one or more servers without reapplying the one or more transformations on the dataset, the transformed dataset including data in the one or more expected formats of data to be received as input data by the one or more machine learning models.

Abstract: Data transformation caching in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: identifying, in dependence upon one or more machine learning models to be executed on the GPU servers, one or more transformations to apply to a dataset; generating, in dependence upon the one or more transformations, a transformed dataset; storing, within one or more of the storage systems, the transformed dataset; receiving a plurality of requests to transmit the transformed dataset to one or more of the GPU servers; and responsive to each request, transmitting, from the one or more storage systems to the one or more GPU servers without re-performing the one or more transformations on the dataset, the transformed dataset.

Abstract: Scaling storage resources in a storage volume, including: monitoring a usage of a volume in a storage pool that includes one or more cloud-based storage systems; determining that the usage of the volume exceeds a threshold usage; and based on the determination, expanding the resources that are included in the storage pool for servicing the volume, including: instantiating one or more new virtual drives that are included in the one or more cloud-based storage systems; and adding the one or more new virtual drives to the storage pool.

Abstract: Improving machine learning models in an artificial intelligence infrastructure includes: storing, within one or more storage systems of an artificial intelligence infrastructure, information describing a dataset and one or more transformations applied to the dataset resulting in a transformed dataset; and storing, within the one or more storage systems, information describing only portions of previous versions of a machine learning model that differ from a current version of the machine learning model, wherein the previous versions used the transformed dataset as input during one or more prior executions by the artificial intelligence infrastructure.

Abstract: Ensuring reproducibility in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: identifying, by a unified management plane, one or more transformations applied to a dataset by the artificial intelligence infrastructure, wherein applying the one or more transformations to the dataset causes the artificial intelligence infrastructure to generate a transformed dataset; storing, within the one or more storage systems, information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset; identifying, by the unified management plane, one or more machine learning models executed by the artificial intelligence infrastructure using the transformed dataset as input; and storing, within the one or more storage systems, information describing one or more machine learning models executed using the transformed dataset as input.

Abstract: Executing a big data analytics pipeline in a storage system that includes compute resources and shared storage resources, including: receiving, from a data producer, a dataset; storing, within the storage system, the dataset; allocating processing resources to an analytics application; and executing the analytics application on the processing resources, including ingesting the dataset from the storage system.

Abstract: Providing for high availability in a data analytics pipeline without replicas, including: creating a data analytics pipeline, wherein each component of the data analytics pipeline is deployed within a container; creating a failover container; detecting that a component within the data analytics pipeline has failed; and responsive to detecting that the component within the data analytics pipeline has failed, deploying the component within the data analytics pipeline that has failed in the failover container.

Abstract: Providing volumes as a service, including: receiving a request to create a volume; determining whether a storage pool that includes one or more cloud-based storage systems includes sufficient resources for provisioning the volume; responsive to determining that the storage pool includes sufficient resources for provisioning the volume, create a volume that is supported by one or more of the cloud-based storage systems in the storage pool; and responsive to determining that the storage pool does not includes sufficient resources for provisioning the volume, expanding the resources that are included in the storage pool.