Abstract: Systems and methods are provided to optimize checkpoint operations for deep learning (DL) model training tasks. For example, a distributed DL model training process is executed to train a DL model using multiple accelerator devices residing on one or more server nodes, and a checkpoint operation is performed to generate and store a checkpoint of an intermediate DL model. A checkpoint operation includes compressing a checkpoint of an intermediate DL model stored in memory of a given accelerator device to generate a compressed checkpoint, and scheduling a time to perform a memory copy operation to transfer a copy of the compressed checkpoint from the memory of the given accelerator device to a host system memory. The scheduling is performed based on information regarding bandwidth usage of a communication link to be utilized to transfer the compressed checkpoint to perform the memory copy operation, wherein the memory copy operation is performed at the scheduled time.

Abstract: An apparatus in one embodiment comprises a host device that includes at least one processor and an associated memory. The host device is configured to implement a plurality of processes each configured to access a shared region of the memory. The host device is further configured to establish a multi-process control group for the shared region, to maintain state information for the multi-process control group, and to track usage of the shared region by the processes based at least in part on the state information. At least a subset of the processes may comprise respective containers implemented utilizing operating system level virtualization of the processor of the host device. The multi-process control group established for the shared region illustratively comprises a coarse-grained control group having a granularity greater than a single page of the shared region.

Abstract: Techniques are provided to decouple data pipeline tasks from an execution flow of a high-performance computing task (e.g., distributed deep model training) in a distributed computing system. For example, a method includes receiving a client request to provision resources for executing a computing job, provisioning accelerator resources of one or more accelerator server nodes in the distributed computing system to perform tasks associated with an execution flow of the computing job, and provisioning a logical nodes within the distributed computing system to compose a data flow pipeline which is configured to perform data flow operations associated with the computing job for providing data to the provisioned accelerator resources to perform the tasks associated with the execution flow of the computing job. The data flow operations include, e.g., data storage input/output operations, data pre-processing operations, and data staging operations, which are decoupled from the execution flow of the computing job.

Abstract: An aspect of optimizing storage of data in a data replication system includes, for a plurality of write requests received from a source site, determining transfer statistics corresponding to each of the write requests and updating a table with the transfer statistics. An aspect also includes grouping pages in the table having common transfer statistics, assigning a unique non-volatile memory express (NVMe) stream identifier (ID) to each of the groups, and identifying grouped pages based on the assigned NVMe stream ID. An aspect further includes selecting a storage optimization technique for each of the groups based on the common transfer statistics and storing data of the write requests for each of the groups according to the selected optimization technique.

Abstract: An apparatus in one embodiment comprises a plurality of container host devices of at least one processing platform. The container host devices implement a plurality of containers for executing applications on behalf of one or more tenants of cloud infrastructure. One or more of the container host devices are each configured to compute distance measures between respective pairs of the containers and to assign the containers to container clusters based at least in part on the distance measures. The distance measures may be computed as respective content-based distance measures between hash identifiers of respective layers of layer structures of the corresponding containers. The apparatus may further comprise an interface configured to present a visualization of the container clusters. User feedback received via the interface is utilized to alter at least one parameter of the computation of distance measures and the assignment of clusters to container clusters.

Abstract: An apparatus in one embodiment comprises a plurality of container host devices of at least one processing platform. The container host devices implement a plurality of containers for executing applications on behalf of one or more tenants of cloud infrastructure. The containers have associated layer structures each characterizing container images of respective different ones of the containers. Movement of container data between different storage devices by at least one of the container host devices for at least one of the containers is controlled based at least in part on one or more characteristics of a corresponding one of the layer structures. For example, controlling movement of container data between the different storage devices may comprise assigning at least one of different prefetching priority weights and different cache-swapping priority weights to different layers of the given layer structure.

Abstract: Systems and methods are provided to configure a replication system to reduce latency of data read access operations in a cloud computing environment. Intelligent routing decision systems and methods are implemented in conjunction with a distributed replication system in a cloud computing environment to direct write requests to a primary site (e.g., production site) by default, and to automatically and intelligently direct read requests to a target site (e.g., primary site or replica site) to reduce data read latency and/or bandwidth. A reduction in read delay is achieved, in part, by intelligently selecting a target site (e.g., data center) in a cloud computing environment which is close or closest in distance to a requesting entity (e.g., user or application) requesting data, thereby reducing a propagation delay and/or latency in accessing and downloading data from the target site.

Abstract: In a distributed data processing system with a set of multiple nodes, a first data shuffle memory pool is maintained at a data shuffle writer node, and a second data shuffle memory pool is maintained at a data shuffle reader node. The data shuffle writer node and the data shuffle reader node are part of the set of multiple nodes of the distributed data processing system. In-memory compression is performed on at least a portion of a data set from the first data shuffle memory pool. At least a portion of the compressed data is transmitted from the first data shuffle memory pool to the second data shuffle memory pool in a peer-to-peer manner. Each of the first data shuffle memory pool and the second data shuffle memory pool may include a hybrid memory configuration.

Abstract: An aspect of optimizing storage of data in a data replication system includes, for a plurality of write requests received from a source site, determining transfer statistics corresponding to each of the write requests and updating a table with the transfer statistics. An aspect also includes grouping pages in the table having common transfer statistics, assigning a unique non-volatile memory express (NVMe) stream identifier (ID) to each of the groups, and identifying grouped pages based on the assigned NVMe stream ID. An aspect further includes selecting a storage optimization technique for each of the groups based on the common transfer statistics and storing data of the write requests for each of the groups according to the selected optimization technique.

Abstract: A method comprises obtaining hardware information of a plurality of dedicated processing resources, wherein the plurality of dedicated processing resources comprises a first dedicated processing resource and a second dedicated processing resource, and the hardware information comprises first hardware information of the first dedicated processing resource and second hardware information of the second dedicated processing resource. The method further comprises generating a first task based on the first hardware information and a second task based on the second hardware information, and allocating the first task to the first dedicated processing resource and the second task to the second dedicated processing resource. For task scheduling in heterogeneous dedicated processing resources (for example, accelerator devices) scenario, the method generates corresponding kernel codes according to different hardware capabilities.

Abstract: A technique versions a configuration of data storage equipment. The technique involves receiving, while the data storage equipment operates in accordance with a first version of the configuration of the data storage equipment, a configuration change command to change the configuration of the data storage equipment. The technique further involves changing the first version of the configuration of the data storage equipment to a second version of the configuration of the data storage equipment in response to the configuration change command. The technique further involves, while the data storage equipment operates in accordance with the second version of the configuration of the data storage equipment, concurrently maintaining access to the first version of the configuration of the data storage equipment and the second version of the configuration of the data storage equipment.

Abstract: A method comprises: determining whether an event triggering processing of data at a storage device occurs, the data being predetermined to be processed at a computing device associated with the storage device; in response to determining that the event occurs, determining available resources of the storage device; and in response to an amount of the available resources exceeding a first predetermined threshold, causing the storage device to process the data and provide the processed data to the computing device. As such, operations to be performed to data may be adaptively allocated, deployed and adjusted on the storage device and the computing device, so that computing capabilities of both the storage device and the computing device may be fully leveraged, and further the computing efficiency of the entire data processing system may be improved.

Abstract: A target data processing method comprises: in response to receiving, from a client, a request for processing target data, obtaining, at a server and from a storage device, the target data, the target data being predetermined to be obtained by the client from the storage device and a first set of operations being predetermined to be performed on the target data; and processing, at the server, the target data by performing the first set of operations on the target data such that at least one dedicated processing resource of the server performs, on the processed target data, a second set of operations associated with the first set of operations. As such, the data moving path may be shortened significantly, the network traffic pressure may be reduced significantly, the efficiency of the target data processing system may be improved, and moreover, the number of dedicated processing resources in the server may be easily scaled.

Abstract: A method is disclosed comprising: generating a plurality of snapshots, each of the snapshots representing a state of one or more storage objects in a storage system; generating a plurality of access pattern records (APRs) for the storage objects, each APR being associated with a different respective one of the plurality of snapshots, each APR indicating a characteristic of workload of the storage system in a period preceding the generation of the APR's associated snapshot; detecting a workload trend for the storage objects, the workload trend being detected by using a workload prediction model that is trained based on the plurality of APRs; and configuring the storage system based on the workload trend, the configuring including allocating additional resources for servicing access requests to the storage objects when the workload trend is a rising workload trend.

Abstract: In a method of tiering data in a computing environment, data service status information is obtained at a first storage environment regarding a data service operation performed on one or more data sets that qualify for storage at a second storage environment based on a given policy, and, based on the obtained data service status information, a determination is made to store the one or more data sets at the second storage environment without performing a reverse operation of the data service operation on the one or more data sets.

Abstract: A time-series data cache is operatively coupled between a time-series analytics application program and a time-series data store, and configured to temporarily store portions of the time-series data. The time-series data store is configured to persistently store time-series data. The time-series data cache is further configured to be responsive to one or more data read requests received from the time-series analytics application program.

Abstract: One or more dynamically occurring events in a distributed data streaming system are monitored. The occurrence of at least one of the one or more dynamically occurring events is evaluated. A checkpoint operation is initiated in the distributed data streaming system based on the evaluation of the occurrence of the at least one dynamically occurring event.

Abstract: Data service status information is obtained at a local storage tier regarding a data service operation performed on one or more data sets that qualify for remote storage based on a given policy. A determination is made whether or not to store at least a portion of the one or more data sets at a remote storage tier based on the obtained data service status information.

Abstract: An apparatus in one embodiment comprises a host device that includes at least one processor and an associated memory. The host device is configured to implement a plurality of processes each configured to access a shared region of the memory. The host device is further configured to establish a multi-process control group for the shared region, to maintain state information for the multi-process control group, and to track usage of the shared region by the processes based at least in part on the state information. At least a subset of the processes may comprise respective containers implemented utilizing operating system level virtualization of the processor of the host device. The multi-process control group established for the shared region illustratively comprises a coarse-grained control group having a granularity greater than a single page of the shared region.

Abstract: Techniques are provided for implementing asynchronous checkpointing of in-memory data in a distributed computing system. For example, a method includes processing a stream of data records by an operator executing on a computing node, maintaining in a system memory, an operator state which is generated in response to the operator processing the stream of data records, and performing an asynchronous checkpointing process. The asynchronous checkpointing process includes enqueuing a checkpoint of the operator state in a first queue, wherein the first queue is maintained in the system memory, and executing a background worker thread to dequeue the checkpoint of the operator state from the first queue and store the checkpoint of the operator state in a data store. The operator continues with processing the stream of data records during the asynchronous checkpointing process.