Abstract: A computer-implemented method of providing data transformation includes installing one or more data transformation plugins in a dataset made accessible for processing an end user's workload. A dataset-specific policy for the accessible dataset is ingested. A data transformation of the accessible dataset is executed by invoking one or more of the data transformation plugins to the accessible dataset based on the dataset-specific policy to generate a transformed dataset. The user's workload is deployed to provide data access for processing using the transformed dataset in accordance with a data governance policy.

Abstract: Aspects of the present disclosure relate to dynamic workload migration in a decentralized hierarchical control plane for management in edge devices and hybrid cloud environments. More specifically, a processing device obtains an indication of a workload associated with a decentralized hierarchical control plane, where the decentralized hierarchical control plane includes a plurality of control nodes in a decentralized hierarchy. The processing device determines, based on data associated with the decentralized hierarchy, a target for migration of the workload. The processing device causes the workload to be migrated to the target.

Abstract: Aspects of the present disclosure relate to a decentralized hierarchical control plane for management in edge devices and hybrid cloud environments. More specifically, a method of the present disclosure includes obtaining, at a device, an indication that the device is to act as a control node in a decentralized hierarchical control plane, where the decentralized hierarchical control plane includes a plurality of control nodes in a decentralized hierarchy. The method includes managing, by a processing device at the device acting as the control node and via the decentralized hierarchical control plane, resources associated with a plurality of devices in the decentralized hierarchy.

Abstract: An embodiment includes detecting a search request of a multi-dimensional search space by a system; responsive to detecting the search request, sampling representative parameters in the multi-dimensional search space based on a sampling threshold. The embodiment includes determining a parameter range of the representative parameters in the multi-dimensional search space. The embodiment also includes transforming a part of the multi-dimensional search space based on a statistical guarantee and the parameter range, where the part of the multi-dimensional search space is a refined space for a solution of the search request.

Abstract: In an approach for storage, search, acquisition, and composition of a digital artifact, a processor obtains the digital artifact in a digital marketplace platform. The digital artifact is a collection of digital data with automatically generated and verifiable provenance and usage data. A processor transforms the digital artifact to define an access privilege. A processor shares the digital artifact in the digital marketplace platform by providing a view of a catalogue including the digital artifact. A processor authorizes a usage request based on the access privilege. A processor rewards a source of the digital artifact based on the usage of the digital artifact.

Abstract: A computer-implemented method, a computer program product, and a computer system for determining optimal data access for deep learning applications on a cluster. A server determines candidate cache locations for one or more compute nodes in the cluster. The server fetches a mini-batch of a dataset located at a remote storage service into the candidate cache locations. The server collects information about time periods of completing a job on the one or more nodes, where the job is executed against fetched mini-batch at the candidate cache locations and the mini-batch at the remote storage location. The server selects, from the candidate cache locations and the remote storage location, a cache location. The server fetches the data of the dataset from the remote storage service to the cache location, and the one or more nodes execute the job against fetched data of the dataset at the cache location.

Abstract: A method, computer system, and a computer program product are provided for query management optimization in a distributed data management system. In one embodiment, at least one query is received. The query is then analyzed and related information associated with the query is obtained. When information exists in a database relating to previously executed queries similar to the received query, that information is obtained. A query execution plan is then formulated using any existing information and information relating to the similarly previously executed queries.

Abstract: A method, computer system, and a computer program product are provided for query management optimization in a distributed data management system. In one embodiment, at least one query is received. The query is then analyzed and related information associated with the query is obtained. When information exists in a database relating to previously executed queries similar to the received query, that information is obtained. A query execution plan is then formulated using any existing information and information relating to the similarly previously executed queries.

Abstract: Embodiments of the present invention provide an approach for automating a configuration of a server infrastructure for cloud applications by leveraging monitoring data from both the infrastructure and the applications that run on it. Specially, input information including an application which has been submitted is received along with a target dataset, a cloud provider and values for a specific performance measure. The application is mapped to a specific class and a performance model is selected based on the class. A set of resource configurations is generated and estimates of a target measure (e.g., run time) are provided for each configuration option using the selected model. A resource configuration option that provides either the best value of the measure or closest to the application objectives is selected and committed to an application deployment file.

Abstract: In an approach for storage, search, acquisition, and composition of a digital artifact, a processor obtains the digital artifact in a digital marketplace platform. The digital artifact is a collection of digital data with automatically generated and verifiable provenance and usage data. A processor transforms the digital artifact to define an access privilege. A processor shares the digital artifact in the digital marketplace platform by providing a view of a catalogue including the digital artifact. A processor authorizes a usage request based on the access privilege. A processor rewards a source of the digital artifact based on the usage of the digital artifact.

Abstract: A computer implemented method for optimizing performance of workflow includes associating each of a plurality of workflow nodes in a workflow with a data cache and managing the data cache on a local storage device on one of one or more compute nodes. A scheduler can request execution of the tasks of a given one of the plurality of workflow nodes on one of the one of more compute nodes that hosts the data cache associated with the given one of the plurality of workflow nodes. Each of the plurality of workflow nodes is permitted to access a distributed filesystem that is visible to each of the plurality of compute nodes. The data cache stores data produced by the tasks of the given one of the plurality of workflow nodes.

Abstract: A method, system, and computer program product for garbage collection of redundant partitions in distributed data management systems are provided. The method stores data across a set of nodes with the data being stored using one or more partitions and the data and the one or more partitions are replicated across the set of nodes. A first partition is determined to be stale at a first node of the set of nodes. The first partition is marked for deletion locally at the first node. A set of deletion votes are determined for the first partition with each node being associated with a deletion vote. The method determines a deletion decision for the first partition on the first node based on the set of deletion votes.

Abstract: Embodiments for processing data with multiple machine learning models are provided. Input data is received. The input data is caused to be evaluated by a first machine learning model to generate a first inference result. The first inference result is compared to at least one quality of service (QoS) parameter. Based on the comparison of the first inference result to the at least one QoS parameter, the input data is caused to be evaluated by a second machine learning model to generate a second inference result.

Abstract: A computer-implemented method, a computer program product, and a computer system for determining optimal data access for deep learning applications on a cluster. A server determines candidate cache locations for one or more compute nodes in the cluster. The server fetches a mini-batch of a dataset located at a remote storage service into the candidate cache locations. The server collects information about time periods of completing a job on the one or more nodes, where the job is executed against fetched mini-batch at the candidate cache locations and the mini-batch at the remote storage location. The server selects, from the candidate cache locations and the remote storage location, a cache location. The server fetches the data of the dataset from the remote storage service to the cache location, and the one or more nodes execute the job against fetched data of the dataset at the cache location.

Abstract: Embodiments for providing intelligent data replication and distribution in a computing environment. Data access patterns of one or more queries issued to a plurality of data partitions may be forecasted. Data may be dynamically distributed and replicated to one or more existing data partitions or additional of the plurality of data partitions according to the forecasting.

Abstract: A computer-implemented method of providing data transformation includes installing one or more data transformation plugins in a dataset made accessible for processing an end user's workload. A dataset-specific policy for the accessible dataset is ingested. A data transformation of the accessible dataset is executed by invoking one or more of the data transformation plugins to the accessible dataset based on the dataset-specific policy to generate a transformed dataset. The user's workload is deployed to provide data access for processing using the transformed dataset in accordance with a data governance policy.

Abstract: A computer implemented method for optimizing performance of workflow includes associating each of a plurality of workflow nodes in a workflow with a data cache and managing the data cache on a local storage device on one of one or more compute nodes. A scheduler can request execution of the tasks of a given one of the plurality of workflow nodes on one of the one of more compute nodes that hosts the data cache associated with the given one of the plurality of workflow nodes. Each of the plurality of workflow nodes is permitted to access a distributed filesystem that is visible to each of the plurality of compute nodes. The data cache stores data produced by the tasks of the given one of the plurality of workflow nodes.

Abstract: Embodiments for managing distributed computing systems are provided. Information associated with operation of a computing node within a distributed computing system is collected. A reliability score for the computing node is calculated based on the collected information. The calculating of the reliability score is performed utilizing the computing node. A remedial action associated with the operation of the computing node is caused to be performed based on the calculated reliability score.

Abstract: Embodiments for providing intelligent data replication and distribution in a computing environment. Data access patterns of one or more queries issued to a plurality of data partitions may be forecasted. Data may be dynamically distributed and replicated to one or more existing data partitions or additional of the plurality of data partitions according to the forecasting.

Abstract: Embodiments for managing distributed computing systems are provided. Information associated with operation of a computing node within a distributed computing system is collected. A reliability score for the computing node is calculated based on the collected information. The calculating of the reliability score is performed utilizing the computing node. A remedial action associated with the operation of the computing node is caused to be performed based on the calculated reliability score.