Abstract: A computer implemented method, computer program product, and system for managing execution of a workflow comprising a set of subworkflows, comprising optimizing the set of subworkflows using a deep neural network, wherein each subworkflow of the set of subworkflows has a set of tasks, wherein each task of the sets of tasks has a requirement of resources of a set of resources; wherein each task of the sets of tasks is enabled to be dependent on another task of the sets of tasks, training the deep neural network by: executing the set of subworkflows, collecting provenance data from the execution, and collecting monitoring data that represents the state of said set of resources, wherein the training causes the neural network to learn relationships between the states of said set of resources, the said sets of tasks, their parameters and the obtained performance, optimizing an allocation of resources of the set of resources to each task of the sets of tasks to ensure compliance with a user-defined quality metric b

Abstract: Techniques are provided for allocation of shared computing resources using source code feature extraction and machine learning techniques. An exemplary method comprises obtaining source code for execution in a shared computing environment; extracting a plurality of discriminative features from the source code; obtaining a trained machine learning model; and generating a prediction of an allocation of one or more resources of the shared computing environment needed to satisfy one or more service level agreement requirements for the source code. The generated prediction is optionally adjusted using a statistical analysis of an error curve, based on one or more error boundaries obtained by the trained machine learning model.

Abstract: Deep reinforcement learning techniques are provided for resource allocation in a shared computing environment.

Abstract: Techniques are provided for adaptive look-ahead configuration for data prefetching. One method comprises, in response to a request for a data item in a storage system: obtaining a size of a look-ahead window for the request based on one of multiple available caching policies; and moving the requested data item and additional data items within the look-ahead window to the cache memory when the requested data item and/or the additional data items within the look-ahead window are not in the cache memory. The multiple available caching policies comprise a caching policy based on characteristics of an input/output workload of the storage system, or a portion thereof; and/or a caching policy based on an input/output workload of at least a portion of the storage system within a prior predefined time window. The look-ahead window size may be varied over time.

Abstract: Techniques are provided for provenance-based software script reuse. One method comprises extracting provenance data from source code including, for example, source code fragments, wherein the extracted provenance data indicates a control flow and a data flow of the source code; encapsulating source code fragments from the source code that satisfy one or more similarity criteria as a reusable source code fragment; and providing a repository of encapsulated reusable source code fragments for reuse during a development of new software scripts. The repository of encapsulated reusable source code fragments optionally comprises a searchable database further including, for example, the provenance data, data annotations, input parameters and generated results for the corresponding source code fragment.

Abstract: A computer implemented method, computer program product, and system for managing execution of a workflow comprising a set of subworkflows, comprising optimizing the set of subworkflows using a deep neural network, wherein each subworkflow of the set of subworkflows has a set of tasks, wherein each task of the sets of tasks has a requirement of resources of a set of resources; wherein each task of the sets of tasks is enabled to be dependent on another task of the sets of tasks, training the deep neural network by: executing the set of subworkflows, collecting provenance data from the execution, and collecting monitoring data that represents the state of said set of resources, wherein the training causes the neural network to learn relationships between the states of said set of resources, the said sets of tasks, their parameters and the obtained performance, optimizing an allocation of resources of the set of resources to each task of the sets of tasks to ensure compliance with a user-defined quality metric b

Abstract: Techniques are provided for allocating shared computing resources using source code feature extraction and cluster-based training of machine learning models. An exemplary method comprises: obtaining a source code corpus with source code segments for execution in a shared computing environment; extracting discriminative features from the source code segments in the source code corpus; obtaining a trained machine learning model, wherein the trained machine learning model is trained using samples of source code segments from clusters derived from clustering the source code corpus based on (i) a term frequency metric, and/or (ii) observed values of execution metrics; and generating, using the trained machine learning model, a prediction of an allocation of resources of the shared computing environment needed to satisfy service level agreement requirements for source code to be executed in the shared computing environment.

Abstract: One example method includes determining a fall through threshold value for a cache, computing a length ‘s’ of a sequence that is close to LRU eviction, and the length ‘s’ is computed when a current fall through metric value is greater than the fall through threshold value, when the sequence length ‘s’ is greater than a predetermined threshold length ‘k,’ performing a first shift of an LRU position to define a protected queue of the cache, initializing a counter with a value of ‘r’, decrementing the counter each time a requested page is determined to be included in the protected queue, until ‘r’=0, and performing a second shift of the LRU position.

Abstract: Managing a cache memory in a storage system includes maintaining a queue that stores data indictive of the read requests for a particular logical storage unit of the storage system in an order that the read requests are received by the storage system, receiving a read request for a particular page of the particular logical storage unit, and removing a number of elements in the queue and resizing the queue in response to the queue being full. Managing the cache memory also includes placing data indicative of the read request in the queue, determining a prefetch metric that varies according to a number of adjacent elements in a sorted version of the queue having a difference that is less than a predetermined value and greater than zero, and prefetching a plurality of pages that come after the particular page sequentially if the prefetch metric is greater than a predefined value.

Abstract: A computer implemented method, computer program product, and system for managing execution of a workflow comprising a set of subworkflows, comprising optimizing the set of subworkflows using a deep neural network, wherein each subworkflow of the set of subworkflows has a set of tasks, wherein each task of the sets of tasks has a requirement of resources of a set of resources; wherein each task of the sets of tasks is enabled to be dependent on another task of the sets of tasks, training the deep neural network by: executing the set of subworkflows, collecting provenance data from the execution, and collecting monitoring data that represents the state of said set of resources, wherein the training causes the neural network to learn relationships between the states of said set of resources, the said sets of tasks, their parameters and the obtained performance, optimizing an allocation of resources of the set of resources to each task of the sets of tasks to ensure compliance with a user-defined quality metric b

Abstract: One example method includes determining a fall through threshold value for a cache, computing a length ‘s’ of a sequence that is close to LRU eviction, and the length ‘s’ is computed when a current fall through metric value is greater than the fall through threshold value, when the sequence length ‘s’ is greater than a predetermined threshold length ‘k,’ performing a first shift of an LRU position to define a protected queue of the cache, initializing a counter with a value of ‘r’, decrementing the counter each time a requested page is determined to be included in the protected queue, until ‘r’=0, and performing a second shift of the LRU position.

Abstract: Managing a cache memory in a storage system includes maintaining a first queue that stores data indictive of the read requests for a particular logical storage unit of the storage system in an order that the read requests are received by the storage system and maintaining a second queue that stores data indictive of the read requests for the particular logical storage unit in a sort order corresponding to page numbers of the read requests, the second queue persisting for a plurality of iterations of read requests. A read request is received and data indicative of the read request is placed in the first queue and in the second queue while maintaining the sort order of the second queue. The second queue is used to determine a prefetch metric that varies according to a number of adjacent elements in the second queue.

Abstract: Methods and apparatus are provided for process monitoring based on large-scale combinations of time series data. An exemplary method comprises generating a model from time series data for a given target time series; determining whether a first difference between measured values and predicted values based on the model exceeds a predefined threshold indicating a target prediction error; in response to a detected target prediction error, performing evaluations of (i) a neighborhood coherence comprising an average of variables of the model weighted by corresponding coefficients on a predefined neighborhood time window, and/or (ii) a second difference between a given value of at least one variable in the model and an average value of the at least one variable based on a training dataset; providing notifications when first predefined criteria based on the evaluations are satisfied; and updating the model when second predefined criteria based on the evaluations are satisfied.

Abstract: Product planning techniques are provided that recommend compositions of product features for weighted heterogeneous consumer segments using regression trees. An exemplary method comprises obtaining historical consumer data comprising product preferences for existing product items for multiple consumer segments; obtaining product features indicating characteristics for each existing product item; prioritizing the consumer segments by obtaining a weight indicating an interest in each consumer segment; computing a total performance metric, for each product item, by calculating a dot product between the consumer segment weights and respective preferences of the consumer segments regarding a given product item; obtaining a regression tree from the existing product items to predict the total performance metric in terms of corresponding product features; and selecting a combination of the product features to be used in future product items based on identified paths in the regression tree.

Abstract: Techniques are provided for simulation-based online workflow optimization.

Abstract: Managing a cache memory in a storage system includes maintaining a first queue that stores data indictive of the read requests for a particular logical storage unit of the storage system in an order that the read requests are received by the storage system and maintaining a second queue that stores data indictive of the read requests for the particular logical storage unit in a sort order corresponding to page numbers of the read requests, the second queue persisting for a plurality of iterations of read requests. A read request is received and data indicative of the read request is placed in the first queue and in the second queue while maintaining the sort order of the second queue. The second queue is used to determine a prefetch metric that varies according to a number of adjacent elements in the second queue.

Abstract: Managing a cache memory in a storage system includes maintaining a queue that stores data indictive of the read requests for a particular logical storage unit of the storage system in an order that the read requests are received by the storage system, receiving a read request for a particular page of the particular logical storage unit, and removing a number of elements in the queue and resizing the queue in response to the queue being full. Managing the cache memory also includes placing data indicative of the read request in the queue, determining a prefetch metric that varies according to a number of adjacent elements in a sorted version of the queue having a difference that is less than a predetermined value and greater than zero, and prefetching a plurality of pages that come after the particular page sequentially if the prefetch metric is greater than a predefined value.

Abstract: A pre-fetching technique determines what data, if any, to pre-fetch on a per-logical storage unit basis. For a given logical storage unit, what, if any, data to prefetch is based at least in part on a collective sequential proximity of the most recently requested pages of the logical storage unit. Determining what, if any, data to pre-fetch for a logical storage unit may include determining a value for a proximity metric indicative of the collective sequential proximity of the most recently requested pages, comparing the value to a predetermined proximity threshold value, and determining whether to pre-fetch one or more pages of the logical storage unit based on the result of the comparison. A data structure may be maintained that includes most recently requested pages for one or more logical storage units. This data structure may be a table.

Abstract: Methods and apparatus are provided for taking snapshots to train prediction models and improve workflow execution.

Abstract: Techniques are provided for dataflow execution time estimation for distributed processing frameworks. An exemplary method comprises: obtaining an input dataset for a dataflow for execution; determining a substantially minimal data unit for a given operation of the dataflow processed by the given operation; estimating a number of rounds required to execute a number of data units in the input dataset using nodes assigned to execute the given operation; determining an execution time spent by the given operation to process one data unit; estimating the execution time for the given operation based on the execution time spent by the given operation to process one data unit and the number of rounds required to execute the number of data units in the input dataset; and executing the given operation with the input dataset. A persistent cost model is optionally employed to record the execution times of known dataflow operations.