Abstract: Parallel training of a machine learning model on a computerized system may be provided. Computing tasks can be assigned to multiple workers of a system. A method may include accessing training data. A parallel training of the machine learning model can be started based on the accessed training data, so as for the training to be distributed through a first number K of workers, K>1. Responsive to detecting a change in a temporal evolution of a quantity indicative of a convergence rate of the parallel training (e.g., where said change reflects a deterioration of the convergence rate), the parallel training of the machine learning model is scaled-in, so as for the parallel training to be subsequently distributed through a second number K? of workers, where K>K??1. Related computerized systems and computer program products may be provided.

Abstract: In an approach for constructing an ensemble model from a set of base learners, a processor performs a plurality of boosting iterations, where: at each boosting iteration of the plurality of boosting iterations, a base learner is selected at random from a set of base learners, according to a sampling probability distribution of the set of base learners, and trained according to a training dataset; and the sampling probability distribution is altered: (i) after selecting a first base learner at a first boosting iteration of the plurality of boosting iterations and (ii) prior to selecting a second base learner at a final boosting iteration of the plurality of boosting iterations. A processor constructs an ensemble model based on base learners selected and trained during the plurality of boosting iterations.

Abstract: A method of dynamically optimizing decision tree inference is provided. The method, which is performed at the computerized system, repeatedly executes one or more decision trees for inference purposes and repeatedly performs an optimization procedure according to two-phase cycles. Each cycle includes two alternating phases, i.e., a first phase followed by a second phase. The decision trees are executed based on a reference data structure, whereby attributes of nodes of the decision trees are repeatedly accessed from the reference data structure during the first phase of each of the cycles. First, the accessed attributes are monitored during the first phase of each cycle, which leads to update statistical characteristics of the nodes. Second, a substitute data structure is configured during the second phase of each cycle based on the updated statistical characteristics. Third, the reference data structure is updated in accordance with the substitute data structure.

Abstract: A continuous machine learning system includes a data generator module, a pipeline search module, a pipeline refinement module, and a pipeline training module. The data generator module obtains raw training data defining a total data size and generates a plurality of data batches from the raw training data. The pipeline search module obtains an initial data batch from among the plurality of data batches and determines a best machine learning model pipeline among a plurality of machine learning model pipelines based on the initial data batch. The pipeline refinement module receives the best machine learning model pipeline and refines the best machine learning model pipeline to generate a refined pipeline that consumes the plurality of data batches. The pipeline training module incrementally trains the refined pipeline using remaining data batches among the plurality of data batches generated after the initial data batch.

Abstract: The invention is notably directed to a computer-implemented method, which aims at jointly identifying an optimal source of computerized resources and optimizing a configuration of the computerized resources. The method comprises configuring a Best-Arm Identification algorithm, in order to (i) associate arms of the algorithm with respective sources of computerized resources and (ii) connect the arms to one or more optimizers. Each of the optimizers is designed to optimize a configuration of such computerized resources. Next, the method iteratively executes the Best-Arm Identification algorithm to progressively eliminate the sources, with a view to eventually identifying one of the sources as an optimal source with an optimized configuration. Several iterations are accordingly performed. During each iteration, each of the arms is pulled and the rewards earned by pulling the arms are computed. Pulling each arm causes to optimize a configuration of computerized resources of a respectively associated source.

Abstract: A tensor representation of a machine learning inferences to be performed is built by forming complementary tensor subsets that respectively correspond to complementary subsets of one or more leaf nodes of one or more decision trees based on statistics of the one or more leaf nodes of the one or more decision trees and data capturing attributes of one or more split nodes of the one or more decision trees and the one or more leaf nodes of the decision trees. The complementary tensor subsets are ranked such that a first tensor subset and a second tensor subset of the complementary tensor subsets correspond to a first leaf node subset and a second leaf node subset of the complementary subsets of the one or more leaf nodes.

Abstract: Accessing a value M identifying M top levels of one or more N decision trees, wherein 1 ? M < Min(L1, ...., LN) and wherein a M top levels defines top nodes for each of the N decision trees, and wherein for each decision tree Ti of the N decision trees. Identifying one or more subtrees subtended by respective subsets of remaining nodes of each decision tree Ti, a remaining nodes including all of the nodes of said each decision tree Ti but its top nodes. Processing each of the K input records through a top nodes of said each decision tree Ti to associate each of the K input records with a single, respective one of the subtrees of each decision tree Ti, wherein K × N associations are obtained in total for the N decision trees and the K input records.

Abstract: Parallel training of a machine learning model on a computerized system is described. Computing tasks of a system can be assigned to multiple workers of the system. Training data can be accessed. The machine learning model is trained, whereby the training data accessed are dynamically partitioned across the workers of the system by shuffling subsets of the training data through the workers. As a result, different subsets of the training data are used by the workers over time as training proceeds. Related computerized systems and computer program products are also provided.

Abstract: Embodiments of the present invention provide computer-implemented methods, computer program products and systems. Embodiments of the present invention can run preemptable tasks distributed according to a distributed environment, wherein each task of a plurality of preemptable tasks has been assigned two or more of the training data samples to process during each iteration. Embodiments of the present invention can, upon verifying that a preemption condition for each iteration is satisfied: preempt any task of the preemptable tasks that have started processing training data samples assigned to it, and update the cognitive model based on outputs obtained from completed tasks, including outputs obtained from both the preempted tasks and completed tasks that have finished processing all training data samples as assigned to it.

Abstract: A system may be configured to perform operations to select a machine learning model. The operations may include training machine learning models with training data of a training data set and obtaining a first value representing a first required runtime for training each machine learning model. The operations may include evaluating, based on the first value, a second value representing a second required runtime for training the machine learning model with a complete training data set. The operations may include calculating a final score for each machine learning model in a group of machine learning models, wherein the calculating is performed on a basis of the second values for the machine learning models, ranking the machine learning models based on the final score to obtain ranks, and selecting the machine learning model that has obtained a highest rank in the ranking.

Abstract: Methods are provided for implementing training of a machine learning model in a processing system, together with systems for performing such methods. A method includes providing a core module for effecting a generic optimization process in the processing system, and in response to a selective input, defining a set of derivative modules, for effecting computation of first and second derivatives of selected functions ƒ and g in the processing system, to be used with the core module in the training operation. The method further comprises performing, in the processing system, the generic optimization process effected by the core module using derivative computations effected by the derivative modules.

Abstract: An approach of accelerating inferences based on decision trees based on accessing one or more decision trees, wherein each decision tree of the decision trees accessed comprises decision tree nodes, including nodes grouped into one or more supersets of nodes designed for joint execution. For each decision tree of the decision trees accessed, the nodes are executed to obtain an outcome for the one or more decision trees, respectively. For each superset of the one or more supersets of said each decision tree, the nodes of each superset are jointly executed by: loading attributes of the nodes of each superset in a respective cache line of the cache memory processing said attributes from the respective cache line until an inference result is returned based on the one or more outcomes.

Abstract: According to one embodiment, a method, computer system, and computer program product for training a cognitive model that involves one or more decision trees as base learners is provided. The present invention may include constructing, by a tree building algorithm, the one or more decision trees, wherein the constructing further comprises associating one or more training examples with one or more leaf nodes of the one or more decision trees and iteratively running a breadth-first search tree builder on one or more of the decision trees to perform one or more tree building operations; and training the cognitive model based on the one or more decision trees.

Abstract: Computer-implemented methods are provided for implementing training of a machine learning model in a heterogeneous processing system comprising a host computer operatively interconnected with an accelerator unit. The training includes a stochastic optimization process for optimizing a function of a training data matrix X, having data elements Xi,j with row coordinates i=1 to n and column coordinates j=1 to m, and a model vector w having elements wj. For successive batches of the training data, defined by respective subsets of one of the row coordinates and column coordinates, random numbers associated with respective coordinates in a current batch b are generated in the host computer and sent to the accelerator unit. In parallel with generating the random numbers for batch b, batch b is copied from the host computer to the accelerator unit.

Abstract: Computer-implemented methods are provided for implementing training of a machine learning model in a heterogeneous processing system that includes a host computer operatively interconnected to an accelerator unit. The training operation involves an iterative optimization process for optimizing a model vector defining the model. Such a method includes, in the host computer, storing a matrix of training data and partitioning the matrix into a plurality of batches of data vectors. For each of successive iterations of the optimization process, a selected subset of the batches is provided to the accelerator unit. In the accelerator unit, each iteration of the optimization process is performed to update the model vector in dependence on vectors in the selected subset for that iteration. In the host computer, batch importance values are calculated for respective batches. The batch importance value is dependent on contributions of vectors in that batch to sub-optimality of the model vector.

Abstract: The present invention is notably directed to a computer-implemented method of training a cognitive model. The cognitive model includes decision trees as base learners. The method is performed using processing means to which a given cache memory is connected, so as to train the cognitive model based on training examples of a training dataset. The cognitive model is trained by running a hybrid tree building algorithm, so as to construct the decision trees and thereby associate the training examples to leaf nodes of the constructed decision trees, respectively. The hybrid tree building algorithm involves a first routine and a second routine. Each routine is designed to access the cache memory upon execution. The first routine involves a breadth-first search tree builder, while the second routine involves a depth-first search tree builder.

Abstract: Embodiments of the present invention provide computer-implemented methods, computer program products and systems. Embodiments of the present invention can run preemptable tasks distributed according to a distributed environment, wherein each task of a plurality of preemptable tasks has been assigned two or more of the training data samples to process during each iteration. Embodiments of the present invention can, upon verifying that a preemption condition for each iteration is satisfied: preempt any task of the preemptable tasks that have started processing training data samples assigned to it, and update the cognitive model based on outputs obtained from completed tasks, including outputs obtained from both the preempted tasks and completed tasks that have finished processing all training data samples as assigned to it.

Abstract: In an approach for constructing an ensemble model from a set of base learners, a processor performs a plurality of boosting iterations, where: at each boosting iteration of the plurality of boosting iterations, a base learner is selected at random from a set of base learners, according to a sampling probability distribution of the set of base learners, and trained according to a training dataset; and the sampling probability distribution is altered: (i) after selecting a first base learner at a first boosting iteration of the plurality of boosting iterations and (ii) prior to selecting a second base learner at a final boosting iteration of the plurality of boosting iterations. A processor constructs an ensemble model based on base learners selected and trained during the plurality of boosting iterations.

Abstract: A method for parallelizing a training of a model using a matrix-factorization-based collaborative filtering algorithm may be provided. The model can be used in a recommender system for a plurality of users and a plurality of items. The method includes providing a sparse training data matrix, selecting a number of user-item co-clusters, and building a user model data matrix by matrix factorization such that a computational load for executing the determining updated elements of the factorized sparse training data matrix is evenly distributed across the heterogeneous computing resources.

Abstract: Parallel training of a machine learning model on a computerized system is described. Computing tasks of a system can be assigned to multiple workers of the system. Training data can be accessed. The machine learning model is trained, whereby the training data accessed are dynamically partitioned across the workers of the system by shuffling subsets of the training data through the workers. As a result, different subsets of the training data are used by the workers over time as training proceeds. Related computerized systems and computer program products are also provided.