Abstract: A system includes a computer(s) coupled to a data storage device(s) that stores a training data repository and a predictive model repository. The training data repository includes retained data samples from initial training data and from previously received data sets. The predictive model repository includes at least one updateable trained predictive model that was trained with the initial training data and retrained with the previously received data sets. A new data set is received. A richness score is assigned to each of the data samples in the set and to the retained data samples that indicates how information rich a data sample is for determining accuracy of the trained predictive model. A set of test data is selected based on ranking by richness score the retained data samples and the new data set. The trained predictive model is accuracy tested using the test data and an accuracy score determined.

Abstract: A system includes a computer(s) coupled to a data storage device(s) that stores a training function repository and a predictive model repository that includes includes updateable trained predictive models each associated with an accuracy score. A series of training data sets are received, being training samples each having output data that corresponds to input data. The training data is different from initial training data that was used with training functions from the repository to train the predictive models initially. Upon receiving a first training data set included in the series and for each predictive model in the repository, the input data in the first training set is used to generate predictive output data that is compared to the output data. Based on the comparison and previous comparisons determined from the initial training data and from previously received training data sets, an updated accuracy score for each predictive model is determined.

Abstract: Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training and retraining predictive models. A series of training data sets for predictive modeling can be received, e.g., over a network from a client computing system. The training data included in the training data sets is different from initial training data that was used with multiple training functions to train multiple trained predictive models stored in a predictive model repository. The series of training data sets are used with multiple trained updateable predictive models obtained from the predictive model repository and multiple training functions to generate multiple retrained predictive models. An effectiveness score is generated for each of the retrained predictive models. A first trained predictive model is selected from among the trained predictive models included in the predictive model repository and the retrained predictive models based on their respective effectiveness scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for suggesting training examples. In one aspect, a method includes receiving a plurality of training examples. A plurality of different types of predictive models are trained using the received training examples, wherein each of the predictive models implements a different machine learning technique. The performance of each trained model is measured. A suggestion score is computed for each training example according to each respective trained model, including weighting each suggestion score by the measured performance of the respective trained model. The computed suggestion scores for each training example are combined to compute an overall suggestion score for each training example, and the training examples are ranked by suggestion scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training and retraining predictive models. A series of training data sets are received and added to a training data queue. In response to a first condition being satisfied, multiple retrained predictive models are generated using the training data queue, multiple updateable trained predictive models obtained from a repository of trained predictive models, and multiple training functions. In response to a second condition being satisfied, multiple new trained predictive models are generated using the training data queue, at least some training data stored in a training data repository and training functions. The new trained predictive models include static trained predictive models and updateable trained predictive models. The repository of trained predictive models is updated with at least some of the retrained predictive models and new trained predictive models.

Abstract: Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training and retraining predictive models. A series of training data sets for predictive modeling can be received, e.g., over a network from a client computing system. The training data included in the training data sets is different from initial training data that was used with multiple training functions to train multiple trained predictive models stored in a predictive model repository. The series of training data sets are used with multiple trained updateable predictive models obtained from the predictive model repository and multiple training functions to generate multiple retrained predictive models. An effectiveness score is generated for each of the retrained predictive models. A first trained predictive model is selected from among the trained predictive models included in the predictive model repository and the retrained predictive models based on their respective effectiveness scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for utilizing predictive models from an application scripting language.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for obtaining a plurality of representations of predictive models, each representation having been received from a different client wherein each representation is associated with a respective user and comprises a description of a respective predictive model, and selecting a model implementation from a plurality of model implementations for each of the obtained representations.

Abstract: Methods, systems, and apparatus, including computer programs stored on a computer storage medium, for training predictive models using large datasets.