Abstract: Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training predictive models. Multiple training data records are received that each include an input data portion and an output data portion. A training data type is determined that corresponds to the training data. For example, a training data type can be determined by inputting the output data portions into one or more trained predictive classifiers. In other example, the training data type can be determined by comparison of the output data portions to data formats. Based on the determined training data type, a set of training functions are identified that are compatible with the training data of the determined training data type. The training data and the identified set of training functions are used to train multiple predictive models.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for multi-label models. One of the methods includes receiving training records, each training record having an input, a first output, and a second output. The method includes generating a first classifier using as input one of the inputs and using as output a corresponding one of the first outputs. The method includes generating a second classifier using as input one of the inputs and using as output a corresponding one of the second outputs. The method includes inputting the inputs into the first classifier and generating first predictive outputs. The method includes inputting the inputs into the second classifier and generating second predictive outputs. The method also includes generating a third classifier using as input the first output and the second output and using as output the first output and the second output of the corresponding training record.

Abstract: A computer-implemented method includes obtaining a database table, the database table including data arranged in a plurality of rows and a plurality of columns, each column of data being associated with a different tag that specifies a category for data in the column, using one or more processors to identify a first predictive model, from a collection of predictive models, that can be applied to the database table to generate a predictive output, in which identifying the first predictive model is based on one or more of the different tags, adding a name associated with the first predictive model to a set of names of predictive models that are compatible with the database table, and providing the set of names of predictive models to a client device.

Abstract: Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training and retraining predictive models. A series of training data sets are received and added to a training data queue. In response to a first condition being satisfied, multiple retrained predictive models are generated using the training data queue, multiple updateable trained predictive models obtained from a repository of trained predictive models, and multiple training functions. In response to a second condition being satisfied, multiple new trained predictive models are generated using the training data queue, at least some training data stored in a training data repository and training functions. The new trained predictive models include static trained predictive models and updateable trained predictive models. The repository of trained predictive models is updated with at least some of the retrained predictive models and new trained predictive models.

Abstract: Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for caching predictive models are described. Records are obtained, each record including a time of a previously submitted predictive request and an identifier of a trained predictive model. A trained scheduling model is generated using the records as training data. A set of identifiers of trained predictive models are determined from a plurality of trained predictive models that are stored in a secondary memory of a computing system. The target time is inputted to the trained scheduling model. In response, a second predictive output is received that comprises the set of identifiers. A set of trained predictive models are obtained that correspond to the set of identifiers from the secondary memory. The set of trained predictive models are stored in a primary memory of the computing system.

Abstract: A system includes a computer(s) coupled to a data storage device(s) that stores a training data repository and a predictive model repository. The training data repository includes retained data samples from initial training data and from previously received data sets. The predictive model repository includes at least one updateable trained predictive model that was trained with the initial training data and retrained with the previously received data sets. A new data set is received. A richness score is assigned to each of the data samples in the set and to the retained data samples that indicates how information rich a data sample is for determining accuracy of the trained predictive model. A set of test data is selected based on ranking by richness score the retained data samples and the new data set. The trained predictive model is accuracy tested using the test data and an accuracy score determined.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for suggesting training examples. In one aspect, a method includes receiving a plurality of training examples. A plurality of different types of predictive models are trained using the received training examples, wherein each of the predictive models implements a different machine learning technique. The performance of each trained model is measured. A suggestion score is computed for each training example according to each respective trained model, including weighting each suggestion score by the measured performance of the respective trained model. The computed suggestion scores for each training example are combined to compute an overall suggestion score for each training example, and the training examples are ranked by suggestion scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for obtaining a plurality of model representations of predictive models, each model representation associated with a respective user and expresses a respective predictive model, and selecting a model implementation for each of the model representations based on one or more system usage properties associated with the user associated with the corresponding model representation.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for anomaly detection. One of the methods includes receiving training data for updating an updateable trained first predictive model. The method includes inputting the training data into a trained second predictive model and generating predictive output data. The method includes based on the predictive output data, detecting an anomaly in the training data as compared to previously received training data. The method includes generating a retrained first predictive model based on the updateable trained first predictive model, a training function and training data that includes the received training data and previously received training data, wherein the received training data is weighted differently than the previously received training data based on the detected anomaly.

Abstract: Methods, systems, and apparatus, for selecting a trained predictive models. A request is received from a client-subscriber computing system for access to a trained predictive model that can generate a predictive output in response to receiving input data having one or more input types. Information that describes each of the trained predictive models in a predictive model repository can be used to determine that one or more models included in the repository match the request. Determining a match can be based (at least in part) on a comparison of the one or more input types to input types included in the information that describes the trained predictive models. Access is provided to at least one of the models to the client-subscriber computing system. The models that match the request are models that were trained using training data provided by a computing system other than the client-subscriber computing system.

Abstract: Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for receiving training data for predictive modeling and executing multiple processes simultaneously to generate multiple trained predictive models using the training data and training functions. After executing the processes for an initial runtime, a convergence status of each process is determined that indicates a likelihood that the training function being executed will converge on the training data. Based on the determination, training functions are identified that are not likely to converge and processes that are executing these training functions are terminated. After an ultimate runtime has expired, processes that are still executing training functions that have not yet converged are terminated. An effectiveness score is generated for each of the trained predictive models that were successfully generated and a trained predictive model is selected based on the effectiveness scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for score normalization. One of the methods includes receiving initial training data, the initial training data comprising initial training records, each initial training record identifying input data as input and a category as output. The method includes generating a first trained predictive model using the initial training data and a training function. The method includes generating intermediate training records by inputting input data of the initial training records to a second trained predictive model, the second trained predictive model generated using the training function, each intermediate training record having a score. The method also includes generating a score normalization model using a score normalization training function and the intermediate training records.

Abstract: A computer-implemented method includes obtaining a database table, the table including multiple rows and multiple columns, in which one or more rows are missing at least one column value, executing a script, using a script engine, in response to obtaining the table, in which executing the script causes one or more values from the rows to be provided as input data to a first predictive model, and processing, using the first predictive model, the input data to obtain output data, the output data including a predicted value for at least one of the missing column values, and populating one or more of the missing column values with the output data to provide a revised database table.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for score normalization. One of the methods includes receiving initial training data, the initial training data comprising initial training records, each initial training record identifying input data as input and a category as output. The method includes generating a first trained predictive model using the initial training data and a training function. The method includes generating intermediate training records by inputting input data of the initial training records to a second trained predictive model, the second trained predictive model generated using the training function, each intermediate training record having a score. The method also includes generating a score normalization model using a score normalization training function and the intermediate training records.

Abstract: Methods, systems, and apparatus, for selecting a trained predictive models. A request is received from a client-subscriber computing system for access to a trained predictive model that can generate a predictive output in response to receiving input data having one or more input types. Information that describes each of the trained predictive models in a predictive model repository can be used to determine that one or more models included in the repository match the request. Determining a match can be based (at least in part) on a comparison of the one or more input types to input types included in the information that describes the trained predictive models. Access is provided to at least one of the models to the client-subscriber computing system. The models that match the request are models that were trained using training data provided by a computing system other than the client-subscriber computing system.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for utilizing predictive models from an application scripting language.

Abstract: Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training and retraining predictive models. A series of training data sets for predictive modeling can be received, e.g., over a network from a client computing system. The training data included in the training data sets is different from initial training data that was used with multiple training functions to train multiple trained predictive models stored in a predictive model repository. The series of training data sets are used with multiple trained updateable predictive models obtained from the predictive model repository and multiple training functions to generate multiple retrained predictive models. An effectiveness score is generated for each of the retrained predictive models. A first trained predictive model is selected from among the trained predictive models included in the predictive model repository and the retrained predictive models based on their respective effectiveness scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for suggesting training examples. In one aspect, a method includes receiving a plurality of training examples. A plurality of different types of predictive models are trained using the received training examples, wherein each of the predictive models implements a different machine learning technique. The performance of each trained model is measured. A suggestion score is computed for each training example according to each respective trained model, including weighting each suggestion score by the measured performance of the respective trained model. The computed suggestion scores for each training example are combined to compute an overall suggestion score for each training example, and the training examples are ranked by suggestion scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for utilizing predictive models from an application scripting language.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for obtaining a plurality of representations of predictive models, each representation having been received from a different client wherein each representation is associated with a respective user and comprises a description of a respective predictive model, and selecting a model implementation from a plurality of model implementations for each of the obtained representations.