Abstract: Methods, systems, and apparatus, for selecting a trained predictive models. A request is received from a client-subscriber computing system for access to a trained predictive model that can generate a predictive output in response to receiving input data having one or more input types. Information that describes each of the trained predictive models in a predictive model repository can be used to determine that one or more models included in the repository match the request. Determining a match can be based (at least in part) on a comparison of the one or more input types to input types included in the information that describes the trained predictive models. Access is provided to at least one of the models to the client-subscriber computing system. The models that match the request are models that were trained using training data provided by a computing system other than the client-subscriber computing system.

Abstract: Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for receiving training data for predictive modeling and executing multiple processes simultaneously to generate multiple trained predictive models using the training data and training functions. After executing the processes for an initial runtime, a convergence status of each process is determined that indicates a likelihood that the training function being executed will converge on the training data. Based on the determination, training functions are identified that are not likely to converge and processes that are executing these training functions are terminated. After an ultimate runtime has expired, processes that are still executing training functions that have not yet converged are terminated. An effectiveness score is generated for each of the trained predictive models that were successfully generated and a trained predictive model is selected based on the effectiveness scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training a predictive model. In one aspect, a method includes receiving over a network predictive modeling training data from a client computing system. The training data and multiple training functions obtained from a repository of training functions are used to train multiple predictive models. A score is generated for each of the trained predictive models, where each score represents an estimation of the effectiveness of the respective trained predictive model. A first trained predictive model is selected from among the trained predictive models based on the generated scores. Access to the first trained predictive model is provided over the network.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for score normalization. One of the methods includes receiving initial training data, the initial training data comprising initial training records, each initial training record identifying input data as input and a category as output. The method includes generating a first trained predictive model using the initial training data and a training function. The method includes generating intermediate training records by inputting input data of the initial training records to a second trained predictive model, the second trained predictive model generated using the training function, each intermediate training record having a score. The method also includes generating a score normalization model using a score normalization training function and the intermediate training records.

Abstract: A computer-implemented method includes obtaining a database table, the table including multiple rows and multiple columns, in which one or more rows are missing at least one column value, executing a script, using a script engine, in response to obtaining the table, in which executing the script causes one or more values from the rows to be provided as input data to a first predictive model, and processing, using the first predictive model, the input data to obtain output data, the output data including a predicted value for at least one of the missing column values, and populating one or more of the missing column values with the output data to provide a revised database table.

Abstract: Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training a predictive model. In one aspect, a method includes receiving over a network predictive modeling training data from a client computing system. The training data and multiple training functions obtained from a repository of training functions are used to train multiple predictive models. A score is generated for each of the trained predictive models, where each score represents an estimation of the effectiveness of the respective trained predictive model. A first trained predictive model is selected from among the trained predictive models based on the generated scores. Access to the first trained predictive model is provided to the client computing system.

Abstract: A method can include the actions of: receiving a feature vector, the feature vector including one or more elements; identifying an element type for each of the one or more elements; selecting, from a set of predictive models, a subset of one or more predictive models based on the element types and one or more performance indicators associated with each predictive model in the set of predictive models; processing the feature vector using the subset of predictive models, each predictive model of the subset of predictive models generating an output based on the feature vector to provide a plurality of outputs; and generating a final output based on the plurality of outputs. Other embodiments may include corresponding systems, apparatus, and computer program products for executing the method.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for score normalization. One of the methods includes receiving initial training data, the initial training data comprising initial training records, each initial training record identifying input data as input and a category as output. The method includes generating a first trained predictive model using the initial training data and a training function. The method includes generating intermediate training records by inputting input data of the initial training records to a second trained predictive model, the second trained predictive model generated using the training function, each intermediate training record having a score. The method also includes generating a score normalization model using a score normalization training function and the intermediate training records.

Abstract: Methods include the actions of storing a first predictive model in computer-readable memory, the first predictive model having been defined based on a first training dataset provided by an owner of the first predictive model and being operable to generate an output based on a query, enabling access to the first predictive model based on permissions defined by the owner, while inhibiting access to the first training dataset, receiving a second training dataset from a user, the second training dataset being distinct from the first training dataset, modifying the first predictive model based on the second training dataset to provide a second predictive model, storing the second predictive model in computer-readable memory, and enabling access to the second predictive model.

Abstract: Methods, systems, and apparatus, for selecting a trained predictive models. A request is received from a client-subscriber computing system for access to a trained predictive model that can generate a predictive output in response to receiving input data having one or more input types. Information that describes each of the trained predictive models in a predictive model repository can be used to determine that one or more models included in the repository match the request. Determining a match can be based (at least in part) on a comparison of the one or more input types to input types included in the information that describes the trained predictive models. Access is provided to at least one of the models to the client-subscriber computing system. The models that match the request are models that were trained using training data provided by a computing system other than the client-subscriber computing system.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for utilizing predictive models from an application scripting language.

Abstract: A system includes a computer(s) coupled to a data storage device(s) that stores a training data repository and a predictive model repository. The training data repository includes retained data samples from initial training data and from previously received data sets. The predictive model repository includes at least one updateable trained predictive model that was trained with the initial training data and retrained with the previously received data sets. A new data set is received. A richness score is assigned to each of the data samples in the set and to the retained data samples that indicates how information rich a data sample is for determining accuracy of the trained predictive model. A set of test data is selected based on ranking by richness score the retained data samples and the new data set. The trained predictive model is accuracy tested using the test data and an accuracy score determined.

Abstract: A system includes a computer(s) coupled to a data storage device(s) that stores a training function repository and a predictive model repository that includes includes updateable trained predictive models each associated with an accuracy score. A series of training data sets are received, being training samples each having output data that corresponds to input data. The training data is different from initial training data that was used with training functions from the repository to train the predictive models initially. Upon receiving a first training data set included in the series and for each predictive model in the repository, the input data in the first training set is used to generate predictive output data that is compared to the output data. Based on the comparison and previous comparisons determined from the initial training data and from previously received training data sets, an updated accuracy score for each predictive model is determined.

Abstract: Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training and retraining predictive models. A series of training data sets for predictive modeling can be received, e.g., over a network from a client computing system. The training data included in the training data sets is different from initial training data that was used with multiple training functions to train multiple trained predictive models stored in a predictive model repository. The series of training data sets are used with multiple trained updateable predictive models obtained from the predictive model repository and multiple training functions to generate multiple retrained predictive models. An effectiveness score is generated for each of the retrained predictive models. A first trained predictive model is selected from among the trained predictive models included in the predictive model repository and the retrained predictive models based on their respective effectiveness scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for suggesting training examples. In one aspect, a method includes receiving a plurality of training examples. A plurality of different types of predictive models are trained using the received training examples, wherein each of the predictive models implements a different machine learning technique. The performance of each trained model is measured. A suggestion score is computed for each training example according to each respective trained model, including weighting each suggestion score by the measured performance of the respective trained model. The computed suggestion scores for each training example are combined to compute an overall suggestion score for each training example, and the training examples are ranked by suggestion scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training and retraining predictive models. A series of training data sets are received and added to a training data queue. In response to a first condition being satisfied, multiple retrained predictive models are generated using the training data queue, multiple updateable trained predictive models obtained from a repository of trained predictive models, and multiple training functions. In response to a second condition being satisfied, multiple new trained predictive models are generated using the training data queue, at least some training data stored in a training data repository and training functions. The new trained predictive models include static trained predictive models and updateable trained predictive models. The repository of trained predictive models is updated with at least some of the retrained predictive models and new trained predictive models.

Abstract: Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training and retraining predictive models. A series of training data sets for predictive modeling can be received, e.g., over a network from a client computing system. The training data included in the training data sets is different from initial training data that was used with multiple training functions to train multiple trained predictive models stored in a predictive model repository. The series of training data sets are used with multiple trained updateable predictive models obtained from the predictive model repository and multiple training functions to generate multiple retrained predictive models. An effectiveness score is generated for each of the retrained predictive models. A first trained predictive model is selected from among the trained predictive models included in the predictive model repository and the retrained predictive models based on their respective effectiveness scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for utilizing predictive models from an application scripting language.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for obtaining a plurality of representations of predictive models, each representation having been received from a different client wherein each representation is associated with a respective user and comprises a description of a respective predictive model, and selecting a model implementation from a plurality of model implementations for each of the obtained representations.

Abstract: Methods, systems, and apparatus, including computer programs stored on a computer storage medium, for training predictive models using large datasets.