Patents Assigned to DataRobot, Inc.
  • Patent number: 11176483
    Abstract: Described herein are systems and methods for providing data sets from a constantly changing database to a streaming machine learning component. In one embodiment, a data streaming sub-system receives multiple incoming streams of data sets, in which each stream is generated in real-time by one of multiple data sources. The streaming sub-system sends data sets, on-the-fly as they are received, to storage in the memory of a database, in which there is a linkage between the storage and the time of arrival or the time of storage, of the data sets. The database receives, from a machine learning component, a request to receive data sets according to a particular time or time period. In response to such request, the database identifies such data sets according to the particular time or time period and sends them to the machine learning component.
    Type: Grant
    Filed: May 3, 2017
    Date of Patent: November 16, 2021
    Assignee: DataRobot Inc.
    Inventors: Swaminathan Sundararaman, Nisha Darshi Talagala, Gal Zuckerman
  • Patent number: 10984367
    Abstract: Systems and techniques for predictive data analytics are described. In a method for selecting a predictive model for a prediction problem, the suitabilities of predictive modeling procedures for the prediction problem may be determined based on characteristics of the prediction problem and/or on attributes of the respective modeling procedures. A subset of the predictive modeling procedures may be selected based on the determined suitabilities of the selected modeling procedures for the prediction problem. A resource allocation schedule allocating computational resources for execution of the selected modeling procedures may be generated, based on the determined suitabilities of the selected modeling procedures for the prediction problem. Results of the execution of the selected modeling procedures in accordance with the resource allocation schedule may be obtained. A predictive model for the prediction problem may be selected based on those results.
    Type: Grant
    Filed: May 5, 2017
    Date of Patent: April 20, 2021
    Assignee: DataRobot, Inc.
    Inventors: Jeremy Achin, Thomas DeGodoy, Timothy Owen, Xavier Conort
  • Patent number: 10671916
    Abstract: Described herein are systems and methods for executing efficiently, in real-time, a plurality of machine learning processes. In one embodiment, a computing platform with multiple compute elements receives multiple data streams, each such stream associated with its own respective machine learning process. Each machine learning process is operative to use its data stream as input to train, in real-time, a respective mathematical model. Each of the processes has peaks and dips in processing demands. The system re-allocates, in real-time, compute elements from the processes with lower processing demands to processes with higher processing demands, thereby handling all of the multiple processes on-the-fly, preventing peak demands from causing the system to stall, and reducing overall the computational resources required by the system.
    Type: Grant
    Filed: September 20, 2016
    Date of Patent: June 2, 2020
    Assignee: DataRobot, Inc.
    Inventors: Swaminathan Sundararaman, Lior Khermosh, Gal Zuckerman
  • Publication number: 20200090075
    Abstract: A method for determining the predictive value of a feature may include: (a) performing predictive modeling procedures associated with respective predictive models, wherein performing each modeling procedure includes fitting the associated model to an initial dataset representing an initial prediction problem; (b) determining a first accuracy score of each of the fitted models, representing an accuracy with which the fitted model predicts an outcome of the initial prediction problem; (c) shuffling values of a feature across observations included in the initial dataset, thereby generating a modified dataset representing a modified prediction problem; (d) determining a second accuracy score of each of the fitted models, representing an accuracy with which the fitted model predicts an outcome of the modified prediction problem; and (e) determining a model-specific predictive value of the feature for each of the fitted models based on the first and second accuracy scores of the fitted model.
    Type: Application
    Filed: June 20, 2019
    Publication date: March 19, 2020
    Applicant: DataRobot, Inc.
    Inventors: Jeremy Achin, Thomas DeGodoy, Xavier Conort, Sergey Yurgenson, Mark L. Steadman, Glen Koundry
  • Patent number: 10558924
    Abstract: A predictive modeling method may include obtaining a fitted, first-order predictive model configured to predict values of output variables based on values of first input variables; and performing a second-order modeling procedure on the fitted, first-order model, which may include: generating input data including observations including observed values of second input variables and predicted values of the output variables; generating training data and testing data from the input data; generating a fitted second-order model of the fitted first-order model by fitting a second-order model to the training data; and testing the fitted, second-order model of the first-order model on the testing data. Each observation of the input data may be generated by (1) obtaining observed values of the second input variables, and (2) applying the first-order predictive model to corresponding observed values of the first input variables to generate the predicted values of the output variables.
    Type: Grant
    Filed: October 23, 2017
    Date of Patent: February 11, 2020
    Assignee: DataRobot, Inc.
    Inventors: Jeremy Achin, Thomas DeGodoy, Timothy Owen, Xavier Conort, Sergey Yurgenson, Mark L. Steadman, Glen Koundry, Hon Nian Chua
  • Publication number: 20200034665
    Abstract: Apparatuses, systems, program products, and methods are disclosed for determining validity of machine learning algorithms for datasets. An apparatus includes a primary training module that is configured to train a first machine learning model for a first machine learning algorithm. An apparatus includes a primary validation module that is configured to validate a first machine learning model to generate an error data set. An apparatus includes a secondary training module that is configured to train a second machine learning model for a second machine learning algorithm using an error data set. A second machine learning algorithm may be configured to predict a suitability of a first machine learning model for analyzing an inference data set. An apparatus includes an action module that is configured to trigger an action in response to a predicted suitability of the first machine learning model not satisfying a predetermined suitability threshold.
    Type: Application
    Filed: July 30, 2018
    Publication date: January 30, 2020
    Applicant: DataRobot, Inc.
    Inventors: SINDHU GHANTA, DREW ROSELLI, NISHA TALAGALA, VINAY SRIDHAR, SWAMINATHAN SUNDARARAMAN, LIOR AMAR, LIOR KHERMOSH, BHARATH RAMSUNDAR, SRIRAM SUBRAMANIAN
  • Publication number: 20190377984
    Abstract: Apparatuses, systems, program products, and method are disclosed for detecting suitability of machine learning models for datasets. An apparatus includes a training evaluation module configured to calculate a first statistical data signature for a training data set of a machine learning system using one or more predefined statistical algorithms. An apparatus includes an inference evaluation module configured to calculate a second statistical data signature for an inference data set of a machine learning system using one or more predefined statistical algorithms. An apparatus includes a score module configured to calculate a suitability score describing the suitability of a training data set to an inference data set as a function of a first and a second statistical data signature. An apparatus includes an action module configured to perform an action related to a machine learning system in response to a suitability score satisfying an unsuitability threshold.
    Type: Application
    Filed: June 6, 2018
    Publication date: December 12, 2019
    Applicant: DataRobot, Inc.
    Inventors: SINDHU GHANTA, DREW ROSELLI, NISHA TALAGALA, VINAY SRIDHAR, SWAMINATHAN SUNDARARAMAN, LIOR AMAR, LIOR KHERMOSH, BHARATH RAMSUNDAR, SRIRAM SUBRAMANIAN
  • Patent number: 10496927
    Abstract: A predictive modeling method may include determining a time interval of time-series data; identifying one or more variables of the data as targets; determining a forecast range and a skip range associated with a prediction problem represented by the data; generating training data and testing data from the time-series data; fitting a predictive model to the training data; and testing the fitted model on the testing data. The forecast range may indicate a duration of a period for which values of the targets are to be predicted. The skip range may indicate a temporal lag between the time period corresponding to the data used to make predictions and the time period corresponding to the predictions. The skip range may separate input data subsets representing model inputs from subsets representing model outputs, and separate test data subsets representing model inputs from subsets representing validation data.
    Type: Grant
    Filed: October 23, 2017
    Date of Patent: December 3, 2019
    Assignee: DataRobot, Inc.
    Inventors: Jeremy Achin, Thomas DeGodoy, Timothy Owen, Xavier Conort, Sergey Yurgenson, Mark L. Steadman, Glen Koundry, Peter Prettenhofer
  • Patent number: 10387900
    Abstract: An apparatus has a memory with processor-executable instructions and a processor operatively coupled to the memory. The apparatus receives datasets including time series data points that are descriptive of a feature of a given entity. The processor determines a time series characteristic based on the data content, and selects, based on the determined characteristic, a set of entrant forecasting models from a pool of forecasting models stored in the memory. Next, the processor trains each entrant forecasting model with the time series data points to produce a set of trained entrant forecasting models. The processor executes each trained entrant forecasting model to generate a set of forecasted values indicating estimations of the feature of the given entity. Thereafter the processor selects at least one forecasting model from the set of trained entrant forecasting models based on computed accuracy evaluations performed over the set of forecasted values.
    Type: Grant
    Filed: April 17, 2017
    Date of Patent: August 20, 2019
    Assignee: DataRobot, Inc.
    Inventors: John Bledsoe, Jeff Gabriel, Jason Montgomery, Ryan Sevey, Matt Steinpreis, Craig Vermeer, Ryan West
  • Patent number: 10366346
    Abstract: A method for determining the predictive value of a feature may include: (a) performing predictive modeling procedures associated with respective predictive models, wherein performing each modeling procedure includes fitting the associated model to an initial dataset representing an initial prediction problem; (b) determining a first accuracy score of each of the fitted models, representing an accuracy with which the fitted model predicts an outcome of the initial prediction problem; (c) shuffling values of a feature across observations included in the initial dataset, thereby generating a modified dataset representing a modified prediction problem; (d) determining a second accuracy score of each of the fitted models, representing an accuracy with which the fitted model predicts an outcome of the modified prediction problem; and (e) determining a model-specific predictive value of the feature for each of the fitted models based on the first and second accuracy scores of the fitted model.
    Type: Grant
    Filed: October 21, 2016
    Date of Patent: July 30, 2019
    Assignee: DataRobot, Inc.
    Inventors: Jeremy Achin, Thomas DeGodoy, Xavier Conort, Sergey Yurgenson, Mark L. Steadman, Glen Koundry
  • Patent number: 10366335
    Abstract: Systems and methods of searching for related data sets are provided. Multivariate data sets can be input as queries into a data set search engine. According to one embodiment, the input data set is automatically reduced to a set of best fit data models of minimum complexity that represent the data set. The data model is then compared to other data models to not only identify similarity between the models, but also to identify the particulars of why the data models are related. Similar data model results can be analyzed to determine the quality of each returned data model based on an information scores. These results can be displayed graphically as a topographical map of nodes and edge. Each node can represent a data model and each edge can reflect the similarity between the nodes.
    Type: Grant
    Filed: September 3, 2013
    Date of Patent: July 30, 2019
    Assignee: DataRobot, Inc.
    Inventor: Michael Schmidt
  • Patent number: 10102483
    Abstract: Various systems and methods provide an intuitive user interface that enables automatic specification of queries and constraints for analysis by ML component. Various implementations provide methodologies for automatically formulating machine learning (“ML”) and optimization queries. The automatic generation of ML and/or optimization queries can be configured to use examples to facilitate formulation of ML and optimization queries. One example method includes accepting input data specifying variables and data values associated with the variables. Within the input data any unspecified data records are identified, and a relationship between the variables specified in the input data and a variable associated with the at least one unspecified data record is automatically determined. The relationship can be automatically determined based on training data contained within the input data. Once a relationship is established a ML problem can be automatically generated.
    Type: Grant
    Filed: November 14, 2016
    Date of Patent: October 16, 2018
    Assignee: DataRobot, Inc.
    Inventor: Michael Schmidt
  • Publication number: 20170243140
    Abstract: Systems and techniques for predictive data analytics are described. In a method for selecting a predictive model for a prediction problem, the suitabilities of predictive modeling procedures for the prediction problem may be determined based on characteristics of the prediction problem and/or on attributes of the respective modeling procedures. A subset of the predictive modeling procedures may be selected based on the determined suitabilities of the selected modeling procedures for the prediction problem. A resource allocation schedule allocating computational resources for execution of the selected modeling procedures may be generated, based on the determined suitabilities of the selected modeling procedures for the prediction problem. Results of the execution of the selected modeling procedures in accordance with the resource allocation schedule may be obtained. A predictive model for the prediction problem may be selected based on those results.
    Type: Application
    Filed: May 5, 2017
    Publication date: August 24, 2017
    Applicant: DataRobot, Inc.
    Inventors: Jeremy Achin, Thomas DeGodoy, Timothy Owen, Xavier Conort
  • Patent number: 9659254
    Abstract: Systems and techniques for predictive data analytics are described. In a method for selecting a predictive model for a prediction problem, the suitabilities of predictive modeling procedures for the prediction problem may be determined based on characteristics of the prediction problem and/or on attributes of the respective modeling procedures. A subset of the predictive modeling procedures may be selected based on the determined suitabilities of the selected modeling procedures for the prediction problem. A resource allocation schedule allocating computational resources for execution of the selected modeling procedures may be generated, based on the determined suitabilities of the selected modeling procedures for the prediction problem. Results of the execution of the selected modeling procedures in accordance with the resource allocation schedule may be obtained. A predictive model for the prediction problem may be selected based on those results.
    Type: Grant
    Filed: July 22, 2016
    Date of Patent: May 23, 2017
    Assignee: DataRobot, Inc.
    Inventors: Jeremy Achin, Thomas DeGodoy, Timothy Owen, Xavier Conort
  • Patent number: 9652714
    Abstract: Systems and techniques for predictive data analytics are described. In a method for selecting a predictive model for a prediction problem, the suitabilities of predictive modeling procedures for the prediction problem may be determined based on characteristics of the prediction problem and/or on attributes of the respective modeling procedures. A subset of the predictive modeling procedures may be selected based on the determined suitabilities of the selected modeling procedures for the prediction problem. A resource allocation schedule allocating computational resources for execution of the selected modeling procedures may be generated, based on the determined suitabilities of the selected modeling procedures for the prediction problem. Results of the execution of the selected modeling procedures in accordance with the resource allocation schedule may be obtained. A predictive model for the prediction problem may be selected based on those results.
    Type: Grant
    Filed: July 22, 2016
    Date of Patent: May 16, 2017
    Assignee: DataRobot, Inc.
    Inventors: Jeremy Achin, Thomas DeGodoy, Timothy Owen, Xavier Conort
  • Publication number: 20160364647
    Abstract: Systems and techniques for predictive data analytics are described. In a method for selecting a predictive model for a prediction problem, the suitabilities of predictive modeling procedures for the prediction problem may be determined based on characteristics of the prediction problem and/or on attributes of the respective modeling procedures. A subset of the predictive modeling procedures may be selected based on the determined suitabilities of the selected modeling procedures for the prediction problem. A resource allocation schedule allocating computational resources for execution of the selected modeling procedures may be generated, based on the determined suitabilities of the selected modeling procedures for the prediction problem. Results of the execution of the selected modeling procedures in accordance with the resource allocation schedule may be obtained. A predictive model for the prediction problem may be selected based on those results.
    Type: Application
    Filed: July 22, 2016
    Publication date: December 15, 2016
    Applicant: DataRobot, Inc.
    Inventors: Jeremy Achin, Thomas DeGodoy, Timothy Owen, Xavier Conort
  • Publication number: 20160335550
    Abstract: Systems and techniques for predictive data analytics are described. In a method for selecting a predictive model for a prediction problem, the suitabilities of predictive modeling procedures for the prediction problem may be determined based on characteristics of the prediction problem and/or on attributes of the respective modeling procedures. A subset of the predictive modeling procedures may be selected based on the determined suitabilities of the selected modeling procedures for the prediction problem. A resource allocation schedule allocating computational resources for execution of the selected modeling procedures may be generated, based on the determined suitabilities of the selected modeling procedures for the prediction problem. Results of the execution of the selected modeling procedures in accordance with the resource allocation schedule may be obtained. A predictive model for the prediction problem may be selected based on those results.
    Type: Application
    Filed: July 22, 2016
    Publication date: November 17, 2016
    Applicant: DataRobot, Inc.
    Inventors: Jeremy Achin, Thomas DeGodoy, Timothy Owen, Xavier Conort
  • Patent number: 9489630
    Abstract: Systems and techniques for predictive data analytics are described. In a method for selecting a predictive model for a prediction problem, the suitabilities of predictive modeling procedures for the prediction problem may be determined based on characteristics of the prediction problem and/or on attributes of the respective modeling procedures. A subset of the predictive modeling procedures may be selected based on the determined suitabilities of the selected modeling procedures for the prediction problem. A resource allocation schedule allocating computational resources for execution of the selected modeling procedures may be generated, based on the determined suitabilities of the selected modeling procedures for the prediction problem. Results of the execution of the selected modeling procedures in accordance with the resource allocation schedule may be obtained. A predictive model for the prediction problem may be selected based on those results.
    Type: Grant
    Filed: May 22, 2015
    Date of Patent: November 8, 2016
    Assignee: DataRobot, Inc.
    Inventors: Jeremy Achin, Thomas DeGodoy, Timothy Owen, Xavier Conort
  • Publication number: 20150339572
    Abstract: Systems and techniques for predictive data analytics are described. In a method for selecting a predictive model for a prediction problem, the suitabilities of predictive modeling procedures for the prediction problem may be determined based on characteristics of the prediction problem and/or on attributes of the respective modeling procedures. A subset of the predictive modeling procedures may be selected based on the determined suitabilities of the selected modeling procedures for the prediction problem. A resource allocation schedule allocating computational resources for execution of the selected modeling procedures may be generated, based on the determined suitabilities of the selected modeling procedures for the prediction problem. Results of the execution of the selected modeling procedures in accordance with the resource allocation schedule may be obtained. A predictive model for the prediction problem may be selected based on those results.
    Type: Application
    Filed: May 22, 2015
    Publication date: November 26, 2015
    Applicant: DataRobot, Inc.
    Inventors: Jeremy Achin, Thomas DeGodoy, Timothy Owen, Xavier Conort