Abstract: An advertising system has limited computing resources to spend evaluating advertisements of advertisers to determine a “best” advertisement to serve to users of a social networking system. The computing resources are allocated (e.g., by varying the number of advertisements that are considered for presentation to a user) based on the neediness of the user and/or the advertiser on a per impression basis. The neediness of a user may be determined by grouping users into groups and determining a yield curve of expected revenue per computing resource used. Then, the revenue may be maximized across impression opportunities for multiple users. The neediness of an advertiser may be determined by biasing the selection of one advertiser's advertisements over another advertiser's advertisements based on an expected revenue, an expected number of interactions of the advertisement, or otherwise maximizing a satisfaction coefficient for the advertiser.

Abstract: An online system identifies an additional feature to evaluate for inclusion in a machine learned model. The additional feature is based on characteristics of one or more dimensions of information maintained by the online system. To generate data for evaluating the additional feature, the online system generates various partitions of stored data, where each partition includes characteristics associated with one or more dimensions on which the additional feature is based. Using values of characteristics in a partition, the online system generates values for the additional feature and includes the values of the additional feature in the partition. Values for the additional feature are generated for various partitions based on the values of characteristics in each partition. The online system combines multiple partitions that include values for the additional feature to generate a training set for evaluating a machine learned model including the additional feature.

Abstract: Based on prior interactions associated with a user, an online system predicts an amount of interaction by the user with an object associated with an advertisement. Using the predicted amount of user interaction, the online system determines an expected value of presenting the advertisement to the user. The advertisement is ranked among other advertisements based on the expected values associated with the advertisements, and one or more advertisements are selected for presentation to the user based on the ranking. An advertisement may also specify a threshold amount of interaction with an associated object as targeting criteria, so the predicted amount of interaction with the object associated with the advertisement may determine if a user is eligible to be presented with the advertisement.

Abstract: An online system identifies an additional feature to evaluate for inclusion in a machine learned model. The additional feature is based on characteristics of one or more dimensions of information maintained by the online system. To generate data for evaluating the additional feature, the online system generates various partitions of stored data, where each partition includes characteristics associated with one or more dimensions on which the additional feature is based. Using values of characteristics in a partition, the online system generates values for the additional feature and includes the values of the additional feature in the partition. Values for the additional feature are generated for various partitions based on the values of characteristics in each partition. The online system combines multiple partitions that include values for the additional feature to generate a training set for evaluating a machine learned model including the additional feature.

Abstract: Some embodiments include a workflow authoring tool that accesses a text string representation of a workflow and a text string representation of at least a data processing operator type. The workflow authoring tool enables definition of one or more data processing operator types that can be referenced in defining the machine learning workflow. When scheduling a workflow, the text string representation of the workflow can be parsed and traversed to generate an interdependency graph of one or more data processing operators. The text string representation of the data processing operator type can identify operator attributes associated with the data processing operator type.

Abstract: An advertising system has limited computing resources to spend evaluating advertisements of advertisers to determine a “best” advertisement to serve to users of a social networking system. The computing resources are allocated (e.g., by varying the number of advertisements that are considered for presentation to a user) based on the neediness of the user and/or the advertiser on a per impression basis. The neediness of a user may be determined by grouping users into groups and determining a yield curve of expected revenue per computing resource used. Then, the revenue may be maximized across impression opportunities for multiple users. The neediness of an advertiser may be determined by biasing the selection of one advertiser's advertisements over another advertiser's advertisements based on an expected revenue, an expected number of interactions of the advertisement, or otherwise maximizing a satisfaction coefficient for the advertiser.

Abstract: Some embodiments include an experiment management interface for a machine learning system. The experiment management interface can manage one or more workflow runs related to building or testing machine learning models. The experiment management interface can receive an experiment initialization command to create a new experiment associated with a new workflow. A workflow can be represented by an interdependency graph of one or more data processing operators. The experiment management interface enables definition of the new workflow from scratch or by cloning and modifying an existing workflow. The workflow can define a summary format for its inputs and outputs. In some embodiments, the experiment management interface can automatically generate a comparative visualization at the conclusion of running the new workflow based on an input schema or an output schema of the new workflow.

Abstract: Some embodiments include a method of machine learner workflow processing. For example, a workflow execution engine can receive an interdependency graph of operator instances for a workflow run. The operator instances can be associated with one or more operator types. The workflow execution engine can assign one or more computing environments from a candidate pool to execute the operator instances based on the interdependency graph. The workflow execution engine can generate a schedule plan of one or more execution requests associated with the operator instances. The workflow execution engine can distribute code packages associated the operator instances to the assigned computing environments. The workflow execution engine can maintain a memoization repository to cache one or more outputs of the operator instances upon completion of the execution requests.

Abstract: The present disclosure is directed to a high-capacity training and prediction machine learning platform that can support high-capacity parameter models (e.g., with 10 billion weights). The platform implements a generic feature transformation layer for joint updating and a distributed training framework utilizing shard servers to increase training speed for the high-capacity model size. The models generated by the platform can be utilized in conjunction with existing dense baseline models to predict compatibilities between different groupings of objects (e.g., a group of two objects, three objects, etc.).

Abstract: Some embodiments include a method of generating a compatibility score for a grouping of objects based on correlations between attributes of the objects. An example grouping is a pair of user and ad. The method may be implemented using a multi-threaded pipeline architecture that utilizes a learning model to compute the compatibility score. The learning model determines correlations between a first object's attributes (e.g., user's liked pages, user demographics, user's apps installed, pixels visited, etc.) and a second object's attributes (e.g., expressed or implied). Example expressed attributes can be targeting keywords; example implied attributes can be object IDs associated with the ad.

Abstract: An online system simplifies modification of features used by machine learned models used by the online system, such as machined learned models with high dimensionality. The online system obtains a superset of features including features used by at least one machine learned model and may include additional features. From the superset of features, the online system generates various groups of features for a machine learned model. The groups of features may be a group including features currently used by the machine learned model, a group including all available features, and one or more intermediate groups. Intermediate groups include various numbers of features from the set selected based on measures of feature impact on the machine learned model associated with various features. A user may select a group of features, test the machine learning model using the selected group, and then launch the tested model based on the results.

Abstract: Some embodiments include a machine learner platform. The machine learner platform can implement a model tracking service to track one or more machine learning models for one or more application services. A model tracker database can record a version history and/or training configurations of the machine learning models. The machine learner platform can implement a platform interface configured to present interactive controls for building, modifying, evaluating, deploying, or compare the machine learning models. A model trainer engine can task out a model training task to one or more computing devices. A model evaluation engine can compute an evaluative metric for a resulting model from the model training task.

Abstract: An advertising platform calculates bids for advertisements and optimizes bids for a plurality of advertisement objectives, where each objective corresponds to a unique user action. The advertising platform identifies an impression opportunity for an advertisement request, computes a bid amount for presenting the advertisement, and provides the computed bid amount to an advertisement selection process. The bid amount is computed based on expected values of user actions associated with the plurality of advertisement objectives and an expected value multiplier of one or more advertisement objectives, where the expected value multiplier of the one or more objectives represents a bound on a range of values for the expected values of the user actions associated with the one or more objectives.

Abstract: The present disclosure is directed to a high-capacity training and prediction machine learning platform that can support high-capacity parameter models (e.g., with 10 billion weights). The platform implements a generic feature transformation layer for joint updating and a distributed training framework utilizing shard servers to increase training speed for the high-capacity model size. The models generated by the platform can be utilized in conjunction with existing dense baseline models to predict compatibilities between different groupings of objects (e.g., a group of two objects, three objects, etc.).

Abstract: Some embodiments include a method of generating a compatibility score for a grouping of objects based on correlations between attributes of the objects. An example grouping is a pair of user and ad. The method may be implemented using a multi-threaded pipeline architecture that utilizes a learning model to compute the compatibility score. The learning model determines correlations between a first object's attributes (e.g., user's liked pages, user demographics, user's apps installed, pixels visited, etc.) and a second object's attributes (e.g., expressed or implied). Example expressed attributes can be targeting keywords; example implied attributes can be object IDs associated with the ad.

Abstract: Some embodiments include a workflow authoring tool that accesses a text string representation of a workflow and a text string representation of at least a data processing operator type. The workflow authoring tool enables definition of one or more data processing operator types that can be referenced in defining the machine learning workflow. When scheduling a workflow, the text string representation of the workflow can be parsed and traversed to generate an interdependency graph of one or more data processing operators. The text string representation of the data processing operator type can identify operator attributes associated with the data processing operator type.

Abstract: Some embodiments include a method of machine learner workflow processing. For example, a workflow execution engine can receive an interdependency graph of operator instances for a workflow run. The operator instances can be associated with one or more operator types. The workflow execution engine can assign one or more computing environments from a candidate pool to execute the operator instances based on the interdependency graph. The workflow execution engine can generate a schedule plan of one or more execution requests associated with the operator instances. The workflow execution engine can distribute code packages associated the operator instances to the assigned computing environments. The workflow execution engine can maintain a memoization repository to cache one or more outputs of the operator instances upon completion of the execution requests.

Abstract: Some embodiments include an experiment management interface for a machine learning system. The experiment management interface can manage one or more workflow runs related to building or testing machine learning models. The experiment management interface can receive an experiment initialization command to create a new experiment associated with a new workflow. A workflow can be represented by an interdependency graph of one or more data processing operators. The experiment management interface enables definition of the new workflow from scratch or by cloning and modifying an existing workflow. The workflow can define a summary format for its inputs and outputs. In some embodiments, the experiment management interface can automatically generate a comparative visualization at the conclusion of running the new workflow based on an input schema or an output schema of the new workflow.

Abstract: Some embodiments include a machine learner platform. The machine learner platform can implement a model tracking service to track one or more machine learning models for one or more application services. A model tracker database can record a version history and/or training configurations of the machine learning models. The machine learner platform can implement a platform interface configured to present interactive controls for building, modifying, evaluating, deploying, or compare the machine learning models. A model trainer engine can task out a model training task to one or more computing devices. A model evaluation engine can compute an evaluative metric for a resulting model from the model training task.

Abstract: An online system identifies an additional feature to evaluate for inclusion in a machine learned model. The additional feature is based on characteristics of one or more dimensions of information maintained by the online system. To generate data for evaluating the additional feature, the online system generates various partitions of stored data, where each partition includes characteristics associated with one or more dimensions on which the additional feature is based. Using values of characteristics in a partition, the online system generates values for the additional feature and includes the values of the additional feature in the partition. Values for the additional feature are generated for various partitions based on the values of characteristics in each partition. The online system combines multiple partitions that include values for the additional feature to generate a training set for evaluating a machine learned model including the additional feature.