Abstract: A computing system automatically selects treatments for users by generating a propensity vector for a set of treatments and selecting a treatment based on the propensity vector. The propensity vector is determined based on one or more computer models that predict user actions responsive to the treatments and the propensity vector is determined based on the value of a treatment parameter. The treatment parameter is perturbed to determine an adjusted propensity vector. Treatments are applied and outcomes determined with the propensities determined by the current value of the treatment parameter, and counterfactuals for the adjusted treatment vector are determined to evaluate the effect of modifying the treatment parameter. When the perturbed treatment parameter value yields improved results in the counterfactual, the current value is modified to improve performance of the model as a whole without requiring retraining of underlying predictive models.

Abstract: An embedding model is trained to learn latent representations of users describing information related to conditional treatment effect for users relative to different potential treatments. The user embeddings may be used to determine the types of situations in which a user responds differently to different conditions or situations. To train this model, a plurality of experiments with users may be performed to determine user responses to different treatment conditions in the experiments. The conditional treatment effect for users in the experiments may be determined, e.g., with counterfactual predictions of a treatment not experienced by a user in the experiment. The embedding model may be trained with decoders that each predict the conditional treatment effect with respect to one of the experiments, enabling a loss for each experiment with respect to the conditional treatment effect to jointly train the embedding model.

Abstract: An online concierge system generates the value for an impression by predicting future behavior by users beyond a current conversion. The predicted future behavior attributes incremental value of subsequent conversions by the user. The online concierge system gathers feature information about the user. Based on experimental data, the online concierge system generates a baseline curve describing expected user behavior for a category of users. Based on feature information of the user, the online concierge system applies a computer model to generate modifiers for the baseline curve to customize the baseline curve for the user. The modified curve is used to predict future actions by the user, and consequently a long-term incremental conversion value for the impression.

Abstract: An online system maintains a shared cache for storing actions assigned to users in different experiment groups. The system receives an indication that a user interacted with an online system, and data associated with the user. The system generates a set of propensities for a set of actions by identifying a first set of features for the user, accessing a first machine learning model, and applying the first machine learning model to the first set of features. The system selects an action based on the set of propensities and presents the action to the user. The system updates a cache of a set of user data and includes the transmitted action. The system receives a second indication and accesses a database to determine a selected action stored in association to the user. The system presents the selected action for a second time to the user.

Abstract: A method for predicting customer long-term behavior using LLM-based modeling is described. The online system receives a representation of a stimulus or treatment that is presented to a user and generates a summary of a simulated user profile. The online system performs an inference task in conjunction with the model serving system or interface system to infer one or more actions that will likely be performed in response to the representation of the stimulus based on the simulated user profile. The online system computes a surrogate measure based on the response received from the model serving system and computes a correlation coefficient between the surrogate measure and a true metric of interest from collected experiment data. Responsive to determining a correlation coefficient greater than a threshold value, the online system predicts the true metric of interest based on the surrogate measure.

Abstract: An off-policy evaluation system performs episodic off-policy evaluations to perform off-policy evaluation (OPE) for multiple, joint episodes. For a single episode, a first machine learning model outputs a propensity for each action for the user and selects a first action for the user from the set of propensities. For a second episode, a second machine learning model outputs a propensity for each action for the user and selects a first action for the user from the set of propensities. The second machine learning model is evaluated by determining an importance weight for the first model and the second model to determine the inverse propensity score of the second machine learning model.

Abstract: A ranking computer model is trained based on grouping a collection of users of an online system into different buckets based on intended likelihoods of presenting a set of content items to the collection of users, wherein a contextual bandit model is employed to compute the intended likelihoods. The online system applies the ranking computer model to generate, based on user data for a user of the online system and contextual data associated with a current session of the user, a ranking score for each content item in a set of content items. The online system selects, based on the ranking score for each content item, one or more content items from the set of content items. The online system causes a device associated with the user to display a user interface with the one or more content items for recommendation to the user.

Abstract: A system generates a set of embeddings for known treatments by applying a machine-learned embedding model to descriptions of the known treatments, where these embeddings form a vector space. The system generates an embedding for a new treatment and mapping it within the vector space, and identifies one or more known treatments with embeddings that exceed a similarity threshold with the new treatment embedding. The system accesses performance data for the selected known treatments to assess user response, and identifies a subset of users for the new treatment based on this performance data. The system also creates a content item that incorporates the new treatment, and transmits instructions to client devices of the targeted users to cause the client devices to display the content item.

Abstract: A computing system automatically selects treatments for users by generating a propensity vector for a set of treatments and selecting a treatment based on the propensity vector. The propensity vector is determined based on one or more computer models that predict user actions responsive to the treatments and the propensity vector is determined based on the value of a treatment parameter. The treatment parameter is perturbed to determine an adjusted propensity vector. Treatments are applied and outcomes determined with the propensities determined by the current value of the treatment parameter, and counterfactuals for the adjusted treatment vector are determined to evaluate the effect of modifying the treatment parameter. When the perturbed treatment parameter value yields improved results in the counterfactual, the current value is modified to improve performance of the model as a whole without requiring retraining of underlying predictive models.

Abstract: An embedding model is trained to learn latent representations of users describing information related to conditional treatment effect for users relative to different potential treatments. The user embeddings may be used to determine the types of situations in which a user responds differently to different conditions or situations. To train this model, a plurality of experiments with users may be performed to determine user responses to different treatment conditions in the experiments. The conditional treatment effect for users in the experiments may be determined, e.g., with counterfactual predictions of a treatment not experienced by a user in the experiment. The embedding model may be trained with decoders that each predict the conditional treatment effect with respect to one of the experiments, enabling a loss for each experiment with respect to the conditional treatment effect to jointly train the embedding model.

Abstract: An online system uses an offline iterative clustering process to evaluate the performance of a set of content selection frameworks. To perform an iteration of the iterative clustering process, an online system clusters the testing example data into a set of clusters. An online system computes a set of framework scores for each of the generated clusters. An online system computes an improvement score for each cluster based on the performance scores of the clusters. To determine whether to perform another iteration, an online system computes an aggregated improvement score based on the improvement scores of the clusters. If an online system determines that the aggregated improvement score does not meet the threshold, an online system performs another iteration of the process above. When an online system finishes the iterative process, an online system outputs the improvement scores of the most-recent iteration.

Abstract: A system accesses user data describing characteristics of a user and generates a content item score for each content item of a plurality of content items. The system generates the content item score by applying a machine-learning model to the user data, and then generates a plurality of content bundles. The system also generates a bundle score for each content bundle based on corresponding content item scores for the content item associated with each content bundle, randomly selects a bundle of the plurality of content bundles based on the generated bundle scores, and transmits the randomly selected bundle to a client device associated with the user for display to the user. Finally, the system applies the model to each of the generated training examples and updates the parameters of the model based on the model output.