Abstract: A computer system allowing users to search for items of interest provides a search query interface. The system receives characters of a search query in the search interface as the user enters the characters and interactively calculates, ranks, and displays a set of possible search query options from which the user can select. To rank the set of possible search query options, the system modifies rankings of candidate search queries based on factors associated with third parties. More specifically, contextual relevance scores are computed for the candidate search queries based on the context, such as a user to whom the search results are provided. These contextual relevance scores are in turn adjusted using factors associated with third parties, such as values calculated based on consideration offered by third parties. Users are shown the search query options, ranked in order of the adjusted relevance scores, as possible query selections.

Abstract: An online system displays items to a user in search results based on appeasement scores for the items, adjusted according to how specific the search query is. The online system receives a search query from a user of an online system. The online system computes a query specificity score, a measure of the specificity of the search query. The online system accesses candidate items from a database that potentially match the search query. For each candidate item, the online system may compute or predict an appeasement score. The online system adjusts the appeasement score based on the query specificity score such that a more specific query weights the appeasement score lower than a less specific query. The online system may then compute a ranking score based on the adjusted appeasement score and display the candidate items to the user based on their ranking scores.

Abstract: A system generates text artifacts using a machine learned language model. The text artifacts may be provided to a search engine for providing to users along with search results. The system iteratively improves the set of text artifacts by performing the following steps. The system updates the prompt used to generate the text artifacts based on the performance of the text artifacts to obtain a new prompt. The system executes the machine learned language model using the new prompt to generate a new set of text artifacts. The system evaluates the new set of text artifacts to determine performance of each of the new set of text artifacts. These steps are repeatedly performed to improve the set of text artifacts.

Abstract: An online system trains a churn prediction model to attribute a churn event to one or more causal events. The churn prediction model receives customer features and online system features as inputs. Various causal events that occur affect one or more online system features. To avoid biasing the churn prediction model using input features that are related to possible causal events, the online system determines customer features and online system features based on customer interactions occurring in different time intervals. The customer features are determined from interactions in a time interval that is earlier than a time interval from which interactions are used to determine online system features. Such time segmenting decorrelates the features input to the model from the events, reducing potential bias from the causal events on the churn prediction model.

Abstract: An embedding model is trained to learn latent representations of users describing information related to conditional treatment effect for users relative to different potential treatments. The user embeddings may be used to determine the types of situations in which a user responds differently to different conditions or situations. To train this model, a plurality of experiments with users may be performed to determine user responses to different treatment conditions in the experiments. The conditional treatment effect for users in the experiments may be determined, e.g., with counterfactual predictions of a treatment not experienced by a user in the experiment. The embedding model may be trained with decoders that each predict the conditional treatment effect with respect to one of the experiments, enabling a loss for each experiment with respect to the conditional treatment effect to jointly train the embedding model.

Abstract: Based on logged information about prior events, an online concierge system generates a set of location metrics that quantify properties of locations such as retailers at which items may be acquired, and residences to which the items are brought. The location metrics can be used for a variety of purposes to aid customers or other users of the online concierge system, such as providing the users with more information (e.g., likely delivery delays) or alternative options (e.g., pricing options), or emphasizing options that the location metrics indicate would be of particular value to the user. To determine whether to emphasize a particular option, the online concierge system applies a machine-learned model that predicts whether emphasizing that option would effect a positive change in user behavior, relative to not emphasizing it.

Abstract: An online system predicts replacement items for presentation to a user using a machine-learning model. The online system receives interaction data describing a user's interaction with the online system. In particular, the interaction data describes an initial item that the user added to their item list. The online system identifies a set of candidate items that could be presented to the user as potential replacements for the initially-added item. The online system applies a replacement prediction model to each of these candidate items to generate a replacement score for the candidate items. The online system selects a proposed replacement item and transmits that item to the user's client device for display to the user. If the user selects the proposed replacement item, the online concierge system replaces the initial item with the proposed replacement item in the user's item list.

Abstract: An online system, such as a concierge service, provides services to users using a set of limited resources. To allocate the limited resources of the system among the users, the system uses a model to predict each user's sensitivity to different levels of service. An allocation module then allocates the limited resources among a set of users based in part on the estimated sensitivities and the supply of available resources.

Abstract: A system accesses user data describing characteristics of a user and generates a content item score for each content item of a plurality of content items. The system generates the content item score by applying a machine-learning model to the user data, and then generates a plurality of content bundles. The system also generates a bundle score for each content bundle based on corresponding content item scores for the content item associated with each content bundle, randomly selects a bundle of the plurality of content bundles based on the generated bundle scores, and transmits the randomly selected bundle to a client device associated with the user for display to the user. Finally, the system applies the model to each of the generated training examples and updates the parameters of the model based on the model output.

Abstract: An online system uses an offline iterative clustering process to evaluate the performance of a set of content selection frameworks. To perform an iteration of the iterative clustering process, an online system clusters the testing example data into a set of clusters. An online system computes a set of framework scores for each of the generated clusters. An online system computes an improvement score for each cluster based on the performance scores of the clusters. To determine whether to perform another iteration, an online system computes an aggregated improvement score based on the improvement scores of the clusters. If an online system determines that the aggregated improvement score does not meet the threshold, an online system performs another iteration of the process above. When an online system finishes the iterative process, an online system outputs the improvement scores of the most-recent iteration.

Abstract: An online concierge system fulfills orders for items offered by retailers and may increase the price of an item offered by a retailer in some instances. The online concierge system applies a markup to an item by applying a pricing policy to a category including the item. To optimize application of pricing policies to categories, the online concierge system categorizes items offered by the retailer and applies an outcome model to combinations of categories and pricing policies. From the output of the outcome model, the online concierge system selects a set of categories and corresponding pricing policies. Using a price adjustment model, the online concierge system determines modifications to one or more of the pricing policies of the set to enforce one or more constraints across multiple pricing policies.

Abstract: An online concierge system uses a new treatment engine to score users for applying treatments of a new treatment type. The new treatment engine uses treatment models to generate treatment lift scores for the user. The new treatment engine applies an aggregation function model to the treatment lift scores to generate an aggregated lift score for the user. If the aggregated lift score exceeds a threshold, the new treatment engine applies a treatment of the new treatment type to the user. The new treatment engine trains the aggregation function model based on training examples used to train the treatment models. For a training example associated with a particular treatment type, the new treatment engine uses a target lift score generated by the treatment model for the treatment type to evaluate the performance of the aggregation function model, and to update the aggregation function model accordingly.

Abstract: An online system may receive, from a content provider, a content presentation campaign that includes one or more objectives. The online system may define a set of one or more policy functions that automatically controls the content presentation campaign. A policy function may control one or more criteria in bidding content slots. The online system may monitor a realized outcome of the content presentation campaign. The online system may apply a reinforcement learning algorithm in adjusting the set of policy functions. The reinforcement learning algorithm adjusts one or more parameters in the set of policy functions to reduce a difference between the realized outcome and the desired outcome set by the content provider. The online system generates an adjusted set of policy functions and uses the adjusted set of policy functions in bidding content slots to present one or more content items provided by the content provider.

Abstract: An online concierge system allows users to purchase items from warehouses and delivers the purchased items to the users. When displaying items offered by a warehouse, the online concierge system predicts an availability of the items at the warehouse using a trained model. When displaying items offered by the warehouse to a user, the online concierge system accounts for the predicted availabilities of different items. For example, the online concierge system determines scores for different items at the warehouse based on relevance to the user and adjusts a score for an item by its predicted availability. The online concierge system uses the adjusted scores for items when displaying items, demoting positions in an interface in which items with lower predicted availabilities are displayed. Additionally, the online concierge system may display a visual indication of a predicted availability of certain items, such as items with less than a threshold predicted availability.