Abstract: An online system leverages stored interactions with items made by users after the online system received queries to determine display of items satisfying the query. For example, the online system trains a model to predict a likelihood of a user performing an interaction with an item displayed after a query was received. As different items receive different amounts of interaction from users, limited historical interaction with certain items may limit accuracy of the model. The online system generates embeddings for previously received queries and uses measures of similarity between embeddings for queries to generate clusters of queries. Previous interactions with queries in a cluster are combined, with the combined data being used for determining display of items in response to a query.

Abstract: Techniques for predicting a wait time for a shopper based on a location the shopper's client device are presented. A system identifies a shopper's current location and uses a machine learning model to predict a wait time until the shopper will receive one or more orders. The machine learning model is trained to use input features including a number of orders received during a current time period for fulfillment near the current location, a number of other shoppers available for fulfilling orders during the current time period near the current location, historical information about a presentation of a plurality of orders to a plurality of shoppers near the current location, and historical information about the shopper and the other nearby available shoppers. The system then sends the predicted wait time to the client device for presentation to the shopper.

Abstract: An online concierge system generates recipe embeddings for recipes including multiple items and user embeddings for users, with the recipe embeddings and user embeddings in a common latent space. To generate the user embeddings and the recipe embeddings, a model includes separate layers for a user model outputting user embeddings and for a recipe model outputting recipe embeddings. When training the model, a weight matrix generates a predicted dietary preference type for a user embedding and for a recipe embedding and adjusts the user model or the recipe model based on differences between the predicted dietary preference type and a dietary preference type applied to the user embedding and to the recipe embedding. Additionally cross-modal layers generate a predicted user embedding from a recipe embedding and generate a predicted recipe embedding from a user embedding that are used to further refine the user model and the recipe model.

Abstract: A system or a method for using machine learning to automatically route user inquiries to a retailer are presented. The system receives an inquiry from a client device associated with a user. The inquiry includes text content and an image. The system uses a natural language model to analyze the received text to identify a first category of items. The system applies the received image to an image recognition model to identify a second category of items contained in the received image. The system then identifies a retailer that carries items in at least one of the first or second category of items, and suggests the retailer to the user via the client device associated with the user. A retail associate at the retailer can then respond to the inquiry via a client device associated with the retailer.

Abstract: An online system displays an ordering interface and, responsive to receiving a request from a client device associated with a user to place an order, retrieves information describing a set of unused credits provided to the user by each of one or more programs. The system identifies a set of the program(s), wherein the set of unused credits provided by each identified program is eligible to be used for acquiring an item in the order. The system accesses and applies a machine-learning model to predict an expiration of the set of unused credits provided to the user by each identified program based on the retrieved information and a current time. The system ranks the set of programs based on the prediction(s), determines a default allocation of a subset of each set of unused credits to the order based on the ranking, and updates the interface to include the default allocation.

Abstract: Self-checkout vehicle systems and methods comprising a self-checkout vehicle having a camera(s), a weight sensor(s), and a processor configured to: (i) identify via computer vision a merchandise item selected by a shopper based on an identifier affixed to the selected item, and (ii) calculate a price of the merchandise item based on the identification and weight of the selected item. Computer vision systems and methods for identifying merchandise selected by a shopper comprising a processor configured to: (i) identify an identifier affixed to the selected merchandise and an item category of the selected merchandise, and (ii) compare the identifier and item category identified in each respective image to determine the most likely identification of the merchandise.

Abstract: An online concierge system selects content for presentation to a user by using a product scoring engine. The product scoring engine generates a user embedding for user data and a query embedding for query data. The product scoring engine generates an anchor embedding based on the user embedding and the query embedding, where the anchor embedding is an embedding in a product embedding space. The product scoring engine compares the anchor embedding to a set of product embeddings to score a set of products for presentation to a user.

Abstract: A wayfinding application executing on a client device receives a current location of the device within a warehouse and accesses a layout of the warehouse describing locations of items included among an inventory of the warehouse. The application identifies a route from the current location to one or more locations within the warehouse associated with one or more target items, generates augmented reality elements including instructions for navigating the route, and sends the elements to a display area of the device. The application detects a location within the warehouse associated with a target item and determines whether the item is at the location based on an image captured by the device. Upon determining it is not at the location, the application alerts a user of the device to a replacement item by generating an augmented reality element that calls attention to it and sending the element to the display area.

Abstract: An online concierge system generates an item graph connecting item nodes with attribute nodes of the items. When the online concierge system receives a search query to identify one or more items from a customer, the online concierge system parses the search query into combinations of terms and identifies item nodes and attribute nodes related to the search query. The online concierge system may determine that no item nodes meet presentation criteria. The online concierge system may determine that a reformulated search query has a higher conversion probability than the search query received from the customer. The online concierge system reformulates the search query. The online concierge system selects item nodes as search results. The online concierge system transmits the search results to the customer.

Abstract: An online system receives a recipe from a customer mobile device. The online system performs natural language processing on the recipe to determine parsed ingredients. For each of one or more of the determined parsed ingredients, the online system maps the parsed ingredient to a generic item. The online system queries a product database with the mapped generic item to obtain one or more products associated with the mapped generic item. The online system applies a machine-learned conversion model to each of the one or more products to determine a conversion likelihood for the product. The conversion model may be trained based on historical data describing previous conversions made by customers presented with an opportunity to add products to an order. The online system selects a product from the one or more products based on the determined conversion likelihoods and adds the selected product to an order.

Abstract: A system may generate a prompt based in part on a search query from a customer client device. The prompt instructs a machine learned model to provide item predictions. And the model was trained by: converting structured data describing items of an online catalog to annotated text data (unstructured data), generating training examples based in part on the annotated text data, and training the model using the training examples. The system may receive item predictions generated by the prompt being applied to the machine learned model, the item predictions may have corresponding item identifiers. The item predictions are processed to identify a recommended item from the item predictions. The processing includes determining item information for the recommended item using an item identifier associated with the recommended item. The item information is provided to the customer client device.

Abstract: An online concierge shopping system fulfills orders using workers who pick items at a warehouse to complete an order and workers to deliver the orders to a customer's location. To optimize the staffing of workers for each task, the system uses a trained model to predict the number of workers needed to achieve an optimal outcome based on an input set of contextual information. The system also schedules specific workers to various shifts using the predicted number of workers needed and then searching a feasibility space for an optimal solution. The trained model may be updated based on performance observations.

Abstract: An online system receives information describing a physical retail store, in which the information includes attributes of physical elements within the store and their arrangement. A request is received from a user to generate a rendering of the store in a virtual reality environment. A profile of the user describing the user's geographic location and a set of historical actions performed by the user are accessed, in which the set of historical actions is associated with one or more of the physical elements. Based on the information describing the store and the profile, the rendering is generated to include virtual reality elements representing a set of the physical elements arranged based on the arrangement of the physical elements, and the rendering is sent for display to the user. When an update to the information describing the store is received, the rendering is updated and sent for display to the user.

Abstract: Disclosed are visual recognition and sensor fusion weight detection system and method. An example method includes: tracking, by a sensor system, objects and motions within a selected area of a store; activating, by the sensor system, a first computing device positioned in the selected area in response to detecting a presence of a customer within the selected area: identifying, by the sensor system, the customer and at least one item carried by the customer; transmitting, by the sensor system, identifying information of the customer and the at least one item to a computing server system via a communication network; measuring, by the first computing device, a weight of the at least one item; transmitting, by the first computing device, the weight to the computing server system via the communication network; and generating, by the computing server system, via the communication network, transaction information of the at least one item.

Abstract: An online concierge system receives orders from users identifying items and a warehouses from which the items are obtained. The online concierge system displays groups of one or more orders to shoppers, allowing a shopper to select a group of orders for fulfillment. When selecting groups of orders to display to shoppers, the online concierge system accounts for costs for fulfilling different groups and displays groups having costs satisfying one or more criteria, while maintaining one or more restrictions on times to fulfill orders. The online concierge system trains a selection prediction model to predict an amount of time for a shopper to select a group of orders and determines an estimated fulfillment time for the group from the predicted amount of time. Accounting for the predicted selection time allows the online concierge system to identify a larger number of groups for which costs of fulfillment are determined.

Abstract: An online concierge system includes a marketplace automation engine for setting various control parameters affecting marketplace operation. The marketplace automation engine applies a hyperparameter learning model to the marketplace state data to predict a set of hyperparameters affecting a set of respective parameterized control decision models. The hyperparameter learning model is trained on historical marketplace state data and a configured outcome objective for the online concierge system. The marketplace automation engine independently applies the set of parameterized control decision models to the marketplace state data using the hyperparameters to generate a respective set of control parameters affecting marketplace operation of the online concierge system. The marketplace automation engine applies the respective set of control parameters to operation of the online concierge system.

Abstract: An online concierge system suggests replacement items when an ordered item may be unavailable. To promote similarity of sources between the replacement item with the ordered item, candidate replacement items are scored, in part, based on a source similarity score based on a source of the candidate replacement item and a source of the ordered item. The source similarity score may be determined by a computer model based on user interactions with item sources. The similarity score may be based on source embeddings that may be determined based on respective item embeddings or may be determined by training source embeddings directly from user-source interactions. The similarity score for a candidate replacement item may be combined with a replacement score indicating the user's likelihood of selecting the candidate replacement item as a replacement to yield a total score for selection as suggestion as a replacement for the ordered item.

Abstract: An online concierge system assigns shoppers to fulfill orders from users. To allocate shoppers, the online concierge system predicts future supply and demand for the shoppers' services for different time windows. To forecast a supply of shoppers, the online concierge system trains a machine learning model that estimates future supply based on access to a shopper mobile application through which the shoppers obtain new assignments by shoppers. The online concierge system also forecasts future orders. The online concierge system estimates a supply gap in a future time period by selecting a target time to accept for shoppers to accept orders and determining a corresponding ratio of number of shoppers and number of orders. The online concierge system may adjust a number of shoppers allocated to the future time period to achieve the determined ratio number of shoppers and number of orders.

Abstract: An online system receives information describing an order placed by a user of the online system and a set of contextual features associated with servicing the order. The online system also retrieves a set of user features associated with the user. The online system accesses a machine learning model trained to predict a tip amount the user is likely to provide for servicing the order and applies the machine learning model to a set of inputs, in which the set of inputs includes the information describing the order, the set of user features, and the set of contextual features. The online system then determines a suggested tip amount for servicing the order based on the predicted tip amount.