Abstract: A system may receive, via a first user interface associated with an online marketplace, a multi-modality request to retrieve a listing for an item, the multi-modality request comprising at least a first image and a first natural language text associated with the item. The system may generate an item embedding based on inputting the first image and the first natural language text to a machine learning model and may generate a first vector associated with the first image and the first natural language text included in the multi-modality request. The system may cause presentation, via the first user interface associated with the online marketplace, of one or more listings for the item retrieved based at least in part on a similarity metric between the first vector and a second vector of a plurality of vectors associated with a plurality of listings.

Abstract: Systems and methods provide determining listings of items based on similarities at least among items and queries in an online shopping system. In particular, the systems and methods determine similarities among items, users, product, messages, reviews, and queries, based on a combination of a machine learning model and similarity index data. The machine learning model (e.g., a Transformer model and a neural network model) generates embedded vector representation of items, queries, and other data in the online shopping systems. The machine learning model may be pre-trained based at least on data associated with items in the online shopping system, and fine-tuned based on a variety of mappings of similarities: item-to-item, user-to-item, query-to-item, and the like. The similarity index data include k-Nearest Neighbor index data for determining items within a range of similarity based on a receive query.