Abstract: Systems and methods for natural language processing are described. Embodiments of the present disclosure receive plain text comprising a sequence of text entities; generate a sequence of entity embeddings based on the plain text, wherein each entity embedding in the sequence of entity embeddings is generated based on a text entity in the sequence of text entities; generate style information for the text entity based on the sequence of entity embeddings; and generate a document based on the style information.

Abstract: Systems and methods for face annotation are described. One or more of the systems and methods include receiving a plurality of annotated images, wherein each annotated image of the annotated images comprises a caption; cropping the annotated image based on a face detection algorithm to obtain a face crop; comparing the face crop to the caption corresponding to the annotated image to obtain a caption similarity score; and filtering the plurality of annotated images based on the caption similarity score to obtain a plurality of annotated face images.

Abstract: Various embodiments use a neural network to analyze images for aspects that characterize the images, to present locations of those aspects on the images, and, additionally, to permit a user to interact with those locations on the images. For example, a user may interact with a visual cue over one of those locations to modify, refine, or filter the results of a visual search, performed on a publication corpus, that uses an input image (e.g., one captured using a mobile device) as a search query.

Abstract: A method and system for color optimization in generated images are described. The method and system include receiving an image generation prompt that includes a text description of target image content and color information describing a target color palette; encoding the image generation prompt to obtain image features that represent the target image content and the target color palette; and generating an image representing the target image content with the target color palette based on the image features.

Abstract: Example embodiments that analyze images to characterize aspects of the images rely on a same neural network to characterize multiple aspects in parallel. Because additional neural networks are not required for additional aspects, such an approach scales with increased aspects.

Abstract: Systems, methods, and computer program products for identifying a candidate product in an electronic marketplace based on a visual comparison between candidate product image visual content and input query image visual content. Embodiments generate and store descriptive image signatures from candidate product images or selected portions of such images. A subsequently calculated visual similarity measure serves as a visual search result score for the candidate product in comparison to an input query image. Any number of images of any number of candidate products may be analyzed, such as for items available for sale in an online marketplace. Image analysis results are stored in a database and made available for subsequent automated on-demand visual comparisons to an input query image. The embodiments enable substantially real time visual based product searching of a potentially vast catalog of items.

Abstract: A multimodal recommendation identification system analyzes data describing a sequence of past content item interactions to generate a recommendation for a content item for a user. An indication of the recommended content item is provided to a website hosting system or recommendation system so that the recommended content item is displayed or otherwise presented to the user. The multimodal recommendation identification system identifies a content item to recommend to the user by generating an encoding that encodes identifiers of the sequence of content items the user has interacted with and generating encodings that encode multimodal information for content items in the sequence of content items the user has interacted with. An aggregated information encoding for a user based on these encodings and a system analyzes the content item sequence encoding and interaction between the content item sequence encoding and the multiple modality encodings to generate the aggregated information encoding.

Abstract: Example embodiments that analyze images to characterize aspects of the images rely on a same neural network to characterize multiple aspects in parallel. Because additional neural networks are not required for additional aspects, such an approach scales with increased aspects.

Abstract: Understanding emojis in the context of online experiences is described. In at least some embodiments, text input is received and a vector representation of the text input is computed. Based on the vector representation, one or more emojis that correspond to the vector representation of the text input are ascertained and a response is formulated that includes at least one of the one or more emojis. In other embodiments, input from a client machine is received. The input includes at least one emoji. A computed vector representation of the emoji is used to look for vector representations of words or phrases that are close to the computed vector representation of the emoji. At least one of the words or phrases is selected and at least one task is performed using the selected word(s) or phrase(s).

Abstract: Systems and methods for image captioning are described. One or more aspects of the systems and methods include generating a training caption for a training image using an image captioning network; encoding the training caption using a multi-modal encoder to obtain an encoded training caption; encoding the training image using the multi-modal encoder to obtain an encoded training image; computing a reward function based on the encoded training caption and the encoded training image; and updating parameters of the image captioning network based on the reward function.

Abstract: Digital content search techniques are described that overcome the challenges found in conventional sequence-based techniques through use of a query-aware sequential search. In one example, a search query is received and sequence input data is obtained based on the search query. The sequence input data describes a sequence of digital content and respective search queries. Embedding data is generated based on the sequence input data using an embedding module of a machine-learning model. The embedding module includes a query-aware embedding layer that generates embeddings of the sequence of digital content and respective search queries. A search result is generated referencing at least one item of digital content by processing the embedding data using at least one layer of the machine-learning model.

Abstract: Systems and methods for image retrieval are described. Embodiments of the present disclosure receive a search query from a user; extract an entity and a color phrase describing the entity from the search query; generate an entity color embedding in a color embedding space from the color phrase using a multi-modal color encoder; identify an image in a database based on metadata for the image including an object label corresponding to the extracted entity and an object color embedding in the color embedding space corresponding to the object label; and provide image information for the image to the user based on the metadata.

Abstract: A query segmentation system segments a search query into a query segment that identifies search results from items stored on a database. The query segmentation system receives a query string, and the query string comprises a plurality of tokens. A first token and a second token are identified from the plurality of tokens, and a first vector and a second vector associated with the first token and the second token respectively are determined. The query segmentation system determines whether to include the first and second tokens in a single query segment based on the first and second vectors. The single query segment is processed to identify the search results from the database.

Abstract: Understanding emojis in the context of online experiences is described. In at least some embodiments, text input is received and a vector representation of the text input is computed. Based on the vector representation, one or more emojis that correspond to the vector representation of the text input are ascertained and a response is formulated that includes at least one of the one or more emojis. In other embodiments, input from a client machine is received. The input includes at least one emoji. A computed vector representation of the emoji is used to look for vector representations of words or phrases that are close to the computed vector representation of the emoji. At least one of the words or phrases is selected and at least one task is performed using the selected word(s) or phrase(s).

Abstract: Digital content search techniques are described. In one example, the techniques are incorporated as part of a multi-head self-attention module of a transformer using machine learning. A localized self-attention module, for instance, is incorporated as part of the multi-head self-attention module that applies local constraints to the sequence. This is performable in a variety of ways. In a first instance, a model-based local encoder is used, examples of which include a fixed-depth recurrent neural network (RNN) and a convolutional network. In a second instance, a masking-based local encoder is used, examples of which include use of a fixed window, Gaussian initialization, and an adaptive predictor.

Abstract: Methods, systems, and computer programs are presented for adding new features to a network service. A method includes receiving an image depicting an object of interest. A category set is determined for the object of interest and an image signature is generated for the image. Using the category set and the image signature, the method identifies a set of publications within a publication database and assigns a rank to each publication. The method causes presentation of the ranked list of publications at a computing device from which the image was received.

Abstract: Methods, systems, and computer programs are presented for adding new features to a network service. A method includes receiving an image depicting an object of interest. A category set is determined for the object of interest and an image signature is generated for the image. Using the category set and the image signature, the method identifies a set of publications within a publication database and assigns a rank to each publication. The method causes presentation of the ranked list of publications at a computing device from which the image was received.

Abstract: In various example embodiments, a system and method for determining an item that has confirmed characteristics are described herein. An image that depicts an object is received from a client device. Structured data that corresponds to characteristics of one or more items are retrieved. A set of characteristics is determined, the set of characteristics being predicted to match with the object. An interface that includes a request for confirmation of the set of characteristics is generated. The interface is displayed on the client device. Confirmation that at least one characteristic from the set of characteristics matches with the object depicted in the image is received from the client device.

Abstract: Methods, apparatuses, and systems for content management and search refinement are described. Embodiments of the present disclosure include search refinement systems configured to identify the most relevant data attributes (e.g., facets) for filtering search results. For example, a user may provide a search query, and a search engine may retrieve a set of search results in addition to providing facets (e.g., data attributes associated with search result objects from the retrieved set of search results) for user search refinement of the retrieved search results. In some embodiments, facets are scored using a significance heuristic, which is used to determine and select facets that provide the most information gain given a set of search results. Selected facets may be presented to a user as filtering options for narrowing a retrieved set of results.

Abstract: Technology is disclosed herein for enhanced similarity search. In an implementation, a search environment includes one or more computing hardware, software, and/or firmware components in support of enhanced similarity search. The one or more components identify a modality for a similarity search with respect to a query object. The components generate an embedding for the query object based on the modality and based on connections between the query object and neighboring nodes in a graph.