Abstract: Systems and methods directed to returning personalized image-based search results are described. In examples, a query including an image may be received, and a personalized item embedding may be generated based on the image and user profile information associated with a user. Further, a plurality of candidate images may be obtained based on the personalized item embedding. The candidate images may then be ranked according to a predicted level of user engagement for a user, and then diversified to ensure visual diversity among the ranked images. A portion of the diversified images may then be returned in response to an image-based search.

Abstract: Aspects of the present disclosure relate to a providing related content recommendations in response to a user search query by supervising the training of pair embeddings using contrastive learning and pairwise co-click signals. The approach combines a two tower model architecture with a cascaded multilayer perceptron model to enable the adoption of variable combinations of input features and more representative learned pair embeddings. The learned embeddings undergo supervised contrastive loss training to generate a related content recommendation model, which is subsequently evaluated using both online and offline metrics. The related content recommendation model can provide results to search queries that improve recommendation quality and increase user engagement, thereby ultimately enhancing long term user experience.

Abstract: Systems and methods directed to returning personalized image-based search results are described. In examples, a query including an image may be received, and a personalized item embedding may be generated based on the image and user profile information associated with a user. Further, a plurality of candidate images may be obtained based on the personalized item embedding. The candidate images may then be ranked according to a predicted level of user engagement for a user, and then diversified to ensure visual diversity among the ranked images. A portion of the diversified images may then be returned in response to an image-based search.

Abstract: The description relates to diversified hybrid image annotation for annotating images. One implementation includes generating first image annotations for a query image using a retrieval-based image annotation technique. Second image annotations can be generated for the query image using a model-based image annotation technique. The first and second image annotations can be integrated to generate a diversified hybrid image annotation result for the query image.

Abstract: A visual search system comprised of a computing device, the computing device including an image processing engine for generating a feature vector representing a user-selected object in an image input, an object detection engine for locating one or more objects in the image input and for determining a category of a user-selected object from objects in the image input, the object detection engine using the category to generate a plurality of attributes for the user-selected object, a product data store for storing a plurality of tables storing one or more attributes associated with a category of the user-selected object, an attribute generation engine for generating a plurality of attribute options for each of the attributes of the user-selected object, and an attribute matching engine for comparing attributes and attribute options of the user-selected object with attributes and attribute options of visually similar products and images.

Abstract: A visual search system detects one or more user-selected objects represented in an image. A first group of attributes for the user-selected objects is identified. A category for the user-selected objects is identified, and a second group of pre-defined attributes associated with the category is retrieved. The first and second groups of attributes are combined into an attributes set. The combined set of attributes are presented to the user. The user selects one or more attributes and a search is performed to identify images similar to the user-selected attributes. The images are ranked and a subset is presented to the user.

Abstract: Systems and methods can be implemented to conduct searches based on images used as queries in a variety of applications. In various embodiments, a set of visual words representing a query image are generated from features extracted from the query image and are compared with visual words of index images. A set of candidate images is generated from the index images resulting from matching one or more visual words in the comparison. A multi-level ranking is conducted to sort the candidate images of the set of candidate images, and results of the multi-level ranking are returned to a user device that provided the query image. Additional systems and methods are disclosed.

Abstract: A visual search engine is described herein. The visual search engine is configured to return information to a client computing device based upon a multimodal query received from the client computing device (wherein the multimodal query comprises an image and text). The visual search engine is further configured to interact with a user of the client computing device to disambiguate information retrieval intent of the user.

Abstract: A visual search engine is described herein. The visual search engine is configured to return information to a client computing device based upon a multimodal query received from the client computing device (wherein the multimodal query comprises an image and text). The visual search engine is further configured to interact with a user of the client computing device to disambiguate information retrieval intent of the user.

Abstract: Representative embodiments disclose mechanisms to perform visual intent classification or visual intent detection or both on an image. Visual intent classification utilizes a trained machine learning model that classifies subjects in the image according to a classification taxonomy. The visual intent classification can be used as a pre-triggering mechanism to initiate further action in order to substantially save processing time. Example further actions include user scenarios, query formulation, user experience enhancement, and so forth. Visual intent detection utilizes a trained machine learning model to identify subjects in an image, place a bounding box around the image, and classify the subject according to the taxonomy. The trained machine learning model utilizes multiple feature detectors, multi-layer predictions, multilabel classifiers, and bounding box regression.

Abstract: A visual search system comprised of a computing device, the computing device including an image processing engine for generating a feature vector representing a user-selected object in an image input, an object detection engine for locating one or more objects in the image input and for determining a category of a user-selected object from objects in the image input, the object detection engine using the category to generate a plurality of attributes for the user-selected object, a product data store for storing a plurality of tables storing one or more attributes associated with a category of the user-selected object, an attribute generation engine for generating a plurality of attribute options for each of the attributes of the user-selected object, and an attribute matching engine for comparing attributes and attribute options of the user-selected object with attributes and attribute options of visually similar products and images.

Abstract: A method for using a speech signal to augment a visual search includes processing the image data to determine an image search intent. Concurrently with processing the image data, the method processes the speech signal to determine at least one speech search intent. The method generates a search query by combining keywords and/or the image from the image search intent with keywords from the speech search intent. The method then performs a search based on the generated query and reports the results of the search. The method generates the image search intent by applying the image data to a knowledge base and generates the speech search intent by converting the speech to text and applying the text to a cognition service.

Abstract: The description relates to diversified hybrid image annotation for annotating images. One implementation includes generating first image annotations for a query image using a retrieval-based image annotation technique. Second image annotations can be generated for the query image using a model-based image annotation technique. The first and second image annotations can be integrated to generate a diversified hybrid image annotation result for the query image.

Abstract: Non-limiting examples of the present disclosure relate to object detection processing of image content that categorically classifies specific objects within image content. Exemplary object detection processing may be utilized to enhance visual search processing including content retrieval and curation, among other technical advantages. An exemplary object detection model is implemented to categorically classify an object. In doing, so an exemplary object detection model may classify objects based on: analysis of specific objects within image content, positioning of the objects within the image content and intent associated with the image content, among other examples. The object detection model generates exemplary categorical classification(s) for specific data objects, which may be propagated to enhance processing efficiency and accuracy during visual search processing.

Abstract: Systems and methods can be implemented to conduct searches based on images used as queries in a variety of applications. In various embodiments, a set of visual words representing a query image are generated from features extracted from the query image and are compared with visual words of index images. A set of candidate images is generated from the index images resulting from matching one or more visual words in the comparison. A multi-level ranking is conducted to sort the candidate images of the set of candidate images, and results of the multi-level ranking are returned to a user device that provided the query image. Additional systems and methods are disclosed.

Abstract: Auxiliary content provided in addition to search results is selected and presented to aid the user in completing tasks and increasing user interaction performance. Auxiliary content is processed utilizing existing search engine categorization and identification mechanisms, thereby facilitating the determination of similarities between the auxiliary content and indexed content that is identified as being responsive to a search query. At least some of the search results identified as being responsive to the search query are compared to auxiliary content to identify similarities, including visual similarities. Similar auxiliary content are selected to aid the user in completing tasks, and such selected auxiliary content is provided with the search results, including in a visually distinct or separated manner.

Abstract: Auxiliary content provided in addition to search results is selected and presented to aid the user in completing tasks and increasing user interaction performance. Auxiliary content is processed utilizing existing search engine categorization and identification mechanisms, thereby facilitating the determination of similarities between the auxiliary content and indexed content that is identified as being responsive to a search query. At least some of the search results identified as being responsive to the search query are compared to auxiliary content to identify similarities, including visual similarities. Similar auxiliary content are selected to aid the user in completing tasks, and such selected auxiliary content is provided with the search results, including in a visually distinct or separated manner.

Abstract: Information from social networks may be used to identify a user's interests and predilections, and the information may be used to affect search results. In one example, social networks have pages that correspond to real entities, such as manufacturers and merchants. Entity pages in social networks are mapped to their corresponding real entities, and information that users leave on the pages (e.g., “likes”, or textual reviews) are extracted to determine users' sentiments about the entities. When users search for products with a search engine, user sentiment is then used to guide the results. Social networks' information about users (e.g., their affinities, such as schools, workplaces, interests) may be used to determine the relevance of specific users' sentiments—e.g., sentiments of users who went to a particular school may be used to influence search results, when the search is requested by someone who went to the same school.

Abstract: A computer system for synchronizing a mobile device with a remote enterprise database includes a replication database. The replication database is mapped to and maintained as a subset image of the remote enterprise database through a bi-directional replication process. A mobile device gateway server is coupled to the replication database. The mobile device gateway includes a concentrator, which is configured to selectively receive replication data from the replication database, transform the received replication data into a format native to the mobile device and to selectively send the received and transformed replication data to the mobile device. The concentrator is also configured to receive mobile device data, transform and to send the received mobile device data to the replication database. The mobile device gateway server acts as a middle tier between one or more enterprise databases and one or more mobile devices or any network-enabled thin client.