Abstract: The subject matter of this specification generally relates to providing content related to text depicted in images. In one aspect, a system includes a data processing apparatus configured to extract text from an image. The extracted text is partitioned into multiple blocks. The multiple blocks are presented as respective first user-selectable targets on a user interface at a first zoom level. A user selection of a first block of the multiple blocks is detected. In response to detecting the user selection of the first block, portions of the extracted text in the first block are presented as respective second user-selectable targets on the user interface at a second zoom level greater than the first zoom level. In response to detecting a user selection of a portion of the extracted text within the first block, an action is initiated based on content of the user-selected text.

Abstract: The subject matter of this specification generally relates to providing content related to text depicted in images. In one aspect, a system includes a data processing apparatus configured to extract text from an image. The extracted text is partitioned into multiple blocks. The multiple blocks are presented as respective first user-selectable targets on a user interface at a first zoom level. A user selection of a first block of the multiple blocks is detected. In response to detecting the user selection of the first block, portions of the extracted text in the first block are presented as respective second user-selectable targets on the user interface at a second zoom level greater than the first zoom level. In response to detecting a user selection of a portion of the extracted text within the first block, an action is initiated based on content of the user-selected text.

Abstract: The subject matter of this specification generally relates to providing content related to text depicted in images. In one aspect, a system includes a data processing apparatus configured to extract text from an image. The extracted text is partitioned into multiple blocks. The multiple blocks are presented as respective first user-selectable targets on a user interface at a first zoom level. A user selection of a first block of the multiple blocks is detected. In response to detecting the user selection of the first block, portions of the extracted text in the first block are presented as respective second user-selectable targets on the user interface at a second zoom level greater than the first zoom level. In response to detecting a user selection of a portion of the extracted text within the first block, an action is initiated based on content of the user-selected text.

Abstract: The subject matter of this specification generally relates to providing content related to text depicted in images. In one aspect, a system includes a data processing apparatus configured to extract text from an image. The extracted text is partitioned into multiple blocks. The multiple blocks are presented as respective first user-selectable targets on a user interface at a first zoom level. A re user selection of a first block of the multiple blocks is detected. In response to detecting the user selection of the first block, portions of the extracted text in the first block are presented as respective second user-selectable targets on the user interface at a second zoom level greater than the first zoom level. In response to detecting a user selection of a portion of the extracted text within the first block, an action is initiated based on content of the user-selected text.

Abstract: The subject matter of this specification generally relates to providing content related to text depicted in images. In one aspect, a system includes a data processing apparatus configured to extract text from an image. The extracted text is partitioned into multiple blocks. The multiple blocks are presented as respective first user-selectable targets on a user interface at a first zoom level. A re user selection of a first block of the multiple blocks is detected. In response to detecting the user selection of the first block, portions of the extracted text in the first block are presented as respective second user-selectable targets on the user interface at a second zoom level greater than the first zoom level. In response to detecting a user selection of a portion of the extracted text within the first block, an action is initiated based on content of the user-selected text.

Abstract: Methods, systems, and apparatus, for determining fine-grained image similarity. In one aspect, a method includes training an image embedding function on image triplets by selecting image triplets of first, second and third images; generating, by the image embedding function, a first, second and third representations of the features of the first, second and third images; determining, based on the first representation of features and the second representation of features, a first similarity measure for the first image to the second image; determining, based on the first representation of features and the third representation of features, a second similarity measure for the the first image to the third image; determining, based on the first and second similarity measures, a performance measure of the image embedding function for the image triplet; and adjusting the parameter weights of the image embedding function based on the performance measures for the image triplets.

Abstract: Methods, systems, and apparatus, for determining fine-grained image similarity. In one aspect, a method includes training an image embedding function on image triplets by selecting image triplets of first, second and third images; generating, by the image embedding function, a first, second and third representations of the features of the first, second and third images; determining, based on the first representation of features and the second representation of features, a first similarity measure for the first image to the second image; determining, based on the first representation of features and the third representation of features, a second similarity measure for the the first image to the third image; determining, based on the first and second similarity measures, a performance measure of the image embedding function for the image triplet; and adjusting the parameter weights of the image embedding function based on the performance measures for the image triplets.

Abstract: Methods, systems, and apparatus, for determining fine-grained image similarity. In one aspect, a method includes training an image embedding function on image triplets by selecting image triplets of first, second and third images; generating, by the image embedding function, a first, second and third representations of the features of the first, second and third images; determining, based on the first representation of features and the second representation of features, a first similarity measure for the first image to the second image; determining, based on the first representation of features and the third representation of features, a second similarity measure for the the first image to the third image; determining, based on the first and second similarity measures, a performance measure of the image embedding function for the image triplet; and adjusting the parameter weights of the image embedding function based on the performance measures for the image triplets.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for analyzing image search result relevance. In one aspect, a method includes receiving result data specifying a search query and responsive image search results that reference images that are responsive to the search query. A determination is made that the search query matches an indexed query. An image relevance model is identified for the indexed query. The image relevance model can output a relevance score adjustment factor for an image search result based on image feature values of the image that is referenced by the search result. A relevance score adjustment factor is determined for each image search result using the identified image relevance model. A relevance score for each image search result is adjusted using the image's image relevance score adjustment factor. The images are ranked based on the adjusted relevance scores.

Abstract: Methods, systems, and apparatus, for determining fine-grained image similarity. In one aspect, a method includes training an image embedding function on image triplets by selecting image triplets of first, second and third images; generating, by the image embedding function, a first, second and third representations of the features of the first, second and third images; determining, based on the first representation of features and the second representation of features, a first similarity measure for the first image to the second image; determining, based on the first representation of features and the third representation of features, a second similarity measure for the the first image to the third image; determining, based on the first and second similarity measures, a performance measure of the image embedding function for the image triplet; and adjusting the parameter weights of the image embedding function based on the performance measures for the image triplets.

Abstract: Methods, systems, and apparatus, for determining fine-grained image similarity. In one aspect, a method includes training an image embedding function on image triplets by selecting image triplets of first, second and third images; generating, by the image embedding function, a first, second and third representations of the features of the first, second and third images; determining, based on the first representation of features and the second representation of features, a first similarity measure for the first image to the second image; determining, based on the first representation of features and the third representation of features, a second similarity measure for the first image to the third image; determining, based on the first and second similarity measures, a performance measure of the image embedding function for the image triplet; and adjusting the parameter weights of the image embedding function based on the performance measures for the image triplets.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing queries made up of images. In one aspect, a method includes indexing images by image descriptors. The method further includes associating descriptive n-grams with the images. In another aspect, a method includes receiving a query, identifying text describing the query, and performing a search according to the text identified for the query.

Abstract: Methods, systems, and apparatus, for determining fine-grained image similarity. In one aspect, a method includes training an image embedding function on image triplets by selecting image triplets of first, second and third images; generating, by the image embedding function, a first, second and third representations of the features of the first, second and third images; determining, based on the first representation of features and the second representation of features, a first similarity measure for the first image to the second image; determining, based on the first representation of features and the third representation of features, a second similarity measure for the first image to the third image; determining, based on the first and second similarity measures, a performance measure of the image embedding function for the image triplet; and adjusting the parameter weights of the image embedding function based on the performance measures for the image triplets.

Abstract: Methods, systems and apparatus for refining image annotations. In one aspect, a method includes receiving, for each image in a set of images, a corresponding set of labels determined to be indicative of subject matter of the image. For each label, one or more confidence values are determined. Each confidence value is a measure of confidence that the label accurately describes the subject matter of a threshold number of respective images to which it corresponds. Labels for which each of the one or more confidence values meets a respective confidence threshold are identified as high confidence labels. For each image in the set of images, labels in its corresponding set of labels that are high confidence labels are identified. Images having a corresponding set of labels that include at least a respective threshold number of high confidence labels are identified as high confidence images.

Abstract: Methods, systems and apparatus for refining image relevance models. In general, one aspect includes receiving a trained image relevance model that generates relevance measures of content feature values of images to a query, identifying a first threshold number of common content feature values for the set of training images, the common content feature values being identified as a set of content feature values that are each shared by at least a portion of the training images, identifying a subset of the set of training images having a quantity of the common content feature values greater than a second threshold number of content features values, and generating a re-trained image relevance model based on content feature values of the set of training images, wherein content feature values of the subset of training images are weighted higher than content feature values of the training images not in the subset.

Abstract: Methods, systems, and apparatus, include computer programs encoded on a computer-readable storage medium, for determining keywords for an image that supports an overlay content item. A method includes identifying, using one or more processors, an image that is to support an overlay content item, the image being presented on a web site and including a portion that is designated as being enabled to receive and display the overlay content item; evaluating pixel data associated with the image including determining one or more labels that are associated with content included within the image; and determining one or more keywords for the image based at least in part on the one or more labels.

Abstract: Methods, systems and apparatus for choosing image labels. In one aspect, a method includes receiving data specifying a first image, receiving text labels for the first image, receiving search results in response to a web search performed using at least some of the text labels as queries, ranking the text labels, at least in part, based on a number of resources referenced by the received search results, wherein at least some of the resources each include an image matching the first image, and selecting an image label for the image from the ranked text labels, the image label being selected based on the ranking.

Abstract: Methods, systems, and apparatus, include computer programs encoded on a computer-readable storage medium, for determining keywords for an image that supports an overlay content item. A method includes identifying, using one or more processors, an image that is to support an overlay content item, the image being presented on a web site and including a portion that is designated as being enabled to receive and display the overlay content item; evaluating pixel data associated with the image including determining one or more labels that are associated with content included within the image; and determining one or more keywords for the image based at least in part on the one or more labels.

Abstract: Methods, systems, and apparatus, include computer programs encoded on a computer-readable storage medium, for determining keywords for an image that supports an overlay content item. A method includes identifying, using one or more processors, an image that is to support an overlay content item, the image being presented on a web site and including a portion that is designated as being enabled to receive and display the overlay content item; evaluating pixel data associated with the image including determining one or more labels that are associated with content included within the image; and determining one or more keywords for the image based at least in part on the one or more labels.

Abstract: Methods, systems and apparatus for choosing image labels. In one aspect, a method includes receiving data specifying a first image, receiving text labels for the first image, receiving search results in response to a web search performed using at least some of the text labels as queries, ranking the text labels, at least in part, based on a number of resources referenced by the received search results, wherein at least some of the resources each include an image matching the first image, and selecting an image label for the image from the ranked text labels, the image label being selected based on the ranking.