Abstract: A visual query such as a photograph, a screen shot, a scanned image, a video frame, or an image created by a content authoring application is submitted to a visual query search system. The search system processes the visual query by sending it to a plurality of parallel search systems, each implementing a distinct visual query search process. These parallel search systems may include but are not limited to optical character recognition (OCR), facial recognition, product recognition, bar code recognition, object-or-object-category recognition, named entity recognition, and color recognition. Then at least one search result is sent to the client system. In some embodiments, when the visual query is an image containing a text element and a non-text element, at least one search result includes an optical character recognition result for the text element and at least one image-match result for the non-text element.

Abstract: Systems and methods are provided for a personalized entity repository. For example, a computing device comprises a personalized entity repository having fixed sets of entities from an entity repository stored at a server, a processor, and memory storing instructions that cause the computing device to identify fixed sets of entities that are relevant to a user based on context associated with the computing device, rank the fixed sets by relevancy, and update the personalized entity repository using selected sets determined based on the rank and on set usage parameters applicable to the user. In another example, a method includes generating fixed sets of entities from an entity repository, including location-based sets and topic-based sets, and providing a subset of the fixed sets to a client, the client requesting the subset based on the client's location and on items identified in content generated for display on the client.

Abstract: Methods and apparatus directed to segmenting content displayed on a computing device into regions. The segmenting of content displayed on the computing device into regions is accomplished via analysis of pixels of a “screenshot image” that captures at least a portion of (e.g., all of) the displayed content. Individual pixels of the screenshot image may be analyzed to determine one or more regions of the screenshot image and to optionally assign a corresponding semantic type to each of the regions. Some implementations are further directed to generating, based on one or more of the regions, interactive content to provide for presentation to the user via the computing device.

Abstract: Systems and methods simulate a hyperlink in regular content displayed on a screen. An example method can include generating, responsive to detecting a simulated hyperlink indication, a centered selection from content displayed on a display of a computing device, providing the centered selection to a simulated hyperlink model that predicts an operation given the centered selection, and initiating the operation using an intent associated with a mobile application. The simulated hyperlink model may also provide, from the centered selection, an intelligent selection used the intent's parameter.

Abstract: A facial recognition search system identifies one or more likely names (or other personal identifiers) corresponding to the facial image(s) in a query as follows. After receiving the visual query with one or more facial images, the system identifies images that potentially match the respective facial image in accordance with visual similarity criteria. Then one or more persons associated with the potential images are identified. For each identified person, person-specific data comprising metrics of social connectivity to the requester are retrieved from a plurality of applications such as communications applications, social networking applications, calendar applications, and collaborative applications. An ordered list of persons is then generated by ranking the identified persons in accordance with at least metrics of visual similarity between the respective facial image and the potential image matches and with the social connection metrics.

Abstract: Systems and methods are provided for detecting and ranking entities identified in screen content displayed on a mobile device. For example, a method includes receiving an image captured from a mobile device display for a mobile application and determining a window that includes a chronological set of images, the images each representing a respective screen captured from a display of a mobile device and having an associated timestamp. The method also includes identifying entities appearing in images in a first portion of the window using text for images in a remaining portion of the window as context to disambiguate ambiguous entity references.

Abstract: Systems and methods are provided for a content-based security for computing devices. An example method includes identifying content rendered by a mobile application, the content being rendered during a session, generating feature vectors from the content and determining that the feature vectors do not match a classification model. The method also includes providing, in response to the determination that the feature vectors do not match the classification model, a challenge configured to authenticate a user of the mobile device. Another example method includes determining a computing device is located at a trusted location, capturing information from a session, the information coming from content rendered by a mobile application during the session, generating feature vectors for the session, and repeating this until a training criteria is met. The method also includes training a classification model using the feature vectors and authenticating a user of the device using the trained classification model.

Abstract: Systems and methods simulate a hyperlink in regular content displayed on a screen. An example method can include generating, responsive to detecting a simulated hyperlink indication, a centered selection from content displayed on a display of a computing device, providing the centered selection to a simulated hyperlink model that predicts an operation given the centered selection, and initiating the operation using an intent associated with a mobile application. The simulated hyperlink model may also provide, from the centered selection, an intelligent selection used the intent's parameter.

Abstract: Implementations of the present disclosure include actions of receiving image data of an image capturing a scene, receiving data describing one or more entities determined from the scene, the one or more entities being determined from the scene, determining one or more actions based on the one or more entities, each action being provided at least partly based on search results from searching the one or more entities, and providing instructions to display an action interface comprising one or more action elements, each action element being to induce execution of a respective action, the action interface being displayed in a viewfinder

Abstract: A facial recognition search system identifies one or more likely names (or other personal identifiers) corresponding to the facial image(s) in a query as follows. After receiving the visual query with one or more facial images, the system identifies images that potentially match the respective facial image in accordance with visual similarity criteria. Then one or more persons associated with the potential images are identified. For each identified person, person-specific data comprising metrics of social connectivity to the requester are retrieved from a plurality of applications such as communications applications, social networking applications, calendar applications, and collaborative applications. An ordered list of persons is then generated by ranking the identified persons in accordance with at least metrics of visual similarity between the respective facial image and the potential image matches and with the social connection metrics.

Abstract: Systems and methods prevent or restrict the mining of content on a mobile device. For example, a method may include identifying a mining-restriction mark in low order bits or high order bits in a frame buffer of a mobile device and determining whether the mining-restriction mark prevents mining of content. Mining includes non-transient storage of a copy or derivations of data in the frame buffer. The method may also include preventing the mining of data in the frame buffer when the mining-restriction mark prevents mining.

Abstract: Implementations of the present disclosure include actions of receiving image data of an image capturing a scene, receiving data describing one or more entities determined from the scene, the one or more entities being determined from the scene, determining one or more actions based on the one or more entities, each action being provided at least partly based on search results from searching the one or more entities, and providing instructions to display an action interface comprising one or more action elements, each action element being to induce execution of a respective action, the action interface being displayed in a viewfinder.

Abstract: Methods and apparatus directed to segmenting content displayed on a computing device into regions. The segmenting of content displayed on the computing device into regions is accomplished via analysis of pixels of a “screenshot image” that captures at least a portion of (e.g., all of) the displayed content. Individual pixels of the screenshot image may be analyzed to determine one or more regions of the screenshot image and to optionally assign a corresponding semantic type to each of the regions. Some implementations are further directed to generating, based on one or more of the regions, interactive content to provide for presentation to the user via the computing device.

Abstract: Systems and methods are provided for providing user assistance on a mobile device. For example, a method may include receiving a first image of a screen captured on a mobile device, determining key content in the first image by performing recognition on the first image, and using the key content as a query to an index that associates key items with screen capture images previously received from the mobile device. The method may further include determining that a search result of the query includes at least one key item with a rank that exceeds a relevancy threshold, determining a portion of a second image from the screen capture images that is associated with the at least one key item, generating annotation data that includes the portion of the second image, and providing the annotation data for display with the screen on the mobile device.

Abstract: Systems and methods for indexing onscreen content on a mobile device are provided. For example, a method can include determining, using an index for a user that associates key items with images of screens captured from a mobile device of the user, that a first key item is responsive to a query initiated at the mobile device, and identifying responsive images, the responsive images being associated with the first key item. The method may also include highlighting, in the responsive images, areas corresponding to the first key item, and providing a search result to the mobile device that includes portions of the responsive images. The portions of the responsive images may be areas corresponding to the first key item in the images.

Abstract: Methods and apparatus directed to segmenting content displayed on a computing device into regions. The segmenting of content displayed on the computing device into regions is accomplished via analysis of pixels of a “screenshot image” that captures at least a portion of (e.g., all of) the displayed content. Individual pixels of the screenshot image may be analyzed to determine one or more regions of the screenshot image and to optionally assign a corresponding semantic type to each of the regions. Some implementations are further directed to generating, based on one or more of the regions, interactive content to provide for presentation to the user via the computing device.

Abstract: Methods, systems, and apparatus for receiving a query image and a user tap location, processing the received query image based on the user tap location, identifying one or more entities associated with the processed query image and in response to receiving (i) the query image, and (ii) the user tap location, providing information about the identified one or more of the entities.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, to generate a scannable document. In one aspect, a method includes receiving a scan request, wherein the scan request includes a plurality of text images; for each text image of the plurality of text images: rectifying the text image to generate a text image with parallel image lines, generating a plurality of word bounding boxes that enclose one or more connected components in the text image, wherein each word bounding box is associated with a respective word, and generating, for each respective word in the text image, a plurality of points that represent the respective word; combining the plurality of text images to form a single text document; and providing the combined image as a scannable document.

Abstract: A visual query is received from a client system, along with location information for the client system, and processed by a server system. The server system sends the visual query and the location information to a visual query search system, and receives from the visual query search system enhanced location information based on the visual query and the location information. The server system then sends a search query, including the enhanced location information, to a location-based search system. The search system receives and provides to the client one or more search results to the client system.

Abstract: Systems and methods prevent or restrict the mining of content on a mobile device. For example, a method may include identifying a mining-restriction mark in low order bits or high order bits in a frame buffer of a mobile device and determining whether the mining-restriction mark prevents mining of content. Mining includes non-transient storage of a copy or derivations of data in the frame buffer. The method may also include preventing the mining of data in the frame buffer when the mining-restriction mark prevents mining.