Abstract: Techniques for facilitating natural language interactions with visual interactive content are described. During a build time, a system analyzes various websites and applications relating to a particular user goal to understand website and application navigation and information relating to the user goal. The learned information is used to store configuration data. During runtime, when a user request performance of an action, the system engages in a dialog with the user to complete the user's goal. The system uses the stored configuration data to determine actions to be performed at a website or application to complete the user's goal, and determines system responses to present to the user to facilitate completion of the goal. Such system responses may request information from the user, may inform the user of information displayed at the website or application, etc.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium determining image relevance responsive to a search query. A method includes, for each resource in a plurality of resources, wherein each resource includes one or more images and text that is separate from each of the one or more images: determining, by a data processing apparatus, and from the text of the resource, resource topics described by the text of the resource. For each of the one or more images, processing the image to determine a set of image topics that describe topics to which content depicted in the image belongs. Determining, by the data process apparatus, one or more topic match scores, wherein each topic match score is a measure of relevance one or more of the image topics of the image to one or more of the resource topics of the resource.

Abstract: Methods, apparatus, and computer-readable media are provided for analyzing a cluster of communications, such as B2C emails, to generate a template for the cluster that defines transient segments and fixed segments of the cluster of communications. More particularly, methods, apparatus, and computer-readable media are provided for generating and/or applying a trained structured machine learning model for a generated template that can be used to determine, for one or more transient segments of subsequent communications, a corresponding probability that a given semantic label is the correct semantic label for extracted content of the transient segment(s).

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating and providing animation snippets. In one aspect, a method includes determining page resources that each include a plurality of persistent visual resources, for each of the determined page resources, generating one or more animation snippets from the persistent visual resources, for a page resource for which one or more animation snippets have been generated, selecting one of the animation snippets for display as a search result that identifies the page resource as being responsive to a search query; and providing animation snippet to a user device from which the query was submitted.

Abstract: Techniques are described herein for generating and applying event data extraction templates. In various implementations, a data extraction template may be applied to structured communications to extract, from each structured communication, event data associated with a transient markup language path indicated in the data extraction template. The data extraction template may include an event-related semantic data type assigned to the transient markup language path and a strength of association between the transient structural path and the event-related semantic data type. Feedback may be obtained concerning event data extracted from one or more of the structured communications. Based on the feedback, the strength of association between the transient markup language path and the event-related semantic data type may be altered.

Abstract: Methods, apparatus, and computer-readable media are provided for generating and applying data extraction templates. In various implementations, a corpus of structured communications such as emails may be grouped into clusters based on one or more similarities between the structured communications. A set of structural paths may be identified from structured communications of a particular cluster. One or more structural paths of the set may be classified as transient wherein a count of occurrences of one or more associated segments of text across the particular cluster satisfies a criterion. One or more transient paths may be assigned a semantic data type and/or a confidentiality designation based on various signals. A data extraction template may be generated to extract, from subsequent structured communications, segments of text associated with transient (and in some cases, non-confidential) structural paths.

Abstract: Methods, apparatus, systems, and computer-readable media are provided for selecting pattern matching segments suitable for electronic communication clustering. A set of pattern matching segments may be identified that match at least one of a corpus of electronic communication addresses. A measure of coverage of each of the set of pattern matching segments across the corpus of electronic communication addresses may be determined. A score associated with each pattern matching segment may be determined based on the measure of coverage and one or more measures of flexibility associated with each of the set of pattern matching segments. One or more of the pattern matching segments may be selected based on the determine scores. A corpus of electronic communications may then be grouped into a plurality of clusters based on a comparison of the one or more selected pattern matching segments to electronic communication addresses associated with the corpus of electronic communications.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating and providing animation snippets. In one aspect, a method includes determining page resources that each include a plurality of persistent visual resources, for each of the determined page resources, generating one or more animation snippets from the persistent visual resources, for a page resource for which one or more animation snippets have been generated, selecting one of the animation snippets for display as a search result that identifies the page resource as being responsive to a search query; and providing animation snippet to a user device from which the query was submitted.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium determining image relevance responsive to a search query. In one aspect, a method includes, for each resource in a plurality of resources, wherein each resource includes one or more images and text that is separate from each of the one or more images: determining, by a data processing apparatus, and from the text of the resource, resource topics described by the text of the resource. For each of the one or more images, processing the image to determine a set of image topics that describe topics to which content depicted in the image belongs. Determining, by the data process apparatus, one or more topic match scores, wherein each topic match score is a measure of relevance one or more of the image topics of the image to one or more of the resource topics of the resource.

Abstract: Methods, apparatus, systems, and computer-readable media are provided for generating and applying data extraction templates. In various implementations, a corpus of plain text communications such as emails may be grouped into clusters based on one or more similarities between the plain text communications. One or more segments of communications of a particular cluster may be classified as transient based on textual pattern matching. One or more other segments of the communications of the particular cluster may be classified as transient based on various criteria. One or more transient segments may be assigned a generic and/or specific semantic data type and/or a confidentiality designation based on various signals. A data extraction template may be generated to extract, from subsequent plain text communications, content associated with transient (and in some cases, non-confidential) segments.

Abstract: Methods, apparatus, systems, and computer-readable media are provided for determining whether communications are attempts at phishing. In various implementations, a potentially-deceptive communication may be matched to one or more templates of a plurality of templates. Each template may represent content shared among a cluster of communications sent by a legitimate entity. In various implementations, it may be determined that an address associated with the communication is not affiliated with one or more legitimate entities associated with the one or more matched templates. In various implementations, the communication may be classified as a phishing attempt based on the determining.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for redacting data from a document collection generated for a set of documents that include personal information. The redaction of the data is based in part on a comparison of the document collection to a set of a personal documents of users for which the users have provided explicit approval to use in the processing of the document collection.

Abstract: Methods, apparatus, systems, and computer-readable media are provided for determining whether communications are attempts at phishing. In various implementations, a potentially-deceptive communication may be matched to one or more templates of a plurality of templates. Each template may represent content shared among a cluster of communications sent by a legitimate entity. In various implementations, it may be determined that an address associated with the communication is not affiliated with one or more legitimate entities associated with the one or more matched templates. In various implementations, the communication may be classified as a phishing attempt based on the determining.

Abstract: Methods and apparatus are described herein for generating and applying event data extraction templates. In various implementations, a set of structural paths may be identified from a corpus of communications. A first structural path of the set of structural paths, associated with a first segment of text, may be classified as transient in response to a determination that a frequency of occurrences of the first segment of text across the corpus satisfies a criterion. Event heuristics may be applied to the communications of the corpus. A determination may be made, based on the applying, that the communications of the corpus are event-related. An event data type may be assigned to the transient structural path based on the applying. An event data extraction template may be generated to extract, from one or more subsequent communications, one or more event-related segments of text associated with the transient structural path.

Abstract: Methods, apparatus, systems, and computer-readable media are provided for determining whether communications are attempts at phishing. In various implementations, a potentially-deceptive communication may be matched to one or more templates of a plurality of templates. Each template may represent content shared among a cluster of communications sent by a trustworthy entity. In various implementations, it may be determined that an address associated with the communication is not affiliated with one or more trustworthy entities associated with the one or more matched templates. In various implementations, the communication may be classified as a phishing attempt based on the determining.

Abstract: Methods, apparatus, and computer-readable media are provided for generating and applying data extraction templates. In various implementations, a corpus of structured communications such as emails may be grouped into clusters based on one or more similarities between the structured communications. A set of structural paths may be identified from structured communications of a particular cluster. One or more structural paths of the set may be classified as transient wherein a count of occurrences of one or more associated segments of text across the particular cluster satisfies a criterion. One or more transient paths may be assigned a semantic data type and/or a confidentiality designation based on various signals. A data extraction template may be generated to extract, from subsequent structured communications, segments of text associated with transient (and in some cases, non-confidential) structural paths.

Abstract: Methods, apparatus, systems, and computer-readable media are provided for determining whether communications are attempts at phishing. In various implementations, a potentially-deceptive communication may be matched to one or more templates of a plurality of templates. Each template may represent content shared among a cluster of communications sent by a trustworthy entity. In various implementations, it may be determined that an address associated with the communication is not affiliated with one or more trustworthy entities associated with the one or more matched templates. In various implementations, the communication may be classified as a phishing attempt based on the determining.

Abstract: Methods, apparatus, systems, and computer-readable media are provided for classifying, or “labeling,” documents such as emails en masse based on association with a cluster/template. In various implementations, a corpus of documents may be grouped into a plurality of disjoint clusters of documents based on one or more shared content attributes. A classification distribution associated with a first cluster of the plurality of clusters may be determined based on classifications assigned to individual documents of the first cluster. A classification distribution associated with a second cluster of the plurality of clusters may then be determined based at least in part on the classification distribution associated with the first cluster and a relationship between the first and second clusters.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for redacting data from a document collection generated for a set of documents that include personal information. The redaction of the data is based in part on a comparison of the document collection to a set of a personal documents of users for which the users have provided explicit approval to use in the processing of the document collection.