Abstract: Described herein are technologies relating to predicting whether a resource is spam based solely upon a Uniform Resource Locator (URL) for the resource. The URL is tokenized in connection with generating a sequence of numerical identifiers for the resource. A score for the URL is computed based upon the sequence of numerical identifiers, where the score is indicative of a probability that the resource pointed to by the URL is spam. generating a score for the URL based upon the sequence of numbers, wherein the score is indicative of a probability that the resource pointed to by the URL is spam. When the score is above a predefined threshold, a label is assigned to the URL that indicates that the resource pointed to by the URL is spam, and an entry for the resource is not included in a search engine index based upon the label assigned to the URL.

Abstract: Described herein are technologies related to inferring information about a webpage based upon semantics of a uniform resource location (URL) of the webpage. The URL is tokenized to create a sequence of tokens. An embedding for the URL is generated based upon the sequence of tokens, wherein the embedding is representative of semantics of the URL. Based upon the embedding for the URL, information about the webpage pointed to by the URL is inferred, the webpage is retrieved, and information is extracted from the webpage based upon the information inferred about the webpage.

Abstract: Described herein are technologies related to inferring information about a webpage based upon semantics of a uniform resource location (URL) of the webpage. The URL is tokenized to create a sequence of tokens. An embedding for the URL is generated based upon the sequence of tokens, wherein the embedding is representative of semantics of the URL. Based upon the embedding for the URL, information about the webpage pointed to by the URL is inferred, the webpage is retrieved, and information is extracted from the webpage based upon the information inferred about the webpage.

Abstract: A link-analyzing system (LAS) extracts information from a markup language (ML) document associated with a web page link. In some implementations, the information that is extracted includes at least: a) address content that is part of the link's destination address; and b) text that is associated with the link but that is not part of the destination address itself. The LAS generates feature information based on the address content and the text, and then uses a classification model to make a classification assessment for the link based on the feature information. In some implementations, the LAS can control a crawling engine based on the classification assessment. In some implementations, the LAS can revise a low-confidence classification assessment based on an examination of the classification assessments of a group of similar links described by the ML document. Other implementations use the above-described functionality to classify other parts of an ML document.

Abstract: Described herein are technologies relating to predicting whether a resource is spam based solely upon a Uniform Resource Locator (URL) for the resource. The URL is tokenized in connection with generating a sequence of numerical identifiers for the resource. A score for the URL is computed based upon the sequence of numerical identifiers, where the score is indicative of a probability that the resource pointed to by the URL is spam. generating a score for the URL based upon the sequence of numbers, wherein the score is indicative of a probability that the resource pointed to by the URL is spam. When the score is above a predefined threshold, a label is assigned to the URL that indicates that the resource pointed to by the URL is spam, and an entry for the resource is not included in a search engine index based upon the label assigned to the URL.

Abstract: Described herein are technologies related to inferring information about a webpage based upon semantics of a uniform resource location (URL) of the webpage. The URL is tokenized to create a sequence of tokens. An embedding for the URL is generated based upon the sequence of tokens, wherein the embedding is representative of semantics of the URL. Based upon the embedding for the URL, information about the webpage pointed to by the URL is inferred, the webpage is retrieved, and information is extracted from the webpage based upon the information inferred about the webpage.