Abstract: A method includes receiving a first request from a first user device to access a first resource that includes data for a second user account for which access to the data is restricted to authorized users, the first request including an authorization token and associated with a first user identifier that identifies a first user; determining that the first user identifier does not identify an authorized user and in response: determining that the first user identifier identifies an authorized user based on the authorization token, and provide the first resource to the first user device; receiving a second request for access to data to the second user account, the second request associated with the first user identifier; and based on the first user identifier being determined to identify authorized user, providing access to the data to the second user account in response to the second request.

Abstract: Various aspects of the subject technology relate to systems, methods, and machine-readable media for providing an updated list of public content items to a search engine. A system is configured to determine whether a sharing property of a content item stored by a first service has been altered, wherein the sharing property indicates whether the content item is available to a different set of users, update a list of content items if the sharing property has been altered, and provide the updated list of content items to a second service.

Abstract: A method includes receiving a first request from a first user device to access a first resource that includes data for a second user account for which access to the data is restricted to authorized users, the first request including an authorization token and associated with a first user identifier that identifies a first user; determining that the first user identifier does not identify an authorized user and in response: determining that the first user identifier identifies an authorized user based on the authorization token, and provide the first resource to the first user device; receiving a second request for access to data to the second user account, the second request associated with the first user identifier; and based on the first user identifier being determined to identify authorized user, providing access to the data to the second user account in response to the second request.

Abstract: A method includes receiving a first request from a first user device to access a first resource that includes data for a second user account for which access to the data is restricted to authorized users, the first request including an authorization token and associated with a first user identifier that identifies a first user; determining that the first user identifier does not identify an authorized user and in response: determining that the first user identifier identifies an authorized user based on the authorization token, and provide the first resource to the first user device; receiving a second request for access to data to the second user account, the second request associated with the first user identifier; and based on the first user identifier being determined to identify authorized user, providing access to the data to the second user account in response to the second request.

Abstract: A computer-implemented method of detecting automated identities, including. The method includes, using one or more computing devices, providing for display to a user a first representation of a first profile for a first identity and a second representation of a second profile for a second identity, providing for display to the user a choice indication mechanism for indicating a judgment about the first representation of the first profile for the first identity and the second representation of the second profile for the second identity, receiving an indication of the judgment from the user about the first representation of the first profile for the first identity and the second representation of the second profile for the second identity, and determining whether at least one of the first identity, the second identity, or the user is more or less likely to be automated, based at least in part on the received indication.

Abstract: Systems and methods for scheduling documents for crawling are disclosed. In some implementations, a method includes obtaining sitemap information for a plurality of websites; and analyzing the sitemap information to identify a website, in the plurality of websites. The website has sitemap information that is at least potentially out of date. The method also includes updating the sitemap information for the identified website by downloading updated sitemap information for the identified website; and scheduling documents for crawling in accordance with the updated sitemap information for the identified website.

Abstract: Various aspects of the subject technology relate to systems, methods, and machine-readable media for providing an updated list of public content items to a search engine. A system is configured to determine whether a sharing property of a content item stored by a first service has been altered, wherein the sharing property indicates whether the content item is available to a different set of users, update a list of content items if the sharing property has been altered, and provide the updated list of content items to a second service.

Abstract: A search engine crawler includes a distributed set of schedulers that are associated with one or more segments of document identifiers (e.g., URLs) corresponding to documents on a network (e.g., WWW). Each scheduler handles the scheduling of document identifiers (for crawling) for a subset of the known document identifiers. Using a starting set of document identifiers, such as the document identifiers crawled (or scheduled for crawling) during the most recent completed crawl, the scheduler removes from the starting set those document identifiers that have been unreachable in each of the last X crawls. Other filtering mechanisms may also be used to filter out some of the document identifiers in the starting set. The resulting list of document identifiers is written to a scheduled output file for use in a next crawl cycle.

Abstract: A search engine crawler includes a scheduler for determining which documents to download from their respective host servers. Some documents, known to be stable based on one or more record from prior crawls, are reused from a document repository. A reuse flag is set in a scheduler record that also contains a document identifier, the reuse flag indicating whether the document should be retrieved from a first database, such as the World Wide Web, or a second database, such as a document repository. A set of such scheduler records are used during a crawl by the search engine crawler to determine which database to use when retrieving the documents identified in the scheduler records.

Abstract: A method of analyzing documents or relationships between documents includes receiving a notification of an available metadata document containing information about one or more network-accessible documents, obtaining a document format indicator associated with the metadata document, selecting a document crawler using the document format indicator, and crawling at least some of the network-accessible documents using the selected document crawler.

Abstract: A method of analyzing documents or relationships between documents includes receiving a notification of an available metadata document containing information about one or more network-accessible documents, obtaining a document format indicator associated with the metadata document, selecting a document crawler using the document format indicator, and crawling at least some of the network-accessible documents using the selected document crawler.

Abstract: Web crawlers crawl websites to access documents of the website for purposes of indexing the documents for search engines. The web crawlers crawl a specified website at a crawl rate that is based on multiple factors. One of the factors is a pre-set crawl rate limit. According to certain embodiments, an owner for a specified website is enabled to modify the crawl rate limit for the specified website when one or more pre-set criteria are met.

Abstract: Methods and systems for a web crawler scheduler that utilizes sitemaps from websites are described. A web crawler scheduling system receives a notification from a website or web server. In response to the notification, the system accesses one or more sitemap(s) for documents associated with the website or web server. The system schedules crawls of the documents based on information identified from the sitemaps. The system crawls at least a subset of the documents scheduled for crawling.

Abstract: A method includes receiving a first request from a first user device to access a first resource that includes data for a second user account for which access to the data is restricted to authorized users, the first request including an authorization token and associated with a first user identifier that identifies a first user; determining that the first user identifier does not identify an authorized user and in response: determining that the first user identifier identifies an authorized user based on the authorization token, and provide the first resource to the first user device; receiving a second request for access to data to the second user account, the second request associated with the first user identifier; and based on the first user identifier being determined to identify authorized user, providing access to the data to the second user account in response to the second request.

Abstract: A method of analyzing documents or relationships between documents includes receiving a notification of an available metadata document containing information about one or more network-accessible documents, obtaining a document format indicator associated with the metadata document, selecting a document crawler using the document format indicator, and crawling at least some of the network-accessible documents using the selected document crawler.

Abstract: A method of analyzing documents or relationships between documents includes receiving a notification of an available metadata document containing information about one or more network-accessible documents, obtaining a document format indicator associated with the metadata document, selecting a document crawler using the document format indicator, and crawling at least some of the network-accessible documents using the selected document crawler.

Abstract: Methods and systems for a web crawler scheduler that utilizes sitemaps from websites are described. A web crawler scheduling system receives a notification from a website or web server. In response to the notification, the system accesses one or more sitemap(s) for documents associated with the website or web server. The system schedules crawls of the documents based on information identified from the sitemaps. The system crawls at least a subset of the documents scheduled for crawling.

Abstract: Web crawlers crawl websites to access documents of the website for purposes of indexing the documents for search engines. The web crawlers crawl a specified website at a crawl rate that is based on multiple factors. One of the factors is a pre-set crawl rate limit. According to certain embodiments, an owner for a specified website is enabled to modify the crawl rate limit for the specified website when one or more pre-set criteria are met.

Abstract: A search engine crawler includes a distributed set of schedulers that are associated with one or more segments of document identifiers (e.g., URLs) corresponding to documents on a network (e.g., WWW). Each scheduler handles the scheduling of document identifiers (for crawling) for a subset of the known document identifiers. Using a starting set of document identifiers, such as the document identifiers crawled (or scheduled for crawling) during the most recent completed crawl, the scheduler removes from the starting set those document identifiers that have been unreachable in each of the last X crawls. Other filtering mechanisms may also be used to filter out some of the document identifiers in the starting set. The resulting list of document identifiers is written to a scheduled output file for use in a next crawl cycle.

Abstract: Methods and systems for a web crawler scheduler that utilizes sitemaps from websites are described. A web crawler scheduling system receives a notification from a website or web server. In response to the notification, the system accesses one or more sitemap(s) for documents associated with the website or web server. The system schedules crawls of the documents based on information identified from the sitemaps. The system crawls at least a subset of the documents scheduled for crawling.