Abstract: A method for performing crowd-sourced native application crawling is disclosed. The method includes determining a list of installed native applications installed on a user device and determining whether a set of crawling conditions are met. The method includes generating a work request in response to the set of crawling conditions being met by the user device and transmitting the work request to a content acquisition server. The work request includes the list of installed native applications. The method includes receiving a crawling task including an application access mechanism corresponding to a state of a native application. The method include launching the native application and setting the state of the native application based on the application access mechanism. The native application issues a content request to a content server. The method further includes receiving the content from the content server and transmitting the content to the content acquisition server.

Abstract: In some embodiments, an entry management module may receive a set of entries, each entry of the set of entries including a time-stamp value and a path identifier to a data object having content that may be changed. An entry selection module may select a first entry from the set of entries based on a first time-stamp value of the first entry, the first time-stamp value identifying a first time being no later than other times identified by time-stamp values of other entries. A crawler module may retrieve first content from a first data object using a first path identifier of the first entry. A freshness module may calculate a first updated timestamp value using a first freshness value, the first freshness value indicating a priority to retrieve possibly changed content from the first data object, the first updated time-stamp value replacing the first time-stamp value.

Abstract: A search system includes a user interface configured to receive information about a first application from a developer of the first application. The search system includes a state access module configured to obtain information about a first type of state of the first application from the developer. The information includes an action performed by the first type of state, a first access URL template, and a designation of at least one parameter for the first access URL template. The first application is configured to display a specific state of the first type of state in response to receiving an access URL formed by instantiating the first access URL template with at least one value for the at least one parameter. The search system includes a search engine configured to, in response to a query, obtain data from the first application according to the information about the first type of state.

Abstract: A search system includes a user interface configured to receive information about a first application from a developer of the first application. The search system includes a state access module configured to obtain information about a first type of state of the first application from the developer. The information includes an action performed by the first type of state, a first access URL template, and a designation of at least one parameter for the first access URL template. The first application is configured to display a specific state of the first type of state in response to receiving an access URL formed by instantiating the first access URL template with at least one value for the at least one parameter. The search system includes a search engine configured to, in response to a query, obtain data from the first application according to the information about the first type of state.

Abstract: A method includes determining, by a processing device of a user device, whether a set of crawling conditions are net by the user device, and generating, by the processing device a work request in response to the set of crawling conditions being met by the user device. The method also includes transmitting, by the processing device, the work request to a content acquisition server, and receiving, by the processing device, one or more crawling tasks from the content acquisition server. For each crawling task, the method further includes, requesting content from a content server based on information contained in the crawling task, receiving the content from the content server, and transmitting the content to the content acquisition server.

Abstract: In some embodiments, an entry management module may receive a set of entries, each entry of the set of entries including a time-stamp value and a path identifier to a data object having content that may be changed. An entry selection module may select a first entry from the set of entries based on a first time-stamp value of the first entry, the first time-stamp value identifying a first time being no later than other times identified by time-stamp values of other entries. A crawler module may retrieve first content from a first data object using a first path identifier of the first entry. A freshness module may calculate a first updated timestamp value using a first freshness value, the first freshness value indicating a priority to retrieve possibly changed content from the first data object, the first updated time-stamp value replacing the first time-stamp value.

Abstract: A method for performing crowd-sourced native application crawling is disclosed. The method includes determining a list of installed native applications installed on a user device and determining whether a set of crawling conditions are met. The method includes generating a work request in response to the set of crawling conditions being met by the user device and transmitting the work request to a content acquisition server. The work request includes the list of installed native applications. The method includes receiving a crawling task including an application access mechanism corresponding to a state of a native application. The method include launching the native application and setting the state of the native application based on the application access mechanism. The native application issues a content request to a content server. The method further includes receiving the content from the content server and transmitting the content to the content acquisition server.