Abstract: Embodiments relate to scraping web content. When scraping data, the target website sometimes redirects to different URLs within its domain. The different URLs represent the same context. Embodiments use a graph ontology to identify which redirected URLs represent the same page.

Abstract: A parsing facility within a service provider infrastructure can navigate through source documents of target web pages and mine a specific list of target data by utilizing multiple parsing frames received from an external computing resource and/or system. The parsing facility receives a series of a plurality of parsing frames at random intermittent intervals. The parsing facility can store each of the plurality of parsing frames within its internal storage and learns the differences between each of the plurality of parsing frames. After learning the differences, the parsing facility can recognize appropriate parsing frames to locate and mine each target data from the source documents. The parsing facility can mine data from source documents by using each of the plurality of parsing frames for every mining cycle, thereby effectively managing the reception and usage of multiple parsing frames without any errors or faults.

Abstract: Proxy servers within a service provider infrastructure are enabled to maintain multiple persistent connections among themselves and to exchange data bi-directionally in an unsolicited manner. Specifically, exit proxy servers are enabled to request their respective proxy supernodes to update the already existing network connection to support Web Socket communication channels. Accordingly, the respective proxy supernodes are enabled to update the network connection with the exit proxy servers to support WebSocket communication channels. A single instance of a proxy supernode and an exit proxy server can maintain multiple Web Socket communication channels with each other. By utilizing the said Web Socket communication channels, the proxy supernode and the exit proxy servers can exchange data with each other simultaneously without any data losses.

Abstract: Systems and methods herein provide for a proxy infrastructure. In the proxy infrastructure, a network element (e.g., a supernode) is connected with a plurality of exit nodes. At one of a plurality of messenger units of the proxy infrastructure, a proxy protocol request is received directly from a client computing device. The proxy protocol request specifies a request and a target. In response the proxy protocol request, a selection is made between one between one of the plurality of exit nodes. A message with the request is sent from the messenger to the supernode connected with the selected exit node. Finally, the message is sent from the supernode to the selected exit node to forward the request to the target.

Abstract: Traffic services for network addresses may be provided within threads executing within a main process for managing the traffic services. The threads may share resources within the main process, reducing the computing resources consumed to provide traffic services to large pools of network addresses. According to one embodiment, a method may include executing a main process for managing traffic services; determining, by the main process, a configuration specifying at least one or more destination addresses; instantiating, by the main process, one or more traffic service (TS) threads for the one or more destination addresses; and/or processing, by the one or more traffic service (TS) threads, inbound traffic for the corresponding one or more destination addresses. Other aspects and embodiments for traffic management are also disclosed.

Abstract: Systems and methods to manage and efficiently implement functional proxy services are disclosed. In the proxy services, a single instance of exit-node is connected to at least two or multiple supernodes at any given time. One of the plurality of supernodes is configured to ping and send diagnostic requests to the connected exit-node through a network. The exit-node is directed to send the pong message and diagnostic response data to a different supernode from among the plurality of supernodes connected to the exit-node. Likewise, a client's request is received by an element of the proxy service provider and forwarded to a specific supernode capable of forwarding the client's request to the exit-node. After performing the client's request, the exit-node returns response data to a different supernode from among the plurality of supernodes connected to the exit-node.

Abstract: Systems and methods herein provide for a proxy infrastructure. In the proxy infrastructure, a network element (e.g., a supernode) is connected with a plurality of exit nodes. At one of a plurality of messenger units of the proxy infrastructure, a proxy protocol request is received directly from a client computing device. The proxy protocol request specifies a request and a target. In response the proxy protocol request, a selection is made between one between one of the plurality of exit nodes. A message with the request is sent from the messenger to the supernode connected with the selected exit node. Finally, the message is sent from the supernode to the selected exit node to forward the request to the target.

Abstract: Systems and methods to intelligently optimize data collection requests are disclosed. In one embodiment, systems are configured to identify and select a complete set of suitable parameters to execute the data collection requests. In another embodiment, systems are configured to identify and select a partial set of suitable parameters to execute the data collection requests. The present embodiments can implement machine learning algorithms to identify and select the suitable parameters according to the nature of the data collection requests and the targets. Moreover, the embodiments provide systems and methods to generate feedback data based upon the effectiveness of the data collection parameters. Furthermore, the embodiments provide systems and methods to score the set of suitable parameters based on the feedback data and the overall cost, which are then stored in an internal database.

Abstract: Proxy servers within a service provider infrastructure are enabled to maintain multiple persistent connections among themselves and to exchange data bi-directionally in an unsolicited manner. Specifically, exit proxy servers are enabled to request their respective proxy supernodes to update the already existing network connection to support Web Socket communication channels. Accordingly, the respective proxy supernodes are enabled to update the network connection with the exit proxy servers to support WebSocket communication channels. A single instance of a proxy supernode and an exit proxy server can maintain multiple Web Socket communication channels with each other. By utilizing the said Web Socket communication channels, the proxy supernode and the exit proxy servers can exchange data with each other simultaneously without any data losses.

Abstract: Systems and methods for coordinating network connectivity and communication between proxy servers, exit-nodes and client modules are disclosed. In one aspect, the proxy infrastructure enables network connectivity between exit-nodes and proxy-nodes without the need of any proxy-gateways or middleware entities to delegate the connections. Proxy-nodes in the proxy infrastructure accept connections with exit-nodes based on geographical proximity, proxy-node metrics, such as server loads and clients' frequent preferences. Further, a single instance of proxy-node can communicate and instruct another instance of proxy-node, to service the client request via a suitable exit-node. Further still, a single instance of proxy-node can communicate and instruct another instance of proxy-node to redirect a suitable exit-node towards the first proxy-node in order to service the client request.

Abstract: Embodiments disclose a system that allows for improved generation of web requests for scraping that, because of the nature of the requests and time and manner they are sent out, appear more organic, as in human generated, than conventional automated scraping systems. The system then manages how a client request to scrape a target website is made to the site, masking the request in a manner that makes it appear to the Web server as if the request is not generated by an automated system. In this way, by appearing more organic, Web servers may be less likely to block requests from the disclosed system or may take longer to block requests from the disclosed system. By avoiding Web servers blocking requests and extending the lifetime of IP proxies before they are blocked, embodiments can use a limited IP proxy address space more efficiently.

Abstract: Systems and methods of task implementation are extended as provided herein and target the web crawling process through a step of submitting a request by a customer to a web crawler. The systems and methods allow a request for a web crawler to be enriched with a customized browsing profile in order to be categorized as an organic human user to obtain targeted content.

Abstract: System and method for efficiently implementing scalable, highly efficient decentralized proxy services through proxy infrastructures situated in different geo-locations. In one aspect, the systems and methods enable users from any geographical location to send requests to the geographically closest proxy infrastructure. One exemplary method described allows proxy infrastructures to gather, classify, and store metadata of exit nodes in its internal database. In another aspect, systems and methods described herein enable proxy infrastructures to select metadata of exit nodes from its internal database and forward requests from a user device to respective proxy servers or proxy supernodes to which the selected exit nodes are connected.

Abstract: A system and method of forming proxy server pools is provided. The method comprises several steps, such as requesting a pool to execute the user's request and retrieving an initial group. The system checks the service history of an initial group, including whether any of the proxy servers in an initial group are exclusive to existing pools. The exclusive proxy servers in an initial group with eligible proxy servers are replaced when needed and new proxy server pools are formed. The system also records the service history of proxy servers and pools before and after the pools are created. The method can also involve predicting the pool health in relation with the thresholds foreseen and replacing the proxy servers below the threshold.

Abstract: Empirical data of exit nodes are continuously monitored and each exit node's overall performance and available capacity are calculated. The empirical data can include monitoring the number of concurrent requests currently being executed by each exit node and the disconnection chronology of each exit node. Further, each exit node is tested by benchmark requests and ping messages and each exit node's quality rate is calculated. Additionally, systems and methods are provided to select an exit node with the highest quality and available capacity value, from a particular pool to route the user request.

Abstract: Systems and methods to intelligently optimize data collection requests are disclosed. In one embodiment, systems are configured to identify and select a complete set of suitable parameters to execute the data collection requests. In another embodiment, systems are configured to identify and select a partial set of suitable parameters to execute the data collection requests. The present embodiments can implement machine learning algorithms to identify and select the suitable parameters according to the nature of the data collection requests and the targets. Moreover, the embodiments provide systems and methods to generate feedback data based upon the effectiveness of the data collection parameters. Furthermore, the embodiments provide systems and methods to score the set of suitable parameters based on the feedback data and the overall cost, which are then stored in an internal database.

Abstract: Embodiments disclose a system that allows for improved generation of web requests for scraping that, because of the nature of the requests and time and manner they are sent out, appear more organic, as in human generated, than conventional automated scraping systems. The system then manages how a client request to scrape a target website is made to the site, masking the request in a manner that makes it appear to the Web server as if the request is not generated by an automated system. In this way, by appearing more organic, Web servers may be less likely to block requests from the disclosed system or may take longer to block requests from the disclosed system. By avoiding Web servers blocking requests and extending the lifetime of IP proxies before they are blocked, embodiments can use a limited IP proxy address space more efficiently.

Abstract: System and method for efficiently implementing scalable, highly efficient decentralized proxy services through proxy infrastructures situated in different geo-locations. In one aspect, the systems and methods enable users from any geographical location to send requests to the geographically closest proxy infrastructure. One exemplary method described allows proxy infrastructures to gather, classify, and store metadata of exit nodes in its internal database. In another aspect, systems and methods described herein enable proxy infrastructures to select metadata of exit nodes from its internal database and forward requests from a user device to respective proxy servers or proxy supernodes to which the selected exit nodes are connected.

Abstract: Systems and methods of task implementation are extended as provided herein and target the web crawling process through a step of submitting a request by a customer to a web crawler. The systems and methods allow a request for a web crawler to be enriched with a customized browsing profile in order to be categorized as an organic human user to obtain targeted content.

Abstract: Embodiments disclose a system that allows for improved generation of web requests for scraping that, because of the nature of the requests and time and manner they are sent out, appear more organic, as in human generated, than conventional automated scraping systems. The system then manages how a client request to scrape a target website is made to the site, masking the request in a manner that makes it appear to the Web server as if the request is not generated by an automated system. In this way, by appearing more organic, Web servers may be less likely to block requests from the disclosed system or may take longer to block requests from the disclosed system. By avoiding Web servers blocking requests and extending the lifetime of IP proxies before they are blocked, embodiments can use a limited IP proxy address space more efficiently.