Abstract: Provided herein are system, apparatus, device, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for caching media resources during a scraping operation. Web resources needed by webpage are stored in a cache that is used by multiple browsers that are scraping the webpage. When an unexpired entry for the web resource is present in the cache, a browser retrieves the web resource and cache instead of making a request from the webpage. This offers a technological improvement of reducing the traffic burden on proxy servers needed to forward the scraping requests and responses.

Abstract: Disclosed herein are system, method, and computer program product embodiments for improving web scraping technology by using machine learning to generate parsing expressions. A system receives a request to identify an element in a first document at a target web page. The system downloads and modifies the first document by adding an index value as an attribute to a tag for the element. A query is submitted to a large language model (LLM), including the modified first document, a description of the element, and a request asking the LLM to identify the element based on the description. The system obtains, from the LLM, the index value assigned to the element. The system generates an expression defining a path to the element in the first document using the index returned by the large language model. The system downloads a second document, and parses data of a second element using the expression.

Abstract: A computer-implemented method is provided that enables remote control of a headless browser. A message from a client to open a connection to a headless browser running on a device remote from the client is received. A command to provision the browser is sent to the headless browser. The command may configure the browser to appear as if the browser is controlled by a human. A command for controlling the headless browser is received from the client. The command is sent to the headless browser for execution. A response to the command is received from the headless browser. Finally, the response is forwarded to the client.

Abstract: Systems and methods to manage and efficiently perform authorization of multiple proxy clients are disclosed. Furthermore, systems and methods to measure and check whether the web traffic of one or more client devices has reached a permissible limit of web traffic assigned by the proxy service provider. Specifically, a proxy is configured to gather and save authorization information of one or more clients within its memory. Therefore, the proxy server can verify and authorize one or more clients by utilizing the data from its memory. Furthermore, the proxy is configured to measure and report the utilized web traffic of one or more client devices to a messaging platform. In another aspect, systems and methods to check whether one or more client devices have reached a permissible amount of web traffic limit are disclosed.

Abstract: Systems and methods to manage and efficiently implement functional proxy services are disclosed. In the proxy services, a single instance of exit-node is connected to at least two or multiple supernodes at any given time. One of the plurality of supernodes is configured to ping and send diagnostic requests to the connected exit-node through a network. The exit-node is directed to send the pong message and diagnostic response data to a different supernode from among the plurality of supernodes connected to the exit-node. Likewise, a client's request is received by an element of the proxy service provider and forwarded to a specific supernode capable of forwarding the client's request to the exit-node. After performing the client's request, the exit-node returns response data to a different supernode from among the plurality of supernodes connected to the exit-node.

Abstract: Disclosed herein are system, method, and computer program product embodiments for improving web scraping technology by dynamically updating scraping parameters. A scrape system may retrieve a webpage addressed at a target URL. The scrape system may compile an object list from the webpage. The scrape system may determine a number of objects in the object list. Based on the determined number of objects, the scrape system may determine a next time to retrieve the webpage addressed at the target URL such that, when the determined number of objects is greater, the next time is sooner. When the determined next time occurs, the scrape system may re-retrieve the webpage addressed at the target URL.

Abstract: Systems and methods herein provide for a proxy infrastructure. In the proxy infrastructure, a network element (e.g., a supernode) is connected with a plurality of exit nodes. At one of a plurality of messenger units of the proxy infrastructure, a proxy protocol request is received directly from a client computing device. The proxy protocol request specifies a request and a target. In response the proxy protocol request, a selection is made between one between one of the plurality of exit nodes. A message with the request is sent from the messenger to the supernode connected with the selected exit node. Finally, the message is sent from the supernode to the selected exit node to forward the request to the target.

Abstract: System and method for efficiently implementing scalable, highly efficient decentralized proxy services through proxy infrastructures situated in different geo-locations. In one aspect, the systems and methods enable users from any geographical location to send requests to the geographically closest proxy infrastructure. One exemplary method described allows proxy infrastructures to gather, classify, and store metadata of exit nodes in its internal database. In another aspect, systems and methods described herein enable proxy infrastructures to select metadata of exit nodes from its internal database and forward requests from a user device to respective proxy servers or proxy supernodes to which the selected exit nodes are connected.

Abstract: This invention discloses selecting a proxy IP device (Proxy-ICDAPIA) by its geographical coordinates and distance from the target web service. A method and system are disclosed, allowing users of the proxy service to specify their preferred Proxy-ICDAPIAs by geographic coordinates in the user's request. Embodiments describe the Proxy-ICDAPIA selection using Geohash areas and strings, also, selecting Proxy-ICDAPIAs from circular geographical areas specified by center coordinates X, Y and radius R. The solution serves for selecting Proxy-ICDAPIAs efficiently in cities and metropolitan areas where geographic density of web services is high, and selecting Proxy-ICDAPIAs by country/city may be not sufficient. Another problem solved when a client uses Proxy-ICDAPIAs selection by coordinates, is an inability to provide a country code. The aforementioned functionality does work without a specified country code, some countries are not included in the pool used when no parameters are provided.

Abstract: Systems and methods to manage and efficiently perform authorization of multiple proxy clients are disclosed. Furthermore, systems and methods to measure and check whether the web traffic of one or more client devices has reached a permissible limit of web traffic assigned by the proxy service provider. Specifically, a proxy is configured to gather and save authorization information of one or more clients within its memory. Therefore, the proxy server can verify and authorize one or more clients by utilizing the data from its memory. Furthermore, the proxy is configured to measure and report the utilized web traffic of one or more client devices to a messaging platform. In another aspect, systems and methods to check whether one or more client devices have reached a permissible amount of web traffic limit are disclosed.

Abstract: Systems and methods to manage and efficiently implement functional proxy services are disclosed. In the proxy services, a single instance of exit-node is connected to at least two or multiple supernodes at any given time. One of the plurality of supernodes is configured to ping and send diagnostic requests to the connected exit-node through a network. The exit-node is directed to send the pong message and diagnostic response data to a different supernode from among the plurality of supernodes connected to the exit-node. Likewise, a client's request is received by an element of the proxy service provider and forwarded to a specific supernode capable of forwarding the client's request to the exit-node. After performing the client's request, the exit-node returns response data to a different supernode from among the plurality of supernodes connected to the exit-node.

Abstract: Systems and methods herein provide for a proxy infrastructure. In the proxy infrastructure, a network element (e.g., a supernode) is connected with a plurality of exit nodes. At one of a plurality of messenger units of the proxy infrastructure, a proxy protocol request is received directly from a client computing device. The proxy protocol request specifies a request and a target. In response the proxy protocol request, a selection is made between one between one of the plurality of exit nodes. A message with the request is sent from the messenger to the supernode connected with the selected exit node. Finally, the message is sent from the supernode to the selected exit node to forward the request to the target.

Abstract: The present invention discloses a method and system of an internet proxy service, configuring the service in an ingress proxy—a virtual single access point receiving requests from a user's device. Users' requests to connect to a target web service on the internet may be directed through a plurality of different proxy service nodes being managed by the ingress proxy node with specific efficient functionalities. The ingress node routes the requests to the target web service through one of the remote proxies (outbound nodes), according to metadata in the user's request. The invention enables more easily/efficiently to select outbound nodes randomly or stick to a selected outbound node. These special functionalities are useful when scraping web data, e.g., it appears technically efficient to use TCP/IP ports. Furthermore, the invention enables one to add other important configurations and functionalities to proxy services using the SOCKS5 protocol.

Abstract: The present invention discloses a method and system of an internet proxy service, configuring the service in an ingress proxy—a virtual single access point receiving requests from a user's device. Users' requests to connect to a target web service on the internet may be directed through a plurality of different proxy service nodes being managed by the ingress proxy node with specific efficient functionalities. The ingress node routes the requests to the target web service through one of the remote proxies (outbound nodes), according to metadata in the user's request. The invention enables more easily/efficiently to select outbound nodes randomly or stick to a selected outbound node. These special functionalities are useful when scraping web data, e.g., it appears technically efficient to use TCP/IP ports. Furthermore, the invention enables one to add other important configurations and functionalities to proxy services using the SOCKS5 protocol.

Abstract: Systems and methods for coordinating network connectivity and communication between proxy servers, exit-nodes and client modules are disclosed. In one aspect, proxy-nodes in a proxy infrastructure accept connections with exit-nodes based on geographical proximity or proxy-node metrics. Further, a proxy-node can communicate and instruct another proxy-node to service the client request via a suitable exit-node. Further still, a proxy-node can communicate and instruct proxy-node to redirect a suitable exit-node towards the first proxy-node in order to service the client request. In another aspect, the proxy-infrastructure enables client modules to connect to proxy-nodes based on geographical proximity, client parameters, and client's behavioral informatics.

Abstract: System and method for efficiently implementing scalable, highly efficient decentralized proxy services through proxy infrastructures situated in different geo-locations. In one aspect, the systems and methods enable users from any geographical location to send requests to the geographically closest proxy infrastructure. One exemplary method described allows proxy infrastructures to gather, classify, and store metadata of exit nodes in its internal database. In another aspect, systems and methods described herein enable proxy infrastructures to select metadata of exit nodes from its internal database and forward requests from a user device to respective proxy servers or proxy supernodes to which the selected exit nodes are connected.

Abstract: Proxy servers within a service provider infrastructure are enabled to maintain multiple persistent connections among themselves and to exchange data bi-directionally in an unsolicited manner. Specifically, exit proxy servers are enabled to request their respective proxy supernodes to update the already existing network connection to support WebSocket communication channels. Accordingly, the respective proxy supernodes are enabled to update the network connection with the exit proxy servers to support WebSocket communication channels. A single instance of a proxy supernode and an exit proxy server can maintain multiple WebSocket communication channels with each other. By utilizing the said WebSocket communication channels, the proxy supernode and the exit proxy servers can exchange data with each other simultaneously without any data losses.

Abstract: ADVANCED RESPONSE PROCESSING IN WEB DATA COLLECTION discloses processor-implemented apparatuses, methods, and systems of processing unstructured raw HTML responses collected in the context of a data collection service, the method comprising, in one embodiment, receiving raw unstructured HTML documents and extracting text data with associated meta information that may comprise style and formatting information. In some embodiments data field tags and values may be assigned to the text blocks extracted, classifying the data based on the processing of Machine Learning algorithms. Additionally, blocks of extracted data may be grouped and re-grouped together and presented as a single data point. In another embodiment the system may aggregate and present the text data with the associated meta information in a structured format. In certain embodiments the Machine Learning model may be a model trained on a pre-created training data set labeled manually or in an automatic fashion.

Abstract: This invention discloses selecting a proxy IP device (Proxy-ICDAPIA) by its geographical coordinates and distance from the target web service. A method and system are disclosed, allowing users to specify Proxy-ICDAPIAs by geographic coordinates. Embodiments describe the Proxy-ICDAPIA selection using Geohash areas, and from circular geographical areas specified by center coordinates and a radius. Proxy-ICDAPIAs are selected in areas where geographic density of web services is high, and selecting Proxy ICDAPIAs by country/city may be insufficient. Another problem solved when a client uses Proxy ICDAPIAs selection by coordinates, is an inability to provide a country code. The aforementioned functionality does work without a specified country code, some countries are not included in the pool used when no parameters are provided.

Abstract: Systems and methods for coordinating network connectivity and communication between proxy servers, exit-nodes and client modules are disclosed. In one aspect, proxy-nodes in a proxy infrastructure accept connections with exit-nodes based on geographical proximity or proxy-node metrics. Further, a proxy-node can communicate and instruct another proxy-node to service the client request via a suitable exit-node. Further still, a proxy-node can communicate and instruct proxy-node to redirect a suitable exit-node towards the first proxy-node in order to service the client request. In another aspect, the proxy-infrastructure enables client modules to connect to proxy-nodes based on geographical proximity, client parameters, and client's behavioral informatics.