Abstract: Methods, systems and computer program products for intelligent data aggregation are described. A data aggregation system receives a request for aggregating data from a target site. The data aggregation system parses the request and dynamically determines what data items need be scraped for a specific request. The data aggregation system controls flow based on a sitemap through life of the request. The sitemap of the target site includes configuration capturing multiple possible navigational flows. Based on the sitemap, the data aggregation system identifies a shortest path to access the data item required by the request. The data aggregation system creates, for each request, a site flow based on the shortest path. The data aggregation system manages and invokes different modules in an agent that follows the site flow to gather data. The data aggregation system executes the agent to retrieve the requested data items.