Abstract: A system and method of improved performant content source crawling. A content source crawling technology that is performant and focused on an initial rapid crawling of a content source to find specific file signatures so that it can determine an inventory of files that have been modified since the last full index event, thereby minimizing the time and computing resources necessary to perform a full crawl on just the select files to update to the search index. A method of Signature Flagging is disclosed and is used to selectively crawl the metadata about contained files and folders found within a content source in order to create that information in an Index within the Shinydocs Search Library.

Abstract: A content management system enables a central server to connect to remote nodes at client sites. Software modules on the remote node is responsible for making necessary calls to the central server in order to create an index of the relevant data (or metadata) and fetch the appropriate binary information and files for the related metadata. Remote nodes are populated with data from the content management system via crawl/synchronize methods, or alternatively a hard drive of the data is initially configured at headquarters whereby data is saved to prevent trafficking data over a potentially unreliable connection over an extended period of time. The hard drive is then installed at the remote site and synchronized with the software module running in high performance enterprise library (HPEL) mode. The HPEL enabled server is configured to crawl and synchronize all the data and pick up differences in data using differential crawls.