Abstract: A method of recognizing graphical objects by subjecting the graphical information gathered through "spying" to a series of rules by which the object becomes understood or recognized as an instance of a standard logical object. Before the rules are applied, graphical objects are first interpreted as primitives including groups of text, lines and images. In order to recognize a graphical object as a logical object, the graphical information is subjected to the rules in an iterative process whereby an understanding of the object is continually refined. As the rules are applied, the results are evaluated to determine whether the graphical object can be "mapped" to a standard logical object such as a textfield or listbox. Once the object is understood as a logical element with which the user is accustomed, it is possible to interact with the object and obtain data from the object as if it were a standard object with a published interface.