Abstract: Reducing graphical user interface (GUI) noise maybe achieved by recording a first execution scenario for control of operation of an application program having a GUI during a recording phase of operation of a cognitive control framework system, setting soft conditions for a search for the application program for the first execution scenario, playing back the application program according to the first execution scenario during a playback phase of operation of the cognitive control framework system, updating the first execution scenario to form a second execution scenario to reduce GUI noise conditions observed during playback, including updating recorded images originally generated by the GUI during the recording phase and updating coordinates for user input data, setting stronger conditions for the search for use in subsequent playbacks; and playing back the application program according to the second execution scenario with the stronger conditions for search.

Abstract: Searching hypotheses for locations of objects in a playback image corresponding to a recorded image generated by a graphical user interface (GUI) of an application program may be accomplished by capturing the playback image, detecting at least one active object in the recorded image, searching subsets of hypotheses from the playback image for an object according to predetermined criteria, recalculating old actions for the object in the playback image by applying actions according to an execution scenario and loading a next set of data, when the object is found, and checking dynamic conditions.

Abstract: Structural content filtration to reduce the number of hypotheses for the location of an active object in a recorded image generated by a graphical user interface (GUI) of an application program may be accomplished by transforming the recorded image and a corresponding playback image, determining a sub-image from the transformed recorded image which corresponds to an object to be searched for in the transformed playback image, determining a set of points on the transformed playback image which have appropriate values for matching the sub-image, and filtering hypotheses on the playback image which are more than a selected distance from any one of the subset of points.

Abstract: Reducing the number of hypotheses for locations of an active object in a saved image generated by a graphical user interface (GUI) of an application program may be accomplished by determining possible triangles for the saved image, wherein vertices of the triangles are points where the active object and any two additional objects of the saved image are located, determining possible pairs of hypotheses corresponding to the two additional objects from a current playback image corresponding to the saved image, determining triangles corresponding to the possible pairs of hypotheses and filtering the detected triangles, determining vertices with weights for similar triangles of the current playback image, and calculating a weight for every hypothesis of the active object and filtering hypotheses of the active object on the current playback image based on a weight bound.

Abstract: Reducing the number of hypotheses for locations of an active object in a saved image generated by a graphical user interface (GUI) of an application program may be accomplished by determining possible triangles for the saved image, wherein vertices of the triangles are points where the active object and any two additional objects of the saved image are located, determining possible pairs of hypotheses corresponding to the two additional objects from a current playback image corresponding to the saved image, determining triangles corresponding to the possible pairs of hypotheses and filtering the detected triangles, determining vertices with weights for similar triangles of the current playback image, and calculating a weight for every hypothesis of the active object and filtering hypotheses of the active object on the current playback image based on a weight bound.

Abstract: Searching hypotheses for locations of objects in a playback image corresponding to a recorded image generated by a graphical user interface (GUI) of an application program may be accomplished by capturing the playback image, detecting at least one active object in the recorded image, searching subsets of hypotheses from the playback image for an object according to predetermined criteria, recalculating old actions for the object in the playback image by applying actions according to an execution scenario and loading a next set of data, when the object is found, and checking dynamic conditions.

Abstract: Structural content filtration to reduce the number of hypotheses for the location of an active object in a recorded image generated by a graphical user interface (GUI) of an application program may be accomplished by transforming the recorded image and a corresponding playback image, determining a sub-image from the transformed recorded image which corresponds to an object to be searched for in the transformed playback image, determining a set of points on the transformed playback image which have appropriate values for matching the sub-image, and filtering hypotheses on the playback image which are more than a selected distance from any one of the subset of points.

Abstract: Reducing graphical user interface (GUI) noise maybe achieved by recording a first execution scenario for control of operation of an application program having a GUI during a recording phase of operation of a cognitive control framework system, setting soft conditions for a search for the application program for the first execution scenario, playing back the application program according to the first execution scenario during a playback phase of operation of the cognitive control framework system, updating the first execution scenario to form a second execution scenario to reduce GUI noise conditions observed during playback, including updating recorded images originally generated by the GUI during the recording phase and updating coordinates for user input data, setting stronger conditions for the search for use in subsequent playbacks; and playing back the application program according to the second execution scenario with the stronger conditions for search.