Abstract: A computer displays a graphical user interface (GUI) that includes a control and receives text instructions in a natural language that describe a location of the control on the GUI. The text instructions instruct an application to perform a user interface (UI) event on the control.

Abstract: Systems, methods, and other embodiments associated with graphical user interface input element identification are described. One example system identifies a graphical user interface (GUI) element through which an input to an application was received based on comparing images generated from data provided by the application. Data that describes the GUI element may then be provided.

Abstract: Systems, methods, and other embodiments associated with graphical user interface (GUI) hierarchy generation are described. One example method includes generating a graph based on an image of a GUI. Nodes in the graph may represent GUI components depicted in the image. Edges in the graph may represent relationships between the GUI components. The example method may include parsing the graph according to a formal graph grammar to produce a GUI hierarchy. The GUI hierarchy may comprise descriptions of groups of the GUI components.

Abstract: Systems, methods, and other embodiments associated with graphical user interface (GUI) component classification are described. One example method includes generating a first vector. The first vector may be generated based on image data describing a GUI component. The example method may also include assigning a GUI component classifier to the GUI component. Assigning the GUI component classifier may comprise comparing the first vector to members of a vector set. Members of the vector set may describe GUI elements. The example method may also include providing the GUI component classifier.

Abstract: Systems, methods, and other embodiments associated with graphical user interface (GUI) component identification are described. One example method includes identifying a set of GUI components. The identifying may be based on a screen image of a GUI. The screen image may comprise data describing a visual representation of the GUI. The example method may also include providing data identifying the set of GUI components.

Abstract: Disclosed is a computer implemented method of detecting a defect in a printed image, the method comprising the steps of: receiving a target image comprising digital image data representing a scan of the printed image; receiving a reference image comprising digital image data representing a reference of the printed image; calculating a structural dissimilarity measure, D, associated with a target pixel located in the target image and a reference pixel located in the reference image; and, determining on the basis of the structural dissimilarity measure whether a defect is present at the target pixel, wherein the structural dissimilarity measure is calculated using a structural measure, s, and a contrast measure, c; the structural measure calculated using a spatial cross-correlation associated with a target region, {right arrow over (x)}, containing the target pixel and a reference region, {right arrow over (y)}, containing the reference pixel, and the contrast measure calculated using a standard deviation assoc