Abstract: Disclosed herein are system, method, and computer program product embodiments for surface automation in black box environments. An embodiment operates by determining scenarios of an application for automation; detecting the scenario during an execution of an application; capturing and storing one or more user interface screenshots of the scenario; identifying and storing user interface information from the user interface screenshot; implementing a sequential set of instructions comprising at least one textual element detection technique and at least one non-textual element detection technique; and executing the sequential set of instructions.

Abstract: Systems, methods, and computer program products are described herein for automatically recognizing user interface (UI) elements. Capturing, by a first application, of user interactions with a graphical user interface (GUI) of a second application is initiated. The GUI includes a plurality of UI elements. The first application is separate and distinct from the second application. A selection of a UI element is received. The selection is iteratively evaluated against a ranked criteria list having a plurality of criteria to identify whether the selection matches at least one criterion of the plurality of criteria. The selection is appended to the ranked criteria list based on the selection matching the at least one criterion. A second selection of another UI element is received from the second application. The first application automatically appends a criterion to the second selection based on the ranked criteria list.