Abstract: Traditional, programmatic automated remote control of computerized devices requires extensive tailoring for each device type and operating system. A visual-symbolic control method enables largely device-agnostic control of any target device with access to display and a means of user input (keyboard, mouse, touchpad, touch-screen, etc). An image-processing daemon analyzes the displayed image and recognizes its component visual entities (windows, icons, buttons, etc.), creates symbolic entities from extracted attributes of the visual entities, and organizes the symbolic entities into a symbolic object model instance. The functional relationships and hierarchies of the visual entities are captured in the arrangement of symbolic entities in the symbolic object model instance. Visual-symbolic control commands act on the symbolic entities, and, where appropriate, the commands are transmitted to the target device as user-like target-input.

Abstract: Traditional, programmatic automated remote control of computerized devices requires extensive tailoring for each device type and operating system. A visual-symbolic control method enables largely device-agnostic control of any target device with access to display and a means of user input (keyboard, mouse, touchpad, touch-screen, etc). An image-processing daemon analyzes the displayed image and recognizes its component visual entities (windows, icons, buttons, etc.), creates symbolic entities from extracted attributes of the visual entities, and organizes the symbolic entities into a symbolic object model instance. The functional relationships and hierarchies of the visual entities are captured in the arrangement of symbolic entities in the symbolic object model instance. Visual-symbolic control commands act on the symbolic entities, and, where appropriate, the commands are transmitted to the target device as user-like target-input.