Abstract: Embodiments are directed to methods and systems for deriving automation instructions. In one scenario, a computer system derives automation instructions by performing the following: rendering a user interface (UI) based on information from an information source and receiving natural language inputs from a user, where the natural language inputs specify an element description and an action type for UI elements rendered in the UI. The method also includes identifying UI elements in the UI that match the element descriptions in the natural language input and whose actions are performable according to their specified action type, and mapping the natural language inputs into executable code using information that corresponds to the identified UI elements.