Abstract: A method and system of UI automation includes receiving a demonstration of an automation to be performed on an application. One or more objects and one or more corresponding labels associated with the demonstration are detected on the application. The demonstration is transformed into one or more natural language instructions. An object is semantically selected during a runtime action based on a large language model (LLM). The semantic selection is reflective of an intention captured in the natural language instruction despite any change in a corresponding label of the object.

Abstract: An example system includes a processor to receive an event trace. The processor can transform the event trace into a semantic story using a generated story template. The processor can input the semantic story into a fine-tuned model. The processor can receive a next skill prediction from the fine-tuned model.