Abstract: A method and system of UI automation includes receiving a demonstration of an automation to be performed on an application. One or more objects and one or more corresponding labels associated with the demonstration are detected on the application. The demonstration is transformed into one or more natural language instructions. An object is semantically selected during a runtime action based on a large language model (LLM). The semantic selection is reflective of an intention captured in the natural language instruction despite any change in a corresponding label of the object.

Abstract: An embodiment includes semantically parsing UI flows to identify subtasks. The embodiment also includes modeling subtasks within a semantic user interface state-action space (UI SAS). The embodiment also includes providing states, actions and their parameters with names that encapsulate each a semantically discrete purpose, thus making them interpretable. The embodiment also includes updating the UI SAS as new UI flows are added. The embodiment also includes generating, via a planning agent, an action sequence as a route in the UI SAS corresponding to a requested task expressed in a user utterance. The embodiment also includes parsing the action sequence for syntactic errors. The embodiment also includes providing feedback to the planning agent to correct the action. The embodiment also includes executing the syntactically validated action sequence, via an execution agent, to perform the requested task within one or more web applications.

Abstract: An example system includes a processor to receive historical data, a formal quality measure, a quality threshold, and a mathematical optimization model. At least part of the mathematical optimization model is generated from the historical data. The processor can measure a quality of the mathematical optimization model using the formal quality measure. The processor can then augment the mathematical optimization model such that the measured quality of the augmented mathematical optimization model exceeds the target quality threshold.

Abstract: A method and system of UI automation includes receiving a demonstration of an automation to be performed on an application. One or more objects and one or more corresponding labels associated with the demonstration are detected on the application. The demonstration is transformed into one or more natural language instructions. An object is semantically selected during a runtime action based on a large language model (LLM). The semantic selection is reflective of an intention captured in the natural language instruction despite any change in a corresponding label of the object.

Abstract: An example system includes a processor to receive historical data, a formal quality measure, a quality threshold, and a mathematical optimization model. At least part of the mathematical optimization model is generated from the historical data. The processor can measure a quality of the mathematical optimization model using the formal quality measure. The processor can then augment the mathematical optimization model such that the measured quality of the augmented mathematical optimization model exceeds the target quality threshold.