Abstract: An action index includes information regarding an actionable graphical user interface (GUI) element of a GUI, an action for the actionable GUI element, a precondition for the action, and a postcondition for the action. A presence of the precondition in the GUI indicates a likelihood of success of the action, and a presence of the postcondition in the GUI, after performing the action, indicates success of the action. An agent performs a task on behalf of a user according to a plan that is based on the action index. The agent comprises an action engine which receives the task and obtains, from a generative artificial intelligence (GAI) model, the plan for completing the task.

Abstract: An annotation tool receives, from a website or application, a graphical user interface (GUI) definition and a GUI image generated by the website or application. The tool identifies an interactive element visible in the GUI image and corresponding code in the GUI definition for the interactive element and infers, from the GUI image and the GUI definition, a description of the interactive element, the description including, in natural language, a purpose of the interactive element. The tool generates an annotation for the interactive element, the annotation comprising the description and updates the website or application based on the annotation. In example scenarios, the annotation is used by an artificial intelligence agent for performing actions on the GUI without referring to the GUI image.

Abstract: The present disclosure relates to systems and methods for creating a copilot. The copilot uses plugins to provide additional features and functionalities to the copilot. The systems and methods use a large language model (LLM) pipeline to generate a knowledge resource used by the plugins and/or an LLM in the copilot to answer queries from a user.

Abstract: The present disclosure relates to systems and methods for creating a copilot. The copilot uses plugins to provide additional features and functionalities to the copilot. The systems and methods use a large language model (LLM) pipeline to generate a knowledge resource used by the plugins and/or an LLM in the copilot to answer queries from a user.

Abstract: The techniques disclosed herein enable an autonomous agent to interpret an input dataset and orchestrate a suite of software modules to perform a computational task on a representation of a chemical material. The input dataset includes a prompt defining a computational task to be performed on a chemical material. Moreover, the input dataset includes data defining a chemical included in the chemical material, molecular descriptors describing the chemical and/or the chemical material, and an external variable. The agent analyzes the benefits and drawbacks of each model within the context of the computational task to determine a technique for performing the computational task. Accordingly, the agent formulates a chain of calls invoking the functionality of data processing tools and models to perform the computational task responsive to the prompt.