Abstract: Using a cognitive artificial intelligence (AI) layer (e.g., a generative AI model and one or more other AI/ML models) to perform robotic process automation (RPA) robot repair is disclosed. Computer vision (CV) and/or optical character recognition (OCR) models may be called by RPA robots when performing activities in RPA workflows to identify graphical elements and text, respectively, in a user interface (UI). However, the CV or OCR model may not find a graphical element or text targeted by the activity of the RPA workflow, or the RPA robot may encounter an exception and otherwise fail when performing the activity. In such cases, a cognitive AI layer including a generative AI model may be used to find the target graphical element or text, or to address the robot failure.

Abstract: Target-based schema identification and semantic mapping for robotic process automation (RPA) are disclosed. When looking at a source, such as a document, a web form, a user interface of a software application, a data file, etc., it is often difficult for software to determine which fields are labels and which are values associated with those labels. Since values have not yet been entered for various labels (e.g., first name, company, customer number, etc.), these labels are easier to detect than when the target also includes various values associated with the labels. A selection of an empty target may be received and target-based schema identification may be performed on the empty target, determining labels and a type of the target. Semantic matching may then be performed between a source and the target. These features may be performed at design time or runtime.

Abstract: Using generative AI to supplement automated information extraction is disclosed. Computer vision (CV) and/or optical character recognition (OCR) models and a generative artificial intelligence (AI) model are used together to extract information (e.g., names, dates, invoice numbers, etc.) from a source. Acceptance threshold(s) may be used to accept predictions for extracted data elements from the models, and the prediction from the generative AI model may be preferred or a human may be tasked with reviewing the element. If no model meets its respective acceptance threshold (whether common or specific to that model), these element(s) may be marked for subsequent human review, or a human can be looped in to correct these element(s). The models may then be retrained using these labeled elements.

Abstract: Target-based schema identification and semantic mapping for robotic process automation (RPA) are disclosed. When looking at a source, such as a document, a web form, a user interface of a software application, a data file, etc., it is often difficult for software to determine which fields are labels and which are values associated with those labels. Since values have not yet been entered for various labels (e.g., first name, company, customer number, etc.), these labels are easier to detect than when the target also includes various values associated with the labels. A selection of an empty target may be received and target-based schema identification may be performed on the empty target, determining labels and a type of the target. Semantic matching may then be performed between a source and the target. These features may be performed at design time or runtime.

Abstract: Graphical element detection using a combined series and delayed parallel execution unified target technique that potentially uses a plurality of graphical element detection techniques, performs default user interface (UI) element detection technique configuration at the application and/or UI type level, or both, is disclosed. The unified target merges multiple techniques of identifying and automating UI elements into a single cohesive approach. A unified target descriptor chains together multiple types of UI descriptors in series, uses them in parallel, or uses at least one technique first for a period of time and then runs at least one other technique in parallel or alternatively if the first technique does not find a match within the time period.

Abstract: Artificial intelligence (AI)-driven, semantic, automatic data transfer between a source and a target using task mining is disclosed. Existing task mining technologies gather all the data pertaining to a screen and/or form, along with the fields and elements present therein. However, semantic meaning may be derived and correlations between fields/elements on the screen may be provided automatically. This may include automatic transformation and validation, and a robotic process automation (RPA) workflow/automation can automatically be generated for a user that performs data copy-and-paste functionality between a source and a target without the user indicating those steps. Also, this functionality may be provided despite the fact that the labels in the source and the target are not exactly the same.

Abstract: Target-based schema identification and semantic mapping for robotic process automation (RPA) are disclosed. When looking at a source, such as a document, a web form, a user interface of a software application, a data file, etc., it is often difficult for software to determine which fields are labels and which are values associated with those labels. Since values have not yet been entered for various labels (e.g., first name, company, customer number, etc.), these labels are easier to detect than when the target also includes various values associated with the labels. A selection of an empty target may be received and target-based schema identification may be performed on the empty target, determining labels and a type of the target. Semantic matching may then be performed between a source and the target. These features may be performed at design time or runtime.

Abstract: Target-based schema identification and semantic mapping for robotic process automation (RPA) are disclosed. When looking at a source, such as a document, a web form, a user interface of a software application, a data file, etc., it is often difficult for software to determine which fields are labels and which are values associated with those labels. Since values have not yet been entered for various labels (e.g., first name, company, customer number, etc.), these labels are easier to detect than when the target also includes various values associated with the labels. A selection of an empty target may be received and target-based schema identification may be performed on the empty target, determining labels and a type of the target. Semantic matching may then be performed between a source and the target. These features may be performed at design time or runtime.

Abstract: Automation of a process running in a first session via robotic process automation (RPA) robot(s) running in a second session is disclosed. In some aspects, a form is displayed in a user session, but one or more attended RPA robots that retrieve and/or interact with data for an application in the first session run in one or more other sessions. In this manner, the operation of the RPA robot(s) may not prevent the user from using other applications or instances when the RPA robot(s) are running, but the data modifications made or facilitated by the RPA robot(s) may be visible to the user in the first session window.

Abstract: Screen response validation of robot execution for robotic process automation (RPA) is disclosed. Whether text, screen changes, images, and/or other expected visual actions occur in an application executing on a computing system that an RPA robot is interacting with may be recognized. Where the robot has been typing may be determined and the physical position on the screen based on the current resolution of where one or more characters, images, windows, etc. appeared may be provided. The physical position of these elements, or the lack thereof, may allow determination of which field(s) the robot is typing in and what the associated application is for the purpose of validation that the application and computing system are responding as intended. When the expected screen changes do not occur, the robot can stop and throw an exception, go back and attempt the intended interaction again, restart the workflow, or take another suitable action.

Abstract: Screen response validation of robot execution for robotic process automation (RPA) is disclosed. Whether text, screen changes, images, and/or other expected visual actions occur in an application executing on a computing system that an RPA robot is interacting with may be recognized. Where the robot has been typing may be determined and the physical position on the screen based on the current resolution of where one or more characters, images, windows, etc. appeared may be provided. The physical position of these elements, or the lack thereof, may allow determination of which field(s) the robot is typing in and what the associated application is for the purpose of validation that the application and computing system are responding as intended. When the expected screen changes do not occur, the robot can stop and throw an exception, go back and attempt the intended interaction again, restart the workflow, or take another suitable action.

Abstract: Graphical element detection using a combined series and delayed parallel execution unified target technique that potentially uses a plurality of graphical element detection techniques, performs default user interface (UI) element detection technique configuration at the application and/or UI type level, or both, is disclosed. The unified target merges multiple techniques of identifying and automating UI elements into a single cohesive approach. A unified target descriptor chains together multiple types of UI descriptors in series, uses them in parallel, or uses at least one technique first for a period of time and then runs at least one other technique in parallel or alternatively if the first technique does not find a match within the time period.

Abstract: Automation of a process running in a first session via robotic process automation (RPA) robot(s) running in a second session is disclosed. In some aspects, a form is displayed in a user session, but one or more attended RPA robots that retrieve and/or interact with data for an application in the first session run in one or more other sessions. In this manner, the operation of the RPA robot(s) may not prevent the user from using other applications or instances when the RPA robot(s) are running, but the data modifications made or facilitated by the RPA robot(s) may be visible to the user in the first session window.

Abstract: Automation of a process running in a first session via robotic process automation (RPA) robot(s) running in a second session is disclosed. In some aspects, a form is displayed in a user session, but one or more attended RPA robots that retrieve and/or interact with data for an application in the first session run in one or more other sessions. In this manner, the operation of the RPA robot(s) may not prevent the user from using other applications or instances when the RPA robot(s) are running, but the data modifications made or facilitated by the RPA robot(s) may be visible to the user in the first session window.

Abstract: Screen response validation of robot execution for robotic process automation (RPA) is disclosed. Whether text, screen changes, images, and/or other expected visual actions occur in an application executing on a computing system that an RPA robot is interacting with may be recognized. Where the robot has been typing may be determined and the physical position on the screen based on the current resolution of where one or more characters, images, windows, etc. appeared may be provided. The physical position of these elements, or the lack thereof, may allow determination of which field(s) the robot is typing in and what the associated application is for the purpose of validation that the application and computing system are responding as intended. When the expected screen changes do not occur, the robot can stop and throw an exception, go back and attempt the intended interaction again, restart the workflow, or take another suitable action.

Abstract: Graphical element detection using a combined series and delayed parallel execution unified target technique that potentially uses a plurality of graphical element detection techniques, performs default user interface (UI) element detection technique configuration at the application and/or UI type level, or both, is disclosed. The unified target merges multiple techniques of identifying and automating UI elements into a single cohesive approach. A unified target descriptor chains together multiple types of UI descriptors in series, uses them in parallel, or uses at least one technique first for a period of time and then runs at least one other technique in parallel or alternatively if the first technique does not find a match within the time period.

Abstract: Automation of a process running in a first session via robotic process automation (RPA) robot(s) running in a second session is disclosed. In some aspects, a form is displayed in a user session, but one or more attended RPA robots that retrieve and/or interact with data for an application in the first session run in one or more other sessions. In this manner, the operation of the RPA robot(s) may not prevent the user from using other applications or instances when the RPA robot(s) are running, but the data modifications made or facilitated by the RPA robot(s) may be visible to the user in the first session window.

Abstract: Graphical element detection using a combination of user interface (UI) descriptor attributes from two or more graphical element detection techniques is disclosed. UI descriptors may be used to compare attributes for a given UI descriptor with attributes of UI elements found at runtime in the UI. At runtime, the attributes for the UI elements found in the UI can be searched for matches with attributes for a respective RPA workflow activity, and if an exact match or a match within a matching threshold is found, the UI element may be identified and interacted with accordingly.

Abstract: Graphical element detection using a combination of user interface (UI) descriptor attributes from two or more graphical element detection techniques is disclosed. UI descriptors may be used to compare attributes for a given UI descriptor with attributes of UI elements found at runtime in the UI. At runtime, the attributes for the UI elements found in the UI can be searched for matches with attributes for a respective RPA workflow activity, and if an exact match or a match within a matching threshold is found, the UI element may be identified and interacted with accordingly.

Abstract: Graphical element detection using a combination of user interface (UI) descriptor attributes from two or more graphical element detection techniques is disclosed. UI descriptors may be used to compare attributes for a given UI descriptor with attributes of UI elements found at runtime in the UI. At runtime, the attributes for the UI elements found in the UI can be searched for matches with attributes for a respective RPA workflow activity, and if an exact match or a match within a matching threshold is found, the UI element may be identified and interacted with accordingly.