Abstract: Systems and methods are provided for detecting an interruption during an automated workflow. An automated workflow may comprise a series of actions to be performed by or with the assistance of a computer. A workflow manager executes a workflow by progressing through a series of workflow states according to instructions associated with the workflow. When the workflow advances to a new state, an interruption detection engine determines whether the state contains an interruption by examining one or more attributes of the workflow state and/or the user interface associated therewith. An interruption detecting engine may examine a document object model and/or utilize computer vision to determine whether an interruption has occurred. When an interruption is detected, a workflow is paused until the interruption is resolved, such as by a user providing a required input. After an interruption has been resolved, the resumes and continues until completion of the workflow.

Abstract: In non-limiting examples of the present disclosure, systems, methods and devices for generating searchable data structures using observational context data are provided. An indication to process observational context data surfaced by a local computing device may be received. The observational context data may comprise display data or audio data. The observational context data may be processed by a context analysis engine. A determination may be made of a current context type occurring on the local computing device. The current context type may be determined from a plurality of context types, each having a semantic data structure comprising a unique set of entity types. A canvas comprising a searchable data structure of the current context type may be generated and populated with objects having entity types corresponding to the current context type. A displayable representation of the canvas may be displayed.

Abstract: Methods and systems are provided that leverage existing information displayed or presented to a user by an app without requiring the integration of new libraries or requiring a user to use a specific app. Rather, methods and systems presented herein identify a task, select an app or multiple apps capable of performing the task, obtain information that may be required by the app or apps, determine a contextual understanding of what is displayed by the selected app or apps, and finally generate and automate the interaction with the app or apps such that the interaction with the app or apps is handled by an automation agent, allowing a user to stay focused on one or more tasks that the user needs to accomplish.