Abstract: Techniques for programmatically controlling computer programs that are configured to be controlled via text-based user interfaces. The techniques include using at least one computer hardware processor to perform: accessing a string specifying content of a user interface (UI) screen of a computer program configured to be controlled via a text-based user interface; generating, based on the string, a two-dimensional (2D) representation of the UI screen; identifying a first controllable UI element in the UI screen at least in part by processing text in the 2D representation of the UI screen; and controlling the computer program by causing, via the first controllable UI element, the computer program to perform at least one action in furtherance of a task.

Abstract: The techniques described herein relate to methods, apparatus, and computer readable media for content-aware image storage. A set of images is accessed from a storage device, each image of the set of images containing at least a part of a screenshot of a display of a computing device used by a user. The set of images is sorted using a similarity metric to obtain a sorted set of images. The sorted set of images are encoded using a video codec to obtain a video encoding the sorted set of images. The video is stored in conjunction with metadata indicating a correspondence between images in the set of images and frames in the video.

Abstract: According to at least one aspect, a system configured to execute a software robot such that interactive actions may be performed by the software robot without interruption is provided. The system comprises at least one hardware processor configured to execute an operating system. The system further comprises at least one non-transitory computer-readable storage medium storing processor-executable instructions configured to execute in a service session in the operating system that, when executed by the at least one hardware processor, causes the at least one hardware processor to: start a first remote desktop session in the operating system, establish a loopback remote desktop connection between the first remote desktop session and the service session, and start a software robot computer program in the first remote desktop session.

Abstract: A system for remotely controlling multiple application programs executing on multiple respective physical computing devices, the system comprising a first computing device, comprising a first processor, and configured to execute a first application program; a second computing device, comprising a second processor, and configured to execute a second application program; and a controller configured to perform: generating, based on first information obtained from the first computing device and second information obtained from the second computing device, a global object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program and the second application program; controlling the first application program to perform the first sub-task at least in part by using the global object hierarchy; and controlling the second application program to perform the second sub-task at least in part by using the global object hierarchy.

Abstract: According to at least one aspect, a system configured to execute a software robot such that interactive actions may be performed by the software robot without interruption is provided. The system comprises at least one hardware processor configured to execute an operating system. The system further comprises at least one non-transitory computer-readable storage medium storing processor-executable instructions configured to execute in a service session in the operating system that, when executed by the at least one hardware processor, causes the at least one hardware processor to: start a first remote desktop session in the operating system, establish a loopback remote desktop connection between the first remote desktop session and the service session, and start a software robot computer program in the first remote desktop session.

Abstract: According to at least one aspect, a system for analyzing computer usage data of a user to identify an underlying task being performed by the user is provided. The system includes a hardware processor and a non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the hardware processor, cause the hardware processor to perform: receiving a plurality of events each indicative of an action performed by a user and contextual information associated with the action performed by the user, clustering the plurality of events to generate a plurality of clustered events, identifying a plurality of sub-tasks in the plurality of clustered events that each comprise a sequence of clustered events, identifying a task in the plurality of clustered events being performed by the user that comprises at least one sub-task, and generating a score for the task indicative of a difficulty of automating the task.

Abstract: A system comprising at least one hardware processor configured to perform: accessing a software robot computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program; generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program; and controlling the first application program to perform the first sub-task. The controlling includes identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task; automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising refreshing the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform the first action.

Abstract: A system comprising at least one hardware processor; at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program, the generating comprising: obtaining a hierarchical representation of the active GUI elements of the application program; generating an object for each of at least some of the active GUI elements represented in the hierarchical representation; and organizing the generated objects into the object hierarchy based on relationships among active GUI elements represented in the hierarchical representation; and controlling, using software robot computer program and the generated object hierarchy, the first application program to perform the first sub-task.

Abstract: According to at least one aspect, a system for analyzing computer usage data of a user to identify an underlying task being performed by the user is provided. The system includes a hardware processor and a non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the hardware processor, cause the hardware processor to perform: receiving a plurality of events each indicative of an action performed by a user and contextual information associated with the action performed by the user, clustering the plurality of events to generate a plurality of clustered events, identifying a plurality of sub-tasks in the plurality of clustered events that each comprise a sequence of clustered events, identifying a task in the plurality of clustered events being performed by the user that comprises at least one sub-task, and generating a score for the task indicative of a difficulty of automating the task.

Abstract: Techniques for executing one or more instances of a computer program using virtual machines, the computer program comprising multiple computer program portions including a first computer program portion. The techniques include determining whether an instance of any of the multiple computer program portions is to be executed; when it is determined that a first instance of the first computer program portion is to be executed, accessing first information specifying a first set of one or more virtual machine resources required for executing the first instance of the first computer program portion; determining whether any one of the plurality of virtual machines has at least the first set of virtual machine resources available; and when it is determined that a first of the plurality of virtual machines has the first set virtual machine resources available, causing the first virtual machine to execute the first instance of the first computer program portion.

Abstract: According to at least one aspect, a system configured to execute a software robot such that interactive actions may be performed by the software robot without interruption is provided. The system comprises at least one hardware processor configured to execute an operating system. The system further comprises at least one non-transitory computer-readable storage medium storing processor-executable instructions configured to execute in a service session in the operating system that, when executed by the at least one hardware processor, causes the at least one hardware processor to: start a first remote desktop session in the operating system, establish a loopback remote desktop connection between the first remote desktop session and the service session, and start a software robot computer program in the first remote desktop session.

Abstract: According to at least one aspect, a system for collecting computer usage information is provided. The system includes a hardware processor, a display coupled to the hardware processor to display a user interface, and a computer-readable storage medium storing processor-executable instructions that cause the hardware processor to receive an indication of an action being performed by a user on the system, cause the system to wait before updating the user interface in response to the action performed by the user, gather contextual information associated with the action while the system is waiting to process the action performed by the user to update the user interface, cause the system to update the user interface in response to the action performed by the user after gathering the contextual information, and store information indicative of the action and the contextual information.

Abstract: A system, comprising at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program; and controlling the first application program to perform a first sub-task. The controlling includes automatically accessing, in the object hierarchy, a first object corresponding to an active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy, the searching comprising modifying the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task.

Abstract: According to at least one aspect, a system is provided. The system comprises at least one hardware processor; and at least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform: monitoring execution of a first computer program configured to control a second computer program to perform a task at least in part by controlling the second computer program, through a graphical user interface (GUI) and/or an application programming interface (API) for the second computer program, to perform at least one action in furtherance of the task; detecting at least one anomaly in the execution of the first computer program; and outputting the detected at least one anomaly in the execution of the first computer program.

Abstract: A system comprising at least one hardware processor; at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program, the generating comprising: obtaining a hierarchical representation of the active GUI elements of the application program; generating an object for each of at least some of the active GUI elements represented in the hierarchical representation; and organizing the generated objects into the object hierarchy based on relationships among active GUI elements represented in the hierarchical representation; and controlling, using software robot computer program and the generated object hierarchy, the first application program to perform the first sub-task.

Abstract: A system comprising at least one computer hardware processor configured to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program; controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions.

Abstract: According to at least one aspect, a system for collecting computer usage information is provided. The system includes a hardware processor, a display coupled to the hardware processor to display a user interface, and a computer-readable storage medium storing processor-executable instructions that cause the hardware processor to receive an indication of an action being performed by a user on the system, gather contextual information associated with the action responsive to the action performed by the user, store information indicative of the action and the contextual information in a volatile memory as an event, determine whether at least one event stored in the volatile memory includes personal information of the user, and write the at least one event stored in the volatile memory to an event log in a non-volatile memory responsive to a determination that the at least one event does not include personal information of the user.

Abstract: According to at least one aspect, a system for analyzing computer usage data of a user to identify an underlying task being performed by the user is provided. The system includes a hardware processor and a non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the hardware processor, cause the hardware processor to perform: receiving a plurality of events each indicative of an action performed by a user and contextual information associated with the action performed by the user, clustering the plurality of events to generate a plurality of clustered events, identifying a plurality of sub-tasks in the plurality of clustered events that each comprise a sequence of clustered events, identifying a task in the plurality of clustered events being performed by the user that comprises at least one sub-task, and generating a score for the task indicative of a difficulty of automating the task.

Abstract: A system comprising at least one computer hardware processor; and at least one computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of a first application program and a second application program; controlling the first application program to perform a first sub-task of a task at least in part by using the object hierarchy; and controlling the second application program to perform a second sub-task of the task at least in part by using the object hierarchy, wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

Abstract: A system for remotely controlling multiple application programs executing on multiple respective physical computing devices, the system comprising a first computing device, comprising a first processor, and configured to execute a first application program; a second computing device, comprising a second processor, and configured to execute a second application program; and a controller configured to perform: generating, based on first information obtained from the first computing device and second information obtained from the second computing device, a global object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program and the second application program; controlling the first application program to perform the first sub-task at least in part by using the global object hierarchy; and controlling the second application program to perform the second sub-task at least in part by using the global object hierarchy.