Display-independent computerized guidance

- KRYON SYSTEMS LTD.

A method for display-independent execution of a computerized guidance script, the method comprising capturing a screenshot of a graphical user interface (GUI) of a computerized application, analyzing the screenshot for matching a control of the screenshot with a control of the guidance script, and executing the guidance script, based on the matching, to operate the computerized application. Additionally, a display-independent computerized guidance engine, comprising a screenshot analysis engine and a guidance script execution engine.

Skip to: Description  ·  Claims  ·  References Cited  · Patent History  ·  Patent History
Description
FIELD OF THE INVENTION

Embodiments of the disclosure relate to display-independent computerized guidance.

BACKGROUND OF THE INVENTION

A Personal Computer (PC) or a mobile device (such as a notebook computer, a Personal Digital Assistant (PDA) device, a cellular phone and/or the like) allows users to utilize various applications, for example, word processing applications, spreadsheet applications, e-mail applications, games and the like. These applications may be able to perform various operations based on the user's command, for example, editing, calculating, formatting, file handling, data sorting, and/or the like.

A Graphical User Interface (GUI) of an application usually includes various controls (sometimes also referred to as “widgets”) that operate different functions of the application. Such controls may include, for example, icons, texts, buttons, input boxes, menus, drop-down lists, sliders, scroll bars, bars and/or any other operable visual element. In various mobile devices, a control may also be embodied in a physical button and not in the GUI of the application itself. For example, a certain button may be used to perform a specific function in an application.

As applications become more and more complex and sophisticated and thus include many different controls, some users find it useful to attend a course or seminar which teaches users how to use one or more applications. Some users require assistance from other users (such as co-workers, customer support representatives and/or the like) in order to be able to complete particular tasks using an application. Furthermore, it may be difficult for the user to find out, or to remember, whether or not an application is capable of performing a particular task, or which sequence of user-initiated steps is required in order to execute a particular task.

Some applications include a “help” function, in which the user may utilize an index of pre-defined topics, or a search based on user-entered keywords, in order to retrieve pre-defined textual and/or visual descriptions which may assist the user in finding how a particular task is performed. Additionally, some applications are associated with a user's manual, or with a batch of Frequently Asked Questions (FAQ), which may further guide the user on how to perform particular tasks.

The foregoing examples of the related art and limitations related therewith are intended to be illustrative and not exclusive. Other limitations of the related art will become apparent to those of skill in the art upon a reading of the specification and a study of the figures.

SUMMARY OF THE INVENTION

The following embodiments and aspects thereof are described and illustrated in conjunction with systems, tools and methods which are meant to be exemplary and illustrative, not limiting in scope.

There is provided, in accordance with an embodiment, a method for display-independent execution of a computerized guidance script, the method comprising: capturing a screenshot of a graphical user interface (GUI) of a computerized application; analyzing the screenshot for matching a control of the screenshot with a control of the guidance script; and executing the guidance script, based on the matching, to operate the computerized application.

There is further provided, in accordance with an embodiment, a display-independent computerized guidance engine, comprising: a screenshot analysis engine for matching a control of a GUI of a computerized application with a control of a guidance script; and a guidance script execution engine for executing the guidance script, based on the matching, to operate the computerized application.

In some embodiments, the matching is performed according to a pre-defined accuracy level.

In some embodiments, the analyzing further comprises matching surroundings of the control of the screenshot with surroundings of the control of the guidance script.

In some embodiments, the method further comprises converting the screenshot to a 1-bit version.

In some embodiments, the method further comprises converting the screenshot to an inverted 1-bit version.

In some embodiments, the executing further comprises activating a control of the computerized application.

In some embodiments, the activating of the control comprises imitating a cursor click.

In some embodiments, the activating of the control comprises imitating keyboard input.

In some embodiments, the method further comprises activating a scroll bar for scrolling to a previously-invisible region of the computerized application.

In some embodiments, said screenshot analysis engine is set to a pre-defined accuracy level.

In some embodiments, said screenshot analysis engine is adapted to match surroundings of the control of the GUI with surroundings of the control of the guidance script.

In some embodiments, said screenshot analysis engine further comprises an image conversion module.

In some embodiments, said image conversion module is adapted to convert a screenshot to a 1-bit version.

In some embodiments, said image conversion module is adapted to convert a screenshot to an inverted 1-bit version.

In some embodiments, said guidance script execution engine is adapted to activate a control of a computerized application.

In some embodiments, said control is selected from a group consisting of: an icon, a text, a button, an input box, a menu, a drop-down list, a slider, a scroll bar and a bar.

In some embodiments, said screenshot analysis engine is adapted to activate a scroll bar for scrolling to a previously-invisible region of a computerized application.

In addition to the exemplary aspects and embodiments described above, further aspects and embodiments will become apparent by reference to the figures and by study of the following detailed description.

BRIEF DESCRIPTION OF THE FIGURES

Exemplary embodiments are illustrated in referenced figures. Dimensions of components and features shown in the figures are generally chosen for convenience and clarity of presentation and are not necessarily shown to scale. It is intended that the embodiments and figures disclosed herein are to be considered illustrative rather than restrictive. The figures are listed below.

FIG. 1 shows a block diagram of a computerized guidance engine;

FIG. 2 shows a flow chart of a method for display-independent execution of a computerized guidance script;

FIG. 3 shows a Graphical User Interface (GUI) of an illustrative computerized application;

FIG. 4 shows a block diagram of a recording engine for a display-independent, computerized guidance script;

FIG. 5 shows a flow chart of a method for recording a display-independent, computerized guidance script;

FIG. 6 shows a block diagram of a display-independent, GUI control recognizer;

FIG. 7 shows a flow chart of a method for display-independent recognition of a GUI control;

FIG. 8 shows a block diagram of a display-independent, scroll bar recognizer; and

FIG. 9 shows a flow chart of a method for display-independent recognition of a scroll bar.

 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

An aspect of some embodiments relates to a guidance engine adapted to provide guidance to a user of a computerized application. The guidance may be performed by executing a guidance script capable of actually operating the application by way of, for example, activating one or more controls of the application. For instance, the guidance script may imitate cursor moves, cursor clicks or double-clicks, keyboard strokes, combination of cursor clicks and keyboard strokes and/or the like in lieu of the user. The term “imitate”, as referred to herein, may refer to an initiation of an action in the computerized application without actually receiving manual input from an input device such as a mouse, a keyboard, a touch screen and/or the like.

Furthermore, the guidance may be provided to the user in a manner independent of various display properties of the application and/or of the computerized device used by the user (this manner hereinafter referred to as “display-independent”). Since an application may be run with different display properties, such as window size, resolution, color, fonts, themes and/or the like, a conventional guidance script which was recorded with one set of display properties may encounter difficulties operating an application which uses a different set of display properties. This may happen, for example, because a certain GUI control operable by the conventional guidance script may appear in a new, unrecognized location and/or form in the actual application on which the script is finally run. The conventional script may therefore simply fail to identify and operate that control.

Accordingly, the present guidance engine may include a screenshot analysis engine for analyzing the GUI of the application, thereby matching at least one control present in the GUI with a corresponding control of a pre-recorded guidance script. Then, a guidance script execution engine may be able to execute the script and operate the necessary control—independent of the display properties of the application on which the script is executed.

An aspect of additional embodiments relates to a recording engine and a method thereof, for recording a display-independent, computerized guidance script. The recording engine may, in addition to recording a sequence of operations in an application, capture and analyze a screenshot of the application's GUI. In the analysis, the recording engine may inspect an area surrounding a determined cursor position, such as a position where a GUI control was activated by the user performing a sequence of operations. The activated GUI control may then be identified, and an image of it and/or metadata associated with it may then be assigned to the respective step of the operation sequence.

An additional aspect relates to a display-independent, GUI control recognizer and a method thereof. The GUI control recognizer may be capable of recognizing and identifying a GUI control in a screenshot of a user interface of an application. The use of the GUI control recognizer may or may not be tied to execution and/or recording of a guidance script. For example, recognition of GUI controls may be used as general-purpose image recognition method, to complement other computerized operations as desired.

Display-Independent Computerized Guidance

Reference is now made to FIG. 1, which shows a block diagram of a computerized guidance engine (hereinafter “guidance engine”) 100, according to an embodiment. Guidance engine 100 may be a software product installed on and/or adapted to be run on a computerized device, such as a Personal Computer (PC), a desktop computer, a mobile computer, a laptop computer, a notebook computer, a tablet computer, a server computer, a handheld computer, a handheld device, a mobile game console, an essentially stationary game console, a Personal Digital Assistant (PDA) device, an on-board device, an off-board device, a cellular phone, a hybrid device (for example a device incorporating functionalities of multiple types of devices, such as PDA functionality and cellular phone functionality), a vehicular device, a non-vehicular device, a mobile or portable device, a non-mobile or non-portable device and/or the like.

Guidance engine 100 optionally includes a screenshot analysis engine 102 and a guidance script execution engine 104, each being a software component and/or module or both being implemented as a single component and/or module. FIG. 2 shows a flow chart of a method 200 (or an “algorithm”) for display-independent execution of a computerized guidance script; operation of guidance engine 100 of FIG. 1 shall now be described through an explanation of method 200 of FIG. 2.

In a block 202, a screenshot 204 of a GUI of a computerized application is captured, such as using screenshot analysis engine 102 (FIG. 1) or a screenshot capture engine (not shown). This may be performed in response to a user request for guidance, made by, for example, pressing a key assigned to activation of guidance engine 100 (FIG. 1), clicking on an icon of the guidance engine, selecting from a list of available guidance scripts, searching for and retrieving a desired guidance script, and/or the like. Alternatively, activity of the user may be tracked, to automatically determine if the user is experiencing difficulties with the operation of the application. If such a difficulty is identified, processing of method 200 may begin, in order to proactively provide the user with guidance as to the operation in which difficulties were encountered.

The capturing may include storing screenshot 204 or GUI 300 as an image, whether in a temporary memory or in a permanent, non-volatile memory. Reference is now parenthetically made to FIG. 3, which shows a GUI 300 of an illustrative computerized application, such as a word processing application. A GUI, such as GUI 300, may include one or more controls (sometimes also referred to as “widgets”) that operate different functions of the application. Such controls may include, for example, icons (such as an icon 302), texts (such as a text 304), buttons (such as a “bold” button 306), input boxes (such as a “font” input box 308 and text area 310), menus, drop-down lists (such as a “font” drop-down list 312), scroll bars (such as a vertical scroll bar 314 and a horizontal scroll bar 316), sliders, bars and/or any other operable visual element.

A different GUI (not shown) may include a user interface of what is often referred to as a “console application” (also “command line”, “command prompt”, “text terminal”, “terminal” and/or the like). A console application may be a software application having a user interface composed mainly of text, and sometimes with the addition of some simple, minimalistic graphic elements and/or symbols. Console applications are usually designed for operation using a keyboard, although sometimes they may also be controlled using a pointing device such as a mouse. Due to their lack of extensive graphic elements, their GUI is often referred to simply as a “user interface” (UI), without the word “graphical”. However, the term GUI, as referred to herein, is intended to include also a UI of a console application.

Back to FIG. 2, a guidance script 208 may be provided. Guidance script 208 may include an executable operation sequence for operating the application. The operation sequence may include one or more step(s) 208a, each step optionally containing an action pertaining to a GUI control (sometimes referred to simply as a “control”) of the application. With reference to GUI 300 of FIG. 3, as a mere example, a first step may include imitating a click on a “tab 1” control 320, a second step may include imitating a click on a “bold” control 306, a third step may include imitating a click on a “tab 2” control 322, and a fourth step may include imitating text typing inside “text area” control 310. Images of one or more GUI controls associated with actions of guidance script 208, such as controls 320, 306, 322 and 310 of FIG. 3, may be provided with the guidance script.

Screenshot 204 may be analyzed in a block 206, for identifying in it a control corresponding to a control of guidance script 208. The analysis may be performed, for example, using screenshot analysis engine 102 (FIG. 1). It may be performed on an original screenshot, such as a color screenshot, and/or on one or more modified screenshots, such as resized screenshots, portions of screenshots, grey-scaled screenshots or black-and-white (1-bit) versions of screenshots, inverted versions of screenshots or of grey-scaled screenshots or of black-and-white versions of screenshots, and/or the like. A modified screenshot may be produced using an image conversion module, a software component adapted to perform the required modification by way of image processing.

The analysis of screenshot 204 may include running an image recognition algorithm adapted to identify graphical and/or textual characteristics of a control in the screenshot, to enable a matching 210 of the control with a corresponding control of a step 208a of guidance script 208. Graphical characteristics of a control may include visible external borderlines of the control. A simple case is when a control has a noticeable borderline. For example, in FIG. 3, a “paste” button 318 can be relatively easily distinguished from its surroundings, since its most peripheral areas include a continuous frame made of connected pixels. Sometimes, a text is associated with a graphical control, such as the word “text” 304 which appears below “paste” button 318 and is associated with it. The algorithm may identify “text” 304 and “paste” button 318 as associated, based on a pre-determined space between them that indicates their association. The space may range, for example, from 1 pixel to several or even dozens of pixels, depending on the situation. The algorithm may further determine if a text is associated with a graphical control by assigning different “weights” to texts based on their position relative to the graphical control. For example, text positioned on the right or left sides of the graphical control may be given a high weight, text positioned above the graphical control may be given a medium weight, and text positioned below the graphical control may be given a low weight. The higher the weight, the more likely it is that the text is associated with the graphical control.

Textual characteristics may be identified Optical Character Recognition (OCR) techniques, such as by locating “islands” of connected pixels, each island typically defining a single character (such as a letter, a number or a symbol); in some cases, such as the case of the letters “i” and “j”, each letter may be defined by two islands, while in further cases, any suitable character may be defined by multiple islands. A whole word may be identified by determining a series of characters positioned adjacently, where a space which follows a letter and which has the width of approximately one or two characters, may indicate a separation between two words. If the space is wider than a pre-determined value, it may indicate that the next word is not associated with the previous word or words, and is part of a separate control.

In the case of a console application UI, it is likely that OCR techniques will be more prevalent than determination of graphical characteristics of elements. That is, as mentioned, console applications usually include mainly text, and OCR techniques may be more suitable for analyzing console application UI in order to identify one or more controls in them.

Following the identification of a control in screenshot 204, it may be matched 210 with a control of guidance script 208, to enable guidance script execution engine 104 (FIG. 1) to successfully execute 212 the guidance script and operate the actual control in the application, such as by clicking on it, by initiating a keyboard stroke or combination that trigger the control, and/or the like. However, in some scenarios, there may exist multiple controls in the application's GUI that look essentially the same. In such scenarios, in order to correctly match one of these GUI controls with the correct control of guidance script 208, it may be necessary to analyze also the surroundings of the control (such as its adjacent controls and/or adjacent graphical elements), both in screenshot 204 and in guidance script 208, so at to increase the chances of a correct matching. Matching controls are often likely to have similar surrounding controls and/or graphical elements.

In an embodiment, it may be possible to pre-define an accuracy level at which the matching is performed. A higher accuracy level is likely to be less tolerant to extreme changes in display properties between screenshot 204 and guidance script 208, whereas a lower accuracy level may still allow matching under display property changes—but may also cause matching mistakes. Optionally, the accuracy level is defined in percentages.

In case the matching is unsuccessful, namely—no corresponding controls of screenshot 204 and guidance script 208 are found, screenshot analysis engine 102 (FIG. 1) may be utilized for automatically activating one or more scroll bars, such as horizontal scroll bar 316 (FIG. 3) and/or vertical scroll bar 314 (FIG. 3), for scrolling to a previously-invisible region of the computerized application. For example, screenshot analysis engine 102 (FIG. 1) may automatically scroll to previously-invisible regions of text area (FIG. 3) for searching for a GUI control which may be a match for a control of guidance script 208. Optionally, after the scrolling uncovers a new region, a screenshot of the GUI may be captured 202 again, and the analysis 206 and matching 210 steps may be repeated for the new region.

During the execution, in block 212, of guidance script 208, one or more help texts may be displayed in order to complement at least one of step(s) 208a with explanations to the user. For example, the execution may be performed in a semi-automatic mode, in which, following an automatic activation of a control, a help text is displayed, asking the user to perform a manual operation. Reference is now made back to FIG. 3, in which an exemplary help text 330 is shown inside an exemplary guide bubble 332. Help text 330 requests the user to enter text into text area 310. After the user enters the text, guidance script 208 (FIG. 2) may proceed to a next step. Guidance script execution engine 104 of FIG. 1 may be adapted to automatically detect when the user finished performing the manual operation. Additionally or alternatively, the user may be presented with a button (not shown), for example, on which he clicks to indicate to guidance script execution engine 104 of FIG. 1 that he completed the manual operation and wishes to resume execution of the guidance script.

Recording a Display-Independent Computerized Guidance Script

Reference is now made to FIG. 4, which shows a block diagram of a recording engine 400 for a display-independent, computerized guidance script, according to an embodiment.

Recording engine 400 may be a software product installed on and/or adapted to be run on a computerized device, such as a Personal Computer (PC), a desktop computer, a mobile computer, a laptop computer, a notebook computer, a tablet computer, a server computer, a handheld computer, a handheld device, a mobile game console, an essentially stationary game console, a Personal Digital Assistant (PDA) device, an on-board device, an off-board device, a cellular phone, a hybrid device (for example a device incorporating functionalities of multiple types of devices, such as PDA functionality and cellular phone functionality), a vehicular device, a non-vehicular device, a mobile or portable device, a non-mobile or non-portable device and/or the like.

Recording engine 400 may be used for recording a guidance script, such as guidance script 208 of FIG. 2, where the guidance script is adapted to be later run on the same or a different computerized device. For example, a software developer or a software developing company may record a guidance script and distribute it to users for running on their computerized devices.

Recording engine 400 optionally includes a screenshot capture engine 402, a screenshot analysis engine 404, and/or an operation sequence recorder 406, each being a software component and/or module or all being implemented as a single component and/or module. FIG. 5 shows a flow chart of a method 500 (or an “algorithm”) for recording a display-independent, computerized guidance script; operation of recording engine 400 of FIG. 4 shall now be described through an explanation of method 500 of FIG. 5.

At an overview of method 500, an operation sequence in the computerized application is manually performed by a user in a block 502. The operation sequence may include one or more step(s) 502a, each step optionally containing an action pertaining to a GUI control of the application. With reference to GUI 300 of FIG. 3, as a mere example, a first step may include manually clicking on a “tab 1” control 320, a second step may include manually clicking on a “bold” control 306, a third step may include manually clicking on a “tab 2” control 322, and a fourth step may include manually typing text inside “text area” control 310.

In a block 504, the operation sequence may be recorded, such as using operation sequence recorder 406 of FIG. 4. The recording may include a recording of actions such as a cursor click, a cursor move path, a keyboard input and/or the like. These actions may be performed either manually (by a user) or automatically, using software adapted to execute actions in the computerized application. Usage of such software may enable faster, more accurate and/or more efficient recording of an operation sequence, without the need for substantial human intervention. During the recording, one or more controls may be assigned, in a block 506, to their respective steps 502a. A guidance script including a recorded operation sequence and a control assigned to each step is produced in a block 516.

The recording of the operation sequence in block 504 and the assigning of each control to a step in block 506, may be performed based on actions on blocks 508-514, as set forth herein:

In a block 508, a screenshot 510 of a GUI of a computerized application is captured, such as using screenshot capture engine 402 (FIG. 4). The capturing may be performed responsive to step 502a, such as following a cursor click, a keyboard input and/or the like by the user.

A position at which a cursor was at when performing the click, may be identified in screenshot 510. For example, with reference to FIG. 3, a cursor 334 is shown positioned next to and clicking on a “tab 4” text 336. Similarly, if a keyboard stroke is the manual operation by the user, a position of a control activated by the stroke may be detected. For example, if the keyboard combination ctrl+4 activates “tab 4” 336, the position of “tab 4” is detected and assumed.

In a block 512, an area surrounding the determined cursor position is analyzed, such as by screenshot analysis engine 404 of FIG. 4. The analysis may enable identification 514 of a control on which the cursor click was performed or which the keyboard stroke activated. Optionally, screenshot analysis engine 404 of FIG. 4 operates similar to screenshot analysis engine 102 FIG. 1. Accordingly, the analysis of block 512 is performed similar to the analysis of block 206 of FIG. 2, as described above.

The identified control and/or its surrounding area may be saved as an image, and, as mentioned above, assigned 506 to step 502a of operation sequence 502 and optionally stored together with guidance script 516.

To summarize method 500, it is capable of producing guidance script 516 which is display-independent, namely—due to the storing of an image together with each step 502a, it is later possible to execute the guidance script on a computerized device having different display properties that the computerized device of which the recording of method 500 is done. That is, the stored image may be matched, during the execution, with an identified control of the GUI of the computerized application on which the guidance script is executed.

Display-Independent Recognition of a GUI Control

Reference is now made to FIG. 6, which shows a block diagram of a display-independent, GUI control recognizer 600, according to an embodiment.

Control recognizer 600 may be a software product installed on and/or adapted to be run on a computerized device, such as a Personal Computer (PC), a desktop computer, a mobile computer, a laptop computer, a notebook computer, a tablet computer, a server computer, a handheld computer, a handheld device, a mobile game console, an essentially stationary game console, a Personal Digital Assistant (PDA) device, an on-board device, an off-board device, a cellular phone, a hybrid device (for example a device incorporating functionalities of multiple types of devices, such as PDA functionality and cellular phone functionality), a vehicular device, a non-vehicular device, a mobile or portable device, a non-mobile or non-portable device and/or the like.

Control recognizer 600 may be used for recognizing and identifying a GUI control in a screenshot of a GUI of a computerized application. The use of control recognizer 600 may or may not be tied to execution and/or recording of a guidance script, as set forth above. For example, recognition of GUI controls may be used as general-purpose image recognition method, to complement other computerized operations as desired.

Control recognizer 600 optionally includes a screenshot capture engine 602, a cursor position identifier 604 and/or a cursor position region analyzer 606, each being a software component and/or module or all being implemented as a single component and/or module. FIG. 7 shows a flow chart of a method 700 (or an “algorithm”) for display-independent recognition of a GUI control; operation of control recognizer 600 of FIG. 6 shall now be described through an explanation of method 700 of FIG. 7.

In a block 702, a screenshot 704 of a GUI of a computerized application is captured, such as using screenshot capture engine 602 (FIG. 6). The capturing is optionally performed during an operation sequence performed by a user, namely, responsive to a cursor click, a keyboard input and/or the like by the user.

In an optional block 706, a position at which a cursor was at when performing the click, if such a click was indeed performed, may be determined. This may be performed by cursor position identifier 604 of FIG. 6. For example, with reference to FIG. 3, a cursor 334 is shown positioned next to and clicking on a “tab 4” text 336. Similarly, if a keyboard stroke was performed by the user, then a position of a control activated by the stroke may be detected. For example, if the keyboard combination ctrl+4 activates “tab 4” 336, the position of “tab 4” is detected and assumed.

In a block 708, if actions of optional block 706 are performed, then an area surrounding the determined cursor position is analyzed, such as by cursor position region analyzer 606 of FIG. 6. The analysis may enable identification of a control on which the cursor click was performed or which the keyboard stroke activated. Optionally, cursor position region analyzer 606 of FIG. 6 operates similar to screenshot analysis engine 102 FIG. 1. Accordingly, the analysis of block 708 is performed similar to the analysis of block 206 of FIG. 2, as described above.

Alternatively, in block 708, if actions of optional block 706 are not performed, then an area of screenshot 704 up to its entirety may be analyzed, such as by cursor position region analyzer 606 of FIG. 6. The analysis may enable identification of one or more controls existing in screenshot 704.

In a block 710, the identified control and/or its surrounding area may be stored as an image. The image of the control may then be used in another computerized process, application and/or the like, as desired. Advantageously, method 700 may enable the analysis of a GUI of a computerized application, in a manner enabling identification and extraction of individual controls from a screenshot of the GUI.

Reference is now made to FIG. 8, which shows a block diagram of a display-independent scroll bar recognizer 800.

Scroll bar recognizer 800 may be a software product installed on and/or adapted to be run on a computerized device, such as a Personal Computer (PC), a desktop computer, a mobile computer, a laptop computer, a notebook computer, a tablet computer, a server computer, a handheld computer, a handheld device, a mobile game console, an essentially stationary game console, a Personal Digital Assistant (PDA) device, an on-board device, an off-board device, a cellular phone, a hybrid device (for example a device incorporating functionalities of multiple types of devices, such as PDA functionality and cellular phone functionality), a vehicular device, a non-vehicular device, a mobile or portable device, a non-mobile or non-portable device and/or the like.

Scroll bar recognizer 800 may be used for recognizing and identifying a scroll bar in a screenshot of a GUI of a computerized application. The use of scroll bar recognizer 800 may or may not be tied to execution and/or recording of a guidance script, as set forth above. For example, recognition of scroll bars may be used as general-purpose image recognition method, to complement other computerized operations as desired.

Scroll bar recognizer 800 optionally includes a screenshot capture engine 802 and a scroll bar analyzer 804, each being a software component and/or module or both being implemented as a single component and/or module. FIG. 9 shows a flow chart of a method 900 (or an “algorithm”) for display-independent recognition of a scroll bar; operation of scroll bar recognizer 800 of FIG. 8 shall now be described through an explanation of method 900 of FIG. 9.

In a block 902, a screenshot 904 of a GUI of a computerized application is captured, such as using screenshot capture engine 802 of FIG. 8.

In a block 906, screenshot 904 is analyzed, in order to identify a scroll bar in a block 908. The analysis may be performed by scroll bar analyzer 804 of FIG. 8. The analysis may include a search for a control (namely, the scroll bar) which complies with the common characteristics of a scroll bar, namely—the existence of a thumb, an elongated trough and one or more arrows. These characteristics may be better understood by returning to FIG. 3. Horizontal scroll bar 316, for example, includes a thumb (sometimes referred to as a “bar”) 316a, which is optionally an element adapted to be dragged (using a cursor and/or a keyboard) along an elongated trough 316d. The dragging of thumb 316a may cause “scrolling”, namely, revelation of previously-invisible areas of text area 310. For example, if thumb 316a is dragged to the right, a previously-invisible right area of text area 310 is revealed. A width (or height, in case of a vertical scroll bar) of thumb 316a is optionally reflective of the degree of zooming applied. A thumb that completely fills a trough indicates that the entire document is being viewed. Alternatively, scrolling may be achieved by clicking on an arrow, such as left arrow 316b or right arrow 316c, of scroll bar 316. Clicking on left arrow 316b, for example, may reveal a previously-invisible left area of text area 310 and may move thumb 316a to the left.

The analysis may include a search for at least two elements of a scroll bar, namely, an arrow and a thumb. Optionally, the search is performed on a 1-bit and/or an inverted 1-bit version of screenshot 904. An arrow may be identified by searching for a continuous island of connected pixels that form a triangular shape. A thumb may be identified by searching in an axis parallel to the pointing direction of the identified arrow. Each element found in that axis may be analyzed, and the existence of a thumb may be determined if an essentially quadrangular island of connected pixels is found. An essentially empty space (represented by an island of connected pixels having the reverse color of the thumb) between the arrow and the thumb may resemble a trough.

Alternatively, the at least two elements searched for may be two arrows of opposing directions, lying on the same axis, which is parallel to their pointing directions.

In a block 910, the identified scroll bar is analyzed, to extract at least one parameter of it in a block 912. The analysis of the scroll bar may include a determination of one or more of the following parameters: a location of the scroll bar, a size of the scroll bar, a location of a thumb of the scroll bar, a size of a thumb of the scroll bar, a size of an arrow of the scroll bar and a location of an arrow of the scroll bar. Each of these sizes may be denoted in pixels, horizontally and/or vertically. Each of these locations may be denoted as X-Y coordinates of the scroll bar, the thumb or the arrow in relation to the entirety of the GUI.

Optionally, following the extraction of the above parameter(s), the scroll bar may be automatically operated for scrolling. For example, its thumb may be dragged and/or its arrows may be clicked for revealing a previously-invisible area of the GUI.

While a number of exemplary aspects and embodiments have been discussed above, those of skill in the art will recognize certain modifications, permutations, additions and sub-combinations thereof. It is therefore intended that the following appended claims and claims hereafter introduced be interpreted to include all such modifications, permutations, additions and sub-combinations as are within their true spirit and scope.

In the description and claims of the application, each of the words “comprise” “include” and “have”, and forms thereof, are not necessarily limited to members in a list with which the words may be associated.

Claims

1. A method for generating a display-independent computerized guidance script, the method comprising:

generating an application specific guidance script adapted to guide a user of a specific application through an operation sequence comprising a series of steps of operation of the specific application, said guidance script including actions pertaining to controls of the application and a control actuation sequence, wherein said generating an application specific guidance script comprises:
i. recording actions executed in the application during a performance of the operation sequence;
ii. recording first screenshots of a user interface of the specific application during the performance of the operation sequence;
iii. identifying within the first screenshots an individual control of the specific application associated with each of one or more of the recorded actions;
iv. extracting a separate image of each identified individual control from the recorded screenshots;
v. assigning each separate extracted image of each of the identified individual controls to a respective step of the operation sequence in the guidance script; and
vi. storing each separate extracted image of each of the identified individual controls together with the respective step of the operation sequence in the guidance script;
and
providing guidance script playback code adapted to cause a computer processor to: capture second screenshots of a client user interface on a computing platform including an instance of the specific application; identify, within the captured second screenshots, control elements of the instance of the specific application correlated to the identified individual controls stored in the guidance script so as to determine a screen location, within the client user interface, of each of the identified control elements; and use the determined screen locations to demonstrate operation of the operation sequence within the specific application, wherein to demonstrate includes displaying a visual description, or imitating, activation of a control element.

2. The method according to claim 1, wherein the guidance script playback code is adapted to cause the computer processor to identify control elements according to a pre-defined accuracy level.

3. The method according to claim 1, wherein the guidance script playback code is adapted to cause the computer processor to match surroundings of the identified individual controls with surroundings of the control elements of the instance of the specific application.

4. The method according to claim 1, wherein to demonstrate further includes displaying help text associated with the control element in a guidance bubble.

5. The method according to claim 1, further comprising converting the first screenshots to inverted 1-bit versions.

6. The method according to claim 1, wherein to demonstrate operation of the operation sequence within the specific application further comprises activating a control of the specific application.

7. The method according to claim 6, wherein the activating of the control comprises imitating a cursor click.

8. The method according to claim 6, wherein the activating of the control comprises imitating keyboard input.

9. The method according to claim 1, wherein the control element is selected from a group consisting of: an icon, a text, a button, an input box, a menu, a drop-down list, a slider, a scroll bar and a bar.

10. The method according to claim 1, the guidance script playback code is further adapted to cause the computer processor to activate a scroll bar for scrolling to a previously-invisible region of the specific application.

11. The method according to claim 1, further comprising converting the screenshot into a converted screenshot, wherein the converted screenshot is selected from the group consisting of: a resized screenshot; a portion of a screenshot; a grey-scaled screenshot; a 1-bit black-and-white screenshot; an inverted version of a screenshot; an inverted version of a grey-scaled screenshot; or an inverted version of a black-and-white version of a screenshot.

12. A computerized system for generating a display-independent computerized guidance script, said system comprising:

first processing circuitry adapted to record: (i) actions executed in a specific application, during a performance of an operation sequence comprising a series of steps of operation of the application; (ii) first screenshots of a user interface of the specific application during the performance of the operation sequence;
screenshot analysis circuitry adapted to process the first screenshots and: (i) identify within the first screenshots an individual control of the specific application associated with each of one or more of the recorded actions; (ii) extract a separate image of each identified individual control from the recorded screenshots
script generation circuitry adapted to generate an application specific guidance script adapted to guide a user of the specific application through the operation sequence, said guidance script including actions pertaining to controls of the application and a control actuation sequence, wherein said generating an application specific guidance script comprises i. assigning each separate extracted image of each of the identified individual controls to a respective step of the operation sequence in the guidance script; and ii. storing each separate extracted image of each of the identified individual controls together with the respective step of the operation sequence in the guidance script;
and
a tangible medium containing guidance script playback code adapted to cause a computer processor to: capture second screenshots of a client user interface on a computing platform including an instance of the specific application; identify, within the captured second screenshots, control elements of the instance of the specific application correlated to the identified individual controls stored in the guidance script so as to determine a screen location, within the client user interface, of each of the identified control elements; and use the determined screen locations to demonstrate operation of the operation sequence within the specific application, wherein to demonstrate includes displaying a visual description, or imitating, activation of a control element.

13. The system according to claim 12, wherein the guidance script playback code is adapted to cause the computer processor to identify control elements according to a pre-defined accuracy level.

14. The system according to claim 12, wherein the guidance script playback code is adapted to cause the computer processor to match surroundings of the identified individual controls with surroundings of the control elements of the instance of the specific application.

15. The system according to claim 12, wherein to demonstrate the operation sequence within the specific application further includes displaying help text associated with the control element in a guidance bubble.

16. The system according to claim 12, wherein to demonstrate operation of the operation sequence within the specific application further comprises activating a control of the specific application.

17. The system according to claim 16, wherein the activating of the control comprises imitating a cursor click.

18. The system according to claim 16, wherein the activating of the control comprises imitating keyboard input.

19. The system according to claim 12, wherein the guidance script playback code is further adapted to cause the computer processor to activate a scroll bar for scrolling to a previously-invisible region of the specific application.

20. A method for display-independent execution of a computerized guidance script, the method comprising:

retrieving from a guidance script repository an application specific guidance script adapted to guide a user of a specific application through an operation sequence comprising a series of steps of operation of the specific application, said guidance script including actions pertaining to controls of the application and a control actuation sequence, wherein said application specific guidance script includes:
i. actions recorded during execution of the operation sequence on a first computing device;
ii. separate images of each of a set of individual controls of the specific application, extracted from first screenshots recorded during the execution of the operation sequence on the first computing device,
wherein each of the separate images is assigned to a respective step of the operation sequence in the guidance script and stored together with the respective step in the guidance script;
capturing second screenshots of a client user interface on a computing platform including an instance of the specific application;
identifying, within the captured second screenshots, control elements of the instance of the specific application correlated to the identified individual controls stored in the guidance script so as to determine a screen location, within the client user interface, of each of the identified control elements; and
using the determined screen locations to demonstrate operation of the operation sequence within the specific application, wherein to demonstrate includes displaying a visual description, or imitating, activation of a control element.
Referenced Cited
U.S. Patent Documents
4622013 November 11, 1986 Cerchio
4648062 March 3, 1987 Johnson
4964077 October 16, 1990 Eisen
5117496 May 26, 1992 Stearns
5121497 June 9, 1992 Kerr
5179654 January 12, 1993 Richards
5252951 October 12, 1993 Tannenbaum
5261820 November 16, 1993 Slye
5287448 February 15, 1994 Nicol
5388993 February 14, 1995 McKiel
5395243 March 7, 1995 Lubin
5432940 July 11, 1995 Potts
5475843 December 12, 1995 Halviatti
5481667 January 2, 1996 Bieniek
5488685 January 30, 1996 Palmer
5546521 August 13, 1996 Martinez
5577186 November 19, 1996 Mann
5581684 December 3, 1996 Dudzik
5602982 February 11, 1997 Judd
5627958 May 6, 1997 Potts
5640537 June 17, 1997 Jessen
5680617 October 21, 1997 Gough
5745738 April 28, 1998 Ricard
5748805 May 5, 1998 Withgott
5790117 August 4, 1998 Halviatti
5825356 October 20, 1998 Habib
5867386 February 2, 1999 Hoffberg
5867678 February 2, 1999 Amro
6020886 February 1, 2000 Jacober
6252594 June 26, 2001 Xia
6307544 October 23, 2001 Harding
6308042 October 23, 2001 Marsh
6339436 January 15, 2002 Amro
6340977 January 22, 2002 Lui
6434532 August 13, 2002 Goldband
6438545 August 20, 2002 Beauregard
6453254 September 17, 2002 Bullwinkel
6563514 May 13, 2003 Samar
6662340 December 9, 2003 Rawat
6687485 February 3, 2004 Hopkins
6690390 February 10, 2004 Walters
6842877 January 11, 2005 Robarts
6937257 August 30, 2005 Dunlavey
7024633 April 4, 2006 Mann
7159188 January 2, 2007 Stabb
7262778 August 28, 2007 Edwards
7441250 October 21, 2008 Katz
7490292 February 10, 2009 Hennum
7543278 June 2, 2009 Klementiev
7546602 June 9, 2009 Hejlsberg
7620895 November 17, 2009 Adkins
7675529 March 9, 2010 Brunner
7861178 December 28, 2010 Lui
7890865 February 15, 2011 Palmer
7930643 April 19, 2011 Chandhoke
7987427 July 26, 2011 Gough
7992101 August 2, 2011 Chaudhri
8205168 June 19, 2012 Van Slembrouck
20020083422 June 27, 2002 Scouten
20020091993 July 11, 2002 Walley
20020186252 December 12, 2002 Himmel
20030001875 January 2, 2003 Black
20030046082 March 6, 2003 Siegel
20030084115 May 1, 2003 Wood
20030142123 July 31, 2003 Malamud
20030163801 August 28, 2003 Thames
20030171926 September 11, 2003 Suresh
20030210260 November 13, 2003 Palmer
20030219707 November 27, 2003 Hopkins
20040143814 July 22, 2004 de Jong
20040258298 December 23, 2004 Chen
20050050470 March 3, 2005 Hudson
20050080588 April 14, 2005 Kobayashi
20050091612 April 28, 2005 Stabb
20050172018 August 4, 2005 Devine
20060005132 January 5, 2006 Herdeg, III
20060053372 March 9, 2006 Adkins et al.
20060080607 April 13, 2006 Cohen
20060095860 May 4, 2006 Wada
20060168548 July 27, 2006 Kelley
20060177136 August 10, 2006 Windl
20060184880 August 17, 2006 Bala
20070122789 May 31, 2007 Yoo
20070266321 November 15, 2007 Bicker
20080034288 February 7, 2008 Landar
20080086700 April 10, 2008 Rodriguez
20080097984 April 24, 2008 Candelore
20080109722 May 8, 2008 Gengler
20080177994 July 24, 2008 Mayer
20080189612 August 7, 2008 Zhang
20080195958 August 14, 2008 Detiege
20080215976 September 4, 2008 Bierner
20080226119 September 18, 2008 Candelore
20080235339 September 25, 2008 Lurey
20090006985 January 1, 2009 Fong
20090322790 December 31, 2009 Behar
20100064238 March 11, 2010 Ludwig
20100070448 March 18, 2010 Omoigui
20100161586 June 24, 2010 Safar
20100175021 July 8, 2010 Ferrara
20100205530 August 12, 2010 Butin
20100325540 December 23, 2010 Biazetti
20110047462 February 24, 2011 Butin
20110047514 February 24, 2011 Butin
20120005611 January 5, 2012 Wey
Foreign Patent Documents
2272208 November 1999 CA
0690426 August 1999 EP
2337612 November 1999 GB
WO 2007/055610 May 2007 WO
2007/085603 August 2007 WO
WO2007/008503 August 2007 WO
Other references
  • “Bit Order”, captured by the Internet Archive Wayback Machine on Feb. 16, 2005, retrieved Jan. 2012 from http://web.archive.org/web/20050216154839/http://www.fileformat.info/mirror/egff/ch06—04.htm, pp. 1-3.
  • “Definition—of—platform—webopedia.pdf”, captured by the Internet Archive Wayback Machine on Jun. 15, 2007, retrieved Jul. 16, 2013 from http://web.archive.org/web/20070615144951/http://www.webopedia.com/TERM/P/PLATFORM.html on Jul. 16, 2013, p. 1.
  • “Wikipedia—cloud—computing.pdf”, captured by the Internet Archive Wayback Machine on Jan. 7, 2009, retreived from http://web.archive.org/web/20090107214626/http://en.wikipedia.org/wiki/Cloud—Computing.html on Jul. 16, 2013, pp. 1,5.
  • Author Unknown, “SHO Player User Guide,” [online] [retrieved Sep. 7, 2011]. Retrieved from the Internet: <URL:http://www.transcensus.com/player—userguide/sho—player—userguide—5.htm>.
  • Final rejection of U.S. Appl. No. 12/546,048 dated Jul. 17, 2012.
Patent History
Patent number: 9405558
Type: Grant
Filed: Aug 24, 2009
Date of Patent: Aug 2, 2016
Patent Publication Number: 20110047462
Assignee: KRYON SYSTEMS LTD. (Tel Aviv)
Inventors: Emma Butin (Tel Aviv), Oren Keinan (Netanya)
Primary Examiner: Aaron Lowenberger
Application Number: 12/546,042
Classifications
Current U.S. Class: Color Image Processing (382/162)
International Classification: G06F 9/44 (20060101); G06F 3/0489 (20130101); G06F 11/34 (20060101);