Testing a Touch-Input Program

Abstract

Methods and systems are disclosed that allow automated testing of an application program that is configured to receive a touch input. A testing mechanism can be configured to identify the touch input that is designed to produce a specified result. The testing mechanism can generate one or more signals simulating the touch input. The testing mechanism can then check the state of the user interface of the application program being tested and determine whether the actual result conforms to the specified result.

Description

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No. 61/357,090, filed Jun. 21, 2010, which is hereby incorporated by reference herein in its entirety.

TECHNICAL FIELD

This subject matter is generally related to software development.

BACKGROUND

A software development process can include a structure for creating and maintaining a software product. A software development process can include multiple stages. Some exemplary software development stages can include design, implementation, testing, and distribution. Some models of software development processes in existence today include waterfall model, spiral model, agile software development, extreme programming (XP), among others.

Modern software development processes for various data processing systems allow for participation of a vast number of diverse developers for a platform. Tools for developing software for the platform can include a publicly available software development kit (SDK) and various rules and guidelines. The SDK can include various libraries and an integrated development environment (IDE). Using the SDK, a developer can develop an application program for the platform. The application program can be distributed to data processing systems that are compatible with the platform, for example, through an application store.

SUMMARY

Methods and systems are disclosed that allow automated testing of an application program that is configured to receive a touch input. A testing mechanism queries the application program's user interface to identify a touch input that is to produce a specified result. The testing mechanism then generates one or more signals simulating the touch input. These signals are input to the application program. The application program responds accordingly, as if a user had actually performed the touch input. For example, the application program modifies its user interface based on the (simulated) touch input. The testing mechanism then queries the application program's user interface again, to determine whether the user interface conforms to the specified result.

These and other embodiments can optionally include one or more of the following features. The application program can include a user interface element that accepts the touch input. The user interface element can include an accessibility component that describes an accessibility attribute of the user interface element. The one or more signals simulating the touch input can be provided at least in part based on the accessibility attribute described in the accessibility component of the user interface element. A result can include an accessibility feature that corresponds to the accessibility attribute described in the accessibility component of the user interface element. The accessibility feature employed is the same one that can be used to provide a voice readout of the application program's user interface.

The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a flowchart illustrating an exemplary process of testing an application program for a device.

FIG. 2 illustrates components of an exemplary system implementing automated testing of a touch-input program.

FIGS. 3A-3D illustrate exemplary techniques of testing various touch screen input programs.

FIGS. 4A and 4B illustrate exemplary techniques of testing programs accepting inputs other than touch screen input.

FIG. 5 illustrates an exemplary user interface of testing a touch-input program.

FIG. 6 is a flowchart illustrating an exemplary process of automatically testing a touch-input program.

FIG. 7 is a flowchart illustrating an exemplary process of testing a touch-input program in an exemplary testing environment.

FIG. 8 is a block diagram illustrating an exemplary device architecture of a mobile device implementing the features and operations described in reference to FIGS. 1-7.

FIG. 9 is a block diagram of an exemplary system architecture for implementing the features and operations described in reference to FIGS. 1-8.

Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

Exemplary Software Development Process

FIG. 1 is a flowchart illustrating an exemplary process 100 of testing a program for a device. In exemplary process 100, a developer acquires (e.g., by downloading) an SDK of a platform, and develops an application program. The application program is to be distributed (e.g., provided for download) to other users of devices compatible with the platform. In this specification, any person who engages in any part of developing the application program can be a developer.

The developer can develop (102) an application program that accepts touch inputs. Developing the application program can include, for example, gathering requirements, designing the application program, writing source code of the application program, compiling the source code into binary code, and linking the binary code into executable code. The touch inputs can include inputs to a touch input component (e.g., a touch-sensitive display, a touch pad, or a touch switch). The touch screen input can include single-touch and multi-touch input. Additionally or alternatively, the touch inputs can include other motion inputs, e.g., shaking, rotating, or generally moving the device on which the application program executes.

The application program, or simply program, that is developed in process 100 can include any computer instructions that are configured to perform user tasks (e.g., tasks that produce results for a user) or system tasks (e.g., tasks that manage computing resources of a computer) or both. The application program can be an application program based on a specified platform. The platform can include libraries that provide functions and application programming interfaces (APIs) supporting the touch inputs. For example, an API can enable the developer to utilize various features of a touch input component, or various features of motion sensors, accelerometers, angular rate sensors, and magnetometers.

The developer can test (104) the application program, for example, by executing and debugging the executable code. The testing can be accomplished using one or more test scripts. The test scripts can include one or more instructions written in a software language that test a part or all of the functionalities of the application program and determine whether the application program meets requirements guiding the design and development of the application program, and performs reliably and cleanly (e.g., without causing memory leaks). In particular, the test scripts can test whether functionalities that accept touch inputs work correctly with corresponding hardware (e.g., a touch input component) of the device. The testing can be conducted on the device, on a host computing device to which the device is tethered or wirelessly connected, or on a computer that executes an emulator of the device.

Once the developer is satisfied with the testing, the developer can submit (106) the application program for review by a system or by a system developer (e.g., a developer responsible for the integrity of the platform). Submitting the application program for review can include uploading the linked binary executable code of the application program and the test script to a server for automatic or manual review. Submitting the application program for review can include submitting one or more test scripts. During the review, a system can automatically execute the submitted test script and test functionality and performance of the submitted application program on one or more types of devices or device emulators.

The developer can receive (108) results of the review by the system. If the application program is not qualified or approved, a message can be sent to the developer. The message can include a statement that the application program did not pass the review process, a list of one or more errors that occurred, and an explanation for each error. The developer can redesign, reimplement, and retest the application program for submission again.

Upon qualification and approval from the review, the application program can be distributed (120). Distributing the application program can include storing the application program in a data store and providing the application program for download by other users (e.g., the general public).

Exemplary System for Testing a Touch-Input Program

FIG. 2 illustrates components of an exemplary system 200 implementing automated testing of a touch-input program. The touch-input program includes a program that can accept a touch input, including an input from a touch input component (e.g., a touch-sensitive display, a touch pad, or a touch switch) or other motion input. Exemplary system 200 can include testing device 202 and testing host 204.

Testing device 202 can include a device on which application program 214 executes. Testing device 202 can include a variety of devices, both mobile and non-mobile. Mobile devices include, but are not limited to: a handheld computer, a personal digital assistant (PDA), a cellular telephone, an electronic tablet, a network appliance, a digital camera, a video camera, a smart phone, an enhanced general packet radio service (EGPRS) mobile phone, a media player, a navigation device, an email device, a game console, or a combination of any two or more of these devices. Testing device 202 can include one or more touch input components (e.g., a touch-sensitive screen, a touch pad, or a touch switch). Testing device 202 can alternatively or additionally include one or more motion or location sensing devices (e.g., an angular rate sensor, an accelerometer, a magnetometer, or a global positioning system (GPS) receiver).

Application program 214 can include various functionalities that utilize the touch input components and motion or location sensing devices of testing device 202. Application program 214 can include a function that is to be invoked upon receiving input from a touch input component, a motion input, or a combination of both. For example, application program 214 can cause testing device 202 to play a sound or speak a word or phrase when a user taps a specified display button, flips testing device 202 upside down, or waves testing device 202 in a specified pattern.

Testing host 204 can be configured to conduct testing of application program 214 in an automatic manner, without actually requiring a user to touch the touch input component of testing device 202 or flip or wave testing device 202. Testing host 204 can include one or more computing devices. Testing device 202 and testing host 204 can connect to one another through a connection cable or connection dock, or through a wired or wireless communications network. In some implementations, testing device can be an emulator (e.g., a virtual device) that executes on testing host 204 or a computing device connected to testing host 204.

Testing host 204 can host test script 206. Test script 206 can generate one or more signals simulating the touch input. The signals simulating the touch input can be fed to testing device 202 as inputs to application program 214. (In another embodiment, not shown, test script 206 is stored on testing device 202 and executes on testing device 202.) Application program 214 can execute in a testing environment where signals simulating the touch inputs replace inputs from various input devices and sensors of testing device 202. For example, if an item is configured to be currently displayed on a touch-sensitive display screen of testing device 202, a function of test script 206 can include generating a signal to the testing environment simulating a touch on the displayed item. Test script 206 can include a series of functions that can send the signals. The functions can be invoked at specified or user-configurable time intervals. The intervals (e.g., number of seconds) can be configured to allow application program 214 to have sufficient time to generate a response to a first touch input. The response can include, for example, displaying a second item on the touch-sensitive display screen. A second touch input can be a touch input on the second item. The confirmation and instruction can be provided using functions supported from an accessibility framework.

In one embodiment, application program 214 makes use of an accessibility framework. The accessibility framework defines a generic object (called an “accessibility object”) that can represent any type of user interface object, such as a window, a control, and even an application program itself. Accessibility objects enable an application to provide information about its user interface and capabilities (e.g., available functions) in a standard manner. Specifically, accessibility objects provide a uniform representation of an application's user interface elements, regardless of the application framework on which the application depends.

The accessibility framework represents an application's user interface as a hierarchy of accessibility objects. An accessibility object provides information about the user interface object that it represents. This information includes, for example, the object's position in the accessibility hierarchy, the object's position on the display, details about what the object is, and what actions the object can perform.

Accessibility objects can be communicated with via a particular application programming interface (the “Accessibility API”). An accessibility object within an application can perform actions, such as actions that correspond to user input like a touch input. Each accessibility object includes information about which actions it supports, if any. For example, the accessibility object representing a button supports the “press” action and sends a request to the actual button user interface element to perform the action.

An accessibility object has attributes associated with it. The number and kind of attributes vary depending on the type of user interface object the accessibility object represents. Attributes have values that can be used to find out about the user interface object. For example, the value of an accessibility object's role attribute indicates what type of user interface object that accessibility object represents.

Technically, an action is an attribute of an accessibility object. However, the accessibility protocol supports actions differently from the way it supports attributes, so actions and attributes are often described separately. An accessibility object can support one or more actions. An action describes how a user interacts with the user interface object that the accessibility object represents. In one embodiment, the accessibility framework defines seven actions that an accessibility object can support: press (a button), increment (the value of a scroller or slider indicator), decrement (the value of a scroller or slider indicator), confirm (the selection of an object), cancel (a selection or an action), raise (a window), and show menu (display a contextual menu associated with an object).

A message can be sent to an accessibility object. A first type of message requests information from the accessibility object. The accessibility object responds to this type of message by returning attribute values. A second type of message requests the performance of an action. The accessibility object responds to this type of message by performing the action.

In one embodiment, the various user interface elements and capabilities of application program 214 are represented by accessibility objects. Test script 206 interacts with the user interface of application program 214 via the Accessibility API. The Accessibility API enables communication with the accessibility objects that represent the user interface elements of application program 214.

Recall that test script 206 can generate one or more signals simulating a touch input. These signals are sent as messages, via the Accessibility API, to accessibility objects that represent user interface elements of application program 214. The messages cause the accessibility objects to perform various actions, thereby sending requests to the actual user interface elements to perform the actions and simulating the user input. Test script 206 can also use messages to obtain information about the state of the user interface of application program 214.

One embodiment of an accessibility framework is described in “Chapter 3: The Mac OS X Accessibility Protocol” within “Accessibility Overview”, by Apple Inc.

Apart from the actions defined in the Accessibility API that are directed to a particular accessibility object, the testing device may support more general APIs for simulating touch events at a system level. For example, a button's accessibility object might accept a “press” action, which tells the button to respond as if it were pressed. This approach sends the action directly to the desired user interface object, but bypasses any OS and application components devoted to routing real user events. The system-level input simulation APIs, on the other hand, allow for more user-event-like actions such as: tap at screen location {x=150, y=320}, without regard to what is at that screen location. These events enter the application being tested in essentially the same way that real user events do, providing testing of those same event-routing code paths. The testing mechanism described here can invoke either the system-level input simulation APIs or the Accessibility action APIs as needed, to best simulate the actions specified in the test script.

Testing host 204 can host event recorder 208. Event recorder 208 can be configured to receive various outputs of test script 206. The outputs of test script 206 can be submitted to the recorder 208. The recorder can record events submitted from testing device 202. The events can include content of the outputs (e.g., a screen shot, an audio recording, or a notification of a vibration) and metadata. The metadata can include, for example, a time that the outputs occurred, a memory status (e.g., a call stack) at the time the outputs occurred, variable values of application program 214, and other information related to a status of application program 214 and testing device 202.

Events recorded by event recorder 208 can be stored in event data store 210. Event analyzer 212 can retrieve data stored in event data store 210 and replay the events. The replay can include presenting various events in a user interface, including screen shots, audio outputs, and physical action of testing device 202. The physical action can be represented using an audio or visual representation. The replay can include displaying various memory states along a timeline.

FIGS. 3A-3D illustrate exemplary techniques of testing various touch screen input programs. Similar techniques apply to testing programs that use different touch input components, such as a touch pad. FIG. 3A illustrates exemplary techniques of testing an application program accepting touch input from a user interface element that supports accessibility functions. Accessibility functions can include functions that provide assistance (e.g., enlarged fonts and/or voice readouts) to people with disabilities (e.g., the visually impaired) for accessing the application program.

The application program “My App” can execute on mobile device 300a. A graphical user interface of the application program can be displayed on touch-sensitive screen 302a of mobile device 300a. The graphical user interface can include window 304. Window 304 can include virtual button 306 “Configure Network” that accepts a touch input (e.g., a tap). Virtual button 306 can be associated with accessibility features.

A test script (e.g., test script 206) can be utilized to test the functions of virtual button 306 and accessibility features associated with virtual button 306. To test a response to a touch input on virtual button 306, the test script can use one or more of the following exemplary functions:

• • hitpoint ( ). The hitpoint function can return a screen position to tap for a specified display element.
  • tap ( ) or tap (object) or tap (location). The tap function can generate a signal simulating a single tap. The tap function can be a method of an identified object, or be performed on the object in the parameter, or be performed at any arbitrary coordinate of the touch-sensitive display. In general, the object in the parameter can include any display or non-display element that can accept a touch input.
  • tapAndHold (duration) or tapAndHold (object, duration) or tapAndHold (location, duration). The tapAndHold function can generate a signal simulating a tap-and-hold. The parameter duration can include a length of time to hold the touch on an element being tapped, in seconds. The default duration value for a tap can be 0. The default value for touch-and-hold gestures (such as drag, pinch open, and pinch close) can be 1.
  • doubleTap ( ) or doubleTap (object) or doubleTap (location). The doubleTap function can generate a signal simulating a double tap.
  • twoFingerTap ( ) or twoFingerTap (object) or twoFingerTap (location). The twoFingerTap function can generate one or more signals simulating a two-finger (or two-stylus) tap.

The exemplary functions can be associated with various display elements (e.g., virtual button 306). The display elements can be identified by one or more display element identifiers. For example, the test script can include a function that returns a main display window, use an element ( ) function to retrieve one or more display elements of the main display window, and identify virtual button 306 from the retrieved elements by name, identifier, or class membership.

The test script can be utilized to test accessibility functions of virtual button 306. For example, virtual button 306 can be configured to invoke dialog balloon 308 upon receiving a single touch. Dialog balloon 308 can include enlarged text of virtual button 306 as an aid to visually impaired users. Additionally or alternatively, virtual button 306 can be configured to invoke a voice over that includes speech (e.g., synthesized or pre-recorded voice) or a Braille output describing the virtual button 306 upon receiving the single touch. The test script can include functions for capturing dialog balloon 308 (e.g., through a screenshot), the voice over (e.g., through recording), or the Braille output for testing the accessibility functions.

FIG. 3B illustrates exemplary techniques of testing an application program accepting touch screen inputs that includes a drag-and-drop. The application program “My App” can execute on mobile device 300b. The application program is configured to display a graphical user interface in which a user can drag-and-drop a display element 310 by tapping (and holding) on display element 310 using a touching means (finger or stylus); while maintaining contact with display element 310 on a display screen 302b, moving the touching means into a display area 314; and releasing display element 310. Upon completion of the drag-and-drop, display element 310 can be displayed in display area 314.

A test script (e.g., test script 206) can be utilized to test the drag-and-drop functions. To test a response to a drag-and-drop input on display element 310, the test script can use one or more of the following exemplary functions: p1 dragInsideWithOptions (options) or dragInsideWithOptions (object, options) or dragInsideWithOptions (location, options). The draglnsideWithOptions function can be used to generate one or more signals simulating the drag-and-drop input.

The options can include a dictionary that specifies characteristics of a gesture (e.g., a movement of a touching means on a touch-sensitive display screen). The characteristics can include one or more of the following:

• • touchCount—a number of touches to use in the specified gesture. For example, the touchCount can represent a number of touching means (fingers or styluses) a user would use to make the specified gesture. The default touchCount value can be 1.
  • duration—a length of time to hold the touch on an element being tapped, in seconds. The default duration value for a tap can be 0. The default value for touch-and-hold gestures (such as drag, pinch open, and pinch close) can be 1.
  • startOffset—a first offset to use in a multiple-point gesture (e.g., a drag-and-drop). The offset can be measured against a current display element within which the drag-and-drop occurs. A default value can be {x:0.0, y:0.0}.
  • endOffset—a last offset to use in a multiple point gesture (e.g., a drag-and-drop). A default value can be {x:0.0, y:0.0}.

The test script can use offsets to achieve finer precision in specifying the hitpoint within a rectangle (rect) for the specified display element. The offset can include a pair of values x and y, each ranging from 0.0 to 1.0. The x and y values can represent, respectively, relative horizontal and vertical positions within the rectangle, with {x:0.0, y:0.0} as the top left and {x:1.0, y:1.0} as the bottom right. For example, {x:0.3, y:0.6} can specify a position just below and to the left of a center; and {x:1.0, y:0.5} can specify a position centered vertically at the far right.

Additionally or alternatively, the test script can use one or more of the following exemplary functions:

• • dragFromToForDuration (fromPointObject, toPointObject, duration).

The fromPointObject parameter can specify a rectangle or point from which the drag action is to begin. The toPointObject parameter can specify a rectangle or point to which the drag action is to end. The duration parameter can specify a length of time, in seconds, between starting and stopping of the drag-and-drop action.

The test script can use the function to simulate the drag-and-drop input that drags display element 310 and drops display element 310 into display area 314. A graphical user interface before the drag-and-drop and a graphical user interface after the drag-and-drop can be recorded.

FIG. 3C illustrates exemplary techniques of testing an application program accepting touch screen inputs that includes pinching and spreading. The application program “My App” can execute on mobile device 300c. The application program is configured to display on touch-sensitive screen 302c a graphical user interface that can accept a pinching input. The pinching input can include a touch screen input where two or more touching means (fingers or styluses) are moving away from each other (pinch-open) or towards each other (pinch-close). The pinch-open and pinch-close inputs can be configured to cause a display element to resize or zoom. For example, display element 326 can be resized to display element 324 upon receiving a pinch-open input. Likewise, display element 324 can be resized to display element 326 upon receiving a pinch-close input. Other functions (e.g., zoom out and zoom in) can be performed in response to the pinch-open and pinch-close inputs.

A test script (e.g., test script 206) can be utilized to test the pinching functions. To test a response to a pinch-open input or pinch-close input on display element 324 or 326, the test script can use one or more of the following exemplary functions:

• • pinchOpenFromToForDuration (fromPointObject, toPointObject, duration). This function can generate a signal simulating a pinch-open gesture from a specified starting screen location to a specified ending screen location, for a specified length of time.
  • pinchCloseFromToForDuration (fromPointObject, toPointObject, duration). This function can generate a signal simulating a pinch-close gesture from a specified starting screen location to a specified ending screen location, for a specified length of time.

A screenshot of the user interface before and after simulated signals of the pinch-open and pinch-close inputs can be recorded. The screenshot can be analyzed to determine, for example, whether display element 324 or 326 has resized correctly.

FIG. 3D illustrates exemplary techniques of testing an application program accepting a flick input. The application program “My App” can execute on mobile device 300d. The application program is configured to display on touch-sensitive screen 302d a graphical user interface that can accept a flick input. The flick input can include a touch screen input that includes a quick movement of a touching means (finger or stylus) to a direction. The flick input can be configured to cause a display element to move to a next element. For example, display element 332 (e.g., a first virtual page) can be configured to curl up to reveal another display element (e.g., a second virtual page) underneath. Alternatively or additionally, the display element can move to the left, right, up, or down to reveal a next display element. In an example accessibility feature, a menu item can be voiced over. A flick input can cause a next menu item to be voiced over.

A test script (e.g., test script 206) can be utilized to test the flicking functions. To test a response to a flick input on display element 332, the test script can use one or more of the following exemplary functions:

• • flickInsideWithOptions (options) or flickInsideWithOptions (object, options) or flickInsideWithOptions (location, options).

The options can include a touchCount, a startOffset, and an endOffset, as described above, and other parameters. The functions can generate one or more signals simulating the flick input.

A screenshot of the user interface before the signals simulating the flick input and a screenshot of the user interface after the signals simulating the flick input can be taken. The screenshots can be analyzed to determine, for example, whether display element 332 has moved in accordance with a specified manner (e.g., according to design). A voice recording can be made if the simulated signals of the flick input cause a new menu item to be voiced over.

An application program accepting a rotation gesture input can be tested. The application program can execute on a mobile device. The application program is configured to display on a touch-sensitive screen a graphical user interface that can accept a rotation gesture input (a type of touch-and-hold gesture). The rotation gesture input can include a touch screen input that includes a rotating movement of one or more touching means (fingers or styluses) in a clockwise or counterclockwise direction (e.g., rotating a finger while it is pressing on the touch screen). For example, the rotation gesture input can be configured to cause a display element to rotate. As another example, the rotation gesture input can be configured to cause a dial-like display element to turn to a new value.

A test script (e.g., test script 206) can be utilized to test the rotation gesture functions. To test a response to a rotation gesture input on a display element, the test script can use one or more of the following exemplary functions:

• • rotateWithOptions (options) or rotateWithOptions (location, options). The rotateWithOptions function can be used to generate one or more signals simulating the rotation gesture input.

The options parameter can include a dictionary that specifies characteristics of a rotation gesture (e.g., a movement of one or more touching means on a touch-sensitive display screen). The characteristics can include one or more of the following:

• • touchCount—a number of touches to use in the specified gesture. For example, the touchCount can represent a number of touching means (fingers or styluses) a user would use to make the specified gesture. A valid touchCount value can range from 1 to 5 (inclusive), and the default touchCount value can be 2.
  • duration—a length of time to hold the touch on an element being rotated, in seconds. The default duration value can be 1.
  • rotation—an amount of rotation, in radians. The default value can be pi (π).
  • centerOffset—an offset to use for the center of the rotation gesture. A default value can be {x:0.0, y:0.0}. This characteristic is not used with a rotateWithOptions function that includes a location parameter.

The location parameter can include a point object at the center of the rotation gesture, with properties for x and y. The relevant coordinates are screen-relative and are adjusted to account for device orientation.

A screenshot of the user interface before the signals simulating the rotate gesture input and a screenshot of the user interface after the signals simulating the rotate gesture input can be taken. The screenshots can be analyzed to determine, for example, whether the display element has moved in accordance with a specified manner (e.g., according to design).

FIGS. 4A and 4B illustrate exemplary techniques of testing programs accepting inputs other than touch screen input. FIG. 4A illustrates exemplary techniques of testing an application program accepting a shake input. The application program “My App” can execute on mobile device 400a. The application program can be configured to provide a response to a motion input of mobile device 400a. The motion input (e.g., a shake) can be detected using a motion sensor, an angular rate sensor, or an accelerometer. The response can include, for example, a voice over of all input items displayed on a display screen, opening or closing a particular user interface, or turning on or turning off mobile device 400a.

A test script (e.g., test script 206) can be utilized to test the application program in response to a shake input. To test the response to the shake input, the test script can use a shake function, which can provide a signal simulating a shake event of mobile device 400a. The shake function can have parameters that can be used to specify a direction, magnitude, and duration of the shake. Screenshots of user interfaces before and after the simulated signals of the shake input can be recorded. A voice recording can be made if the simulated signals of the shake input cause a voice over.

FIG. 4B illustrates exemplary techniques of testing an application program accepting a change of orientation input. The application program “My App” executes on mobile device 400b and can be configured to provide a response to a change of orientation of mobile device 400b. An orientation of mobile device 400b can include, for example, a portrait mode (e.g., upside up or upside down), a landscape mode (e.g., landscape left or landscape right), and a face mode (e.g., face up or face down). The orientation can additionally or alternatively include an orientation of mobile device 400b in a global reference frame. For example, the orientation can include a pointing direction or heading of an axis of the mobile device (e.g., “north west” or “bearing 315 degrees”). The change of orientation can be detected using an accelerometer, an angular rate sensor, a motion sensor, a magnetometer, or a combination of the sensors.

The application program “My App” executing on mobile device 400b can alternatively or additionally be configured to provide a response to a change of location of mobile device 400b. The change of location can be detected using a GPS device, a wireless triangulation device, a baseband processor, a magnetometer, or a combination of the sensors. A test script (e.g., test script 206) can be utilized to test behavior of the application program in response to a change of orientation or change of location. To test a response to a change of orientation or change of location input the test script can use one or more of the following exemplary functions:

• • getDeviceOrientation. This function can retrieve an orientation of mobile device 400b.
  • setDeviceOrientation (orientation). This function can set an orientation of mobile device 400b.
  • getDeviceLocation ( ). This function can get a location of mobile device 400b.
  • setDeviceLocation (location). This function can set a location of mobile device 400b.

The functions can generate one or more signals simulating a change in orientation or location. Screenshots of user interfaces before and after the simulated signals of the change of orientation or change of location can be recorded. A recording can be made if the simulated signals of the change of orientation or change of location input cause a voice over. A change of user interfaces (e.g., a change of a map display or a change of a compass dial display) can be compared against the simulated change of orientation or change of location to determine whether the application program “My App” behaves correctly.

FIG. 5 illustrates an exemplary user interface 500 of testing a touch-input program. Exemplary user interface 500 can be displayed on a display device coupled to a testing host (e.g., testing host 204 of FIG. 2) or a testing device (e.g., testing device 202 of FIG. 2).

User interface 500 can include instruments section 502. Instruments section 502 can present for display various testing projects (e.g., groups of scripts, application programs, and associated devices). In the example given, a test of a touch-input program to be executed on example device “Device 1” is being conducted.

User interface 500 can include script control section 504. Script control section 504 can include control elements that enable a user to select one or more automated test scripts (e.g., test script 206 of FIG. 2). The test scripts can be written using various programming languages (e.g., JavaScript). Script control section 504 can include controls for opening one or more editing windows of the test scripts.

User interface 500 can include screenshot control section 506. Screenshot control section 506 can include controls that can configure a time of taking screenshots. The screenshots can be taken and recorded upon request or at predetermined screenshot time intervals or at screenshot points specified in the test script.

User interface 500 can include time control sections 508a and 508b. Time control sections 508a and 508b can include controls for configuring an inspection time range for the testing. The inspection time range can specify a beginning time and an end time for inspecting behavior of the application program being tested. The beginning time and end time can be offsets from the beginning of the application program execution.

User interface 500 can include script log section 510. Script log section 510 can include a set of events triggered by the test script. The set of events can be organized by time. A selection of a particular event can cause data related to the event to be displayed in event display section 512.

Event display section 512 can display one or more events selected by the user. The event can be displayed in association with event data. The event data can include stack data, variable data (e.g., values of variables at time of the event), and memory status. The event data can include one or more screenshots (e.g., screenshot 514) of the event. For example, the screenshot can include a display element (e.g., display element 324 of FIG. 3C) before a pinch-close input is applied to the display element. Additionally or alternatively, the event data can include an audio clip (e.g., a voice over) associated with an event.

As mentioned above, an application program that is configured to receive a touch input can be tested in an automated manner. A testing mechanism queries the application program's user interface (e.g., by sending messages to accessibility objects using the Accessibility API) to identify a touch input that is to produce a specified result. The testing mechanism then generates one or more signals simulating the touch input. These signals are input to the application program (e.g., by sending messages via the Accessibility API to various accessibility objects). The application program responds accordingly, as if a user had actually performed the touch input. For example, the application program modifies its user interface based on the (simulated) touch input. The testing mechanism then queries the application program's user interface again, to determine whether the user interface conforms to the specified result.

FIG. 6 is a flowchart illustrating exemplary process 600 of automatically testing a touch-input application program. For convenience, process 600 will be described in reference to a system implementing process 600. The system can include a device that executes the touch-input program, or a computer connected to the device, or both.

The system can interrogate (602) the user interface of the application program to identify a touch input operable for producing a specified result in the application program. The application program can include a user interface element that accepts a touch input. In some implementations, the touch input includes a gesture of one or more touching means (fingers or styluses) on a display screen of the device. The user interface element can include one or more accessibility components (e.g., accessibility objects) that describe one or more accessibility attributes of the user interface element. The values of these attributes provide information about the user interface object and can be queried by the system.

The system can generate (604) one or more signals simulating the touch input. The one or more signals simulating the touch input can be provided at least in part based on the accessibility attribute described in the accessibility component of the user interface element. For example, the signals can correspond to the tap or flick that is to elicit the specified result.

The system interrogates (606) the user interface again, checking for the expected changes. The system can also receive events logged by the script (e.g., failure to simulate the touch input because the specified user interface element does not exist or an explicit script verification of one or more facets of the current user interface state) (not shown). An event can include an accessibility feature that corresponds to the accessibility attribute described in the accessibility component of the user interface element.

The system can determine (608) that the actual result conforms to the specified result. Determining that the actual result conforms to the specified result can include comparing screenshots of the actual result with design specifications of the touch-input program.

In some implementations, the system can provide a library of signals simulating various touch inputs. A test script for testing the touch-input application program can include serial or parallel calls to the library to generate one or more signals simulating sequential or concurrent touch inputs.

FIG. 7 is a flowchart illustrating an exemplary process of testing a touch-input program in an exemplary testing environment.

A testing host can receive (702) a first event. The first event can include the test script logging a simulated touch input to an application program executing on a device. The device can be a device connected to the testing host through a communications network (e.g., a local area network (LAN) or a wide area network (WAN)). The first event can be generated from a test script. The simulated touch input for the application program can be produced by the testing script, or by a testing tool kit in response to an execution of the testing script. The simulated touch input can include a single touch input, a multi touch input, a gesture, or a physical movement of the device. The simulated user input is associated with a timestamp.

The testing host can receive (704) a second event from the test script, verifying that the application program user interface is in an expected state following the touch input. The event can include a screenshot of the application program.

The testing host can acquire (706) a status of the device. The status can correspond to the response to the simulated touch input. The status can include a call stack, a memory status, including memory failure status, or a memory leak status.

The testing host can provide (708) for display a monitoring interface. Providing for display the monitoring interface includes providing for display, based on a timestamp, the visual representation of the first event, the visual representation of the second event, and visual representations of other events in a timeline.

Exemplary Mobile Device Architecture

FIG. 8 is a block diagram illustrating exemplary device architecture 800 of a mobile device implementing the features and operations described in reference to FIGS. 1-7. The mobile device can be, for example, a handheld computer, a personal digital assistant, a cellular telephone, an electronic tablet, a network appliance, a camera, a smart phone, an enhanced general packet radio service (EGPRS) mobile phone, a network base station, a media player, a navigation device, an email device, a game console, or a combination of any two or more of these data processing devices or other data processing devices.

The mobile device can include memory interface 802, one or more data processors, image processors and/or processors 804, and peripherals interface 806. Memory interface 802, one or more processors 804 and/or peripherals interface 806 can be separate components or can be integrated in one or more integrated circuits. Processors 804 can include one or more application processors (APs) and one or more baseband processors (BPs). The application processors and baseband processors can be integrated in one single process chip. The various components in the mobile device, for example, can be coupled by one or more communication buses or signal lines.

Sensors, devices, and subsystems can be coupled to peripherals interface 806 to facilitate multiple functionalities. For example, motion sensor 810, light sensor 812, and proximity sensor 814 can be coupled to peripherals interface 806 to facilitate orientation, lighting, and proximity functions of the mobile device. Location processor 815 (e.g., GPS receiver) can be connected to peripherals interface 806 to provide geopositioning. Electronic magnetometer 816 (e.g., an integrated circuit chip) can also be connected to peripherals interface 806 to provide data that can be used to determine the direction of magnetic North. Thus, electronic magnetometer 816 can be used as an electronic compass. Accelerometer 817 can also be connected to peripherals interface 806 to provide data that can be used to determine change of speed and direction of movement of the mobile device. An angular rate sensor (e.g., a Micro-Electro-Mechanical System (MEMS) gyro) can be connected to peripherals interface 806 to provide data that can be used to determine a rotational velocity of the mobile device.

Camera subsystem 820 and an optical sensor 822, e.g., a charged coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) optical sensor, can be utilized to facilitate camera functions, such as recording photographs and video clips.

Communication functions can be facilitated through one or more wireless communication subsystems 824, which can include radio frequency receivers and transmitters and/or optical (e.g., infrared) receivers and transmitters. The specific design and implementation of the communication subsystem 824 can depend on the communication network(s) over which a mobile device is intended to operate. For example, a mobile device can include communication subsystems 824 designed to operate over a GSM network, a GPRS network, an EDGE network, a Wi-Fi or WiMax network, and a Bluetooth network. In particular, the wireless communication subsystems 824 can include hosting protocols such that the mobile device can be configured as a base station for other wireless devices.

Audio subsystem 826 can be coupled to a speaker 828 and a microphone 830 to facilitate voice-enabled functions, such as voice recognition, voice replication, digital recording, and telephony functions.

I/O subsystem 840 can include touch screen controller 842 and/or other input controller(s) 844. Touch-screen controller 842 can be coupled to a touch screen 846 or pad. Touch screen 846 and touch screen controller 842 can, for example, detect contact and movement or break thereof using any of a plurality of touch sensitivity technologies, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch screen 846.

Other input controller(s) 844 can be coupled to other input/control devices 848, such as one or more buttons, rocker switches, thumb-wheel, infrared port, USB port, and/or a pointer device such as a stylus. The one or more buttons (not shown) can include an up/down button for volume control of speaker 828 and/or microphone 830.

In one implementation, a pressing of the button for a first duration may disengage a lock of the touch screen 846; and a pressing of the button for a second duration that is longer than the first duration may turn power to the mobile device on or off. The user may be able to customize a functionality of one or more of the buttons. The touch screen 846 can, for example, also be used to implement virtual or soft buttons and/or a keyboard.

In some implementations, the mobile device can present recorded audio and/or video files, such as MP3, AAC, and MPEG files. In some implementations, the mobile device can include the functionality of an MP3 player. The mobile device may, therefore, include a pin connector that is compatible with the MP3 player. Other input/output and control devices can also be used.

Memory interface 802 can be coupled to memory 850. Memory 850 can include high-speed random access memory and/or non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and/or flash memory (e.g., NAND, NOR). Memory 850 can store operating system 852, such as Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks. Operating system 852 may include instructions for handling basic system services and for performing hardware dependent tasks. In some implementations, operating system 852 can include a kernel (e.g., UNIX kernel).

Memory 850 may also store communication instructions 854 to facilitate communicating with one or more additional devices, one or more computers and/or one or more servers. Memory 850 may include graphical user interface instructions 856 to facilitate graphic user interface processing; sensor processing instructions 858 to facilitate sensor-related processing and functions; phone instructions 860 to facilitate phone-related processes and functions; electronic messaging instructions 862 to facilitate electronic-messaging related processes and functions; web browsing instructions 864 to facilitate web browsing-related processes and functions; media processing instructions 866 to facilitate media processing-related processes and functions; GPS/Navigation instructions 868 to facilitate GPS and navigation-related processes and instructions; camera instructions 870 to facilitate camera-related processes and functions; magnetometer data 872 and calibration instructions 874 to facilitate magnetometer calibration. The memory 850 may also store other software instructions (not shown), such as security instructions, web video instructions to facilitate web video-related processes and functions, and/or web shopping instructions to facilitate web shopping-related processes and functions. In some implementations, the media processing instructions 866 are divided into audio processing instructions and video processing instructions to facilitate audio processing-related processes and functions and video processing-related processes and functions, respectively. An activation record and International Mobile Equipment Identity (IMEI) or similar hardware identifier can also be stored in memory 850. Memory 850 can include testing instructions 876 that can perform one or more functions as described above.

Each of the above identified instructions and applications can correspond to a set of instructions for performing one or more functions described above. These instructions need not be implemented as separate software programs, procedures, or modules. Memory 850 can include additional instructions or fewer instructions. Furthermore, various functions of the mobile device may be implemented in hardware and/or in software, including in one or more signal processing and/or application specific integrated circuits.

Exemplary System Architecture

FIG. 9 is a block diagram of an exemplary system architecture 900 for implementing the features and operations described in reference to FIGS. 1-8. Other architectures are possible, including architectures with more or fewer components. In some implementations, architecture 900 includes one or more processors 902 (e.g., dual-core Intel® Xeon® Processors), one or more output devices 904 (e.g., LCD), one or more network interfaces 906, one or more input devices 908 (e.g., mouse, keyboard, touch-sensitive display, touch pad, touch switch) and one or more computer-readable mediums 912 (e.g., RAM, ROM, SDRAM, hard disk, optical disk, flash memory, etc.). These components can exchange communications and data over one or more communication channels 910 (e.g., buses), which can utilize various hardware and software for facilitating the transfer of data and control signals between components.

The term “computer-readable medium” refers to any medium that participates in providing instructions to processor 902 for execution, including without limitation, non-volatile media (e.g., optical or magnetic disks), volatile media (e.g., memory) and transmission media. Transmission media includes, without limitation, coaxial cables, copper wire and fiber optics.

Computer-readable medium 912 can further include operating system 914 (e.g., Mac OS® server, Windows® NT server), network communication module 916, database interface 920, test script 930, event recorder 940, event data store 950, and event analyzer 960, as described in reference to FIGS. 1-8. Operating system 914 can be multi-user, multiprocessing, multitasking, multithreading, real time, etc. Operating system 914 performs basic tasks, including but not limited to: recognizing input from and providing output to devices 908, 904; keeping track of and managing files and directories on computer-readable mediums 912 (e.g., memory or a storage device); controlling peripheral devices; and managing traffic on the one or more communication channels 910. Network communications module 916 includes various components for establishing and maintaining network connections (e.g., software for implementing communication protocols, such as TCP/IP, HTTP, etc.). Database interface 920 can include interface to various data stores such as event data store 950.

Architecture 900 can be included in any device capable of hosting a database application program. Architecture 900 can be implemented in a parallel processing or peer-to-peer infrastructure or on a single device with one or more processors. Software can include multiple software components or can be a single body of code.

The described features can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language (e.g., Objective-C, Java), including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

Suitable processors for the execution of a program of instructions include, by way of example, both general and special purpose microprocessors, and the sole processor or one of multiple processors or cores, of any kind of computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer will also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, ASICs (application-specific integrated circuits).

To provide for interaction with a user, the features can be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user and a keyboard and a pointing device such as a mouse or a trackball by which the user can provide input to the computer.

The features can be implemented in a computer system that includes a back-end component, such as a data server, or that includes a middleware component, such as an application server or an Internet server, or that includes a front-end component, such as a client computer having a graphical user interface or an Internet browser, or any combination of them. The components of the system can be connected by any form or medium of digital data communication such as a communication network. Examples of communication networks include, e.g., a LAN, a WAN, and the computers and networks forming the Internet.

The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

A number of implementations have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. For example, pseudo code of exemplary functions is provided. Functions that implement the features described in this specification can have any names or parameters. Accordingly, other implementations are within the scope of the following claims.

Claims

1. A method, comprising:

identifying a touch input operable for producing a specified result in an application program;

generating one or more signals simulating the touch input;

receiving, in response to the one or more signals, a result from the application program; and

determining that the received result conforms with the specified result.

2. The method of claim 1, where the application program includes a user interface element that accepts the touch input.

3. The method of claim 2, where the user interface element includes an accessibility component that describes an accessibility attribute of the user interface element.

4. The method of claim 3, where the one or more signals simulating the touch input are provided at least in part based on the accessibility attribute described in the accessibility component of the user interface element.

5. The method of claim 3, where the received result includes an accessibility feature that corresponds to the accessibility attribute described in the accessibility component of the user interface element.

6. The method of claim 1, where the touch input includes a gesture of one or more touching means on a touch input component.

7. The method of claim 6, where the touch input component comprises a touch-sensitive display or a touch pad.

8. The method of claim 1, further comprising:

providing a library of signals simulating touch inputs; and

providing a script that includes calls to the library.

9. A method executed on a computer, comprising:

receiving a first event, the first event including a simulated touch input to an application program executing on a device;

receiving a second event from the device, the second event including a response to the simulated touch input, the response produced by a verification step performed by a test script;

acquiring a status of the device, the status corresponding to the response to the simulated touch input produced; and

providing for display a monitoring interface, the monitoring interface including a visual representation of the first event, a visual representation of the second event, and one or more visual representations of the status associated with the second event.

10. The method of claim 9, where the device is a mobile device connected to the computer through a wide area network (WAN).

11. The method of claim 9, further comprising:

executing the test script, the test script producing the simulated touch input to the application program.

12. The method of claim 9, where acquiring the status of the device comprises:

acquiring a memory status, the memory status including at least one element of a group containing a call stack, a memory failure status, and a memory leak status.

13. The method of claim 9, where receiving the second event from the device includes receiving a screenshot from the device.

14. The method of claim 9, where the simulated touch input includes a simulated multi-touch input.

15. The method of claim 9, where:

the simulated touch input is associated with a timestamp; and

providing for display the monitoring interface includes providing for display, based on the timestamp, the visual representation of the first event, the visual representation of the second event, and visual representations of other events in a timeline.

16. A computer program product tangibly stored on a storage device, operable to cause data processing apparatus to perform operations comprising:

identifying a touch input operable for producing a specified result in an application program;

generating one or more signals simulating the touch input; and

determining whether the user interface conforms to the specified result.

17. The product of claim 16, where the application program includes a user interface element that accepts the touch input.

18. The product of claim 17, where the user interface element includes an accessibility component that describes an accessibility attribute of the user interface element.

19. The product of claim 18, where the one or more signals simulating the touch input are provided at least in part based on the accessibility attribute described in the accessibility component of the user interface element.

20. A system, comprising:

one or more computers configured to perform operations comprising: identifying a touch input operable for producing a specified result in an application program; generating one or more signals simulating the touch input; and determining whether the user interface conforms with the specified result.