|Publication number||US7712074 B2|
|Application number||US 10/302,453|
|Publication date||May 4, 2010|
|Filing date||Nov 21, 2002|
|Priority date||Nov 21, 2002|
|Also published as||US20040100502, US20100281457|
|Publication number||10302453, 302453, US 7712074 B2, US 7712074B2, US-B2-7712074, US7712074 B2, US7712074B2|
|Original Assignee||Bing Ren|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (10), Non-Patent Citations (8), Referenced by (12), Classifications (8), Legal Events (2)|
|External Links: USPTO, USPTO Assignment, Espacenet|
The present invention pertains to the field of application software interfaces. More specifically the invention pertains to improving automated interactions with user interfaces to software applications.
In normal usage, software applications interact with other systems or users. This interaction can occur through Input/Output (I/O) devices coupled with a hardware device running the software application or as communications from other software applications. The I/O devices provide information from a variety of sources including networking devices, machinery controlled by the software applications, and hardware devices used by users of a system to facilitate data input and output to and from the system. Examples of such hardware devices are keyboards, displays and mice. These hardware devices are for real-time input and output of typed and graphical information with a software application.
There are times when real time user interaction with a software application is not desired. Reasons for this include the error prone nature of user interaction as well as the high cost of providing users to operate software. Thus, it is desirable to automate the operation of software applications in certain environments.
For example, as software applications are developed or troubleshot, changes are made to the software application. After these changes are made, it is frequently desirable to test the software application's functionality using such techniques as regression and functional testing to ensure that, as a result of the changes made, other aspects of the operation of the software application have not been adversely affected. To ensure the repeatability of the results of such testing, it is desirable to automate the process of performing regression and functional testing.
In addition to automating software testing, there are several other uses for removing real time interaction with software applications. For example, when developing a product demonstration for use in trade shows, it is desirable to be able to have a product perform an automated execution of a software application, without requiring user attendance, to provide a canned demonstration of the software application. Another example where automated operation of a software application is desirable is in the testing of systems by remote means. By executing locally stored tests from a remote machine, a remote user can determine potential problems with a local software application. A plethora of other scenarios exist where the automated interaction with a software application is desirable.
Currently, methods for automating the interaction with user interfaces of software applications operate by indicating pixel locations on a screen for an input action to be applied to the software application. First, while a user interacts with the software application, a separate recording application will record screen locations, in pixels, of mouse/key actions made by the user. Examples of such actions are mouse moves, mouse presses, mouse releases, mouse clicks and key actions such as presses or releases. This pixel and action information is stored in files in the system containing the software application. Then, during playback of these files by a replay application, the replay application follows the stored pixel information and applies the appropriate actions at the appropriate pixel location as indicated by the stored values. The underlying assumption in this method is that screen locations of programs/applications during playback will be the same screen locations as they were during the record stage. However, this is not necessarily the case. For example, when an application playback is performed by a different machine than the machine with which the record was performed, which uses a different display resolution, the actual software location on the screen may be at a different pixel location. This will likely cause an error to occur during the playback as an input action is applied to an incorrect window in the software application.
Examples of the present invention are illustrated in the accompanying drawings. The drawings do not, however, limit the scope of the present invention. Similar references in the drawings indicate similar elements.
Methods for improving the automated interactions with a component-based user interface application are described below. In the following description, for purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without those specific details. In other instances, well-known features are omitted or simplified in order not to obscure the present invention. Further, the phrase “in one embodiment” is used repeatedly. In general, the phrase “in one embodiment” does not refer to the same embodiment although it may. The terms “comprising”, “including”, and “having”, as used herein, are synonymous. Finally, while the present invention is explained with respect to a Java virtual machine-based user interface, the present invention is applicable to any component-based user interface application, such as those developed using Visual-Basic and Visual-C++.
The present invention improves automated interactions with a component-based graphical user interface (GUI). In various embodiments, the present invention determines “social relationships” among objects in a GUI. Thus, given a reference object, social relationships to this reference object may be used to identify a target object for a particular automated interaction. As an example, being provided a top level component in a Java-based GUI application as a reference object, and social relationship information relative to this top level component, it is possible to identify a target Java component. Note that in Java, the “object” are “components”. The target Java component is the component to which a desired action is to be performed. Thus, a top level component is provided as a starting point for the identification. Next, socially identifying information which includes information regarding a target object's relationship to the top level component is obtained. The socially identifying information is used to identify uniquely the target component by identifying the target component relative to other components, including possibly the top component itself, in existence in the top level component. When the proper target component is identified, an action is then applied to the target component.
In contrast, as previously alluded to, prior approaches to automated interaction used pixel counts to identify the locations where input actions to a software application are to occur. Pixel-based locations, however, tend to be unreliable over time and from system to system. The present invention is not dependent upon pixel locations. Rather, the present invention identifies target objects from a software application itself and, therefore, does not have the problems associated with the prior pixel-based approaches.
In an exemplary embodiment, user interactions with a GUI are captured and recorded to a file. Each recorded interaction includes socially descriptive information identifying the target component for the respective interaction. An example of recorded user interactions might include a recordation of the login sequence of a user to an NT workstation. In such a case, the present invention would thus record the mouse clicks and keystrokes executed by the user logging in to the workstation. Alternatively, a user may manually create an interactions file to be applied to target components along with the appropriate socially descriptive information to identify the respective target components.
In either case, another part of the invention is used to read the interactions file, determine the appropriate target components for the respective interactions based upon the socially descriptive information, and then apply the interactions to the target components. For example, using the interactions file, the present invention can identify a particular text field in a GUI window, enter text in the text field, then identify a “submit” button in the GUI window next to the text field, and enter a mouse click on the “submit” button, just as if a user had entered the interactions in real time.
Generating interactions files, also known as script files, for use by the methods to automate the interaction with the software applications can be an automated process itself. As previously mentioned there are several ways of obtaining the script files for use in the playback role (e.g. playing back the actions recorded in the script files). A user could, for example, manually develop these script files after becoming familiar with the syntax of the script files. However, as manually writing files can be a laborious, time-consuming process, it is desirable to provide a user of the system with an ability to capture a set of movements, or interactions, and have them recorded directly to a script file. For example, it would be desirable to have a tool automatically generate the sequence of actions to enable a user to perform a login of the system as described subsequently in connection with
Applying Actions to Target Components
In a typical component-based graphical user interface (GUI), components, or objects, may include windows, text fields, action buttons, menus, menu items, hyperlinks, or any number of other interactive objects. These objects are often organized in a hierarchy. For instance, the highest level of the hierarchy may be a single window, referred to as a “desktop” in some operating systems. That window may include one or more interactive objects, such as sub-windows, at the second level of the hierarchy. Some of those objects may include more objects at the third level of the hierarchy, which may include more objects at the fourth level. Objects at each level may include sub-objects at the next lower level. The hierarchy may include many levels. At any instant in time, a user may be able to display only a small part of the entire hierarchy.
As part of the task of determining an object to which an action is to be applied, the scripting files that were created manually or with the aid of a utility, as previously discussed, are read.
In any case, once the immediate parent object has been identified, peer objects can be examined and, using the peer location information, the target object can be determined from its relationship to the peers as identified by the signature information, at 310. For example, a peer object, identifiable by name, may have a particular spatial orientation with respect to the target object. This spatial orientation may be used to identify the target object to the left or right of the peer object by checking the location of each object. In another embodiment, an unnamed target object is identified by having specific information regarding the target object's “rank” among other unnamed objects. In various embodiments, peers are ranked by spatial distance from a parent, by location within a parent based on a predefined pattern, or by any other objective means. Once the target object has been identified by the signature information, an input action is applied to the target object, at 320.
When determining a target component, the use of names to fully identify a target component relative to a top level component may not be possible. There may be missing links in terms of unnamed components in the hierarchical chain. Signature-based identification can also be used to identify these “missing links” in an ancestral chain leading to a particular target object. For instance, the name of the target object may be known, and the name of the top-level grandparent of the target object may be known, but the parent of the target object between the top-level grandparent and the target object may not have a name, or may not have an unique name, or the name may simply be unknown. In any case, the parent object can be uniquely identified by signature information. For instance, the signature could be based on the position of the parent with respect to the grandparent, the position of the parent with respect to one or more of its peers, or any other objective standard. In various embodiments, an entire hierarchical chain leading to a target object can be defined in terms of signature information, name information, or a combination thereof.
If the component cannot be identified from only the parent information, then additional information is checked. In the illustrated embodiment of
If, in this embodiment, the component is still not identifiable, then yet additional information is checked. The next type of information that is checked is peer information, at 635. Peer information is useful to help determine the identity of unnamed components when more than one component of the same type exists within the same parent. As an example, in the present embodiment, when multiple text fields are to be filled out in a window, multiple JTextField components may exist in a parent container. Peer information may include the number of components of the same type and a ranking among the number of components of the target component. The ranking, in this embodiment, describes the relative location from top to bottom and left to right. Thus, a component appearing above another component of the same type will have a higher rank. In this embodiment, a component at the same height but left of another component will also have a higher rank. Such a ranking can be used to distinguish between components. If the component can be identified from the aggregate of information from parent, description and peer information, at 640, then a component match has occurred, and the proper component identification information is returned, at 660. The identified component is then further processed as described subsequently.
Finally in this embodiment, if the component is not identified from the above referenced signature information, then label information is checked, at 645. Frequently, for objects with no name in a user interface, there exists a corresponding label component that will inform a user as to the desired input for the unnamed component. Examples of these label components are text labels describing the desired content of a text field or an identifying label for a radio button. In such a case, the label is likely to be unique for the given context and will have a spatial reference with respect to the desired component. If the component can be identified from the aggregate of information from parent, description, peer and label information, at 650, then a component match has occurred, and the proper component identification information is returned, at 660. The identified component is then further processed as described subsequently. Otherwise, an error is reported that the signature information provided was not sufficient to identify the component, at 655.
The first action 822 is a keyboard input of “root” into the text entry box associated with the label “username”. This text entry box is a target component of reserved Java type JTextField 832, which corresponds to field 1315, in
Once an object has been properly identified, its location on a display device can be identified. Using this identified location, the input can be applied. As mentioned, an advantage of the present method over previous pixel based methods, lies in the identification of the components and then, for the given system, obtaining the position of the appropriate input location is performed through the appropriate interaction with the system. For example, in one embodiment, once a component in a java operating environment has been identified, methods inherent to the component can be used to determine the component's location. Specifically, in this embodiment, the getLocationOnScreen( ) routine can be called to identify a location on a screen. The getLocationOnScreen routine returns to the caller the absolute location where the bounding box for the component is located on the screen. As previously noted, if pixel information is provided as an indicia of the location for applying an input action, when changing between platforms for screen types, the pixel information may likely be an incorrect indicia of the proper input location for the action. In this embodiment, by utilizing the getLocationOnScreen, the coordinates of the desired location are provided independent of the hardware platform or any other variable.
As described above, test cases are utilized for performing actions on components. As seen in the example associated with
Exemplary Operating Environment
These elements perform their conventional functions known in the art. In particular, mass storage 1406 and system memory 1414 are used to store permanent and working copies of the utilities to perform the automated software interaction. The permanent copy may be pre-loaded into mass storage 1406 in a factory, loaded from distribution medium (not shown), or downloaded from a remote distribution source (not shown). Distribution medium may be a tape, a CD, a DVD or other storage medium of the like. The constitutions of these elements are known. Any one of a number of implementations of these elements known in the art may be used to form computer system 1400.
Certain embodiments may include additional components, may not require all of the above components, or may combine one or more components. Those skilled in the art will be familiar with a variety of alternative implementations.
Embodiments of the present invention provide a method of automating interactions with a graphical user interface. Actions are applied to an object in the graphical user interface by identifying the target object through socially identifying information and applying an action to the identified object.
By identifying a component through identifying information and applying the action based upon this information, the actions can be repeatedly and successfully applied at the specific pixel locations of the target object even as the pixel locations change from application to application, and from platform to platform. This allows the playback method to be used on different computers without error in contrast to prior art pixel-based methods that potentially apply input actions outside the pixel location of the target object as the pixel location of the target object changes.
The present invention can be embodied in the form of methods as well as apparatuses for practicing those methods. The present invention can also be embodied in the form of program code embodied in tangible media, such as punched cards, magnetic tape, floppy disks, hard disk drives, CD-ROMs, flash memory cards, or any other machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention. The present invention can also be embodied in the form of program code, for example, whether stored in a storage medium, loaded into and/or executed by a machine, or transmitted over some transmission medium or carrier, such as over electrical wiring or cabling, through fiber optics, or via electromagnetic radiation, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention. When implemented upon a general-purpose processor, the program code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits.
It is to be understood that the present invention is not limited to the embodiments described above, but encompasses any and all embodiments within the scope of the subjoined claims.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US6046740 *||Feb 6, 1998||Apr 4, 2000||Seque Software, Inc.||Application testing with virtual object recognition|
|US6115719 *||Nov 20, 1998||Sep 5, 2000||Revsoft Corporation||Java compatible object oriented component data structure|
|US6698012 *||Sep 15, 2000||Feb 24, 2004||Nortel Networks Limited||Method and system for testing behavior of procedures|
|US6901588 *||Apr 17, 2000||May 31, 2005||Codemesh, Inc.||Sharing components between programming languages by use of polymorphic proxy|
|US7000224 *||Apr 13, 2000||Feb 14, 2006||Empirix Inc.||Test code generator, engine and analyzer for testing middleware applications|
|US7165240 *||Jun 20, 2002||Jan 16, 2007||International Business Machines Corporation||Topological best match naming convention apparatus and method for use in testing graphical user interfaces|
|US7210117 *||Dec 20, 2000||Apr 24, 2007||National Instruments Corporation||System and method for programmatically generating a graphical program in response to program information|
|US7272822 *||Sep 17, 2002||Sep 18, 2007||Cisco Technology, Inc.||Automatically generating software tests based on metadata|
|US20020078255 *||Oct 16, 2001||Jun 20, 2002||Shankar Narayan||Pluggable instantiable distributed objects|
|US20030084429 *||Oct 26, 2001||May 1, 2003||Schaefer James S.||Systems and methods for table driven automation testing of software programs|
|1||*||"An automated oracle for verifying GUI objects", Takahashi, Jul. 2001, pp. 83-88. Online retrieved at .|
|2||*||"Coverage criteria for GUI testing", Memon et al., Sep. 2001, pp. 256-267. Online retrieved at .|
|3||*||"The state of the art in automating usability evaluation of user interfaces", Ivory et al., Dec. 2001, pp. 470-516. Online retrieved at .|
|4||*||"User-based testing of GUI sequences and their interactions", White et al., Nov. 2001, pp. 54-63. Online retrieved at .|
|5||*||"An automated oracle for verifying GUI objects", Takahashi, Jul. 2001, pp. 83-88. Online retrieved at <http://delivery.acm.org/10.1145/510000/505494/p83-takahashi.pdf>.|
|6||*||"Coverage criteria for GUI testing", Memon et al., Sep. 2001, pp. 256-267. Online retrieved at <http://delivery.acm.org/10.1145/510000/503244/p256-memon.pdf>.|
|7||*||"The state of the art in automating usability evaluation of user interfaces", Ivory et al., Dec. 2001, pp. 470-516. Online retrieved at <http://delivery.acm.org/10.1145/510000/503114/p470-ivory.pdf>.|
|8||*||"User-based testing of GUI sequences and their interactions", White et al., Nov. 2001, pp. 54-63. Online retrieved at <http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=989458&isnumber=21326>.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US8046754 *||Jul 31, 2007||Oct 25, 2011||International Business Machines Corporation||Methods and apparatus for customizing user-interface control in existing application|
|US8307357 *||Mar 28, 2008||Nov 6, 2012||International Business Machines Corporation||Methods and apparatus for customizing user-interface control in existing application|
|US8402434||Oct 21, 2008||Mar 19, 2013||International Business Machines Corporation||Graphical user interface (GUI) script generation and documentation|
|US8667467 *||Jul 26, 2010||Mar 4, 2014||Sap Aktiengesellschaft||Dynamic test scripts|
|US8793578||Jul 11, 2011||Jul 29, 2014||International Business Machines Corporation||Automating execution of arbitrary graphical interface applications|
|US20070271549 *||Jul 31, 2007||Nov 22, 2007||International Business Machines Corporation||Methods and Apparatus for Customizing User-Interface Control in Existing Application|
|US20080178083 *||Mar 28, 2008||Jul 24, 2008||International Business Corporation||Methods and Apparatus for Customizing User-Interface Control in Existing Application|
|US20090044110 *||Oct 21, 2008||Feb 12, 2009||International Business Machines Corporation||Graphical User Interface (GUI) Script Generation and Documentation|
|US20110214107 *||Mar 1, 2010||Sep 1, 2011||Experitest, Ltd.||Method and system for testing graphical user interfaces|
|US20120023485 *||Jul 26, 2010||Jan 26, 2012||Sap Ag||Dynamic Test Scripts|
|US20120124495 *||Nov 23, 2009||May 17, 2012||Nitsan Amichai||System and method for object relationship identification in a user interface|
|US20150067498 *||Aug 11, 2014||Mar 5, 2015||Fujitsu Limited||Scenario generation method, scenario execution method, scenario generation device, and scenario execution device|
|U.S. Classification||717/105, 715/700, 717/125|
|International Classification||G06F9/44, G09G5/00, G06F3/00|
|Nov 21, 2002||AS||Assignment|
Owner name: MENTOR GRAPHICS CORPORATION, OREGON
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:REN, BING;REEL/FRAME:013519/0559
Effective date: 20021119
Owner name: MENTOR GRAPHICS CORPORATION,OREGON
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:REN, BING;REEL/FRAME:013519/0559
Effective date: 20021119
|Oct 11, 2013||FPAY||Fee payment|
Year of fee payment: 4