Mercury WinRunner Evaluation

"The organization has created a suite of manual test cases using a text editor but is finding it difficult to maintain, use, and execute these test cases efficiently as the test organization's role grows. The test cases have proven effective in detecting defects before they reach production but the time required to manage and execute these test cases is now impacting the return on investment. Solution - invest in a test automation tool or suite of tools."

Evaluation

The first thing an organization must accomplish is to catalogue what needs or requirements the Testing Software is expected satisfy. There are three categories or "points-of-view" that must be addressed: Management / Organization, Test Architecture, and End-User.

Management / Organization Perspective

Needs Analysis
Management clearly stated the objective for purchasing the Test Automation solution was:
"The selected Test Automaton tool shall enable end-users to author, maintain, and execute automated test cases in a web-enabled, shareable environment. Furthermore the test automation tool shall support test case design, automation, and execution .best practices. as defined by the Test Organization. Minimum acceptable ROI is 5 hours saved for every hour currently invested."
Findings
General: Mercury WinRunner enabled the end-users to organize, author, and maintain a set of automated test cases and supporting software libraries in a shareable environment. Mercury WinRunner did support test case design, automation, and execution "best practices" as defined by the Test Organization but the framework to perform these tasks was certainly not "out-of-the-box". Mercury WinRunner provides the toolbox but it is the responsibility of the users to build and support the testing framework or purchase a testing framework that is compatible with Mercury WinRunner.
Development: Mercury WinRunner provides a software development environment designed to meet the needs of Test Automation Engineers. The programming language, Test Script Language (TSL), has the look and feel of a simplified version of C. Experienced Automation Engineers or developers should have little difficulty in becoming proficient TSL developers. The actual organization of the TSL code is left up to the Engineers - the test organization must take responsibility for organizing and maintaining this code base.
Mapping: Mercury WinRunner provides an effective GUI mapping utility that enables a skilled test automation engineer to map and maintain information on the application interface.
Wizards: Mercury WinRunner is a fully mature Tier 1 test automation development tool. It comes with a large catalogue of automation wizards - most of these wizards are aimed at the play and record test automation paradigm.
Maintenance: Mercury WinRunner is a test automation development studio, which allows for both development and maintenance of TSL code. Management and control of the software is left to the testing organization - therefore maintenance can become burdensome if the testing organization does not implement adequate configuration management practices.
Summary: Mercury WinRunner met the needs of the testing organization but additional investment in a Testing Framework and a clear development paradigm must be implemented with the automation tool. If the testing organization does not implement a clear development paradigm and testing framework any value derived from test automation will be lost to the long-term maintenance burden.

Test Architecture

Needs Analysis
Management has defined the immediate organizational goal but the long-term architectural necessities must be defined by the testing organization. The Testing Organization clearly stated the Architectural requirements for the Test Automation tool were:
"The selected Test Automation tool shall have a history of operational success in the appropriate environments with a well established end-user community to draw upon. The tool must support enterprise wide collaboration over several simultaneous engagements / projects and smooth software upgrade path. The minimum acceptable ROI of 5 hours saved for every hour currently invested must be maintained across the enterprise and during any integration or upgrade activities."
Findings
Summary: Mercury WinRunner enabled the end-users to organize, author, and maintain a set of automated test scripts across the enterprise. WinRunner is used and supported by large geographically dispersed user community - Mercury supplies an open forum where this community can share solutions. Integration is available for configuration management tools, test management tools, requirement management tools, and commercial testing frameworks - including Mercury's own Mercury Quality Center. Mercury's WinRunner is a well established toolset which several vendors from inside and outside the testing tool space has chosen to integrate with.

End-User "Automation Engineer"

Needs Analysis
The End-User needs analysis detailed product capabilities as they apply to the testing process - this list of requirements extended for several pages and included several test automation challenges. In brief the End-User's stated that the Test Automation solution shall:

1. Support the creation, implementation, and execution of Automated Test Cases.
2. Support enterprise wide, controlled access to Test Automation (Web enabled preferred).
3. Support Data Driven Automated Test Cases.
4. Support Keyword enabled Test Automation.
5. Enable Test Automation and verification of Web, GUI, .NET, and Java applications.
6. Support the integration of other toolsets via a published API or equivalent capacity.

Findings
Summary: The Test Automation Engineers found that WinRunner supplied the basic framework required to meet all the organization's automation needs. An enterprise wide test automation solution was implemented once WinRunner was combined with a disciplined approach to Test Case Design and software control.
Lessons Learned
From their automation work with WinRunner the Test Automation engineers learned three (3) valuable lessons:
1. WinRunner is very dependent on the accuracy and robustness of the GUI map. The most effective way to construct and test a GUI map was to map against the most recent version and then test the map against earlier versions of the application. This helped the engineers identify object characteristics that would remain stable between releases - the engineers then used these object characteristics to map all objects of the same type. These mapping characteristics could be captured in the general startup script for all engineers.
2. WinRunner comes with the capacity to execute a standard startup script when it is invoked. This script could be used to standardize several operational aspects of WinRunner: GUI map configuration, standard functions, and shared libraries or scripts. This startup script could also be manipulated to allow each engineer to load shared libraries, GUI maps, and functions and to load their own versions of the same elements. In other words the shared libraries were loaded first and then (when desired) the engineer could load their versions of these libraries or functions allowing them to develop against a common code base without impacting other engineers.
3. WinRunner allows a group of engineers to work together by creating a common GUI map and script libraries. This enables the creation and maintenance of a toolkit that can be used and enhanced by the entire team. Once the automation solution has matured this shared resource can be managed using most configuration management tools - protecting the automation investment.

Evaluation Summary

Mercury WinRunner is Tier 1 Test Automation solution that met the need of our Testing Organization. Once the organization had moved into the Test Automation space the architectural limitations of WinRunner. were recognized and compensated for by applying a supporting test framework. It should be noted that there are several competing testing frameworks that integrate smoothly with WinRunner. It should also be noted that there are several Test Automation tools - including Mercury's own QuickTestPro (QTP) - that provide a more comprehensive testing framework. The testing organization was able to maintain a return-on-investment of eight hours saved for every hour invested for each test cycle.