Mercury TestDirector - Evaluation

"The organization has meticulously tracked Test Requirements and Test Cases using spreadsheets but is finding this to be a cumbersome process as the test organization grows. It has been shown that this process has reduced the number of defects reaching the field but the cost of maintaining the approach is now impacting its effectiveness. Solution - invest in a test management tool or suite of tools."

Evaluation

The first thing an organization must accomplish is to catalogue what needs or requirements the software is expected satisfy. There are three categories or "points-of-view" that were addressed during the evaluation process: Management / Organization, Test Architecture, and End-User.

Needs Analysis: Management / Organization Perspective

Management clearly stated the objective for purchasing the Test Management software was: "The selected Test Management system shall enable end-users to author and maintain requirements and test cases in a web-enabled, shareable environment. Furthermore the test management tool shall support test management "best practices" as defined by the Test Organization. Minimum acceptable ROI is 4 hours saved for every hour currently invested."

Findings: Management / Organization Perspective

General: Mercury TestDirector enabled the end-users to organize, author, and maintain a hierarchy of Requirements, Test Cases, and defects in a web-enabled, shareable environment. The solution supported both the functional decomposition of the application under test (AUT) and the System / Business decomposition of the AUT. In both cases the solution supplied an effective non-technical interface for the business analyst and testers to author, update, report on, and maintain test artifacts. Requirements: Mercury TestDirector supplied an effective non-technical interface for the business analyst to author, update, report on, and maintain requirements. There does seem to be an intrinsic limitation to the "bill-of-material" approach to requirement captures and organization - it is two-dimensional. This limitation would not impact most organizations but if a complex relationship exists between the requirements or other artifacts then Mercury TestDirector would not easily meet the need (i.e. multi-dimensional relationships). Test Cases: Mercury TestDirector once again supplied an effective non-technical interface for the business to author, update, report on, and maintain Test Cases. The standard infrastructure supplied by Mercury TestDirector. does not enforce test case design and test case organization best practices but it does support them. Test Case design is basically a free-form text exercise with a very "thin" organizational overlay in the form of Steps - any standards or design discipline would have to come from the Test Designer. The newest Mercury TestDirector. enhancement Mercury Business Process Testing does seem to go a long way to address this but it was not part of the initial evaluation. Test Execution: Mercury TestDirector once again supplied a non-technical interface for the business to author, update, report on, and maintain Test Sets. The standard infrastructure supplied by Mercury TestDirector. does not enforce test set management and test set organization best practices but it does support them. Mercury TestDirector Test Lab does allow for integration with several test automation tools and supplies an open API to allow the purchaser the ability to integrate with almost any test automation tool. Test Lab, in our opinion, was the weakest link in the tool - management and maintenance of the Test Sets is basically a free-form folder approach. Defect Tracking: Mercury TestDirector once again supplied a non-technical interface for the business to author, update, report on, and maintain Defects. The standard infrastructure supplied by Mercury TestDirector does not enforce Defect management and Defect organization best practices but it does support them. Mercury TestDirector does supply an adequate Defect Tracking tool for most organizations - once again the approach is somewhat two-dimensional and might be found wanting if the test organization wanted to maintain more complex relationships between Defects or other test artifacts. Summary: From a Management / Organization Perspective Mercury TestDirector lives up to its reputation as Tier 1 Test Management tool. There are architectural choices that were made that enhance usability at the expense of functionality but Mercury TestDirector will meet the needs of most Testing Organizations.

Needs Analysis: Test Architecture

An Architectural framework has not been defined by the Test Organization therefore a general set of Architectural guidelines was applied during the evaluation. The Test Management application shall:

1. Have a record of integrating successfully with all Tier 1 testing software vendors.
2. Have a history of operational success in the appropriate environments.
3. Have an established end-user community that is accessible to any end-user.
4. Support enterprise wide collaboration.
5. Support customization.
6. Support several (1 to n) simultaneous engagements / projects.
7. Provide a well-designed, friendly, and intuitive user interface.
8. Provide a smooth migration / upgrade path from one iteration of the product to the next.
9. Provide a rich online-help facility and effective training mechanisms (tutorials, courseware, etc.).

Findings: Test Architecture

Have a record of integrating successfully with all Tier 1 testing software vendors: Mercury TestDirector provides an open, well-defined API that currently supports integration with several tool-sets. Other vendors supply (usually at no cost) integration with Mercury TestDirector. We must give Mercury the highest marks here - if you cannot integrate another providers tool with Mercury TestDirector it will not be the fault of Mercury TestDirector. Have a history of operational success in the appropriate environments: Mercury TestDirector has a long history of being a fully functional web-enabled application that will operate in any windows environment. It should be noted that Mercury TestDirector functionality within a Unix based environment was not part of this evaluation and there are known issues with Mercury TestDirector within the context of a Unix environment. Have an established end-user community that is accessible to any end-user: Mercury TestDirector has a large, established, and supportive user community that is fully supported by Mercury Interactive. During our evaluation we found that all our preliminary questions could be answered through the knowledge base supplied by this community and even more surprising the answers were not always in favor of the tool - highest marks must be given to Mercury Interactive for supporting a free and open forum on their solution. Support enterprise wide collaboration: Mercury TestDirector is a fully functional web-enabled application that supports a concurrent user license model, which fully supports enterprise collaboration. Support customization: Mercury TestDirector supports customization of the application display elements and the data base model through an intuitive interface. Mercury TestDirector supports customization / integration with other tools and applications through its fully published API. During the evaluation it was almost "too easy" to customize the interface and available data elements - perhaps a tutorial on cost benefit analysis should be included in the tool set. .What is the on-going maintenance cost of each additional field or element? Support several (1 to n) simultaneous engagements / projects: Mercury TestDirector can support several simultaneous engagements / projects by allowing the user to create a separate database instance for each project. The issue with this approach is that it makes the re-integration of several projects back into a common baseline a manually intensive process. Provide a well-designed, friendly, and intuitive user interface: Mercury TestDirector hierarchical tree or "bill-of-materials" interface provides an extremely friendly interface. During the evaluation novice users became familiar with, and comfortable using, the solution in one or two days. The only complaint was that there is no .undo. key or undo option. Provide a smooth migration / upgrade path from one iteration of the product to the next: The evaluation did not provide an opportunity to validate the migration / upgrade path from one iteration of Mercury TestDirector to the next. The knowledge base found at the Mercury Interactive web site does indicate that there were issues in the past (over 2 years ago) but recent upgrades seem to have proceeded with little difficulty. Provide a rich online-help facility and effective training mechanisms: Mercury TestDirector does provide a rich online-help facility and an adequate tutorial. During the evaluation we found most users did not use or require the online-help facility for their day-to-day task due to the intuitive nature of the interface.

Needs Analysis: End-User

The End-User needs analysis should be a detailed catalogue of product requirements. These requirements were evaluated on a simple pass / fail criteria. The Test Management solution shall:

1. Support the authoring of Test Requirements.
2. Support the maintenance of Test Requirements.
3. Support enterprise wide controlled access to Test Requirements (Web enabled preferred).
4. Support discrete grouping or partitioning of Test Requirements.
5. Support Traceability of requirements to Test Cases and Defects.
6. Support "canned" and "user defined" queries against Test Requirements.
7. Support "canned" and "user defined" reports against Test Requirements.
8. Support coverage analysis of Test Requirements against Test Cases.
9. Support the integration of other toolsets via a published API or equivalent capacity.
10. Support the creation of Defects.
11. Support the maintenance of Defects.
12. Support the tracking of Defects.
13. Support enterprise wide controlled access to Defects (Web enabled preferred).
14. Support integration with all Tier 1 and 2 Test Management tools that support integration.
15. Enable structured and ad-hoc searches for existing Defects.
16. Enable the categorization of Defects.
17. Enable customization of Defect content.
18. Support "canned" and customized reports.

Findings: End-User

Mercury TestDirector certainly met or even exceeded the user community's expectations during the initial evaluation and eventual implementation. The one functional area that the users believed needed improvement was in the area of Reporting. Once the user community became accustomed to the application they found that the standard reports and the custom reporting capabilities did not meet their expectations.

Evaluation Summary

Mercury TestDirector is a Tier 1 Test Management solution that will meet the needs of most testing organizations. The strength of the application lies in its ability to allow novice users to quickly become proficient in its use and in its ability to quickly convey information. The weakness of the application lies in its somewhat simplistic approach to several aspects of the Test Management space. It should be noted that most organizations will find Mercury TestDirector. a suitable solution for all their Test Management needs only organizations with the most complex testing requirements will find issues with Mercury TestDirector - most of these issues can be addressed by using the Mercury TestDirector open API to integrate the appropriate toolset or application extension.