Evaluating Testing Software & Tools

Once a testing organization reaches a certain size, level of maturity, or workload the requirement to purchase / build testing software or aides becomes apparent. There are several classes of testing tools available today that make the testing process easier, more effective, and more productive. Choosing the appropriate tool to meet the testing organization's long-term and short-term goals can be a challenging and frustrating process. Following a few simple guidelines and applying a common-sense approach to software acquisition and implementation will lead to a successful implementation of the appropriate tool and a real return on investment (ROI).
One of the simplest questions to ask when looking at testing software is "What is ROI?" The simplest answer is "Anything that reduces the hours required to accomplish any given task". Testing tools should be brought into an organization to improve the efficiency of a proven testing process - the value of the actual process has already been established within the organization or within the industry.
Example: Test Management
The organization has meticulously tracked Test Requirements and Test Cases using spreadsheets but is finding this to be a cumbersome process as the test organization grows. It has been shown that this process has reduced the number of defects reaching the field but the cost of maintaining the approach is now impacting its effectiveness. Solution - invest in a test management tool or suite of tools.
Example: Test Automation
The organization has created a suite of manual test cases using a text editor but is finding it difficult to maintain, use, and execute these test cases efficiently as the test organization's role grows. The test cases have proven effective in detecting defects before they reach production but the time required to manage and execute these test cases is now impacting the return on investment. Solution - invest in a test automation tool or suite of tools.
Example: Defect Management
The test organization has implemented a defect tracking process using e-mail and a relational database but is now finding that defects are being duplicated and mishandled as the volume of defects grows. Solution - upgrade the current in-house solution or invest in a defect management tool.

Needs Analysis

The first thing an organization must accomplish is to catalogue what needs or requirements the Testing Software is expected satisfy. For an organization that is new to the acquisition process this can be a rather intimidating exercise. There are three categories or "points-of-view" that must be addressed: Management / Organization, Test Architecture, and End-User.
Needs Analysis: Management / Organization
Management or the test organization needs to clearly state what the objective is for purchasing testing software. The mission or goal that will be met by acquiring the test software and the expected ROI in terms of person-hours once the tool has been fully implemented. This can be accomplished by creating a simple mission statement and a minimum acceptable ROI. It should be noted that any ROI of less than 3 (hours) to 1 (current hour) should be considered insufficient because of the impact of introducing a new business process into the testing organization. This should be a concise statement on the overall goal (1 to 3 sentences) not a dissertation or catalogue of the products capabilities.
Example: Test Management
The selected Test Management system shall enable end-users to author and maintain requirements and test cases in a web-enabled, shareable environment. Furthermore the test management tool shall support test management .best practices. as defined by the Test Organization. Minimum acceptable ROI is 4 hours saved for every hour currently invested.
Example: Test Automation
The selected Test Automaton tool shall enable end-users to author, maintain, and execute automated test cases in a web-enabled, shareable environment. Furthermore the test automation tool shall support test case design, automation, and execution "best practices" as defined by the Test Organization. Minimum acceptable ROI is 5 hours saved for every hour currently invested.
Example: Defect Management
The selected Defect Management tool shall enable end-users to author, maintain, and track/ search defects in a web-enabled, e-mail-enabled, shareable environment. Furthermore the defect management tool shall support authoring, reporting, and tracking .best practices. as defined by the Test Organization. Minimum acceptable ROI is 4 hours saved for every hour currently invested.

Needs Analysis: Test Architecture

Management has defined the immediate organizational goal but the long-term architectural necessities must be defined by the testing organization. When first approaching the acquisition of testing software test organizations have usually not invested much effort in defining an overall test architecture. Lack of an overall Test Architecture can lead to product choices that may be effective in the short-term but lead to additional long-term costs or even replacement of a previously selected toolset. If an Architectural framework has been defined then the Architectural needs should already be clearly understood and documented - if not then a general set of Architectural guidelines can be applied. The selected Testing Software and tool vendor shall:

1. Have a record of integrating successfully with all Tier 1 testing software vendors.
2. Have a history of operational success in the appropriate environments.
3. Have an established end-user community that is accessible to any end-user.
4. Support enterprise wide collaboration.
5. Support customization.
6. Support several (1 to n) simultaneous engagements / projects.
7. Provide a well-designed, friendly, and intuitive user interface.
8. Provide a smooth migration / upgrade path from one iteration of the product to the next.
9. Provide a rich online-help facility and effective training mechanisms (tutorials, courseware, etc.).

The general architectural requirements for any tool will include more objectives than the nine listed above but it is important to note that any objective should be applied across the entire toolset.

Needs Analysis: End-User

The End-User needs analysis should be detailed dissertation or catalogue of the envisioned product capabilities as they apply to the testing process - probably a page or more of requirements itemized or tabulated in such a way as to expedite the selection process. This is where the specific and perhaps unique product capabilities are listed. The most effective approach is to start from a set of general requirements and then extend into a catalogue of more specific/related requirements.
Example: Test Management
The Test Management solution shall:

1. Support the authoring of Test Requirements.
2. Support the maintenance of Test Requirements.
3. Support enterprise wide controlled access to Test Requirements (Web enabled preferred).
4. Support discrete grouping or partitioning of Test Requirements.
5. Support Traceability of requirements to Test Cases and Defects.
6. Support .canned. and .user defined. queries against Test Requirements.
7. Support .canned. and .user defined. reports against Test Requirements.
8. Support coverage analysis of Test Requirements against Test Cases.
9. Support the integration of other toolsets via a published API or equivalent capacity.
10. And so on.

The key here is to itemize the requirements to a sufficient level of detail and then apply these requirements against each candidate.
Example: Test Automation
The Test Automation solution shall:

1. Support the creation, implementation, and execution of Automated Test Cases.
2. Support enterprise wide, controlled access to Test Automation (Web enabled preferred).
3. Support Data Driven Automated Test Cases.
4. Support Keyword enabled Test Automation.
5. Integrate with all Tier 1 and 2 Test Management tools that support integration.
6. Integrate with all Tier 1 and 2 Defect Management tools that support integration.
7. Enable Test Case Design within a non-technical framework.
8. Enable Test Automation and verification of Web, GUI, .NET, and Java applications.
9. Support the integration of other toolsets via a published API or equivalent capacity.
10. And so on.

Once again the key is to itemize the requirements to a sufficient level of detail. It is not necessary that all the requirements are "realistic" in terms of what is available - looking to the future can often lead to choosing the tool that eventually does provide the desired ability.
Example: Defect Management
The Defect Management solution shall:

1. Support the creation of Defects.
2. Support the maintenance of Defects.
3. Support the tracking of Defects.
4. Support enterprise wide controlled access to Defects (Web enabled preferred).
5. Support integration with all Tier 1 and 2 Test Management tools that support integration.
6. Enable structured and ad-hoc searches for existing Defects.
7. Enable the categorization of Defects.
8. Enable customization of Defect content.
9. Support "canned" and customized reports.
10. And so on.

In all cases understanding of the basic needs will change as you proceed through the process of defining and selecting appropriate Testing Software. In all cases ensure that a particular vendor is not re-defining the initial goal but becoming an educated consumer in any given product space will lead to a redefinition of the basic requirements that should be recognized and documented.

Identify Candidates

Identifying a list of potential software candidates can be accomplished by investigating several obvious sources: Generic Web Search, Quality Assurance and Testing On-line forums, QA and Testing e-magazines, and co-workers. Once a list of potential software candidates has been created an assessment of currently available reviews can be done - with an eye for obvious marketing ploys. It is also important to note which products command the largest portion of the existing market and which product has the fastest growth rate - this relates to the availability of skilled end-users and end-user communities. Review the gathered materials against the needs analysis and create a short list (3 to 5) of candidates for assessment.

Assess Candidates

If you have been very careful and lucky your first encounter with the Vendors Sales force will occur at this time. This can be a frustrating experience if you are purchasing a relatively small number of licenses or an intimidating one if you are going to be placing an order for a large number of licenses. Being vague as to the eventual number of licenses can put you in the comfortable middle ground.
Assessments of any Testing Software should be accomplished onsite with a full demo version of the software. When installing any new Testing Software: install on a typical end-user system, check for .dll. file conflicts, check for registry entry issues, check for file conflicts, and ensure the software is operational. Record any issues discovered during the initial installation and seek clarification and resolution from the vendor.
Once the Testing Software has been installed assess the software against the previous needs analysis - first performing any available online tutorials and then applying the software to your real-world situation. Record any issues discovered during the assessment process and seek clarification and resolution from the vendor. Any additional needs discovered during an assessment should be recorded and applied to all candidates.
The assessment process itself will lead to the assessment team gaining skills in the product space. It is always wise to do one final pass of all candidates once the initial assessment is completed. Each software candidate can now be graded against the needs/ requirements and a final selection made.

Implementation

Implementation is obviously not part of the selection process but is a common point of failure. Test organizations will often invest in testing software but not in the wherewithal to successfully use it. Investing hundreds of thousands of dollars in software but not investing capital in onsite training and consulting expertise is a recipe for disaster. The software vendor should supply a minimum level of training for any large purchase and be able to supply or recommend onsite consultants / trainers that will ensure the test organization can take full advantage of the purchased software as quickly as possible. By bringing in the right mix of training, consulting, and vendor expertise the test organization can avoid much of the disruption any change in process brings and quickly gain the benefits that software can provide.