Test Tools & Aids: Reviews, Test Management, Test Automation, & Defect Tracking

There are several classes of testing tools available today that make the testing process easier, more effective, and more productive. When properly implemented these tools can provide a test organization with substantial gains in testing efficiency. Test tools need to fit into the overall testing architecture and should be viewed as process enablers - not as the "answer". Test organizations will often look to tools to facilitate: Reviews, Test Management, Test Design, Test Automation, and Defect Tracking. It is quite common for a testing tool or family of tools to address one or more of these needs but for convenience they will be addressed from a functional perspective not a "package" perspective.
It is important to note that, as with any tool, improper implementation or ad-hoc implementation of a testing tool can lead to a negative impact on the testing organization. Ensure a rigorous selection process is adhered to when selecting any tool including: needs analysis, on-site evaluation, and an assessment of return on investment (ROI).

Reviews

Reviews and technical inspections are the most cost effective to way detect and eliminate defects in any project. This is also one of the most underutilized testing techniques; consequently there are very few tools available to meet this need. Any test organization that is beginning to realize the benefits of Reviews and Inspections but is encountering scheduling issues between participants should be looking to an on-line collaboration tool. There are several on-line collaboration tools available for the review and update of documents but only one (that I'm aware of) that actually addresses the science and discipline of Reviews - ReviewPro^TM. by Software Development Technologies. I normally do not "plug" a particular tool but when the landscape is sparse I believe some recognition is in order.

Test Management

Test Management encompasses a broad range of activities and deliverables. The Test Management aid or set of aids selected by an organization should integrate smoothly with any communication (i.e. e-mail, network, etc.) and automation tools that will be applied during the testing effort. Generic management tools will often address the management of resources, schedules, and testing milestones but not the activities and deliverables specific to the testing effort: Test Requirements, Test Cases, Test Results, and Analysis. For more on Test Management see Testing & The Role of a Test Lead / Manager.
Test Requirements
Requirements or Test Requirements often become the responsibility of the testing organization. In order for any set of requirements to be useful to the testing organization they must be: maintainable, testable, consistent, and traceable to the appropriate test cases. The Requirements Management tool must be able to fulfill these needs within the context of the testing teams operational environment. For more on requirements see Testing & The Role of a Test Designer / Tester.
Test Cases
The testing organization is responsible for authoring and maintaining Test Cases. A Test Case authoring and management tool should enable the test organization to: catalogue, author, maintain, manually execute, and auto-execute automated tests. These Test Cases need to be traceable to the appropriate requirements and the results recorded in such a manner as to support coverage analysis. The key integration requirements when looking to Test Case management tools are: Does it integrate with the Test Requirement Aide (?), Does it integrate with the Test Automation tools being used (?), and will it support coverage analysis (?). For more on test cases see Test Deliverables: Test Plan, Test Case, Defect-Fault, and Status Report.
Test Results & Analysis
A Test Management suite of tools and aids needs to be able to report on several aspects of the testing effort. There is an immediate requirement for test case results - Which test case steps passed and which failed? There will be periodic status reports that will address several aspects of the testing effort: Test Cases executed / not executed, Test Cases passed / failed, Requirements tested / not tested, Requirements verified / not verified, and Coverage Analysis. The reporting mechanism should also support the creation of custom or ad-hoc reports that are required by the Test Organization.

Test Automation

Several Test Automation frameworks have been implemented over the years by commercial vendors and testing: Record & Playback, Extended Record & Playback, and Load / Performance. For more on Test Automation see Testing & The Role of a Test Automation Engineer.
Record & Playback
Record and Playback frameworks were the first commercially successful testing solutions. The tool simply records a series of steps or actions against the application and allows a playback of the recording to verify that the behavior of the application has not changed.
Extended Record & Playback
It became quickly apparent that a simple Record and Playback paradigm was not very effective and did not make test automation available to non-technical users. Vendors extended the Record & Playback framework to make the solution more robust and transparent. These extensions included: Data Driven, Keyword, and Component Based solutions.
Load / Performance
Load / Performance test frameworks provide a mechanism to simulate transactions against the application being tested and to measure the behavior of the application while it is under this simulated load. The Load / Performance tool enable the tester to: load, measure, and control the application.

Defect Tracking

The primary purpose of testing is to detect defects in the application before it is released into production; furthermore defects are arguably the only product the testing team produces that is seen by the project team. The Defect Management tools must enable the test organization to: author, track, maintain, trace to Test Cases, and trace to Test Requirements any defects found during the testing effort. The Defect Management tool also needs to support both scheduled and ad-hoc analysis and reporting on defects. For more information on defects see Testing & The Role of a Test Designer / Tester and Test Deliverables: Test Plan, Test Case, Defect-Fault, and Status Report.
Schedule
The test schedule defines when and by whom testing activities will be performed. The information gathered for the body of the Test Plan is used here in combination with the available resource pool to determine the test schedule. Experience from previous testing efforts along with a detailed understanding of the current testing goals will help make the test schedule as accurate as possible. There are several planning / scheduling tools available that make the plan easier to construct and maintain.

Test Case

Test Cases are the formal implementation of a test case design. The goal of any given test case or set of test cases is to detect defects in the system being tested. A Test Case should be documented in a manner that is useful for the current test cycle and any future test cycles - at a bare minimum each test case should contain: Author, Name, Description, Step, Expected Results, and Status (see - "Testing and The Role of a Test Designer Tester").
Test Case Name
The name or title should contain the essence of the test case including the functional area and purpose of the test. Using a common naming convention that groups test cases encourages reuse and help prevents duplicate test cases from occurring.
Test Case Description
The description should clearly state what sequence of business events to be exercised by the test case. The Test Case description can apply to one or more test cases; it will often take more than one test case to fully test an area of the application.
Test Case Step
Each test case step should clearly state the navigation, data, and events required to accomplish the step. Using a common descriptive approach encourages conformity and reuse. Keywords offer on of the most effective approaches to Test Case design and can be applied to both manual and automated test cases (see - "Keyword based Test Automaton").
Expected Results
The expected behavior of the system after any test case step that requires verification / validation - this could include: screen pop-ups, data updates, display changes, or any other discernable event or transaction on the system that is expected to occur when the test case step is executed.
Status
The operational status of the test case - Is it ready to be executed?

Defect-Fault

The primary purpose of testing is to detect defects in the application before it is released into production; furthermore defects are arguably the only product the testing team produces that is seen by the project team. Document defects in a manner that is useful in the defect remediation process - at a bare minimum each defect should contain: Author, Name, Description, Severity, Impacted Area, and Status (see - "Testing and The Role of a Test Designer Tester").
Defect Name
The name or title should contain the essence of the defect including the functional area and nature of the defect.
Defect Description
The description should clearly state what sequence of events leads to the defect and when possible a screen snapshot or printout of the error.
How to replicate
The defect description should provide sufficient detail for the triage team and the developer fixing the defect to duplicate the defect.
Defect severity
The severity assigned to a defect is dependent on: phase of testing, impact of the defect on the testing effort, and the Risk the defect would present to the business if the defect was rolled-out into production.
Impacted area
The Impacted area can be referenced by functional component or functional area of the system - often both are used.

Status Report

A test organization and members of the testing team will be called upon to create Status Reports on a daily, weekly, monthly, and project bases. The content of any status report should remain focused on the testing objective, scope, and schedule milestones currently being addressed. It is useful to state each of these at the beginning of each status report and then publish the achievements or goals accomplished during the current reporting period and those that will be accomplished during the next reporting period. Any known risks that will directly impact the testing effort need to be itemized here, especially any "showstoppers" that will prevent any further testing of one or more aspects of the system.
Reporting Period
The period covered in the current status report with references to any previous status reports that should be reviewed.
Mission Statement
The objective of the current testing effort needs to be clearly stated and understood by the testing team and any other organization involved in the deployment.
Current Scope
The components of the system being tested (hardware, software, middleware, etc.) need to be clearly defined as being "In Scope" and any related components that are not being tested need to be clearly itemized as "Out of Scope".
Schedule Milestones
Any schedule milestones being worked on during the current reporting period need to be listed and their current status clearly stated. Milestones that were scheduled but not addressed during the current reporting period need to be raised as Risks.
Risks
Risks are factors that could negatively impact the current testing effort. An itemized list of risks that are currently impacting the testing effort should be drawn up and their impact on the testing effort described.