Testing One must know

What is the relationship between product maturity and testing?

Acceptance	- Product is ready for deployment.
System	- Product is ready to be tested as an integrated whole or system.
Function	- Functional testing can be performed against delivered components.
Unit	- Developer can test code as an un-integrated unit.
Design Review	- Product concept can be captured and reviewed.
% Construction	- How much more construction is required to complete the product.
% Product	- How much of the product has been constructed.

How can the Testing Organization help prevent defects from occurring?

There are really two sides to testing Verification and Validation . unfortunately the meaning of these terms has been defined differently by several governing / regulatory bodies. To put it more succinctly there is testing that can be performed before the product is constructed / built and there are types of testing that can be performed after the product has been constructed / built.
Preventing defects from occurring involves testing before the product is constructed / built. There are several methods for accomplishing this goal. The most powerful and cost effective being Reviews. Reviews can be either formal / technical reviews or peer reviews. Formal product development life cycles will provide the testing team with useful materials / deliverables for the review process. When properly implemented any effective development paradigm should supply these deliverables. For example:

• Cascade
  • Requirements
  • Functional Specifications
• Agile or Extreme
  • High level Requirements
  • Storyboards

Testing needs to be involved in this Review process and any defects need to be recorded and managed.

How and when can the Testing Organization detect software defects?

The Testing Organization can detect software defects after the product or some operational segment of it has been delivered. The type of testing to be performed depends on the maturity of the product at the time. The classic hierarchy or sequence of testing is:

• Design Review
• Unit Testing
• Function Testing
• System Testing
• User Acceptance Testing

The Testing Team should be involved in at least three of these phases: Design Review, Function Testing, and System Testing.
Functional Testing involves the design, implementation, and execution of test cases against the functional specification and / or functional requirements for the product. This is where the testing team measures the functional implementation against the product intent using well-formulated test cases and notes any discrepancies as defects (faults). For example testing to ensure the web page allows the entry of a new forum member . in this case we are testing to ensure the web page functions as an interface.
System Testing follows much the same course (Design, Implement, execute and defect) but the intent or focus is very different. While Functional Testing focuses on discrete functional requirements System Testing focuses on the flow through the system and the connectivity between related systems. For example testing to ensure the application allows the entry, activation, and recovery of a new forum member . in this case we are testing to ensure the system supports the business. There are several types of System Testing, what is required for any given release should be determined by the Scope:

• Security
• Performance
• Integration

What are the minimum set of measurements and metrics?

The single most important deliverable the testing team maintains are defects. Defects are arguably the only product the testing team produces that are seen and understood by the project as a whole. This is where the faults against the system are recorded and tracked -- at a bare minimum each defect should contain:

• Defect Name / Title
• Defect description . What requirement is not being met?
• Detail instructions on how to replicate the defect.
• Defect severity.
• Impacted functional area.
• Defect Author.
• Status (Open, Work-in-Progress, Fixed, Closed)

This will then provide the data for a minimal set of metrics:

• Number of defects raised
• Distribution of defects in terms of severity
• Distribution of defects in terms of functional area

From this baseline the measurements and metrics a testing organization maintains are dependent on its maturity and mission statement. The SEI (Software Engineering Institute) Process Maturity Levels apply to testing as much as they do to any Software Engineering discipline:

1. Initial: (Anarchy) Unpredictable and poorly controlled.
2. Repeatable: (Folklore) Repeat previously mastered tasks.
3. Defined: (Standards) Process characterized, fairly well understood.
4. Managed: (Measurement) Process measured and controlled.
5. Optimizing: (Optimization) Focus on process improvement.

How disciplined the testing organization needs to become and what measurements and metrics are required are dependent on a cost benefit analysis by the Test Lead / Manager. What makes sense in terms of the stated goals and previous performance of the testing organization?

How to grow and maintain a Testing Organization?

Managing or leading a testing team is probably one of the most challenging positions in the IT industry. The team is usually understaffed, lacks appropriate tooling, and financing. Deadlines don.t move but the testing phase is continually being pressured by product delays. Motivation and retention of key testing personnel under these conditions is critical . How do you accomplish this seemly impossible task? I can only go by my personal experience both as a lead and a team member:

• If the timelines are impacted modify the Test Plan appropriately in terms of Scope.
• Clearly communicate the situation to the testing team and project management.
• Keep clear lines of communication to Development and project management.
• Whenever possible sell, sell, sell the importance and contributions of the Testing Team.
• Ensure the testing organization has clearly defined roles for each member of the team and a well-defined career path.
• Measure and communicate testing return on investment -- if the detected defect would have reached the field what would have been the cost.
• Explain testing expenditures in terms of investment (ROI) not cost.
• Finally, never lose your cool -- Good luck.