Six facts about software application risks

Similar to SDLC (software development lifecycle management), there is RLC or Risk lifecycle management in a software application in which there are different stages involved. The different stages could be risk identification, risk assessment, impact analysis, countermeasure identification, countermeasure assessment, risk plan etc. There are certain facts about Risk:

1. All Risks identified or perceived in a software application do not necessarily happen in real application usage scenario: This is a proven fact that all risks identified or perceived from an application during its pre-launch stage do not happen during post launch real-life usage stage. Some risks perceived may not happen ever. And some unidentified risks may appear later. Whatever is the case, it is always good to identify the risks that may occur during its usage, the more realistic the better. It is not important that they happen in real scenario, more important is to plan how to cope up if at all they happen.
2. All risks have an impact: All risks have an impact – large, medium or small, but they have. It is the impact that makes its severity high, medium or low and accordingly a plan is prepared to handle the risk, when it happens.
3. Same risk in different circumstances will have different impact: The same risk will vary in terms of its severity under different circumstances of usage, user base, geographic location, type of application etc.
4. No application is 100% risk free, whatsoever countermeasures are taken for it, and only thing that gets done with the countermeasures is lowering of risk: A risk plan to countermeasure a risk never fool-proofs a risk’s impact, only it helps in lowering its impact to a certain level.
5. Risk Impact Cost vs. Countermeasure cost: It is very important to have an analysis of both before deciding on the plan. Some risk may be very severe but its countermeasure cost could be unaffordable.
6. The biggest risk in any application is identification of wrong risks, impact, and plan: Identification of wrong risk with right estimation of impact and countermeasure is useless. Equally useless is identification of right risk with wrong impact analysis (thereby underestimating or overestimating the impact) and arriving at a wrong countermeasure. Right risk identification with right impact analysis but with wrong countermeasure also is a waste of efforts.

Eight Checkpoints for Testers during Software Testing

Testing is a process of facilitating development team/ project team in improving the quality of software before it is released to the customer for use. Some key essential steps are always there that need to be followed by the Testers during Software Testing to streamline the process. The most important checkpoints for testers during software testing, in my opinion, would be:
(1)- Complete document of customer and business requirements specifying all the requirements for the development of the product.
(2)- Latest completed build and URL of the application to hit for Testing.
(3)- Software requirements for installing the application on PCs for testing or for database connectivity.
(4)-Training/Demo of the project by development team to Testing Team so as to understand the flow/functionality of the software.
(5)- Scope of testing should be made clear by the development team/Head.
(6)- If the build comes for retesting, then it should be accompanied by the revised document which includes the updated changes incorporated in the software.
(7)- Clarity regarding which member of development team should be contacted in case of any clarification required during the testing phase regarding the functionality of the module or if testers encounter a showstopper in the Software.
(8)- After release of the bugs list to development team, how much time they will require for fixing the bugs.

Twenty ways to ensure complete coverage of software testing

To ensure complete coverage of software testing, testing team must be careful about certain activities that are part of the process. If the software testing is not complete as per the business and customer requirements, it could have a severe adverse effect during or post implementation of the software at the customer site. The more is the coverage, the less are the chances of any bugs passing to the implementation phase. So, to ensure the complete coverage of software product and to find the maximum possible bugs or defects, testers have to ensure following steps:
1. Ensure that the documents defining the business and customer requirements are complete and correct. How to do that can be taken up in a separate topic.
2. Ensure that the development team has clearly understood the documents.
3. Ensure that testers themselves have thoroughly read and understood the documents.
4. Prepare a clear cut scope of testing based on product documents.
5. The strategy and Test Planning is as per system requirements.
6. Decide test methodology and test tools (if any), and test schedule.
7. Prepare Test Cases based on business rules and customer requirements.
8. Ensure that the test cases are extensive and sensible to cover the complete requirements testing.
9. Ensure that during testing no changes in the test environment (coding etc.) is done by development team.
10. Ensure that development team representatives (1 or all) are present during the complete testing.
11. Create Test Scenarios based on test cases.
12. Observe the result of each test case and record it accordingly.
13. Prepare a comprehensive and detailed test report explaining each of the test case, scenario and its result elaborately.
14. Ensure that all bugs reported should make sense (no duplication/overlapping of scenarios and no repetition of bugs reported)
15. Ensure that the complete testing finishes as per schedule.
16. The final report submitted should clearly state the areas not covered under testing, reason for the same and its impact on the product.
17. Simulate any bugs that are not clear to development team.
18. Ensure that you have a tentative plan from the development team when they are fixing all bugs and submitting it back to testing team.
19. Verify all bugs fixed and ensure that the development team is sitting with testers during verification.
20. Prepare the final report and submit back to Development team, giving the status of each bug fixed, as verified as fixed or not fixed. Report any new bugs arrived in the software while fixing these bugs.

Types of Test Automation Frameworks

In a previous posting, we examined the evolution of automation frameworks.
How are frameworks being implemented today by various QA organizations? Here’s a basic summary of the types of test automation currently in use:
Ad-Hoc

• Scripting developed in reactionary mode to test a single issue or fix
• Test case steps are part of each Action script: high maintenance, low reusability
• Contains some data inputs stored in test script’s datasheet, but not true data-driven

Data-Driven

• Scripts are an assembly of function calls
• Data for test cases read in from external source (e.g., Excel spreadsheet, ODBC database)
• Results can be captured externally per script execution (i.e., spreadsheet, database)

Keyword-Driven

• Test cases are expressed as sequence of keyword-prompted actions
• A Driver script executes Action scripts which call functions as prompted by keywords
• No scripting knowledge necessary for developing and maintaining test cases

Model-Driven

• Descriptive programming is used to respond to dynamic applications (e.g., websites)
• Actually, this is a method which can used within other solution types
• Objects defined by parameterized code (i.e., regular expressions, descriptive programming)
• Custom functions used to enhance workflow capabilities

3rd-Party: HP (Mercury) Quality Center with QuickTest Pro and Business Process Testing

• Similar to keyword-driven but controlled via HP Quality Center database
• Can be used for collaborative work efforts: Business Analysts, Testers, Automaters
• Begins with high-level test requirements:
• Business Requirements defined
• Application Areas (shared resources) defined
• Business Components defined and grouped under Application Areas
• Test steps defined
• Tests can be defined as Scripted Components (i.e., QTP scripts with Expert Mode)
• Business Process Tests and Scripted Components are collected under Test Plan
• Test Runs are organized from Test Plan components and executed from Test Lab
• Test Runs can be scheduled and/or executed on remote/virtual test machines (with QTP)
• Defects can be generated automatically or entered manually per incident
• Dashboard available for interactive status monitoring

Intelligent Query-Driven

• Agile
• Object Oriented
• Constructed as a layered framework
• Test data is compiled at runtime using data-mining techniques

Of course, a combination of these techniques could also be used based on the scope and depth of test requirements.

Functional Testing of Web Services: Part IV

Two other web service test tools which deserve honorable mention in our comparison tests were: soapUI, available as Open Source or in a pro version distributed by eviware; and HP’s QuickTest Professional Web Services Add-In.
soapUI
This tool does not require a detailed knowledge of complicated technologies like .NET. Although the user interface is not as intuitive as that of some other tools, a tester who has some experience in programming can learn to create, organize and perform test activities fairly quickly.
soapUI allows structuring your test project into test suites that contain test cases, which can contain test steps. This structure is well-managed: you can add, modify, delete and change the order of every item in the structure. soapUI provides the tools to manage and run your test cases, and to view the SOAP responses to your test requests. You can even include limited load testing scenarios. For added flexibility, soapUI supports Groovy Script, a scripting language similar to Java.
QuickTest Pro Web-Services Add-In
QuickTest Pro with the Web Services Add-In offers a wizard to create a test object that represents the Web Service and port you want to test, and inserts the relevant steps directly into your test or component. This accelerates the process of designing a basic test that checks the operations that your Web service supports. You then update the wizard-generated steps of your test or component by replacing the generated argument values with known valid values, updating the expected values, and selecting the nodes you want to check in your checkpoints.
A QTP XML Warehouse setting is used to store request data, and a QTP Object Repository is used to store responses as checkpoints for verification.
The wizard generates a generic XML structure as a place holder for the expected XML return values. However, before you can actually run your test or component, you must replace the default values with the appropriate values for your test. A valid SOAP request can be imported into the XML Warehouse as a separate step, and a separate XML checkpoint would need to be created for each test case. Defining tests is a little cumbersome and the UI is a little clumsy, but if you’re already using QTP for functional testing, including web service regression tests in your test suites is an easy option.

Test Automation Metrics: Tracking Progress

Did you ever have one of those days/weeks/months when your boss walks nervously into your office and says in that ‘I-just-came-out-of-a-very-long-and-gruesome-meeting’ tone of voice, “Well, , how’s that automation project coming all?” You figure that just answering “Fine” probably is not enough in this situation, since the fate of the company — and certainly your job — hangs in the balance. Oh, if only you had been tracking your progress all along!
Well, have no fear, the answer is here!
In a previous blog entry, I stated that “automated testing is the programmatic execution of test cases.” That gives us the starting point for our metrics: manual test case count. Generally, these counts fall into two categories: strategically Planned Test Cases, and actually written Manual Test Cases.
Most automation effort tends to be focused on coverting written manual test cases to automated test scripts, so that gives us our third metric, Automated Test Cases.
Progress can be gauged by the ratio of actual to expected. Let’s asemble our metrics into a table and see how this plays out:

As shown above, the progress of automation is expressed as the ratio between (Automated Test Cases)/(Manual Test Cases). This implies a typical dependency of automation on the creation of separate, detailed manual tests created by subject matter experts. If your organization follows a different sort of QA process, where the person doing the automation is also writing the manual tests and converting them directly into automated test scripts, the formula could be altered to express the ratio of (Automated Test Cases)/(Planned Test Cases).

Creating a Solid Test Automation Framework with QTP

HP’s QuickTest Pro can itself be used to build a test automation framework. Listed below are some of the characteristics you should take into consideration during the design phase.
Test automation scripts and functions should:

• Be Reusable and Repeatable
• Require Low Maintenance
• Allow Unattended Execution
• Support Platform Independence - IE 6 or IE 7, shouldn’t matter
• Use Data Abstraction - Keyword-Driven Framework

HP QuickTest Pro scripts should use:

• Environment Variables
• Test execution ‘Driver’ to call ‘Actions’
• ‘Action’ to call functions - i.e., function call for each action
• Imported test data (Excel) - common format
• Results reporting - QTP + Quality Center; Excel “Step Status”

VBScript functions can be:

• Simple - Specific purpose; True / False or 0 / 1 return code
• Complex - Multiple steps (e.g., login); “Fuzzy” return code that maps to business rules
• Resilient - include Error Handling

Handling Objects in the Repository is done by:

• Object changes/additions per app release
• Replace with descriptive programming whenever possible
• Functions using regular expressions (e.g., popup windows: IE 6 vs IE 7)
• Web app objects usually classifiable by type, html tag, or class (QTP test object properties)
• Reusable methods: look before you write!

Introduction to HP QuickTest Pro Objects

HP QuickTest Professional uses an Object Repository to store information about the various fields and controls used to build the user interface of a software application. This repository is essentially a database of the names and properties of all the objects encountered during test script creation.
“Essentially all configuration and run functionality provided via the QuickTest interface is in some way represented in the QuickTest automation object model via objects, methods, and properties.” – HP (Mercury) QuickTest Professional User’s Guide
A QuickTest Pro 9.x Object Repository file structure looks something like this:

Here’s an example of the object definition for an ‘Add’ button contained in the repository above:

Note that the value of the “name” property defaults to what was discovered during recording, but the value of any property can be changed by a QTP scripter to a more recognizable value - including the logical name - as part of the repository maintenance process.
By comparison, these properties are very similar to object definitions found using the Microsoft IE Dev Toolbar.
Application object:

Object properties discovered with Microsft IE Developer Toolbar:

Note also that whatever works in VBScript for object handling works in QTP - but not necessarily vice-versa!

function junction: File System Object( Functional testing, HP QuickTest Professional)

Writing functions that make use of the methods exposed by the Windows file system object might seem trivial at first, but take my word for it, it is a worthwhile activity. If you use HP QuickTest Pro for test automation, or even need to do some system maintenance with VBScript, these simple scripts will become an invaluable part of your toolkit. You will come to depend on their simplicity and robustness when they are used for mundane tasks within your test automation scripts.
This first function returns the name of a file contained in a long path statement.
For example, if FullSpec = “C:\Folder1\Folder2\Folder3\MyFile.xls”, the function will return the value “MyFile.xls”:

‘————————————————————–
Function GetLongFileName(FullSpec)
‘returns file name with the extension from full path
‘assuming last element is file name
Dim fso
Set fso = CreateObject(”Scripting.FileSystemObject”)
GetLongFileName = fso.GetFileName(FullSpec)
End Function

Conversely, this next function returns the full path to a file from the fully-pathed file name.
For example, if FullSpec = “C:\Folder1\Folder2\Folder3\MyFile.xls”, the function will return the value “C:\Folder1\Folder2\Folder3″:

‘————————————————————–
Function GetParentPath(FullSpec)
‘returns just the path portion of string
‘assuming last element is file name
‘ Note: function parses out trailing backslash
Dim fso
Set fso = CreateObject(”Scripting.FileSystemObject”)
GetParentPath = fso.GetParentFolderName(FullSpec)
End Function

And as another twist, the function below returns the name of the file without the extension.
For example, if FullSpec = “C:\Folder1\Folder2\Folder3\MyFile.xls”, the function will return the value “MyFile”. This is useful as part of a routine for renaming converted files:

‘————————————————————–
Function GetFileBase(filespec)
‘returns file name without extension from full path
‘assuming last element is file name
Dim fso
Set fso = CreateObject(”Scripting.FileSystemObject”)
GetFileBase = fso.GetBaseName(filespec)
End Function

Of course, we can do much more than parse file names. Here’s a function that will add a new folder to a specified path, after verifying that the path is found and that the folder doesn’t already exist:

‘————————————————————–
Function AddNewFolder(FullPath, FolderName)
‘creates a subfolder under the specified path
Dim fso, f, fc, nf
If FolderName = “” Then
FolderName = “New Folder”
End If
Set fso = CreateObject(”Scripting.FileSystemObject”)
If (fso.FolderExists(FullPath)) Then
Set f = fso.GetFolder(FullPath)
Set fc = f.SubFolders
Else
AddNewFolder = “Path ” & FullPath & ” not found!”
Exit Function
End If
If (fso.FolderExists(FullPath & “\” & FolderName)) Then
AddNewFolder = “Folder ” & FolderName & ” exists”
Exit Function
Else
Set nf = fc.Add(FolderName)
AddNewFolder = 0 ‘ “Folder ” & FolderName & ” added”
End If
End Function

Don’t worry if your functions seem too small and simple: in the end, they really work better that way. If they do one thing, and do it well, they become a sturdy link in the chain of actions that can make up a test case!

How to Implement a Functional Test Automation Methodology

A lot of people talk about using a “Methodology”, but what does that really mean? There are many complicated meanings for the word “methodology” itself, in Wikipedia and elsewhere. Personally, I have always taken it to mean “writing down (the -ology part) the way things are done (the method part) for a given process.”
OK, so I sit down, write up how everything is or should be done for functional test automation, and I have my methodology. What do I do with this scholarly work? Why, implement it, of course! Here are the steps I have used successfully in many automation engagements.
Discover
* Conduct Discovery Session(s) with subject matter experts, QA analysts, testers
* Establish Functional Test Goals
* Define Application(s) Under Test
* Review Requirements, Design Specifications, and Manual Functional Tests
* Identify Business Processes to Automate
* Identify Test Resources (Tools, Staff, Skills, Environments)
* Create Test Plan
* Develop Detailed Project Plan
Develop
* Exercise AUT (that’s Application Under Test, for you newbies)
* Build Test Data
* Create Business Component Tests
* Define Test Plan Components
* Customize Test Scripts and Function Libraries
* Create Test Sets and Parameters
* Dry-Run Test Sets (Test the Tests)
Execute
* Verify Test Readiness (App Build Complete?)
* Validate Test Data
* Execute Test Cycles (Iterations)
* Review Functional Test Results
* Identify Defects
Analyze
* Analyze defects discovered
* Submit Defects to Development for Resolution
* Retest Cycle for Closure
* Validate results with stakeholders
Report
* Perform Test Coverage Analysis
* Test Execution Metrics
* Present Report(s) to stakeholders
Transition
* Knowledge Transfer
* Framework Support and Maintenance (Ongoing)
- each section includes Key Terms, Roles & Responsibilities, and Deliverables
- effort (number of tasks) diminishes as method progresses
If you are not a newbie (an oldbie?), then you might recognize that this is a kind of mashup between standard QA processes and test automation. Since test automation is really just another method for executing tests, this is a natural fit. As with all good things, a solid QA process is reusable in different situations, including automation.

Automation Test Data Management (TDM)

Test Data Management (TDM) is fundamental to the success of automated testing. For example, consider that one of the most beneficial forms of test automation is data-driven testing, which gives testers the ability to input and manipulate massive amounts of data in a relatively short period of time. If the data is bad, then running the tests could produce a mountain of unreliable results, and a whole lot of wasted time, money and effort. It pays to get data management right, especially with test automation.
Associated with test data creation are issues of capacity (i.e., disk space), data verification, data confidentiality, and prolonged test durations. If other forms of testing, such as manual or performance, are taking place in the same environment, there can be issues of concurrency, with tests failing simply because key data has changed “behind the scenes” during test execution, with records locked or altered unexpectedly.
Checking both visible test results and the effects of test execution on the database are essential to successful automation. Every automated test must start with a known data state, and end with the data in a predictable state — or even in its pre-test state.
Some questions that will help in planning a test data strategy include:

• How will data be created and entered into the system?
• Will production data be copied? If so, how will privacy of information be ensured?
• Will data need to be created from scratch?
• Who will be responsible for test data?
• What volume of data is needed to test the application or system, and how frequently will it need to be refreshed?
• Should date be refreshed completely, or incrementally?

In some cases, the test automation tools can be leveraged to load all the pre-test data and thus create the initial state of the database prior to executing functional tests.

Developing Effective Test Plans for Automation

An application test plan should contain a minimum set of optimized test cases with maximum test coverage of all critical application functions. It should be executed using a tool that easily adapts to changing data and requirements.
In order to create an effective automation test, it is first necessary to review the application test plan provided by the application owner to evaluate its suitability for automation. You don’t want to automate a “BAD” test case. Consider the exact intent of the test plan and determine if you can create an effective test case (same coverage) more simply with more reliable automation. It is not acceptable to simply write all the automation scripts directly from the manual test plans. This has the same inherent limitation as doing record/replay for every script: the test is unreliable.
Design test cases and test scripts to be modular. Instead of using one script to perform multiple functions, break the script into separate functions.
Design test cases and test scripts to be generic in terms of process and repeatable in terms of data. Read test data from a separate source: keep the scripts free of test data so that when you do have to change the process or the data, you only have to maintain one item in one place.
Consider the goal of the test
Don’t just blindly follow a manual test plan. See if there is a simple way to accomplish the objective stated in the test plan.
Focus on modularity and reusability. Create a set of evaluation criteria for functions to be considered when using the automated test tool. These criteria may include:

• Repeatability of tests
• Criticality/Risk of applications
• Simplicity of operation
• Ease of automation
• Level of documentation of the function (requirements, specifications, etc.)

Targeting Test Plans for Automation
A test plan representing a good candidate for automation would have the following characteristics:

• Contains a repeatable sequence of actions
• The sequence of actions is repeated many times
• It is technically feasible to automate the sequence of actions (tool is capable, no external hardware actions)
• The behavior of the application under test is the same with automation as without
• Testing involves non-UI aspects of the application (almost all non-UI functions can and should be executed using automated tests)
• The same tests must be run on multiple hardware configurations
• The same tests must be run with varied combinations of other applications to verify compatibility (i.e., Interoperability Testing)

Automated Software Testing: In The Beginning

Like other living things, software QA processes evolve over time. From its humble beginnings, test automation has undergone a similar transformation.
The first stage of test automation was the ‘Record and Playback’ age. Tools were marketed for their ability to record a typical user session and then faithfully play it back using the same objects and inputs. Good marketing, bad practice, because as soon as the application changed the recording stopped working. But it did get people interested in automation.
The next stage could be called the ‘Script Modularity’ age. The recording concept was retained, but now it was linked to a scripting language that allowed a tool expert to create modular, reusable scripts to perform the actions required in a test case. These scripts could be maintained as separate modules that corresponded roughly to the modules of an application, making it easier to change the test code when the application code changed. Easier, but still not efficient. Complex applications would require complex scripts, which usually require more expertise to maintain.
And what about handling all that test data hardcoded into each script? The mind boggled.
Luckily it didn’t boggle too long, which led to the next stage, the ‘Data-Driven’ age. Tools were constructed that allowed access to large pools of external test data, so that these modular scripts could process iteration after iteration of data input. They could churn through mountains of data, as often as desired. What could be better? Well, there were still maintenance issues, as the number of test scripts still grew in direct, or sometimes geometric, proportion to the growth of applications. Additional tools were created just to manage the test execution tools as the asset inventory climbed ever higher. And all those tool experts, they were getting expensive. But that’s the price of progress, right?
Wrong! Evolution usually favors simplicity, since knobby bits tend to break off and fail, sometimes endangering the entire species. Another level of abstraction was necessary, and this ushered in the ‘Keyword-Driven’ age. The test actions were generalized and stored in function libraries, objects were either inventoried in repositories or identified descriptively by type, and testers who were experts in application testing no longer needed to be test tool experts to execute their automated tests. By choosing from a list of keywords linked to functions, they could now describe their tests in their own terms. Test tool script maintenance was simplified down to occasionally updating the few assets that were required to process the keywords, which meant fewer tool experts (a.k.a, ‘knobby bits’ — lol). Truly, a Golden Age.
Of course, evolution doesn’t stop there. There are many experiments in progress today: business process testing, model-driven testing, intelligent query-driven testing, to name a few. The goal seems to be to find the toolset that provides the greatest test coverage with the least amount of maintenance.


And certainly the field of artificial intelligence will have a major impact on sofware testing in the future.