QA Discussions Forum

Test Automation using HP QTP 9.X(Note: I asked this question in Linked Group FYI I'm posting here. If you got any better idea, very one is welcomed to discuss.)

Question: I have a question regarding the automation testing.
Take project as web application of Insurance company .
when exactly an automation will start in the project.?How can you judge that automation is required at this particular moment in the project ?

Responses: 

Michael Weinstock

With QTP, you need a stable front end (GUI) on the app before you start automation testing. If the GUI is undergoing significant change, you are likely to run into maintenance issues with automated tests as the objects on the GUI will be changing.

That said, just like with any sort of testing, there is nothing to stop you planning your automation testing ahead of time.

In terms of working out what to automate, when I teach a QTP class I usually suggest to my students that as a rule of thumb, if a test is going to need to be run more than 50 times, then it is a good candidate for being automated as the ROI will increase.

Regards,

Michael Weinstock
Melbourne, Australia

 _________________________________________________________________________________

Vijay Kalkundri • - Automation can start from the very beginning of the project.If we have have a prototype of the web application then the automation engineer can start work on automation. The Test automation will have various activities like Feasilibility analysis,Tool selection, skill set development, training on the tool, Framework design, script development, Script verification, Script execution and bug reporting.
- Automation is basically driven by the bussiness needs. When i mean the bussiness needs it basically refers to cost and benifits . One needs to ask himself the following questions :
- Does the project demand the reduction of the lifecycle cost of the software product ?
- Does the project some kinds of testing activities that cannot be run manually such as (Memory leak, concurrent, load, ) ?
- Does the project/product span over releases, where the tester will have to run through rounds and rounds of regression tests. In that case there is a likelihood of mundaneness being introduced into the work of the test engineer ? The test engineer can be used more effectively to do other complex area.

If answers to all the questions is yes then there is a need for the Test Automation in the project. 
___________________________________________________________________________________
Va • Vijay thank you for your answer. Could you please tell an example. Below are more ?'s
1) Take a scenario like "A web page/Win Application has 3 text boxes, 2 buttons, 1 lable control" My questions is can we automate this scenario. If yes, then how can we do that.
2) As per Michael can't we automate more than 50 times and less than 50 times. If yes, can you please explain with an example.
3) What is the major difference between Automation and Manual testing. 
___________________________________________________________________________________
Vijay Kalkundri •
Following are my view on the questions :
1) It can definitely be automated. But i will suggest you to do the following steps
- Do a feasibility analysis on the existing manual test cases so that you can decide on which test cases you are going to automate.
- Get a buy in from the lead/manage/management before you start of the automation.
- Decide on the tool that you want to use. ( Open source or purchased tool)
- Scripting language that you/ your team is comfortable with.
- Build a framework .
- Script the case and verify the same.
2) If there is a cost and time benefit that your project is going to deliver then it is defenetly should be automated. The best example is suppose you have started automation on a banking project which needs to be tested for lot of ( say 100 scenarios) and the life span of the project is 1 and half year.Then it can be a candidate for automation.
3) Major difference what i feel from the Automation testing over the manual testing are :
- Can be run unattended
- Can be used to cover lot of scenarios, which manually would be tedious.
- Some of the tests cannot be run manually . This has to be automated ( such as concurrency, Load, Stress, Volume, memory leak).
- Can be tested for lot of combination of the data.
- Can be used to generate the test results that need minimal analysis.
These are some of the difference there are lot more .. :)
 ___________________________________________________________________________________
Mohamed Amine M'BARKI
Q: when exactly an automation will start in the project.?
My Answer: Before starting to automate you must know whether or not we really need to automate the application... for that you have to meet the actors of the project: a tester a business analyst, an IT developer etc.. who know very well the application and who are lead to implement the test Plan and the test strategy etc... during the meeting you have to answer the following criteria:
1- did all expected results reproducible, or predictable? (Desired Answer: yes)
2- did the insurance application stable with expected behavior? (A: stable)
3- Is it necessary to have human intervention? (A: No)
4- How long take a test to last? (A: Not a long time)
5- What are the hard tests to execute manually? .....
6- What are the technologies used to built th insurance application? (A: the technos must be compliant with the QTP addins you have)
7- Tests are predictable, reusable? (A: Yes, Yes)
So when you have all the answers then you can decide to do automation.
About a previous post on number of execution, the formula to calculate the ROI for a test is:
ROI = [(Nb of executions * (time to execute the test manualy - time to execute it with automation) - time spent to design the test]/time spent to design the test
The aim is to have an ROI > > > than 1, and this is possible if we have a very important number of executions..
 
Kind Regards,
Amine  
____________________________________________________________________________________

Mark Ferguson • 1 - 'Record & Playback will only record the actions on the screen' - In most cases this is simply not a true statement. A lot of tools (like QTP, SilkTest, Rational Functional Tester et al) will also interrogate the GUI objects and remember their attributes. There are still some tools that essentially 'screen scrape' (EggPlant, VNC Robot come to mind) but they are the exception rather than the rule. On a Windows platform, APIs like IAccessible make writing tools that get 'under the bonnet' relatively easy nowadays.

2 - 'Unique technology' is not the same as a unique technique. The technique of 'fuzzy' matching object attributes has been available in a lot of GUI automation tools for years. I'm sure that Original Software has its own implementation of this technique and if you push linguistic boundaries you could perhaps claim 'unique technology'. However if you ask the question, 'Do other tools have a resilience to changes in the underlying object attributes?', the answer is usually 'Yes'. 
____________________________________________________________________________________
Yaron Peri-Glass • Going back to the original questions ;)
When can/should you start automating?

The answer really depends on the automation approach you take.

1. Using a basic RECORD-REPLAY approach, your test suite will be indeed very sensitive to GUI changes. So, to be effective, you will need to wait for the GUI to get stable.

But, even though this is only one approach, it is still very common (unfortunately), so some people take it for granted.

2. A different approach is to focus on a REUSABLE test suite.
This can be achieved by various techniques, like KDT, scripting, Smart Recording and more. If you take this path, you can start building your test suite at day 1, and keep relatively low maintenance even when GUI is changing.
(We are following this way with our regression from day 1, and results are very good)

Also, regarding the factor of 50X mentioned above
This is really a very safe number to use, but it applies mainly in a RECORD-REPLAY approach.
If you build your test suite in a REUSABLE way, the ROI for each reusable component is much higher. You no longer count how many times a test case is executed, but rather how many test cases are using it. And this number, typically, is much higher.

Bottom line is
If you choose a REUSABLE approach you can effectively automate a much wider part of the test cases, and do it from day 1. 
_________________________________________________________________________________

Sid (Sudheer) Sagar • Not sure why you are still at Record and Play level. Keyword driven Frameworks have made creation of test scrips user friendly, enabling the non-technical guys to participate in the dev process and also give them a meaningful role in maintenance. Second, functionality like, Parametrization to check the same process with diff variables is helpful where the company size is large and employee number is high. Imagine a company has 50K employees and want to check their credentials every 15 days. Can you think of Manual testing? So, the size matters. Finally, QTP integrates with Quality Management tools like HP Quality Center from where QTP scripts can be fired to record the test results which in turn reflect the health of the project in Test Coverage Matrix.

50 or 100 does not matter, but re-usability, critical and the time span provided to complete the test can impact your decision.

______________________________________________________
______________________________________________________
Question: Web Application Automation Using QTP?
 I am trying to create a framework that will be used for automating test cases for a web application.
My company uses QTP, but then it is a keyword driven, record and playback testing tool hence only after the application has been implemented, I can go about automating the test cases.
But in one particular white paper found in http://www.sqa-test.com/w_paper1.html, says that keyword driven test tool uses the best of both data driven testing and also record and playback tools.
It also says that using a tool like QTP which is keyword driven, one can automate (for example) test cases for a web application even before the application is really created, by recording it on an another dummy application.
My doubt is it really possible?

Also I need to desperately create a framework which can be used to automate the test cases using the business requirements, since the web application will be huge (~700 screens), so if it is possible to create test cases which are common to few screens and if it can be done before the development, it would lessen a lot of burden on the testers.
Can someone please help me in figuring out a way to achieve this?
Even if you can point me in the right path it would be great.
But let me try and explain again what exactly I want more clearly.

In case of VBScript, I can for example write a script that will validate the fields of a screen (Name field, Date Field, Password field, etc).
So this script can be Run & Writtern as follows:

MasterScript.vbs "Runtest_ValidateFields" "NameOfApp_MyWebApp"

////////***********MasterScript.vbs:***********//////////

Var TestToRun, ApplicationToTest

TestToRun = Arg[1]
ApplicationToTest = Arg[2]

If TestToRun == Runtest_ValidateFields
Call ValidateFields.vbs ApplicationToTest.html ApplicationToTest.xls

Else
...
...

End If

End

/////////***********************************///////////////

Now this ValidateFields.vbs will take the field names and field types of the application we are testing from the excel sheet (I've attached the excel sheet for your reference).

Now using this kind of a framework I can execute ValidateFields.vbs for any web application regardless.
I only need to specify the field names present and need to code the ValidateFields.vbs in a smart dynamic way.

Now coming back to my question,

Is is possible to do something as reusable and as dynamic as this using QTP (I record the test scenario using a dummy web application which will output the business functionality and then run it on a different web applications).

Answer:

But first I suggest that you click here to go over our Yaron Assa's Framework Manager.
See if this is suitable to your needs, and if you'll need more support we'll see how it's possible to make progress.

http://www.advancedqtp.com/content/knowledge-base/projects/frameworkmanager/

___________________________________________________________________________________

How to automate testing efforts for a project being developed in Agile method?

Olga Magalinskaia • I'm currently working in Agile and we're using NUnit and Silverlight unit test frameworks for unit testing and TestComplete for UI testing

Samuel Oosterhuis • The implementation of a robust testing methodology will be different for every project, but at two larger companies that I have worked used a system where the developers write their own unit tests, and there is a separate group writing the system tests.
This requires the development group to have a solid infrastructure in place for the unit tests, and the automation group writing the system tests to work closely with the developers for the tests they write (especially true in Agile groups.) This all ties together much better if you are using a Continuous Integration system.
The tools will be different for different types of software testing, but make sure to use something that the developers are comfortable using or it will be much harder to get solid buy-in from both groups.

Hariharan Ramakrishnan • I guess your question is, How to introduce Automation when the project is developed using Agile method.
If this assumption is correct, then my answer would be, Initially the project should be done manually for each iteration and in 2 or 3 iterations you would be able to come up with good amount of work for automation based on the repetition and tediousness. Then propose the Automation activity and complete it. From my experience I was able to save atleast 30 Man days effort Per month in 6 months due course using automation. (This was done in a 4 member team with 3 Functional Testers and 1 Automation Tester) The same was presented to the Stakeholders.


Karlin Rasam • I agree with Hariharan that initially the project should be done manually till certain features of the product are stable. Then you can propose an automation activity alongside of manual testing. Some of the automation scenarios that I can think of in an Agile model are, Build verification or Smoke testing to quickly determine the stability of the new build before you actually begin huge manual scenarios. Configuration testing which involves you to run your automation script on different system configuration. Regression testing which helps you to easily identify bugs in a non-changing feature. Also, reusable functions or scripts can be developed which can be used for a different module in an end to end testing.


Hardeep Rai • For automation application being developed under agile development, It is very beneficial for legacy features those will not be much modified in future.
To encounter the issues of frequent UI changes in new features those are under development. Automation scripts should be designed in such a way that maintenance/update is less time consuming. Try using lot of reusable action so that code is reused and we need to update at single place once there is change in flow or Object properties.
Process can be implemented with development team, To send a list of flow/object property changes in advance to QA team so that scripts can updated advance before build is delivered to QA.

Anthony Gomez • You have to have some sort of stable components to automate. Otherwise you're just chasing your tail :) Agree with others that you have to be 2-3 sprints in before there is anything you can do.
Regression testing is key here for unchanged components.


Debasis Biswal • You can use BPT component module of QC10 to leverage your automation effort for projects developed in Agile method.The idea here is to automate individual business component.Once you autoamted all the business component attached to your test case,the test case got autoamted fully.

Shanker Lolakapuri • Agree with Biswal, I am working on BPT approach a huge product implementation. Spint 1-3 (depending on no of sprints) develop business components and once application is stable you can start automation scenarios. Any unexpected changes can be easily maintanied as major functionality is divided into components.


Murali Krishna • Agree with All! I believe introduction of Automation in Agile depends on the type of functionality, coverage and especially duration of Iteration. We can plan according to how much percentage would be beneficiary if we go for automation.
One more point which i would want to emphasis there are situations & types of functionality which we may have to go directly to Automation level (eg. service level Automation) with less documentation (may be test scenarios).

George Wilson • Good topic and one close to my heart - gave a presentation on it a couple of weeks ago. You can find the slides here. www.origsoft.com/tmf-agile-slides I believe, however you do it, you have to have regression testing within each sprint. Anything else is too late, you have missed the boat.


Patrick Giasson • TDD is often used in Agile process and it may be used as well with automation. In the case we don't use Record&Playback then we can develop the tests with the agreed expectations. Most of the tests will fail on the first run and then investigation will be made to fix the tests and/or the application under tests to eventually get the required quality level.


Tim Bower • Patrick - Doesn't this add to the testing load as you are really doing two parallel developments, One for the AUT and one for the automated tests. In my mind there are two uses for automation in any environment. The first is to automate repetitive tasks within the sprint/project. The second is to run regression tests on an application in subsequent sprints to validate the existing functionality has not been adversely impacted. The first use means that a lot of the test assets (scripts) could get thrown away but some will make it through to a 'regression' pack. However, if you cannot create the test assets quickly and easily then this is not feasible. And as a result of this there are no good candidates for regression tests.
I understand the concept of creating test automation without the AUT but have never come across a situation where it really works.
TDD is a great approach, almost as good as ATDD (Acceptance Test Driven Development). What we need is a way of very quickly 'translating' the test criteria into test automation so that automated tests can be created and used within a sprint without impact on the workload and those that are chosen for regression must be available in the very next sprint.