Software Automated Testing Guidelines

8/8/2019 Software Automated Testing Guidelines

1/5

Software Automated Testing Guidelines

Malik Jahan Khan, Abdul Qadeer, Shafay Shamail

Department of Computer ScienceLahore University of Management Sciences (LUMS)Lahore Pakistan

[email protected], [email protected], [email protected]

AbstractToday software marketplace is competitive andvolatile like never before. Commercial softwarecomplexity is increasing while time pressure torelease high quality software is also on the rise.On one hand more defects are probable due toincreased software complexity while on theother hand organizations cannot afford to let slipmany bugs in the field. In an effort to test an

application reliably, quickly and thoroughly,automated software testing has emerged as aviable solution to meet todays challenges.Although it is true that automated softwaretesting tools have become very powerful, itshould be kept in mind that proper induction andutilization of tool is also nothing less than achallenge. The primary focus of this paper is todescribe the common pitfalls and to recommendguidelines to maximize the success probabilityof tool induction.

1. Introduction

Most of todays enterprise software is developed

using some variant of agile process like RUP [8].The main idea of these processes is to break-up thewhole big project into many small manageable parts.Each part is released to the client, while thesubsequent releases integrate with the older one.These kinds of processes have their own benefits forall the stakeholders.

Once a project is complete and the whole productis fully deployed, the product goes into themaintenance phase. Software in general andenterprise applications in specific should not resistchanging. Due to this reason these days software arebuilt using pluggable components. Any componentcould be changed anytime as the needs arise. There

could be numerous reasons for the change. We arenot discussing the reasons to initiate these changesbut one thing is obvious that in todays volatileworld change is inevitable in the software.

In the above two scenarios, it is notable thatsoftware testing team has to do lot of regressiontesting. In scenario one, whenever a new iteration iscomplete and it is merged with the previous release,the inspection team has to thoroughly test the newfunctionality while running the regression test on the

previous release, to make sure that integration issmooth. In a big enterprise application the amount ofsoftware, to be regression tested, increases. Inscenario two, mostly the client changes are too smallas compared to the size of the whole project. Againthe result is that once the change is implemented,inspection team has a lot of testing work as far asregression testing is concerned. Although software

organizations try to circumvent this situation byintelligently analyzing and segregating softwareparts which could potentially malfunction and henceare good candidate of regression testing. But thismethod has its potential risks.

Iterative development and changes initiated byclient are not the only situations when testing teamhas lot of work to do. Different organizationsperform different testing cycles on their releases.Similarly after each bug fix, localized regression isperformed. So the workload of inspection team isever increasing [5].

In this paper, we are not discussing that how an

organization could choose from its repertoire ofchoices. Our focus is to present guidelines for anorganization, which has already decided to go forautomated testing. Automated software testing iscomparatively a newer approach of testing and lot ofmyths surrounds this technique. In section 2, wehave discussed various testing alternatives.Automated testing is one of them. In section 3, wehave discussed some benefits of automated testing.In section 4, a few challenges of automation havebeen discussed briefly. In rest of this paper, aprescription or guidelines for an organization whichhas already arrived at the decision to use automatedtesting, have been suggested in detail.

2. Testing Alternatives

To meet a certain quality level, an organizationcould have the following alternatives at its disposal:

2.1Commit their inspection team to worklonger and harder

Our local software industry is notorious for longerworking hours. Ask any software engineer and he


2/5

will tell you his wish to keep a balance between hispersonal life and work. It is very common fororganizations to commit their current humanresource to work longer and harder. But this couldonly serve as a short term solution. There is alwaysthe hazard that people will burn out and the turnoverrate could increase due to people leaving the

organization. In any case there is always highprobability of missed bugs, slipped into the field.

2.2Increase the personnel in the inspectionteam

The second option is to increase the testingpersonnel in the inspection department.Organizations need to conduct cost benefit analysisbefore hiring new resources. Hiring experiencedresources is more costly while hiring fresh graduatesand training them in-house has its own cost andrisks. The direct costs are in terms of pays andtrainings. The in-direct cost is that the experienced

people have to devote their time to train newresources. There is always some risk associated, ifnew resources are put on work without giving themenough time to understand the business domain ofthe software.

2.3Give more testing time to the same teamThe third option is to give the same testing teammore time, so that they could test properly. But itwould be against the spirit of an agile process ifsystem testing is consuming a big chunk of the totaliteration time. Software business is very competitivethese days and organizations are looking ways to

expedite the software delivery. So this option is nomore an option for most of the organizations.

2.4Do not do anything and rely on whatevertesting is possible in given time

Option four is again not viable. These days,organizations have to be proactive to stay profitablein the business. It should be noted that using thisoption will result in decreased product quality byslipping the bugs in the shipped release. Later, thesebugs would be reported by the client. These fielddefects not only reduce customer satisfaction butalso put the organization under pressure as they now

have to take care of bigger backlog of field bugsalong with new release. So neglecting problems isno solution because problems will keep on comingback with more intensity.

2.5Use automated testingThe fifth option is to use the automated testing.Automated testing is a technique to test software bymeans of a software tool, which records the userbehavior and later it could replay the recorded scriptto mimic or simulate the human behavior. The

recorded script could be enhanced by means ofprogramming. There are different automation toolsfor white box and black box testing. For exampleRationals Purifier is used to detect any memoryleaks in C / C++ code. Mercury InteractivesWinRunner and Quick Test Professional (QTP) areexamples of GUI based functional black box testing.

3. Benefits of Automated Testing

Automated testing is a very powerful technique. Butto extract this power proper induction and usage oftool is mandatory. Let us discuss the major benefitsof an automated testing tool.

3.1FastAlthough automated testing tools mimic or simulatea human user but they do it very quickly ascompared to a human being. In manual testing therate of test case execution varies due to many humanfactors like a person is not equally productive the

whole day. On the other hand an automated toolcould run the script with the same high speed eachtime, it is executed. It dramatically decreases thetesting time and hence utilizing the inspection timeto the fullest.

3.2ReliableIn manual testing, there is always a chance of humanerror. Automated testing tools, on the other hand,run the scripts reliably each time. Exact same stepsare followed every time, the script is run.

3.3RepeatableThe recorded script is repeatable. One can test howa website or application reacts after repeatedexecution of the same operations. Repeatedexecution becomes very cumbersome in manualtesting.

3.4ProgrammableIn almost every script, the recorded script isenhanced by means of programming elements.These programming elements enhance power of thescripts. Now they can verify a larger range andvariety of functional requirements.

3.5ComprehensiveOne can build a suite of tests that covers everyfeature in a website or application, hence, increasingthe code and data coverage. It is always desirable totest the complete functionality of the software.

3.6ReusableOne can reuse tests on different versions of awebsite or application, even if the user-interfacechanges.


3/5

4. Challenges of Automation

Though automated testing has a lot of benefit, but italso has some associated challenges. Following listincludes some of the major challenges ofautomation. Our suggested guidelines will be helpfulto face these challenges [3]:

i. Selection of Test Toolii. Customization of Tool

iii. Selection of Automation Leveliv. Development and Verification of Scriptv. Implementation of Test Management System

5. Guidelines [2], [7], [1]

Many expectations are not completely met inautomated testing, but it really adds a lot of benefits,even done by right people in right environment atright time [6]. Cost/benefit analysis is veryimportant before going for automated testing.Adapting automated testing for trivial and smallcases may be quite expensive. So, experienced

decision makers play an important role here.Following suggested guidelines will be quite helpfulto adapt a successful automation.

5.1Technical and Managerial DecisionIncorporating test automation could be atechnological decision at the beginning butultimately technical managers have to defend theirproposal in front of organizational management.Different people have their own point of views aboutdifferent things. The gap between different peoplecould be wider depending on differentcircumstances. So it is very important to discuss the

automation decision inside out to reach a commonunderstanding among stake holders. Time, cost andresource requirements should be crystal clear amongthe management levels.

5.2Planning Automated TestingAppropriate plan is required to introduce automatedtesting in the organization. This plan should havemeasurable milestones, shared with the uppermanagement as well.

5.3Market SurveyMarket survey is necessary. This market survey has

two dimensions. Firstly the organization shouldthoroughly know the kind of software they want toapply automation on. They should also getautomation success and experience of otherorganizations, working in similar domain. Mostlyother organizations dont provide such data. Socompany has to invest either for a related consultantor they need to buy the industry survey reports. Itshould be noted that there are many automatedsoftware testing vendors in the market and it maynot be feasible for the organization to try everyproduct to decide among the tool. It is highly

recommended that an organization try to buy acomparison report of automated tools. In market,every vendor claims that its tool is easy to use,friendly, and reliable and your non-technical peoplecan easily learn and use it [9]. But you need toevaluate after a comprehensive survey and thenselect the really stable tool.

5.4Proof of ConceptOnce the tool choice is narrowed down (not morethan two), the next task is proof of concept. It isvery important to make a dedicated team to work onproof of concept. Their task is to verify the toolcapabilities by having first hands on experience onthe organization software. Few automated softwareare available for limited time trial version. Sometimes, organizations need to buy at least one licensefor this purpose. This phase is very crucial becausecompany decision greatly depends on the results ofthis study. So it goes without saying that this group

has the responsibility to appropriately selecting theportions of software to be tested using the tool,which are representative of whole application.

5.5Trained PeopleTo conduct proof of concept, company need somepeople who know automated testing. So companycould either hire people or could train their resourcesfor automated testing. To train people could be acostly option, but to conduct proof of concept,company needs people who are with theorganization for quite some time and theyunderstand the company software domain well.

5.6Automated Test ScriptWriting automated test script is just like any otherdevelopment activity. The difference is that therequirement gathering and high level design is notneeded. Design, coding and testing of automatedtest script code is very important. Dontunderestimate this point because consider howsarcastic it would be that an organization inducted anautomation tool to increase their testing efficiencybut they ended up with more bugs slipped into thefield because the automated script wasnt able todetect it due to a bug in its script. This is a majorcost of test automation. Again the test script writerisnt the best person to test it.

5.7Properties of Test ScriptTest script needs all the qualities of a good softwarelike maintainability, proper documentation etc. Thisis because if test scripts could not be changedquickly and easily, then they are almost useless. Allthis needs experienced test script writers. It isrecommended that the organizations identify theportions of the program, which are least probable tochange and yet important for regression test. If they


4/5

couldnt find such functionality and they dont haveexperienced test script writers, it is recommendednot to do test automation at this point.

5.8Significance of Test CaseIt is a myth that automated testing can catch more

bugs as compared to manual testing. It is necessaryto understand that an automated test script couldonly be as efficient in finding bugs as the originaltest case. Test script is a translation of a test case. Ifa test case skips something, the automated test scriptwill miss it.

5.9Manual Test CasesFor test automation, detailed test script along withtest data is necessary. Some times, a level ofabstraction hides the detail of test case because thetesting engineer has the domain knowledge. Ifautomated test script writer is someone else, the

proper detail is necessary. If the organization couldtrain all of their test team for automation, so thatthey could write the test script for their own testcases, that is the ideal case. If it is not possible andthere are only few people who know automationwell and they have to automate everyones test casesthen it is recommended to increase the quality anddetail of manual test cases. The automated test scriptshould be tested by the person, who originally wrotethe test case.

5.10Significance of Automation ToolGive your automation tool and yourself sufficienttime before judging its usefulness. Although it is

very important to quantitatively measure theefficiency due to test tool but the conclusions shouldbe made after sufficient time. The organizationshould rely on quick tutorial based tool learningapproaches. Vendors provide very good tutorialswith tools. The best bet is to put the people to writethe real test script after learning most important andbasic features. Advanced features could be learnedas needed. It decreases training cost. The automatedtest scripts may not have all those nice features likemaintainability in the beginning, but after sometime, organization would have valuable work force.One important factor is to hire people in inspectionteam with right qualification, so that they haveprogramming background. This thing reallyincreases the speed of script coding.

5.11Requirements ChangeWidespread application change could make all thetest scripts useless and they may need to change. Ifsome part of software is highly probable to bechanged, it shouldnt be automated.

5.12Incorporating IntelligenceAutomated testing tools can not mimic humanintelligence. When a test engineer manually executesthe steps of a test case, s/he could intelligentlychange his/her actions as needed. Incorporatingsuch customization in the script is very difficult, but

not impossible.

5.13Exhaustive TestingExhaustive testing is yet not possible. Althoughcode and data coverage is greatly increased, but yettesting each branch of code with each permutation ofdata is not possible.

5.14Time to Run Automated Test ToolIn GUI-based automated test tools, it is difficult tostart writing scripts if code is yet not ready. Thatmeans, it may not be possible to use the tool fromthe very beginning of a project. The best situation is

when first release is manually tested and deployed.Then while release two is under development, theautomated scripts of release one should be writtenfor the purposes of regression testing.

5.15TestabilityTestability becomes one of the most important traitsafter the induction of a costly tool (The cost of asophisticated tool is around $4000 per license).Testability is the quality of a software that how easyor difficult it is to test the software. Many third partycomponents may not be recognized by the tool. Inthese conditions organization should provide the

developers with a list of GUI gadgets, which couldbe recognized by the automation tool.

5.16Automation SuccessThe Software Test Life Cycle (STLC) is theroadmap to automation success. STLC parallels theSoftware Development Life Cycle (SDLC), andincludes phases planning, analysis, design,construction, testing (initial test cycles), bug fixesand re-testing (final testing and implementation) [4].Lack of planning earlier is one of the biggest reasonsof automation failures. A lot of automation workshould be done before the software is available.Start Early is the big lesson, practitionersrecommend for successful automation [4].

5.17Part of CultureMake the automation part of culture. Though, itcant be guaranteed to experience a successfulautomation in the first go, however, experience andlearn leads to successful adaptation of thisphenomenon. Finally, this success will makeautomation part of the organizational culture. The


5/5

complex and hardly controllable future projects needa pre-existing culture of automation.

The computer which runs automated scripts maynot be used in parallel for other activities. It is amisconception that a test engineer will manually testapplication while the tool is running its script in the

background. When the automated tool mimics theuser behavior by clicking the gadgets, the operatingsystem thinks that its the active window and hencebrings that window in the foreground. That meansthat a test engineer may not do something else, whiletest script is running in the background. So eitherseparate machines are required or test engineer mayneed to observe running script. Some times, un-attended scripts are run at the close of business torun over night. But again writing unattended scriptsis more difficult than ordinary scripts.

6. Conclusions

Every organization is unique and has its own needs.It is difficult to recommend one size fit allsolution. But following the guidelines given in thispaper increases the probability of success and couldmake the automation experience satisfactory for allthe stakeholders. These guidelines might prove to bequite effective when organizations customize themaccording to their own business architecture,available human resources, nature of the project,level of quality needed and budget constraints.

Acknowledgements

We are highly thankful to Tauqeer Hussain, TariqFayyaz and Zeeshan Ali Rana for their valuable help

and guidance towards completing this work.

References

[1] AutoTester Inc. Best practices guide toautomated testing, A document of basicdescription of Automated Testing,http://www.autotester.com/content/pdfs/BPG-pdf.PDF, 2002.

[2] Dustin, E.Lessons in test automation, SoftwareTesting & Quality Engineering,September/October 1999 pp. 16-21.

[3] Hughes Software Systems Ltd. Test Automation,http://www.hssworld.com/whitepapers/whitepaper_pdf/test_automation.pdf, December 2002.

[4] Isenberg, H. M. The Practical Organization of Automated Software Testing,http://www.automated-testing.com/PATfinal.htm, 2005.

[5] Kan, S. H. Metrics and Models in SoftwareEngineering, Second Edition, PearsonEducation, 2002.

[6] Kerry, Automated Software Testing Aperspective,http://www.testingstuff.com/autotest.pdf

[7] Pettichord, B. Seven steps to test automationsuccess, http://www.io.com/~wazmo/papers/seven_steps.html, June 2001.

[8] Pressman, R. S. Software Engineering, APractitioners Approach, Sixth Edition,McGraw Hill Education, 2005.

[9] Zambelich, K. Totally Data-Driven AutomatedTesting, A White Paper of Automated TestingSpecialists Inc., http://www.sqa-test.com/w_paper1.html, 1998.

Documents

Software Automated Testing Guidelines