A Practitioner's Guide to Software Test Design @Team LiBdahlan.unimal.ac.id/files/ebooks/2004 A... · A Practitioner's Guide to Software Test Design contains today's important current

APractitioner'sGuidetoSoftwareTestDesignbyLeeCopeland ISBN:158053791x

ArtechHouse©2004

Thistextpresentsalltheimportanttestdesigntechniquesinasingleplaceandinaconsistent,andeasy-to-digestformat.Itenablesyoutochoosethebesttestcasedesign,findsoftwaredefects,developoptimalstrategies,andmore.

TableofContentsAPractitioner'sGuidetoSoftwareTestDesign

Preface

Chapter1 - TheTestingProcessChapter2 - CaseStudiesSectionI-BlackBoxTestingTechniquesChapter3 - EquivalenceClassTestingChapter4 - BoundaryValueTestingChapter5 - DecisionTableTestingChapter6 - PairwiseTestingChapter7 - State-TransitionTestingChapter8 - DomainAnalysisTestingChapter9 - UseCaseTestingSectionII-WhiteBoxTestingTechniquesChapter10 - ControlFlowTestingChapter11 - DataFlowTestingSectionIII-TestingParadigmsChapter12 - ScriptedTestingChapter13 - ExploratoryTestingChapter14 - TestPlanningSectionIV-SupportingTechnologiesChapter15 - DefectTaxonomiesChapter16 -WhentoStopTesting

SectionV-SomeFinalThoughtsAppendixA - Brown&DonaldsonCaseStudyAppendixB - StatelessUniversityRegistrationSystemCaseStudyBibliography

Index

ListofFigures

ListofTables

ListofExamples

BackCover

Here’sacomprehensive,up-to-dateandpracticalintroductiontosoftwaretestdesign.Thisinvaluablebookpresentsalltheimportanttestdesigntechniquesinasingleplaceandinaconsistent,andeasy-to-digestformat.Animmediatelyusefulhandbookfortestengineers,developers,qualityassuranceprofessionals,andrequirementsandsystemsanalysts,itenablesyouto:choosethebesttestcasedesign,findsoftwaredefectsinlesstimeandwithfewerresources,anddevelopoptimalstrategiesthathelpreducethelikelihoodofcostlyerrors.Italsoassistsyouinestimatingtheeffort,timeandcostofgoodtesting.

Numerouscasestudiesandexamplesofsoftwaretestingtechniquesareincluded,helpingyoutofullyunderstandthepracticalapplicationsofthesetechniques.Fromwell-establishedtechniquessuchasequivalenceclasses,boundaryvalueanalysis,decisiontables,andstate-transitiondiagrams,tonewtechniqueslikeusecasetesting,pairwisetesting,andexploratorytesting,thebookisanindispensableresourcefortestingprofessionalsseekingtoimprovetheirskillsandanexcellentreferenceforcollege-levelcoursesinsoftwaretestdesign.

AbouttheAuthor

LeeCopelandisaninternationallyknownconsultantinsoftwaretesting,withover30yearsofexperienceasaninformationsystemsprofessional.Hehasheldanumberoftechnicalandmanagerialpositionswithcommercialandnonprofitorganizationsintheareasofsoftwaredevelopment,testing,andprocessimprovement.HehastaughtseminarsandconsultedextensivelythroughouttheUnitedStatesandinternationally.

APractitioner'sGuidetoSoftwareTestDesignLeeCopeland

ArtechHousePublishersBoston•London

LibraryofCongressandBritishCIPinformationavailableonrequest685CantonStreetNorwood,MA02062(781)769-9750www.artechhouse.com

46GillinghamStreetLondonSW1V1AH+44(0)207596-8750

Copyright©2004STQEPublishing

Allrightsreserved.Nopartofthisbookshallbereproduced,storedinaretrievalsystem,ortransmittedbyanymeans,electronic,mechanical,photocopying,recording,orotherwisewithoutwrittenpermissionfromthepublisher.InternationalStandardBookNumber:1-58053-791-X

PrintedintheUnitedStatesofAmerica

FirstPrinting:November2003

Trademarks

Alltermsmentionedinthisbookthatareknowntobetrademarksorservicemarkshavebeenappropriatelycapitalized.ArtechHousePublishersandSTQEPublishingcannotattesttotheaccuracyofthisinformation.Useofaterminthisbookshouldnotberegardedasaffectingthevalidityofanytrademarkorservicemark.

WarningandDisclaimer

Everyefforthasbeenmadetomakethisbookascompleteandaccurateaspossible,butnowarrantyorfitnessisimplied.Theinformationprovidedisonan"asis"basis.Theauthorsandthepublishershallhaveneitherliabilitynorresponsibilitytoanypersonorentitywithrespecttoanylossordamagesarisingfromtheinformationcontainedinthisbook.

Dedication

TomywifeSuzanne,andourwonderfulchildrenandgrandchildrenShawnandMartha

http://www.artechhouse.com

AndrewandCassandra

David

CathleenKatelynnandKiley

MelissaandJayRoss,Elizabeth,andMiranda

BrianandHeatherCassidyandCaden

ThomasandJeni

Carrie

Sundari

Rajan

andtoWayne,Jerry,Dani,Ron,andRayannefortheirencouragementovertheyears.

LeeCopelandisaninternationallyknownconsultantinsoftwaretesting,withover30yearsofexperienceasaninformationsystemsprofessional.Hehasheldanumberoftechnicalandmanagerialpositionswithcommercialandnonprofitorganizationsintheareasofsoftwaredevelopment,testing,andprocessimprovement.HehastaughtseminarsandconsultedextensivelythroughouttheUnitedStatesandinternationally.

AsaconsultantforSoftwareQualityEngineering,Leetravelstheworldpromotingeffectivesoftwaretestingtohisclients.Inaddition,heistheprogramchairforSTAREASTandSTARWEST,theworld'spremierconferencesonsoftwaretesting.

PrefaceAPractitioner'sGuidetoSoftwareTestDesigncontainstoday'simportantcurrenttestdesignapproachesinoneuniquebook.Untilnow,softwaretestershadtosearchthroughanumberofbooks,periodicals,andWebsitestolocatethisvitalinformation.

ImportanceofTestDesign

"Theactofcareful,complete,systematic,testdesignwillcatchasmanybugsastheactoftesting....Personally,Ibelievethatit'sfarmoreeffective."

-BorisBeizer

Thebookfocusesonlyonsoftwaretestdesign,notrelatedsubjectssuchastestplanning,testmanagement,testteamdevelopment,etc.Whilethoseareimportantinsoftwaretesting,theyhaveoftenovershadowedwhattestersreallyneed—themorepracticalaspectsoftesting,specificallytestcasedesign.Otherexcellentbookscanguideyouthroughtheoverallprocessofsoftwaretesting.OneofmyfavoritesisSystematicSoftwareTestingbyRickCraigandStefanJaskiel.

APractitioner'sGuidetoSoftwareTestDesignillustrateseachtestdesignapproachthroughdetailedexamplesandstep-by-stepinstructions.Theseleadthereadertoaclearunderstandingofeachtestdesigntechnique.

Today'sTestingChallengesForanysystemofinterestingsizeitisimpossibletotestallthedifferentlogicpathsandallthedifferentinputdatacombinations.Oftheinfinitenumberofchoices,eachoneofwhichisworthyofsomeleveloftesting,testerscanonlychooseaverysmallsubsetbecauseofresourceconstraints.Thepurposeofthisbookistohelpyouanalyze,design,andchoosesuchsubsets,toimplementthoseteststhataremostlikelytodiscoverdefects.

Itisvitaltochoosetestcaseswisely.Missingadefectcanresultinsignificantlossestoyourorganizationifadefectivesystemisplacedintoproduction.

APractitioner'sGuidetoSoftwareTestDesigndescribesasetofkeytestdesignstrategiesthatimproveboththeefficiencyandeffectivenessofsoftwaretesters.

StructureandApproachAPractitioner'sGuidetoSoftwareTestDesignexplainsthemostimportanttestdesigntechniquesinusetoday.Someofthesetechniquesareclassicsandwellknownthroughoutthetestingcommunity.Somehavebeenaroundforawhilebutarenotwellknownamongtestengineers.Stillothersarenotwidelyknown,butshouldbebecauseoftheireffectiveness.Thisbookbringstogetherallthesetechniquesintoonevolume,helpingthetestdesignerbecomemoreefficientandeffectiveintesting.

Eachtestdesigntechniqueisapproachedfromapractical,ratherthanatheoreticalbasis.Eachtestdesigntechniqueisfirstintroducedthroughasimpleexample,thenexplainedindetail.Whenpossible,additionalexamplesofitsusearepresented.Thetypesofproblemsonwhichtheapproachcanbeused,alongwithitslimitations,aredescribed.Eachtestdesigntechniquechapterendswithasummaryofitskeypoints,alongwithexercisesthereadercanuseforpractice,andreferencesforfurtherreading.Testerscanusethetechniquespresentedimmediatelyontheirprojects.

ANotefromtheAuthor

Iloveagooddoubleintegralsign

asmuchasthenexttester,butwe'regoingtoconcentrateonthepractical,notthetheoretical.

Eachtestdesignapproachisdescribedinaself-containedchapter.Becausethechaptersarefocused,concise,andindependenttheycanberead"outoforder."Testerscanreadthechaptersthataremostrelevanttotheirworkatthemoment.

AudienceThisbookwaswrittenspecificallyfor:

Softwaretestengineerswhohavetheprimaryresponsibilityfortestcasedesign.Thisbookdetailsthemostefficientandeffectivemethodsforcreatingtestcases.

Softwaredeveloperswho,withtheadventofExtremeProgrammingandotheragiledevelopmentmethods,arebeingaskedtodomoreandbettertestingofthesoftwaretheywrite.Manydevelopershavenotbeenexposedtothedesigntechniquesdescribedinthisbook.

Testanddevelopmentmanagerswhomustunderstand,atleastinprinciple,theworktheirstaffperforms.Notonlydoesthisbookprovideanoverviewofimportanttestdesignmethods,itwillassistmanagersinestimatingtheeffort,time,andcostofgoodtesting.

Qualityassuranceandprocessimprovementengineerswhoarechargedwithdefiningandimprovingtheirsoftwaretestingprocess.

Instructorsandprofessorswhoaresearchingforanexcellentreferenceforacourseinsoftwaretestdesigntechniques.

AppreciationThefollowingreviewershaveprovidedinvaluableassistanceinthewritingofthisbook:AnneMeilof,ChuckAllison,DalePerry,DannyFaught,DorothyGraham,GeoffQuentin,JamesBach,JonHagar,PaulGerrard,RexBlack,RickCraig,RobertRose-Coutré,SidSnook,andWayneMiddleton.Mysincerethankstoeachofthem.Anyfaultsinthisbookshouldbeattributeddirectlytothem.(Justkidding!)

SomeFinalCommentsThisbookcontainsanumberofreferencestoWebsites.Thesereferenceswerecorrectwhenthemanuscriptwassubmittedtothepublisher.Unfortunately,theymayhavebecomebrokenbythetimethebookisinthereaders'hands.

Ithasbecomestandardpracticeforauthorstoincludeapithyquotationonthetitlepageofeachchapter.Unfortunately,thepracticehasbecomesoprevalentthatallthegoodquotationshavebeenused.Justforfun,Ihavechoseninsteadtoincludeoneachchaptertitlepageawinningentryfromthe2003Bulwer-LyttonFictionContest(http://www.bulwer-lytton.com).Since1982,theEnglishDepartmentatSanJoseStateUniversityhassponsoredthisevent,acompetitionthatchallengeswriterstocomposetheopeningsentencetotheworstofallpossiblenovels.ItwasinspiredbyEdwardGeorgeBulwer-LyttonwhobeganhisnovelPaulCliffordwith:

"Itwasadarkandstormynight;therainfellintorrents—exceptatoccasionalintervals,whenitwascheckedbyaviolentgustofwindwhichsweptupthestreets(foritisinLondonthatourscenelies),rattlingalongthehousetops,andfiercelyagitatingthescantyflameofthelampsthatstruggledagainstthedarkness."

MyappreciationtoDr.ScottRiceofSanJoseStateUniversityforpermissiontousetheseexemplaryillustrationsofbadwriting.Hopefully,nothinginthisbookwillwinthisprestigiousaward.

http://www.bulwer-lytton.com

AcknowledgementsThecaricatureofMickJaggerisownedandcopyrightedbyMartinO'Loughlinandusedbypermission.

ClipArtcopyrightbyCorelCorporationandusedunderalicensingagreement.

ReferencesBeizer,Boris(1990).SoftwareTestingTechniques(SecondEdition).VanNostrandReinhold.

Craig,RickD.andStefanP.Jaskiel(2002).SystematicSoftwareTesting.ArtechHousePublishers.

Chapter1:TheTestingProcess

OverviewTheflockofgeeseflewoverheadina'V'formation—notinanold-fashioned-lookingTimesNewRomankindofa'V',branchedoutslightlyatthetwooppositearmsatthetopofthe'V',norinamoremodern-looking,straightandcrisp,linearArialsortof'V'(althoughsincetheywereflying,Arialmighthavebeenappropriate),butinaslightlyasymmetric,tiltingoff-to-one-sidesortofitalicizedCourierNew-like'V'—andLaFonteknewthathewasjustthetypeofmantoknowthedifference.[1]

—JohnDotson

[1]Ifyouthinkthisquotationhasnothingtodowithsoftwaretestingyouarecorrect.Foranexplanationpleaseread"SomeFinalComments"inthePreface.

TestingWhatistesting?Whilemanydefinitionshavebeenwritten,atitscoretestingistheprocessofcomparing"whatis"with"whatoughttobe."AmoreformaldefinitionisgivenintheIEEEStandard610.12-1990,"IEEEStandardGlossaryofSoftwareEngineeringTerminology"whichdefines"testing"as:

"Theprocessofoperatingasystemorcomponentunderspecifiedconditions,observingorrecordingtheresults,andmakinganevaluationofsomeaspectofthesystemorcomponent."

The"specifiedconditions"referredtointhisdefinitionareembodiedintestcases,thesubjectofthisbook.

KeyPoint Atitscore,testingistheprocessofcomparing"whatis"with"whatoughttobe."

RickCraigandStefanJaskielproposeanexpandeddefinitionofsoftwaretestingintheirbook,SystematicSoftwareTesting.

"Testingisaconcurrentlifecycleprocessofengineering,usingandmaintainingtestwareinordertomeasureandimprovethequalityofthesoftwarebeingtested."

Thisviewincludestheplanning,analysis,anddesignthatleadstothecreationoftestcasesinadditiontotheIEEE'sfocusontestexecution.

Differentorganizationsanddifferentindividualshavevariedviewsofthepurposeofsoftwaretesting.BorisBeizerdescribesfivelevelsoftestingmaturity.(Hecalledthemphasesbuttodayweknowthepoliticallycorrecttermis"levels"andtherearealwaysfiveofthem.)

Level0-"There'snodifferencebetweentestinganddebugging.Otherthaninsupportofdebugging,testinghasnopurpose."Defectsmaybestumbleduponbutthereisnoformalizedefforttofindthem.

Level1-"Thepurposeoftestingistoshowthatsoftwareworks."Thisapproach,whichstartswiththepremisethatthesoftwareis(basically)correct,mayblindustodiscoveringdefects.GlenfordMyerswrotethatthoseperformingthetestingmaysubconsciouslyselecttestcasesthatshouldnotfail.Theywillnotcreatethe"diabolical"testsneededtofinddeeplyhiddendefects.

Level2-"Thepurposeoftestingistoshowthatthesoftwaredoesn'twork."Thisisaverydifferentmindset.Itassumesthesoftwaredoesn'tworkandchallengesthetestertofinditsdefects.Withthisapproach,wewillconsciouslyselecttestcasesthatevaluatethesysteminitsnooksandcrannies,atitsboundaries,andnearitsedges,usingdiabolicallyconstructedtestcases.

Level3-"Thepurposeoftestingisnottoproveanything,buttoreducetheperceivedriskof

notworkingtoanacceptablevalue."Whilewecanproveasystemincorrectwithonlyonetestcase,itisimpossibletoeverproveitcorrect.Todosowouldrequireustotesteverypossiblevalidcombinationofinputdataandeverypossibleinvalidcombinationofinputdata.Ourgoalsaretounderstandthequalityofthesoftwareintermsofitsdefects,tofurnishtheprogrammerswithinformationaboutthesoftware'sdeficiencies,andtoprovidemanagementwithanevaluationofthenegativeimpactonourorganizationifweshippedthissystemtocustomersinitspresentstate.

Level4-"Testingisnotanact.Itisamentaldisciplinethatresultsinlow-risksoftwarewithoutmuchtestingeffort."Atthismaturitylevelwefocusonmakingsoftwaremoretestablefromitsinception.Thisincludesreviewsandinspectionsofitsrequirements,design,andcode.Inaddition,itmeanswritingcodethatincorporatesfacilitiesthetestercaneasilyusetointerrogateitwhileitisexecuting.Further,itmeanswritingcodethatisself-diagnosing,thatreportserrorsratherthanrequiringtesterstodiscoverthem.

CurrentChallengesWhenIaskmystudentsaboutthechallengestheyfaceintestingtheytypicallyreply:

Notenoughtimetotestproperly

Toomanycombinationsofinputstotest

Notenoughtimetotestwell

Difficultyindeterminingtheexpectedresultsofeachtest

Nonexistentorrapidlychangingrequirements

Notenoughtimetotestthoroughly

Notrainingintestingprocesses

Notoolsupport

Managementthateitherdoesn'tunderstandtestingor(apparently)doesn'tcareaboutquality

Notenoughtime

Thisbookdoesnotcontain"magicpixiedust"thatyoucanusetocreateadditionaltime,betterrequirements,ormoreenlightenedmanagement.Itdoes,however,containtechniquesthatwillmakeyoumoreefficientandeffectiveinyourtestingbyhelpingyouchooseandconstructtestcasesthatwillfindsubstantiallymoredefectsthanyouhaveinthepastwhileusingfewerresources.

TestCasesTobemosteffectiveandefficient,testcasesmustbedesigned,notjustslappedtogether.Theword"design"hasanumberofdefinitions:

1. Toconceiveorfashioninthemind;invent:designagoodreasontoattendtheSTARtestingconference.Toformulateaplanfor;devise:designamarketingstrategyforthenewproduct.

2. Toplanoutinsystematic,usuallydocumentedform:designabuilding;designatestcase.

3. Tocreateorcontriveforaparticularpurposeoreffect:agamedesignedtoappealtoallages.

4. Tohaveasagoalorpurpose;intend.

5. Tocreateorexecuteinanartisticorhighlyskilledmanner.

KeyPoint

Tobemosteffectiveandefficient,testcasesmustbedesigned,notjustslappedtogether.

Eachofthesedefinitionsappliestogoodtestcasedesign.Regardingtestcasedesign,RogerPressmanwrote:

"Thedesignoftestsforsoftwareandotherengineeringproductscanbeaschallengingastheinitialdesignoftheproductitself.Yet...softwareengineersoftentreattestingasanafterthought,developingtestcasesthat'feelright'buthavelittleassuranceofbeingcomplete.Recallingtheobjectivesoftesting,wemustdesignteststhathavethehighestlikelihoodoffindingthemosterrorswithaminimumamountoftimeandeffort."

Welldesignedtestcasesarecomposedofthreeparts:

Inputs

Outputs

Orderofexecution

KeyPoint Testcasesconsistofinputs,outputs,andorderofexecution.

Inputs

Inputsarecommonlythoughtofasdataenteredatakeyboard.Whilethatisasignificantsourceofsysteminput,datacancomefromothersources—datafrominterfacingsystems,datafrominterfacingdevices,datareadfromfilesordatabases,thestatethesystemisinwhenthedataarrives,andtheenvironmentwithinwhichthesystemexecutes.

Outputs

Outputshavethissamevariety.Oftenoutputsarethoughtofasjustthedatadisplayedonacomputerscreen.Inaddition,datacanbesenttointerfacingsystemsandtoexternaldevices.Datacanbewrittentofilesordatabases.Thestateortheenvironmentmaybemodifiedbythesystem'sexecution.

Alloftheserelevantinputsandoutputsareimportantcomponentsofatestcase.Intestcasedesign,determiningtheexpectedoutputsisthefunctionofan"oracle."

Anoracleisanyprogram,process,ordatathatprovidesthetestdesignerwiththeexpectedresultofatest.Beizerlistsfivetypesoforacles:

KiddieOracles-Justruntheprogramandseewhatcomesout.Ifitlooksaboutright,itmustberight.

RegressionTestSuites-Runtheprogramandcomparetheoutputtotheresultsofthesametestsrunagainstapreviousversionoftheprogram.

ValidatedData-Runtheprogramandcomparetheresultsagainstastandardsuchasatable,formula,orotheraccepteddefinitionofvalidoutput.

PurchasedTestSuites-Runtheprogramagainstastandardizedtestsuitethathasbeenpreviouslycreatedandvalidated.Programslikecompilers,Webbrowsers,andSQL(StructuredQueryLanguage)processorsareoftentestedagainstsuchsuites.

ExistingProgram-Runtheprogramandcomparetheoutputtoanotherversionoftheprogram.

OrderofExecution

Therearetwostylesoftestcasedesignregardingorderoftestexecution.

Cascadingtestcases-Testcasesmaybuildoneachother.Forexample,thefirsttestcaseexercisesaparticularfeatureofthesoftwareandthenleavesthesysteminastatesuchthatthesecondtestcasecanbeexecuted.Intestingadatabaseconsiderthesetestcases:

1. Createarecord

2. Readtherecord

3. Updatetherecord

4. Readtherecord

5. Deletetherecord

6. Readthedeletedrecord

Eachofthesetestscouldbebuiltontheprevioustests.Theadvantageisthateachtestcaseistypicallysmallerandsimpler.Thedisadvantageisthatifonetestfails,thesubsequenttestsmaybeinvalid.

Independenttestcases-Eachtestcaseisentirelyselfcontained.Testsdonotbuildoneachotherorrequirethatothertestshavebeensuccessfullyexecuted.Theadvantageisthatanynumberoftestscanbeexecutedinanyorder.Thedisadvantageisthateachtesttendstobelargerandmorecomplexandthusmoredifficulttodesign,create,andmaintain.

TypesOfTestingTestingisoftendividedintoblackboxtestingandwhiteboxtesting.

Blackboxtestingisastrategyinwhichtestingisbasedsolelyontherequirementsandspecifications.Unlikeitscomplement,whiteboxtesting,blackboxtestingrequiresnoknowledgeoftheinternalpaths,structure,orimplementationofthesoftwareundertest.

Whiteboxtestingisastrategyinwhichtestingisbasedontheinternalpaths,structure,andimplementationofthesoftwareundertest.Unlikeitscomplement,blackboxtesting,whiteboxtestinggenerallyrequiresdetailedprogrammingskills.

Anadditionaltypeoftestingiscalledgrayboxtesting.Inthisapproachwepeekintothe"box"undertestjustlongenoughtounderstandhowithasbeenimplemented.Thenwecloseuptheboxanduseourknowledgetochoosemoreeffectiveblackboxtests.

TestingLevelsTypicallytesting,andthereforetestcasedesign,isperformedatfourdifferentlevels:

UnitTesting-Aunitisthe"smallest"pieceofsoftwarethatadevelopercreates.Itistypicallytheworkofoneprogrammerandisstoredinasinglediskfile.Differentprogramminglanguageshavedifferentunits:InC++andJavatheunitistheclass;inCtheunitisthefunction;inlessstructuredlanguageslikeBasicandCOBOLtheunitmaybetheentireprogram.

KeyPoint Theclassicaltestinglevelsareunit,integration,system,andacceptance.

IntegrationTesting-Inintegrationweassembleunitstogetherintosubsystemsandfinallyintosystems.Itispossibleforunitstofunctionperfectlyinisolationbuttofailwhenintegrated.AclassicexampleisthisCprogramanditssubsidiaryfunction:/*mainprogram*/voidoops(int);intmain(){oops(42);/*calltheoopsfunctionpassinganinteger*/return0;}

/*functionoops(inaseparatefile)*/#include<stdio.h>voidoops(doublex){/*expectsadouble,notanint!*/printf("%f\n",x);/*Willprintgarbage(0ismostlikely)*/}

Iftheseunitsweretestedindividually,eachwouldappeartofunctioncorrectly.Inthiscase,thedefectonlyappearswhenthetwounitsareintegrated.Themainprogrampassesanintegertofunctionoopsbutoopsexpectsadoublelengthintegerandtroubleensues.Itisvitaltoperformintegrationtestingastheintegrationprocessproceeds.

SystemTesting-Asystemconsistsofallofthesoftware(andpossiblyhardware,usermanuals,trainingmaterials,etc.)thatmakeuptheproductdeliveredtothecustomer.Systemtestingfocusesondefectsthatariseatthishighestlevelofintegration.Typicallysystemtestingincludesmanytypesoftesting:functionality,usability,security,internationalizationandlocalization,reliabilityandavailability,capacity,performance,backupandrecovery,portability,andmanymore.Thisbookdealsonlywithfunctionalitytesting.Whiletheothertypesoftestingareimportant,theyarebeyondthescopeofthisvolume.

AcceptanceTesting-Acceptancetestingisdefinedasthattesting,whichwhencompletedsuccessfully,willresultinthecustomeracceptingthesoftwareandgivingustheirmoney.Fromthecustomer'spointofview,theywouldgenerallylikethemost

exhaustiveacceptancetestingpossible(equivalenttothelevelofsystemtesting).Fromthevendor'spointofview,wewouldgenerallyliketheminimumleveloftestingpossiblethatwouldresultinmoneychanginghands.Typicalstrategicquestionsthatshouldbeaddressedbeforeacceptancetestingare:Whodefinestheleveloftheacceptancetesting?Whocreatesthetestscripts?Whoexecutesthetests?Whatisthepass/failcriteriafortheacceptancetest?Whenandhowdowegetpaid?

Notallsystemsareamenabletousingtheselevels.Theselevelsassumethatthereisasignificantperiodoftimebetweendevelopingunitsandintegratingthemintosubsystemsandthenintosystems.InWebdevelopmentitisoftenpossibletogofromconcepttocodetoproductioninamatterofhours.Inthatcase,theunit-integration-systemlevelsdon'tmakemuchsense.ManyWebtestersuseanalternatesetoflevels:

Codequality

Functionality

Usability

Performance

Security

TheImpossibilityOfTestingEverythingInhismonumentalbookTestingObject-OrientedSystems,RobertBinderprovidesanexcellentexampleoftheimpossibilityoftesting"everything."Considerthefollowingprogram:intblech(intj){j=j-1;//shouldbej=j+1j=j/30000;returnj;}

Notethatthesecondlineisincorrect!Thefunctionblechacceptsanintegerj,subtractsonefromit,dividesitby30000(integerdivision,wholenumbers,noremainder)andreturnsthevaluejustcomputed.Ifintegersareimplementedusing16bitsonthiscomputerexecutingthissoftware,thelowestpossibleinputvalueis-32768andthehighestis32767.Thusthereare65,536possibleinputsintothistinyprogram.(Yourorganization'sprogramsareprobablylarger.)Willyouhavethetime(andthestamina)tocreate65,536testcases?Ofcoursenot.Sowhichinputvaluesdowechoose?Considerthefollowinginputvaluesandtheirabilitytodetectthisdefect.

Input(j) ExpectedResult ActualResult

1 0 0

42 0 0

40000 1 1

-64000 -2 -2

Oops!Notethatnoneofthetestcaseschosenhavedetectedthisdefect.Infactonlyfourofthepossible65,536inputvalueswillfindthisdefect.Whatisthechancethatyouwillchooseallfour?Whatisthechanceyouwillchooseoneofthefour?WhatisthechanceyouwillwinthePowerballlottery?Isyouranswerthesametoeachofthesethreequestions?

SummaryTestingisaconcurrentlifecycleprocessofengineering,using,andmaintainingtestwareinordertomeasureandimprovethequalityofthesoftwarebeingtested.(CraigandJaskiel)

Thedesignoftestsforsoftwareandotherengineeringproductscanbeaschallengingastheinitialdesignoftheproductitself.Yet...softwareengineersoftentreattestingasanafterthought,developingtestcasesthat'feelright'buthavelittleassuranceofbeingcomplete.Recallingtheobjectivesoftesting,wemustdesignteststhathavethehighestlikelihoodoffindingthemosterrorswithaminimumamountoftimeandeffort.(Pressman)

Blackboxtestingisastrategyinwhichtestingisbasedsolelyontherequirementsandspecifications.Whiteboxtestingisastrategyinwhichtestingisbasedontheinternalpaths,structure,andimplementationofthesoftwareundertest.

Typicallytesting,andthereforetestcasedesign,isperformedatfourdifferentlevels:Unit,Integration,System,andAcceptance.

Practice1. Whichfourinputstotheblechroutinewillfindthehiddendefect?Howdidyou

determinethem?Whatdoesthissuggesttoyouasanapproachtofindingotherdefects?

References

Beizer,Boris(1990).SoftwareTestingTechniques(SecondEdition).VanNostrandReinhold.

Binder,RobertV.(2000).TestingObject-OrientedSystems:Models,Patterns,andTools.Addison-Wesley.

Craig,RickD.andStefanP.Jaskiel(2002).SystematicSoftwareTesting.ArtechHousePublishers.

IEEEStandard610.12-1990,IEEEStandardGlossaryofSoftwareEngineeringTerminology,1991.

Myers,Glenford(1979).TheArtofSoftwareTesting.JohnWiley&Sons.

Pressman,RogerS.(1982).SoftwareEngineering:APractitioner'sApproach(FourthEdition).McGraw-Hill.

Chapter2:CaseStudiesTheyhadbutonelastremainingnighttogether,sotheyembracedeachotherastightlyasthattwo-flavorentwinedstringcheesethatisorangeandyellowish-white,theorangeprobablybeingablandCheddarandthewhite...Mozzarella,althoughitcouldpossiblybeProvoloneorjustplainAmerican,asitreallydoesn'ttastedistinctlydissimilarfromtheorange,yettheywouldhaveyoubelieveitdoesbycoloringitdifferently.

—MariannSimms

WhyCaseStudies?Twocasestudiesareprovidedintheappendicesofthisbook.AppendixAdescribes"Brown&Donaldson,"anonlinebrokeragefirm.AppendixBdescribesthe"StatelessUniversityRegistrationSystem."Examplesfromthesecasestudiesareusedtoillustratethetestcasedesigntechniquesdescribedinthisbook.Inaddition,someofthebook'sexercisesarebasedonthecasestudies.Thefollowingsectionsbrieflydescribethecasestudies.ReadthedetailedinformationinAppendixAandBwhenrequired.

Brown&DonaldsonBrown&Donaldson(B&D)isafictitiousonlinebrokeragefirmthatyoucanusetopracticethetestdesigntechniquespresentedinthisbook.B&DwasoriginallycreatedforSoftwareQualityEngineering'sWeb/eBusinessTestingcourse(formoredetailsseehttp://www.sqe.com).

ScreenshotsofvariouspagesareincludedinAppendixA.Referencewillbemadetosomeofthesethroughoutthebook.TheactualB&DWebsiteisfoundathttp://bdonline.sqe.com.AnyresemblancetoanyactualonlinebrokerageWebsiteispurelycoincidental.

YoucanactuallytrytheB&DWebsite.First-timeuserswillneedtocreateaBDonlineaccount.Thisaccountisnotreal—anytransactionsrequestedorexecutedviathisaccountwillnotoccurintherealworld,onlyinthefictitiousworldofB&D.Onceyouhavecreatedanaccount,youwillbypassthisstepandloginwithyourusernameandpassword.Whilecreatinganewaccountyouwillbeaskedtosupplyanauthorizationcode.Theauthorizationcodeiseight1s.

ThisWebsitealsocontainsanumberofdownloadabledocumentsfromtheB&Dcasestudy,whichcanbeusedtoassistyouindevelopingtestplansforyourownWebprojects.

http://www.sqe.com

http://bdonline.sqe.com

StatelessUniversityRegistrationSystemEverystatehasastateuniversity.ThiscasestudydescribesanonlinestudentregistrationsystemforthefictitiousStatelessUniversity.PleasedonotattempttocashoutyourstocksfromBrown&DonaldsontoenrollatStatelessU.

ThedocumentinAppendixBdescribestheplanneduserinterfacefortheStatelessUniversityRegistrationSystem(SURS).Itdefinestheuserinterfacescreensintheorderinwhichtheyaretypicallyused.Itstartswiththeloginscreen.Thenitprovidesthedatabaseset-upfields,theaddition/change/deletionofstudents,theaddition/change/deletionofcourses,andtheaddition/change/deletionofclasssections.Thefinaldataentryscreenprovidestheselectionofspecificcoursesectionsforeachstudent.Additionaladministrativefunctionsarealsodefined.

SectionI:BlackBoxTestingTechniques

ChapterList

Chapter3:EquivalenceClassTesting

Chapter4:BoundaryValueTesting

Chapter5:DecisionTableTesting

Chapter6:PairwiseTesting

Chapter7:State-TransitionTesting

Chapter8:DomainAnalysisTesting

Chapter9:UseCaseTesting

PartOverview

DefinitionBlackboxtestingisastrategyinwhichtestingisbasedsolelyontherequirementsandspecifications.Unlikeitscomplement,whiteboxtesting,blackboxtestingrequiresnoknowledgeoftheinternalpaths,structure,orimplementationofthesoftwareundertest(SUT).

Thegeneralblackboxtestingprocessis:

Therequirementsorspecificationsareanalyzed.

ValidinputsarechosenbasedonthespecificationtodeterminethattheSUTprocessesthemcorrectly.InvalidinputsmustalsobechosentoverifythattheSUTdetectsthemandhandlesthemproperly.

Expectedoutputsforthoseinputsaredetermined.

Testsareconstructedwiththeselectedinputs.

Thetestsarerun.

Actualoutputsarecomparedwiththeexpectedoutputs.

AdeterminationismadeastotheproperfunctioningoftheSUT.

ApplicabilityBlackboxtestingcanbeappliedatalllevelsofsystemdevelopment—unit,integration,system,andacceptance.

Aswemoveupinsizefrommoduletosubsystemtosystemtheboxgetslarger,withmorecomplexinputsandmorecomplexoutputs,buttheapproachremainsthesame.Also,aswemoveupinsize,weareforcedtotheblackboxapproach;therearesimplytoomanypathsthroughtheSUTtoperformwhiteboxtesting.

DisadvantagesWhenusingblackboxtesting,thetestercanneverbesureofhowmuchoftheSUThasbeentested.Nomatterhowcleverordiligentthetester,someexecutionpathsmayneverbeexercised.Forexample,whatistheprobabilityatesterwouldselectatestcasetodiscoverthis"feature"?if(name=="Lee"&&employeeNumber=="1234"&&employmentStatus=="RecentlyTerminatedForCause"){sendLeeacheckfor$1,000,000;}

KeyPoint

Whenusingblackboxtesting,thetestercanneverbesureofhowmuchofthesystemundertesthasbeentested.

Tofindeverydefectusingblackboxtesting,thetesterwouldhavetocreateeverypossiblecombinationofinputdata,bothvalidandinvalid.Thisexhaustiveinputtestingisalmostalwaysimpossible.Wecanonlychooseasubset(oftenaverysmallsubset)oftheinputcombinations.

InTheArtofSoftwareTesting,GlenfordMyersprovidesanexcellentexampleofthefutilityofexhaustivetesting:Howwouldyouthoroughlytestacompiler?Bywritingeverypossiblevalidandinvalidprogram.Theproblemissubstantiallyworseforsystemsthatmustrememberwhathashappenedbefore(i.e.,thatremembertheirstate).Inthosesystems,notonlymustwetesteverypossibleinput,wemusttesteverypossiblesequenceofeverypossibleinput.

KeyPoint

Eventhoughwecan'ttesteverything,formalblackboxtestingdirectsthetestertochoosesubsetsofteststhatarebothefficientandeffectiveinfindingdefects.

AdvantagesEventhoughwecan'ttesteverything,formalblackboxtestingdirectsthetestertochoosesubsetsofteststhatarebothefficientandeffectiveinfindingdefects.Assuch,thesesubsetswillfindmoredefectsthanarandomlycreatedequivalentnumberoftests.Blackboxtestinghelpsmaximizethereturnonourtestinginvestment.

References

Myers,GlenfordJ.(1979).TheArtofSoftwareTesting.JohnWiley&Sons.

Chapter3:EquivalenceClassTestingOnthefourthdayofhisexplorationoftheAmazon,Byronclimbedoutofhisinnertube,checkedthelatestnewsonhispersonaldigitalassistant(hereafterPDA)outfittedwithwirelesstechnology,andrealizedthatthegnawinghefeltinhisstomachwasnotfear—no,hewasnotafraid,ratherelated—norwasittension—no,hewasactuallyratherrelaxed—soitwasinallprobabilityaparasite.

—ChuckKeelan

IntroductionEquivalenceclasstestingisatechniqueusedtoreducethenumberoftestcasestoamanageablelevelwhilestillmaintainingreasonabletestcoverage.Thissimpletechniqueisusedintuitivelybyalmostalltesters,eventhoughtheymaynotbeawareofitasaformaltestdesignmethod.Manytestershavelogicallydeduceditsusefulness,whileothershavediscovereditsimplybecauseoflackoftimetotestmorethoroughly.

Considerthissituation.Wearewritingamoduleforahumanresourcessystemthatdecideshowweshouldprocessemploymentapplicationsbasedonaperson'sage.Ourorganization'srulesare:

0–16Don'thire

16–18Canhireonapart-timebasisonly

18–55Canhireasafull-timeemployee

55–99Don'thire[*]

[*]Note:Ifyou'vespottedaproblemwiththeserequirements,don'tworry.Theyarewrittenthiswayforapurposeandwillberepairedinthenextchapter.

Observation

WiththeserulesourorganizationwouldnothavehiredDoogieHouser,M.D.orCol.HarlanSanders,onetooyoung,theothertooold.

Shouldwetestthemoduleforthefollowingages:0,1,2,3,4,5,6,7,8,...,90,91,92,93,94,95,96,97,98,99?Ifwehadlotsoftime(anddidn'tmindthemind-numbingrepetitionandwerebeingpaidbythehour)wecertainlycould.Iftheprogrammerhadimplementedthismodulewiththefollowingcodeweshouldtesteachage.(Ifyoudon'thaveaprogrammingbackgrounddon'tworry.Theseexamplesaresimple.Justreadthecodeanditwillmakesensetoyou.)If(applicantAge==0)hireStatus="NO";If(applicantAge==1)hireStatus="NO";…If(applicantAge==14)hireStatus="NO";If(applicantAge==15)hireStatus="NO";If(applicantAge==16)hireStatus="PART";If(applicantAge==17)hireStatus="PART";If(applicantAge==18)hireStatus="FULL";If(applicantAge==19)hireStatus="FULL";…If(applicantAge==53)hireStatus="FULL";If(applicantAge==54)hireStatus="FULL";

If(applicantAge==55)hireStatus="NO";If(applicantAge==56)hireStatus="NO";…If(applicantAge==98)hireStatus="NO";If(applicantAge==99)hireStatus="NO";

Giventhisimplementation,thefactthatanysetoftestspassestellsusnothingaboutthenexttestwecouldexecute.Itmaypass;itmayfail.

Luckily,programmersdon'twritecodelikethis(atleastnotveryoften).Abetterprogrammermightwrite:If(applicantAge>=0&&applicantAge<=16)hireStatus="NO";If(applicantAge>=16&&applicantAge<=18)hireStatus="PART";If(applicantAge>=18&&applicantAge<=55)hireStatus="FULL";If(applicantAge>=55&&applicantAge<=99)hireStatus="NO";

Giventhistypicalimplementation,itisclearthatforthefirstrequirementwedon'thavetotest0,1,2,...14,15,and16.Onlyonevalueneedstobetested.Andwhichvalue?Anyonewithinthatrangeisjustasgoodasanyotherone.Thesameistrueforeachoftheotherranges.Rangessuchastheonesdescribedherearecalledequivalenceclasses.Anequivalenceclassconsistsofasetofdatathatistreatedthesamebythemoduleorthatshouldproducethesameresult.Anydatavaluewithinaclassisequivalent,intermsoftesting,toanyothervalue.Specifically,wewouldexpectthat:

Ifonetestcaseinanequivalenceclassdetectsadefect,allothertestcasesinthesameequivalenceclassarelikelytodetectthesamedefect.

Ifonetestcaseinanequivalenceclassdoesnotdetectadefect,noothertestcasesinthesameequivalenceclassislikelytodetectthedefect.

KeyPoint

Agroupoftestsformsanequivalenceclassifyoubelievethat:Theyalltestthesamething.

Ifonetestcatchesabug,theothersprobablywilltoo.

Ifonetestdoesn'tcatchabug,theothersprobablywon'teither.

CemKanerTestingComputerSoftware

Thisapproachassumes,ofcourse,thataspecificationexiststhatdefinesthevariousequivalenceclassestobetested.Italsoassumesthattheprogrammerhasnotdonesomethingstrangesuchas:If(applicantAge>=0&&applicantAge<=16)

hireStatus="NO";If(applicantAge>=16&&applicantAge<=18)hireStatus="PART";If(applicantAge>=18&&applicantAge<=41)hireStatus="FULL";//strangestatementsfollowIf(applicantAge==42&&applicantName=="Lee")hireStatus="HIRENOWATHUGESALARY";If(applicantAge==42&&applicantName<>"Lee")hireStatus="FULL";//endofstrangestatements

If(applicantAge>=43&&applicantAge<=55)hireStatus="FULL";If(applicantAge>=55&&applicantAge<=99)hireStatus="NO";

Usingtheequivalenceclassapproach,wehavereducedthenumberoftestcasesfrom100(testingeachage)tofour(testingoneageineachequivalenceclass)—asignificantsavings.

Now,arewereadytobegintesting?Probablynot.Whataboutinputvalueslike969,-42,FRED,and&$#!@?Shouldwecreatetestcasesforinvalidinput?Theansweris,asanygoodconsultantwilltellyou,"itdepends."Tounderstandthisanswerweneedtoexamineanapproachthatcameoutoftheobject-orientedworldcalleddesign-by-contract.

NoteAccordingtotheBible,theageofMethuselahwhenhediedwas969years(Gen5:27).ThankstotheGideonswhomadethisdataeasilyaccessibleinmyhotelroomwithouttheneedforahighspeedInternetconnection.

Inlaw,acontractisalegallybindingagreementbetweentwo(ormore)partiesthatdescribeswhateachpartypromisestodoornotdo.Eachofthesepromisesisofbenefittotheother.

Inthedesign-by-contractapproach,modules(called"methods"intheobject-orientedparadigm,but"module"isamoregenericterm)aredefinedintermsofpre-conditionsandpost-conditions.Post-conditionsdefinewhatamodulepromisestodo(computeavalue,openafile,printareport,updateadatabaserecord,changethestateofthesystem,etc.).Pre-conditionsdefinewhatthatmodulerequiressothatitcanmeetitspost-conditions.Forexample,ifwehadamodulecalledopenFile,whatdoesitpromisetodo?Openafile.WhatwouldlegitimatepreconditionsofopenFilebe?First,thefilemustexist;second,wemustprovidethename(orotheridentifyinginformation)ofthefile;third,thefilemustbe"openable,"thatis,itcannotalreadybeexclusivelyopenedbyanotherprocess;fourth,wemusthaveaccessrightstothefile;andsoon.Pre-conditionsandpostconditionsestablishacontractbetweenamoduleandothersthatinvokeit.

Testing-by-contractisbasedonthedesign-by-contractphilosophy.Itsapproachistocreatetestcasesonlyforthesituationsinwhichthepre-conditionsaremet.Forexample,wewould

nottesttheopenFilemodulewhenthefiledidnotexist.Thereasonissimple.Ifthefiledoesnotexist,openFiledoesnotpromisetowork.Ifthereisnoclaimthatitwillworkunderaspecificcondition,thereisnoneedtotestunderthatcondition.

ForMoreInformation

SeeBertrandMeyer'sbookObject-OrientedSoftwareConstructionformoreondesign-by-contract.

Atthispointtestersusuallyprotest.Yes,theyagree,themoduledoesnotclaimtoworkinthatcase,butwhatifthepreconditionsareviolatedduringproduction?Whatdoesthesystemdo?Dowegetamisspelledwordonthescreenorasmokingcraterwhereourcompanyusedtobe?

Adifferentapproachtodesignisdefensivedesign.Inthiscasethemoduleisdesignedtoacceptanyinput.Ifthenormalpreconditionsaremet,themodulewillachieveitsnormalpostconditions.Ifthenormalpre-conditionsarenotmet,themodulewillnotifythecallerbyreturninganerrorcodeorthrowinganexception(dependingontheprogramminglanguageused).Thisnotificationisactuallyanotheroneofthemodule'spostconditions.Basedonthisapproachwecoulddefinedefensivetesting:anapproachthattestsunderbothnormalandabnormalpre-conditions.

InsightAstudentinoneofmyclasses,let'scallhimFred,saidhedidn'treallycarewhichdesignapproachwasbeingused,hewasgoingtoalwaysusedefensivetesting.WhenIaskedwhy,hereplied,"Ifitdoesn'twork,whowillgettheblame-thoseresponsibleorthetesters?"

Howdoesthisapplytoequivalenceclasstesting?Dowehavetotestwithinputslike-42,FRED,and&$#!@?Ifweareusingdesign-by-contractandtesting-by-contracttheanswerisNo.Ifweareusingdefensivedesignandthusdefensivetesting,theanswerisYes.Askyourdesignerswhichapproachtheyareusing.Iftheyanswereither"contract"or"defensive,"youknowwhatstyleoftestingtouse.Iftheyanswer"Huh?"thatmeanstheyarenotthinkingabouthowmodulesinterface.Theyarenotthinkingaboutpre-conditionandpost-conditioncontracts.Youshouldexpectintegrationtestingtobeaprimesourceofdefectsthatwillbemorecomplexandtakemoretimethananticipated.

TechniqueThestepsforusingequivalenceclasstestingaresimple.First,identifytheequivalenceclasses.Second,createatestcaseforeachequivalenceclass.Youcouldcreateadditionaltestcasesforeachequivalenceclassifyouhavethetimeandmoney.Additionaltestcasesmaymakeyoufeelwarmandfuzzy,buttheyrarelydiscoverdefectsthefirstdoesn'tfind.

Insight

Astudentinoneofmyclasses,let'scallherJudy,feltveryuncomfortableabouthavingonlyonetestcaseforeachequivalenceclass.Shewantedatleasttwoforthatwarmandfuzzyfeeling.Iindicatedthatifshehadthetimeandmoneythatapproachwasfinebutsuggestedtheadditionaltestswouldprobablybeineffective.Iaskedhertokeeptrackofhowmanytimestheadditionaltestcasesfounddefectsthatthefirstdidnotandletmeknow.IneverheardfromJudyagain.

Differenttypesofinputrequiredifferenttypesofequivalenceclasses.Let'sconsiderfourpossibilities.Let'sassumeadefensivetestingphilosophyoftestingbothvalidandinvalidinput.Testinginvalidinputsisoftenagreatsourceofdefects.

Ifaninputisacontinuousrangeofvalues,thenthereistypicallyoneclassofvalidvaluesandtwoclassesofinvalidvalues,onebelowthevalidclassandoneaboveit.ConsidertheGoofyMortgageCompany(GMC).Theywillwritemortgagesforpeoplewithincomesbetween$1,000/monthand$83,333/month.Anythingbelow$1,000/monthyoudon'tqualify.Anythingover$83,333/monthyoudon'tneedGMC,justpaycash.

Foravalidinputwemightchoose$1,342/month.Forinvalidswemightchoose$123/monthand$90,000/month.

Figure3-1:Continuousequivalenceclasses Ifaninputconditiontakesondiscretevalueswithinarangeofpermissiblevalues,therearetypicallyonevalidandtwoinvalidclasses.GMCwillwriteasinglemortgageforonethroughfivehouses.(Remember,it'sGoofy.)Zeroorfewerhousesisnotalegitimateinput,norissixorgreater.Neitherarefractionalordecimalvaluessuchas21/2or3.14159.

Figure3-2:Discreteequivalenceclasses Foravalidinputwemightchoosetwohouses.Invalidscouldbe-2and8.

GMCwillmakemortgagesonlyforaperson.Theywillnotmakemortgagesforcorporations,trusts,partnerships,oranyothertypeoflegalentity.

Figure3-3:Singleselectionequivalenceclasses Foravalidinputwemustuse"person."Foraninvalidwecouldchoose"corporation"or"trust"oranyotherrandomtextstring.Howmanyinvalidcasesshouldwecreate?Wemusthaveatleastone;wemaychooseadditionaltestsforadditionalwarmandfuzzyfeelings.

GMCwillmakemortgagesonCondominiums,Townhouses,andSingleFamilydwellings.TheywillnotmakemortgagesonDuplexes,MobileHomes,Treehouses,oranyothertypeofdwelling.

Figure3-4:Multipleselectionequivalenceclass Forvalidinputwemustchoosefrom"Condominium,""Townhouse,"or"SingleFamily."Whiletherulesayschooseonetestcasefromthevalidequivalenceclass,amorecomprehensiveapproachwouldbetocreatetestcasesforeachentryinthevalidclass.Thatmakessensewhenthelistofvalidvaluesissmall.But,ifthiswerealistofthefiftystates,theDistrictofColumbia,andthevariousterritoriesoftheUnitedStates,

wouldyoutesteveryoneofthem?Whatifthelistwereeverycountryintheworld?Thecorrectanswer,ofcourse,dependsontherisktotheorganizationif,astesters,wemisssomethingthatisvital.

Now,rarelywillwehavethetimetocreateindividualtestsforeveryseparateequivalenceclassofeveryinputvaluethatentersoursystem.Moreoften,wewillcreatetestcasesthattestanumberofinputfieldssimultaneously.Forexample,wemightcreateasingletestcasewiththefollowingcombinationofinputs:

KeyPoint

Rarelywillwehavethetimetocreateindividualtestsforeveryseparateequivalenceclassofeveryinputvalue.

Table3-1:Atestcaseofvaliddatavalues.

MonthlyIncome NumberofDwellings Applicant DwellingTypes Result

$5,000 2 Person Condo Valid

Eachofthesedatavaluesisinthevalidrange,sowewouldexpectthesystemtoperformcorrectlyandforthetestcasetoreportPass.

Itistemptingtousethesameapproachforinvalidvalues.

Table3-2:Atestcaseofallinvaliddatavalues.Thisisnotagoodapproach.


$100 8 Partnership Treehouse Invalid

Ifthesystemacceptsthisinputasvalid,clearlythesystemisnotvalidatingthefourinputfieldsproperly.Ifthesystemrejectsthisinputasinvalid,itmaydosoinsuchawaythatthetestercannotdeterminewhichfielditrejected.Forexample:ERROR:653X-2.7INVALIDINPUT

Inmanycases,errorsinoneinputfieldmaycanceloutormaskerrorsinanotherfieldsothesystemacceptsthedataasvalid.Abetterapproachistotestoneinvalidvalueatatimetoverifythesystemdetectsitcorrectly.

Table3-3:Asetoftestcasesvaryinginvalidvaluesonebyone.


$100 1 Person SingleFam Invalid

$1,342 0 Person Condo Invalid

$1,342 1 Corporation Townhouse Invalid

$1,342 1 Person Treehouse Invalid

Foradditionalwarmandfuzzyfeelings,theinputs(bothvalidandinvalid)couldbevaried.

Table3-4:Asetoftestcasesvaryinginvalidvaluesonebyonebutalsovaryingthevalidvalues.


$100 1 Person SingleFamily Invalid

$1,342 0 Person Condominium Invalid

$5,432 3 Corporation Townhouse Invalid

$10,000 2 Person Treehouse Invalid

Anotherapproachtousingequivalenceclassesistoexaminetheoutputsratherthantheinputs.Dividetheoutputsintoequivalenceclasses,thendeterminewhatinputvalueswouldcausethoseoutputs.Thishastheadvantageofguidingthetestertoexamine,andthustest,everydifferentkindofoutput.Butthisapproachcanbedeceiving.Inthepreviousexample,forthehumanresourcessystem,oneofthesystemoutputswasNO,thatis,Don'tHire.Acursoryviewoftheinputsthatshouldcausethisoutputwouldyield{0,1,...,14,15}.Notethatthisisnotthecompleteset.Inaddition{55,56,...,98,99}shouldalsocausetheNOoutput.It'simportanttomakesurethatallpotentialoutputscanbegenerated,butdon'tbefooledintochoosingequivalenceclassdatathatomitsimportantinputs.

ExamplesExample1

ReferringtotheTradeWebpageoftheBrown&DonaldsonWebsitedescribedinAppendixA,considertheOrderTypefield.ThedesignerhaschosentoimplementthedecisiontoBuyorSellthroughradiobuttons.Thisisagooddesignchoicebecauseitreducesthenumberoftestcasesthetestermustcreate.Hadthisbeenimplementedasatextfieldinwhichtheuserentered"Buy"or"Sell"thetesterwouldhavepartitionedthevalidinputsas{Buy,Sell}andtheinvalidsas{Trade,Punt,...}.Whatabout"buy","bUy","BUY"?Arethesevalidorinvalidentries?Thetesterwouldhavetoreferbacktotherequirementstodeterminetheirstatus.

Insight Letyourdesignersandprogrammersknowwhentheyhavehelpedyou.They'llappreciatethethoughtandmaydoitagain.

Withtheradiobuttonimplementationnoinvalidchoicesexist,sononeneedtobetested.Onlythevalidinputs{Buy,Sell}needtobeexercised.

Example2

Again,referringtotheTradeWebpage,considertheQuantityfield.Inputtothisfieldcanbebetweenoneandfournumericcharacters(0,1,...,8,9)withavalidvaluegreaterorequalto1andlessthanorequalto9999.Asetofvalidinputsis{1,22,333,4444}whileinvalidinputsare{-42,0,12345,SQE,$#@%}.

InsightVeryoftenyourdesignersandprogrammersuseGUIdesigntoolsthatcanenforcerestrictionsonthelengthandcontentofinputfields.Encouragetheiruse.Thenyourtestingcanfocusonmakingsuretherequirementhasbeenimplementedproperlywiththetool.

Example3

OntheTradepagetheuserentersatickerSymbolindicatingthestocktobuyorsell.Thevalidsymbolsare{A,AA,AABC,AAC,...,ZOLT,ZOMX,ZONA,ZRAN).Theinvalidsymbolsareanycombinationofcharactersnotincludedinthevalidlist.Asetofvalidinputscouldbe{A,AL,ABE,ACES,AKZOY)whileasetofinvalidscouldbe{C,AF,BOB,CLUBS,AKZAM,42,@#$%).

ForMoreInformation

ClickontheSymbolLookupbuttonontheB&DTradepagetoseethefulllistofstocksymbols.

Example4

Rarelywillwecreateseparatesetsoftestcasesforeachinput.Generallyitismoreefficienttotestmultipleinputssimultaneouslywithintests.Forexample,thefollowingtestscombineBuy/Sell,Symbol,andQuantity.


Buy/Sell Symbol Quantity Result

Buy A 10 Valid

Buy C 20 Invalid

Buy A 0 Invalid

Sell ACES 10 Valid

Sell BOB 33 Invalid

Sell ABE -3 Invalid

ApplicabilityandLimitationsEquivalenceclasstestingcansignificantlyreducethenumberoftestcasesthatmustbecreatedandexecuted.Itismostsuitedtosystemsinwhichmuchoftheinputdatatakesonvalueswithinrangesorwithinsets.Itmakestheassumptionthatdatainthesameequivalenceclassis,infact,processedinthesamewaybythesystem.Thesimplestwaytovalidatethisassumptionistoasktheprogrammerabouttheirimplementation.

Equivalenceclasstestingisequallyapplicableattheunit,integration,system,andacceptancetestlevels.Allitrequiresareinputsoroutputsthatcanbepartitionedbasedonthesystem'srequirements.

SummaryEquivalenceclasstestingisatechniqueusedtoreducethenumberoftestcasestoamanageablesizewhilestillmaintainingreasonablecoverage.

Thissimpletechniqueisusedintuitivelybyalmostalltesters,eventhoughtheymaynotbeawareofitasaformaltestdesignmethod.

Anequivalenceclassconsistsofasetofdatathatistreatedthesamebythemoduleorthatshouldproducethesameresult.Anydatavaluewithinaclassisequivalent,intermsoftesting,toanyothervalue.

Practice1. ThefollowingexercisesrefertotheStatelessUniversityRegistrationSystemWeb

sitedescribedinAppendixB.Definetheequivalenceclassesandsuitabletestcasesforthefollowing:

1. ZIPCode—fivenumericdigits.

2. State—thestandardPostOfficetwo-characterabbreviationforthestates,districts,territories,etc.oftheUnitedStates.

3. LastName—onethroughfifteencharacters(includingalphabeticcharacters,periods,hyphens,apostrophes,spaces,andnumbers).

4. UserID—eightcharactersatleasttwoofwhicharenotalphabetic(numeric,special,nonprinting).

5. StudentID—eightcharacters.Thefirsttworepresentthestudent'shomecampuswhilethelastsixareauniquesix-digitnumber.Validhomecampusabbreviationsare:AN,Annandale;LC,LasCruces;RW,RiversideWest;SM,SanMateo;TA,Talbot;WE,Weber;andWN,Wenatchee.

ReferencesBeizer,Boris(1990).SoftwareTestingTechniques.VanNostrandReinhold.

Kaner,Cem,JackFalkandHungQuocNguyen(1999).TestingComputerSoftware(SecondEdition).JohnWiley&Sons.


Chapter4:BoundaryValueTestingThePrincelookeddownatthemotionlessformofSleepingBeauty,wonderinghowhersupplelipswouldfeelagainsthisownandcontemplatingwhetherornotanAltoidwasstrongenoughtostandupagainstthekindofmorningbreathonlyahundredyears'napcouldcreate.

—LynneSella

IntroductionEquivalenceclasstestingisthemostbasictestdesigntechnique.Ithelpstesterschooseasmallsubsetofpossibletestcaseswhilemaintainingreasonablecoverage.Equivalenceclasstestinghasasecondbenefit.Itleadsustotheideaofboundaryvaluetesting,thesecondkeytestdesigntechniquetobepresented.

Inthepreviouschapterthefollowingrulesweregiventhatindicatehowweshouldprocessemploymentapplicationsbasedonaperson'sage.Theruleswere:

0–16 Don'thire


18–55Canhireasafull-timeemployee

55–99Don'thire

Noticetheproblemattheboundaries—the"edges"ofeachclass.Theage"16"isincludedintwodifferentequivalenceclasses(asare18and55).Thefirstrulesaysdon'thirea16-year-old.Thesecondrulesaysa16-year-oldcanbehiredonapart-timebasis.

Boundaryvaluetestingfocusesontheboundariessimplybecausethatiswheresomanydefectshide.Experiencedtestershaveencounteredthissituationmanytimes.Inexperiencedtestersmayhaveanintuitivefeelthatmistakeswilloccurmostoftenattheboundaries.Thesedefectscanbeintherequirements(asshownabove)orinthecodeasshownbelow:

KeyPoint

Boundaryvaluetestingfocusesontheboundariesbecausethatiswheresomanydefectshide.

If(applicantAge>=0&&applicantAge<=16)hireStatus="NO";If(applicantAge>=16&&applicantAge<=18)hireStatus="PART";If(applicantAge>=18&&applicantAge<=55)hireStatus="FULL";If(applicantAge>=55&&applicantAge<=99)hireStatus="NO";

Ofcourse,themistakethatprogrammersmakeiscodinginequalitytestsimproperly.Writing>(greaterthan)insteadof≥(greaterthanorequal)isanexample.

Themostefficientwayoffindingsuchdefects,eitherintherequirementsorthecode,isthroughinspection.GilbandGraham'sbook,SoftwareInspection,isanexcellentguidetothisprocess.However,nomatterhoweffectiveourinspections,wewillwanttotestthecodetoverifyitscorrectness.

Perhapsthisiswhatourorganizationmeant:

0–15Don'thire


18–54Canhireasfull-timeemployees

55–99Don'thire

Whataboutages-3and101?Notethattherequirementsdonotspecifyhowthesevaluesshouldbetreated.Wecouldguessbut"guessingtherequirements"isnotanacceptablepractice.

Thecodethatimplementsthecorrectedrulesis:If(applicantAge>=0&&applicantAge<=15)hireStatus="NO";If(applicantAge>=16&&applicantAge<=17)hireStatus="PART";If(applicantAge>=18&&applicantAge<=54)hireStatus="FULL";If(applicantAge>=55&&applicantAge<=99)hireStatus="NO";

Theinterestingvaluesonorneartheboundariesinthisexampleare{-1,0,1},{15,16,17},{17,18,19},{54,55,56},and{98,99,100}.Othervalues,suchas{-42,1001,FRED,%$#@}mightbeincludeddependingonthemodule'sdocumentedpreconditions.

TechniqueThestepsforusingboundaryvaluetestingaresimple.First,identifytheequivalenceclasses.Second,identifytheboundariesofeachequivalenceclass.Third,createtestcasesforeachboundaryvaluebychoosingonepointontheboundary,onepointjustbelowtheboundary,andonepointjustabovetheboundary."Below"and"above"arerelativetermsanddependonthedatavalue'sunits.Iftheboundaryis16andtheunitis"integer"thenthe"below"pointis15andthe"above"pointis17.Iftheboundaryis$5.00andtheunitis"USdollarsandcents"thenthebelowpointis$4.99andtheabovepointis$5.01.Ontheotherhand,ifthevalueis$5andtheunitis"USdollars"thenthebelowpointis$4andtheabovepointis$6.

KeyPoint

Createtestcasesforeachboundaryvaluebychoosingonepointontheboundary,onepointjustbelowtheboundary,andonepointjustabovetheboundary.

Notethatapointjustaboveoneboundarymaybeinanotherequivalenceclass.Thereisnoreasontoduplicatethetest.Thesamemaybetrueofthepointjustbelowtheboundary.

Youcould,ofcourse,createadditionaltestcasesfartherfromtheboundaries(withinequivalenceclasses)ifyouhavetheresources.Asdiscussedinthepreviouschapter,theseadditionaltestcasesmaymakeyoufeelwarmandfuzzy,buttheyrarelydiscoveradditionaldefects.

Boundaryvaluetestingismostappropriatewheretheinputisacontinuousrangeofvalues.ReturningagaintotheGoofyMortgageCompany,whataretheinterestingboundaryvalues?Formonthlyincometheboundariesare$1,000/monthand$83,333/month(assumingtheunitstobeUSdollars).

Figure4-1:Boundaryvaluesforacontinuousrangeofinputs.

Testdatainputof{$999,$1,000,$1,001}onthelowendand{$83,332,$83,333,$83,334}onthehighendarechosentotesttheboundaries.

BecauseGMCwillwriteamortgageforonethroughfivehouses,zeroorfewerhousesisnotalegitimateinputnorissixorgreater.Theseidentifytheboundariesfortesting.

Figure4-2:Boundaryvaluesforadiscreterangeofinputs.

Rarelywillwehavethetimetocreateindividualtestsforeveryboundaryvalueofeveryinputvaluethatentersoursystem.Moreoften,wewillcreatetestcasesthattestanumberofinputfieldssimultaneously.

Table4-1:Asetoftestcasescontainingcombinationsofvalid(ontheboundary)valuesandinvalid(offtheboundary)points.

MonthlyIncome NumberofDwellings Result Description

$1,000 1 Valid Minincome,mindwellings

$83,333 1 Valid Maxincome,mindwellings

$1,000 5 Valid Minincome,maxdwellings

$83,333 5 Valid Maxincome,maxdwellings

$1,000 0 Invalid Minincome,belowmindwellings

$1,000 6 Invalid Minincome,abovemaxdwellings

$83,333 0 Invalid Maxincome,belowmindwellings

$83,333 6 Invalid Maxincome,abovemaxdwellings

$999 1 Invalid Belowminincome,mindwellings

$83,334 1 Invalid Abovemaxincome,mindwellings

$999 5 Invalid Belowminincome,maxdwellings

$83,334 5 Invalid Abovemaxincome,maxdwellings

Plotting"monthlyincome"onthex-axisand"numberofdwellings"onthey-axisshowsthe"locations"ofthetestdatapoints.

Figure4-3:Datapointsontheboundariesanddatapointsjustoutsidetheboundaries.

Notethatfouroftheinputcombinationsareontheboundarieswhileeightarejustoutside.Alsonotethatthepointsoutsidealwayscombineonevalidvaluewithoneinvalidvalue(justoneunitloweroroneunithigher).

ExamplesBoundaryvaluetestingisapplicabletothestructure(lengthandcharactertype)ofinputdataaswellasitsvalue.Considerthefollowingtwoexamples:

Example1

ReferringtotheTradeWebpageoftheBrown&DonaldsonWebsitedescribedinAppendixA,considertheQuantityfield.Inputtothisfieldcanbebetweenoneandfournumericcharacters(0,1,...,8,9).Asetofboundaryvaluetestcasesforthelengthattributewouldbe{0,1,4,5}numericcharacters.

Example2

Again,ontheTradepage,considertheQuantityfield,butthistimeforvalueratherthanstructure(lengthandcharactertype).WhetherthetransactionisBuyorSell,theminimumlegitimatevalueis1souse{0,1,2}forboundarytesting.Theupperlimitonthisfield'svalueismorecomplicated.IfthetransactionisSell,whatisthemaximumnumberofsharesthatcanbesold?Itisthenumbercurrentlyowned.Forthisboundaryuse{sharesOwned-1,sharesOwned,sharesOwned+1}.IfthetransactionisBuy,themaximumvalue(numberofsharestobepurchased)isdefinedasshares=(accountBalance-commission)/sharePrice

assumingafixedcommission.Use{shares-1,shares,shares+1}astheboundaryvaluetestcases.

ApplicabilityandLimitationsBoundaryvaluetestingcansignificantlyreducethenumberoftestcasesthatmustbecreatedandexecuted.Itismostsuitedtosystemsinwhichmuchoftheinputdatatakesonvalueswithinrangesorwithinsets.

Boundaryvaluetestingisequallyapplicableattheunit,integration,system,andacceptancetestlevels.Allitrequiresareinputsthatcanbepartitionedandboundariesthatcanbeidentifiedbasedonthesystem'srequirements.

SummaryWhileequivalenceclasstestingisuseful,itsgreatestcontributionistoleadustoboundaryvaluetesting.

Boundaryvaluetestingisatechniqueusedtoreducethenumberoftestcasestoamanageablesizewhilestillmaintainingreasonablecoverage.

Boundaryvaluetestingfocusesontheboundariesbecausethatiswheresomanydefectshide.Experiencedtestershaveencounteredthissituationmanytimes.Inexperiencedtestersmayhaveanintuitivefeelthatmistakeswilloccurmostoftenattheboundaries.

Createtestcasesforeachboundaryvaluebychoosingonepointontheboundary,onepointjustbelowtheboundary,andonepointjustabovetheboundary."Below"and"above"arerelativetermsanddependonthedatavalue'sunits.

Practice1. ThefollowingexercisesrefertotheStatelessUniversityRegistrationSystemWeb

sitedescribedinAppendixB.Definetheboundaries,andsuitableboundaryvaluetestcasesforthefollowing:

1. ZIPCode—fivenumericdigits.

2. FirstconsiderZIPCodejustintermsofdigits.Then,determinethelowestandhighestlegitimateZIPCodesintheUnitedStates.Forextracredit[1],determinetheformatofpostalcodesforCanadaandthelowestandhighestvalidvalues.

3. LastName—onethroughfifteencharacters(includingalphabeticcharacters,periods,hyphens,apostrophes,spaces,andnumbers).Forextracredit[2]createafewverycomplexLastNames.Canyoudeterminethe"rules"forlegitimateLastNames?Foradditionalextracredit[3]useaphonebookfromanothercountry—tryFinlandorThailand.

4. UserID—eightcharactersatleasttwoofwhicharenotalphabetic(numeric,special,nonprinting).

5. CourseID—threealphacharactersrepresentingthedepartmentfollowedbyasix-digitintegerwhichistheuniquecourseidentificationnumber.Thepossibledepartmentsare:

PHY-Physics

EGR-Engineering

ENG-English

LAN-Foreignlanguages

CHM-Chemistry

MAT-Mathematics

PED-Physicaleducation

SOC-Sociology

[1]Thereactuallyisnoextracredit,sodoitforfun.




Gilb,TomandDorothyGraham(1993).SoftwareInspection.Addison-Wesley.ISBN0-201-63181-4.


Chapter5:DecisionTableTestingI'dstumbledontosolvingmyfirstmurdercase,havingfoundmyselftheonlyeyewitness,yetnomatterhowfranticallyIpleadedwithJohnLawthattheperpwasrightinfrontofthemandtheverydamethey'dbeengrilling-thesultrybutdeviousMissKitwinkle,whoplayedthegrievingpatsythewayaconcertpianistplayerplaysapiano-thecopsjustkeptsmilingandstuffingcrackersinmybeak.

—ChrisEsco

IntroductionDecisiontablesareanexcellenttooltocapturecertainkindsofsystemrequirementsandtodocumentinternalsystemdesign.Theyareusedtorecordcomplexbusinessrulesthatasystemmustimplement.Inaddition,theycanserveasaguidetocreatingtestcases.

Decisiontablesareavitaltoolinthetester'spersonaltoolbox.Unfortunately,manyanalysts,designers,programmers,andtestersarenotfamiliarwiththistechnique.

TechniqueDecisiontablesrepresentcomplexbusinessrulesbasedonasetofconditions.Thegeneralformis:

Table5-1:Thegeneralformofadecisiontable.

Rule1 Rule2 … Rulep

Conditions

Condition-1

Condition-2

…

Condition-m

Actions

Action-1

Action-2

…

Action-n

Conditions1throughmrepresentvariousinputconditions.Actions1throughnaretheactionsthatshouldbetakendependingonthevariouscombinationsofinputconditions.Eachoftherulesdefinesauniquecombinationofconditionsthatresultintheexecution("firing")oftheactionsassociatedwiththatrule.Notethattheactionsdonotdependontheorderinwhichtheconditionsareevaluated,butonlyontheirvalues.(Allvaluesareassumedtobeavailablesimultaneously.)Also,actionsdependonlyonthespecifiedconditions,notonanypreviousinputconditionsorsystemstate.

Perhapsaconcreteexamplewillclarifytheconcepts.Anautoinsurancecompanygivesdiscountstodriverswhoaremarriedand/orgoodstudents.Let'sbeginwiththeconditions.Thefollowingdecisiontablehastwoconditions,eachoneofwhichtakesonthevaluesYesorNo.

Table5-2:Adecisiontablewithtwobinaryconditions.

Rule1 Rule2 Rule3 Rule4

Conditions

Married? Yes Yes No No

GoodStudent? Yes No Yes No

Notethatthetablecontainsallcombinationsoftheconditions.Giventwobinaryconditions(YesorNo),thepossiblecombinationsare{Yes,Yes},{Yes,No},{No,Yes},and{No,No}.Eachrulerepresentsoneofthesecombinations.Asatesterwewillverifythatallcombinationsoftheconditionsaredefined.Missingacombinationmayresultindevelopingasystemthatmaynotprocessaparticularsetofinputsproperly.

Nowfortheactions.Eachrulecausesanactionto"fire."Eachrulemayspecifyanactionuniquetothatrule,orrulesmayshareactions.

Table5-3:Addingasingleactiontoadecisiontable.


Conditions

Married? Yes Yes No No

GoodStudent? Yes No Yes No

Actions

Discount($) 60 25 50 0

Decisiontablesmayspecifymorethanoneactionforeachrule.Again,theserulesmaybeuniqueormaybeshared.

Table5-4:Adecisiontablewithmultipleactions.


Conditions

Condition-1 Yes Yes No No

Condition-2 Yes No Yes No

Actions

Action-1 DoX DoY DoX DoZ

Action-2 DoA DoB DoB DoB

Inthissituation,choosingtestcasesissimple—eachrule(verticalcolumn)becomesatestcase.TheConditionsspecifytheinputsandtheActionsspecifytheexpectedresults.

Whilethepreviousexampleusessimplebinaryconditions,conditionscanbemorecomplex.

Table5-5:Adecisiontablewithnon-binaryconditions.


Conditions

Condition-1 0–1 1–10 10–100 100–1000

Condition-2 <5 5 6or7 >7

Actions



Inthissituationchoosingtestcasesisslightlymorecomplex—eachrule(verticalcolumn)becomesatestcasebutvaluessatisfyingtheconditionsmustbechosen.Choosingappropriatevalueswecreatethefollowingtestcases:

Table5-6:Sampletestcases.

TestCaseID Condition-1 Condition-2 ExpectedResult

TC1 0 3 DoX/DoA

TC2 5 5 DoY/DoB

TC3 50 7 DoX/DoB

TC4 500 10 DoZ/DoB

Ifthesystemundertesthascomplexbusinessrules,andifyourbusinessanalystsordesignershavenotdocumentedtheserulesinthisform,testersshouldgatherthisinformationandrepresentitindecisiontableform.Thereasonissimple.Giventhesystembehaviorrepresentedinthiscompleteandcompactform,testcasescanbecreateddirectlyfromthedecisiontable.

Intesting,createatleastonetestcaseforeachrule.Iftherule'sconditionsarebinary,asingletestforeachcombinationisprobablysufficient.Ontheotherhand,ifaconditionisarangeofvalues,considertestingatboththelowandhighendoftherange.InthiswaywemergetheideasofBoundaryValuetestingwithDecisionTabletesting.

KeyPoint Createatleastonetestcaseforeachrule.

Tocreateatestcasetablesimplychangetherowandcolumnheadings:

Table5-7:Adecisiontableconvertedtoatestcasetable.

TestCase1 TestCase2 TestCase3 TestCase4

Inputs

Condition-1 Yes Yes No No

Condition-2 Yes No Yes No

ExpectedResults



ExamplesDecisionTabletestingcanbeusedwheneverthesystemmustimplementcomplexbusinessrules.Considerthefollowingtwoexamples:Example1

ReferringtotheTradeWebpageoftheBrown&DonaldsonWebsitedescribedinAppendixA,considertherulesassociatedwithaBuyorder.

Table5-8:AdecisiontablefortheBrown&DonaldsonBuyorder.

Rule1 Rule2 Rule3 Rule4 Rule5 Rule6 Rule7 Rule8

Conditions

ValidSymbol No No No No Yes Yes Yes Yes

ValidQuantity No No Yes Yes No No Yes Yes

SufficientFunds No Yes No Yes No Yes No Yes

Actions

Buy? No No No No No No No Yes

Admittedly,theoutcomeisreadilyapparent.Onlywhenavalidsymbol,validquantity,andsufficientfundsareavailableshouldtheBuyorderbeplaced.Thisexamplewaschosentoillustrateanotherconcept.

Examinethefirstfourcolumns.IftheSymbolisnotvalid,noneoftheotherconditionsmatter.Oftentableslikethisarecollapsed,rulesarecombined,andtheconditionsthatdonotaffecttheoutcomearemarked"DC"for"Don'tCare."Rule1nowindicatesthatiftheSymbolisnotvalid,ignoretheotherconditionsanddonotexecutetheBuyorder.

Table5-9:Acollapseddecisiontablereflecting"Don'tCare"conditions.

Rule1 Rule2 Rule3 Rule4 Rule5

Conditions

ValidSymbol No Yes Yes Yes Yes

ValidQuantity DC No No Yes Yes

SufficientFunds DC No Yes No Yes

Actions

Buy? No No No No Yes

NotealsothatRule2andRule3canbecombinedbecausewhetherSufficientFundsare

availabledoesnotaffecttheaction.

Table5-10:Afurthercollapseddecisiontablereflecting"Don'tCare"conditions.


Conditions

ValidSymbol No Yes Yes Yes

ValidQuantity DC No Yes Yes

SufficientFunds DC DC No Yes

Actions

Buy? No No No Yes

Whilethisisanexcellentideafromadevelopmentstandpointbecauselesscodeiswritten,itisdangerousfromatestingstandpoint.Itisalwayspossiblethatthetablewascollapsedincorrectlyorthecodewaswrittenimproperly.Theun-collapsedtableshouldalwaysbeusedasthebasisforourtestcasedesign.

Example2

ThefollowingscreenisfromtheStatelessUniversityRegistrationSystem.Itisusedtoenternewstudentsintothesystem,tomodifystudentinformation,andtodeletestudentsfromthesystem.

Figure5-1:SURSStudentDatabaseMaintenanceScreen.

Toenteranewstudent,entername,address,andtelephoneinformationontheupperpartofthescreenandpressEnter.ThestudentisenteredintothedatabaseandthesystemreturnsanewStudentID.Tomodifyordeleteastudent,entertheStudentID,selecttheDeleteorModifyradiobuttonandpressEnter.Thedecisiontablereflectingtheserulesfollows:

Table5-11:AdecisiontableforStatelessUniversityRegistrationSystem.


Rule1

Rule2

Rule3

Rule4

Rule5

Rule6

Rule7

Rule8

Rule9

Rule10

Rule11

Rule12

Rule13

Rule14

Rule

Conditions

EnteredStudentdata

No No No No No No No No Yes Yes Yes Yes Yes Yes Yes

EnteredStudentID

No No No No Yes Yes Yes Yes No No No No Yes Yes Yes

SelectedModify No No Yes Yes No No Yes Yes No No Yes Yes No No Yes

SelectedDelete No Yes No Yes No Yes No Yes No Yes No Yes No Yes

Actions

Createnewstudent

No No No No No No No No Yes No No No No No

ModifyStudent No No No No No No Yes No No No Yes No No No

DeleteStudent No No No No No Yes No No No No No No No No

Rules1through8indicatethatnodatawasenteredaboutthestudent.Rules1through4indicatethatnoStudentIDwasenteredforthestudent,thusnoactionispossible.Rules5through8indicatetheStudentIDwasentered.InthesecasescreatinganewStudentisnotproper.Rule5doesnotrequesteithermodificationordeletionsoneitherisdone.Rules6and7requestonefunctionandsotheyareperformed.NotethatRule8indicatesthatbothmodificationanddeletionaretobeperformedsonoactionistaken.

Rules9through16indicatethatdatawasenteredaboutthestudent.Rules9through12indicatethatnoStudentIDwasenteredsotheserulesrefertoanewstudent.Rule9createsanewstudent.Rule10deletesthestudent.Rule11allowsmodificationofthestudent'sdata.Rule12requeststhatbothmodificationanddeletionaretobeperformedsonoactionistaken.Rules13through16supplystudentdataindicatinganewstudentbutalsoprovideaStudentIDindicatinganexistingstudent.Becauseofthiscontradictoryinput,noactionistaken.Often,errormessagesaredisplayedinthesesituations.

ApplicabilityandLimitationsDecisionTabletestingcanbeusedwheneverthesystemmustimplementcomplexbusinessruleswhentheserulescanberepresentedasacombinationofconditionsandwhentheseconditionshavediscreteactionsassociatedwiththem.

SummaryDecisiontablesareusedtodocumentcomplexbusinessrulesthatasystemmustimplement.Inaddition,theyserveasaguidetocreatingtestcases.

Conditionsrepresentvariousinputconditions.Actionsaretheprocessesthatshouldbeexecuteddependingonthevariouscombinationsofinputconditions.Eachruledefinesauniquecombinationofconditionsthatresultintheexecution("firing")oftheactionsassociatedwiththatrule.

Createatleastonetestcaseforeachrule.Iftherule'sconditionsarebinary,asingletestforeachcombinationisprobablysufficient.Ontheotherhand,ifaconditionisarangeofvalues,considertestingatboththelowandhighendoftherange.

Practice1. AttendingStatelessUniversityisanexpensiveproposition.Afterall,theyreceiveno

statefunding.Likemanyotherstudents,thoseplanningonattendingapplyforstudentaidusingFAFSA,theFreeApplicationforFederalStudentAid.Thefollowinginstructionsweretakenfromthatform.ExaminethemandcreateadecisiontablethatrepresentstheFAFSArules.(Note:Youcan'tmakeupstufflikethis.)

StepFour:Whoisconsideredaparentinthisstep?

Readthesenotestodeterminewhoisconsideredaparentforpurposesofthisform.AnswerallquestionsinStepFouraboutthem,evenifyoudonotlivewiththem.

Areyouanorphan,orareyouorwereyou(untilage18)award/dependentofthecourt?IfYes,skipStepFour.Ifyourparentsarebothlivingandmarriedtoeachother,answerthequestionsaboutthem.Ifyourparentiswidowedorsingle,answerthequestionsaboutthatparent.Ifyourwidowedparentisremarriedasoftoday,answerthequestionsaboutthatparentandthepersonwhomyourparentmarried(yourstepparent).Ifyourparentsaredivorcedorseparated,answerthequestionsabouttheparentyoulivedwithmoreduringthepast12months.(Ifyoudidnotlivewithoneparentmorethantheother,giveanswersabouttheparentwhoprovidedmorefinancialsupportduringthelast12months,orduringthemostrecentyearthatyouactuallyreceivedsupportfromaparent.)Ifthisparentisremarriedasoftoday,answerthequestionsontherestofthisformaboutthatparentandthepersonwhomyourparentmarried(yourstepparent).

ReferencesBeizer,Boris(1990).SoftwareTestingTechniques(SecondEdition).VanNostrandReinhold.


Chapter6:PairwiseTestingAntonwasattractedtoAngelalikeamothtoaflame-notjustanymoth,butoneofthegiantsilkmothsofthegenusHyalophora,perhapsHyalophoraeuryalus,whosegreatred-brownwingswithwhitebasalandpostmedianlinesflapalmostlanguorouslyuntiloneignitesintheflame,fanningtheconflagrationtoevergreaterheightsuntilburningdowntothehirsutethoraxandabdomen,thefat-ladencontentsofwhichprovideasatisfyingsizzletoendtheagony.

—AndrewAmlen

IntroductionAstheyusedtosayonMontyPython,"Andnowforsomethingcompletelydifferent."

Considerthesesituations:

AWebsitemustoperatecorrectlywithdifferentbrowsers—InternetExplorer5.0,5.5,and6.0,Netscape6.0,6.1,and7.0,Mozilla1.1,andOpera7;usingdifferentplug-ins—RealPlayer,MediaPlayer,ornone;runningondifferentclientoperatingsystems—Windows95,98,ME,NT,2000,andXP;receivingpagesfromdifferentservers—IIS,Apache,andWebLogic;runningondifferentserveroperatingsystems—WindowsNT,2000,andLinux.

WebCombinations

8browsers

3plug-ins

6clientoperatingsystems

3servers

3serverOS

1,296combinations.

Abankhascreatedanewdataprocessingsystemthatisreadyfortesting.Thisbankhasdifferentkindsofcustomers—consumers,veryimportantconsumers,businesses,andnon-profits;differentkindsofaccounts—checking,savings,mortgages,consumerloans,andcommercialloans;theyoperateindifferentstates,eachwithdifferentregulations—California,Nevada,Utah,Idaho,Arizona,andNewMexico.

BankCombinations

4customertypes

5accounttypes

6states

120combinations.

Inanobject-orientedsystem,anobjectofclassAcanpassamessagecontainingaparameterPtoanobjectofclassX.ClassesB,C,andDinheritfromAsotheytoocansendthemessage.ClassesQ,R,S,andTinheritfromPsotheytoocanbepassedastheparameter.ClassesYandZinheritfromXsotheytoocanreceivethemessage.

OOCombinations

4senders

5parameters

3receivers

60combinations.

InsightStudentsinmyclassesoftenhaveaverydifficulttimethinkingofbadwaystodothings.Cultivatetheskillofchoosingpoorly.Itwillbeinvaluableinevaluatingothers'ideas.

CanYouBelieveThis?

Astudentinoneofmyclassessharedthisstory:Hisorganizationusesaprocesstheycall"Post-InstallationTestPlanning."Itsoundsimpressiveuntilyoudecipherit.WhateverteststheyhappentorunthathappentopassaredocumentedastheirTestPlan.

Whatdotheseverydifferentsituationsallhaveincommon?Eachhasalargenumberofcombinationsthatshouldbetested.Eachhasalargenumberofcombinationsthatmayberiskyifwedonottest.Eachhassuchalargenumberofcombinationsthatwemaynothavetheresourcestoconstructandrunallthetests,therearejusttoomany.Wemust,somehow,selectareasonablysizedsubsetthatwecouldtestgivenourresourceconstraints.Whataresomewaysofchoosingsuchasubset?Thisliststartswiththeworstschemesbutdoesimprove:

Don'ttestatall.Simplygiveupbecausethenumberofinputcombinations,andthusthenumberoftestcases,isjusttoogreat.

Testallcombinations[once],butdelaytheprojectsoitmissesitsmarketwindowsothateveryonequitsfromstress,orthecompanygoesoutofbusiness.

Chooseoneortwotestsandhopeforthebest.

Choosetheteststhatyouhavealreadyrun,perhapsaspartofprogrammer-ledtesting.Incorporatethemintoaformaltestplanandrunthemagain.

Choosetheteststhatareeasytocreateandrun.Ignorewhethertheyprovideusefulinformationaboutthequalityoftheproduct.

Makealistofallthecombinationsandchoosethefirstfew.

Makealistofallthecombinationsandchoosearandomsubset.

Bymagic,chooseaspeciallyselected,fairlysmallsubsetthatfindsagreatmanydefects—morethanyouwouldexpectfromsuchasubset.

Thislastschemesoundslikeawinner(butitisalittlevague).Thequestionis—whatisthe"magic"thatallowsustochoosethat"speciallyselected"subset?

Insight Randomselectioncanbeaverygoodapproachtochoosingasubsetbutmostpeoplehaveadifficulttimechoosingtrulyrandomly.

Theanswerisnottoattempttotestallthecombinationsforallthevaluesforallthevariablesbuttotestallpairsofvariables.Thissignificantlyreducesthenumberofteststhatmustbecreatedandrun.Considerthesignificantreductionsintesteffortintheseexamples:

Ifasystemhadfourdifferentinputparametersandeachonecouldtakeononeofthreedifferentvalues,thenumberofcombinationsis34whichis81.Itispossibletocoverallthepairwiseinputcombinationsinonlyninetests.

Ifasystemhadthirteendifferentinputparametersandeachonecouldtakeononeof

threedifferentvalues,thenumberofcombinationsis313whichis1,594,323.Itispossibletocoverallthepairwiseinputcombinationsinonlyfifteentests.

Ifasystemhadtwentydifferentinputparametersandeachonecouldtakeononeoftendifferentvalues,thenumberofcombinationsis1020.Itispossibletocoverallthepairwiseinputcombinationsinonly180tests.

Thereismuchanecdotalevidenceaboutthebenefitofpairwisetesting.Unfortunately,thereareonlyafewdocumentedstudies:

InacasestudypublishedbyBrownlieofAT&Tregardingthetestingofalocal-areanetwork-basedelectronicmailsystem,pairwisetestingdetected28percentmoredefectsthantheiroriginalplanofdevelopingandexecuting1,500testcases(laterreducedto1,000becauseoftimeconstraints)andtook50percentlesseffort.

AstudybytheNationalInstituteofStandardsandTechnologypublishedbyWallaceandKuhnonsoftwaredefectsinrecalledmedicaldevicesreviewedfifteenyearsofdefectdata.Theyconcludedthat98percentofthereportedsoftwareflawscouldhavebeendetectedbytestingallpairsofparametersettings.

KuhnandReillyanalyzeddefectsrecordedintheMozillaWebbrowserdatabase.Theydeterminedthatpairwisetestingwouldhavedetected76percentofthereportederrors.

Whydoespairwisetestingworksowell?Idon'tknow.Thereisnounderlying"softwarephysics"thatrequiresit.Onehypothesisisthatmostdefectsareeithersingle-modedefects(thefunctionundertestsimplydoesnotworkandanytestofthatfunctionwouldfindthedefect)ortheyaredouble-modedefects(itisthepairingofthisfunction/modulewiththatfunction/modulethatfailseventhoughallotherpairingsperformproperly).Pairwisetestingdefinesaminimalsubsetthatguidesustotestforallsingle-modeanddouble-modedefects.Thesuccessofthistechniqueonmanyprojects,bothdocumentedandundocumented,isagreatmotivationforitsuse.

NotePairwisetestingmaynotchoosecombinationswhichthedevelopersandtestersknowareeitherfrequentlyusedorhighlyrisky.Ifthesecombinationsexist,usethepairwisetests,thenaddadditionaltestcasestominimizetheriskofmissinganimportantcombination.

TechniqueTwodifferenttechniquesareusedtoidentifyallthepairsforcreatingtestcases—orthogonalarraysandtheAllpairsalgorithm.

OrthogonalArrays

Whatareorthogonalarrays?TheoriginoforthogonalarrayscanbetracedbacktoEuler,thegreatmathematician,intheguiseofLatinSquares.GenichiTaguchihaspopularizedtheiruseinhardwaretesting.AnexcellentreferencebookisQualityEngineeringUsingRobustDesignbyMadhavS.Phadke.

Considerthenumbers1and2.Howmanypaircombinations(combinationstakentwoatatime)of'1'and'2'exist?{1,1},{1,2},{2,1}and{2,2}.Anorthogonalarrayisatwo-dimensionalarrayofnumbersthathasthisinterestingproperty—chooseanytwocolumnsinthearray.Allthepairwisecombinationsofitsvalueswilloccurineverypairofcolumns.Let'sexamineanL4(23)array:

Table6-1:L4(23)OrthogonalArray

1 2 3

1 1 1 1

2 1 2 2

3 2 1 2

4 2 2 1

Thegraycolumnheadingsandrownumbersarenotpartoftheorthogonalarraybutareincludedforconvenienceinreferencingthecells.Examinecolumns1and2—dothefourcombinationsof1and2allappearinthatcolumnpair?Yes,andintheorderlistedearlier.Nowexaminecolumns1and3—dothefourcombinationsof1and2appearinthatcolumnpair?Yes,althoughinadifferentorder.Finally,examinecolumns2and3—dothefourcombinationsappearinthatcolumnpairalso?Yestheydo.TheL4(23)arrayisorthogonal;thatis,chooseanytwocolumns,allthepairwisecombinationswilloccurinallthecolumnpairs.

ImportantNote

Asatesteryoudonothavetocreateorthogonalarrays,allyoumustdoislocateoneofthepropersize.Books,Websites,andautomatedtoolswillhelpyoudothis.

Anoteaboutthecurious(butstandard)notation:L4meansanorthogonalarraywithfourrows,(23)isnotanexponent.Itmeansthatthearrayhasthreecolumns,eachwitheithera1ora2.

Figure6-1:Orthogonalarraynotation Let'sconsideralargerorthogonalarray.Giventhenumbers1,2and3,howmanypaircombinationsof1,2,and3exist?{1,1},{1,2},{1,3},{2,1},{2,2},{2,3},{3,1},{3,2},and{3,3}.BelowisanL9(34)array:


1 2 3 4

1 1 1 1 1

2 1 2 2 2

3 1 3 3 3

4 2 1 2 3

5 2 2 3 1

6 2 3 1 2

7 3 1 3 2

8 3 2 1 3

9 3 3 2 1

Examinecolumns1and2—dotheninecombinationsof1,2,and3allappearinthatcolumnpair?Yes.Nowexaminecolumns1and3—dotheninecombinationsof1,2,and3appearinthatcolumnpair?Yes,althoughinadifferentorder.Examinecolumns1and4—dotheninecombinationsappearinthatcolumnpairalso?Yestheydo.Continueonbyexaminingotherpairsofcolumns—2and3,2and4,andfinally3and4.TheL9(34)arrayisorthogonal;thatis,chooseanytwocolumns,allthecombinationswilloccurinallofthecolumnpairs.

Tool

TherdExperttoolfromPhadkeAssociatesimplementstheorthogonalarrayapproach.Seehttp://www.phadkeassociates.com

Notethatnotallcombinationsof1s,2s,and3sappearinthearray.Forexample,{1,1,2},{1,2,1},and{2,2,2)donotappear.Orthogonalarraysonlyguaranteethatallthepaircombinationsexistinthearray.Combinationssuchas{2,2,2}aretriples,notpairs.

http://www.phadkeassociates.com

ThefollowingisanL18(35)orthogonalarray.Ithasfivecolumns,eachcontaininga1,2,or3.Examinecolumns1and2forthepair{1,1}.Doesthatpairexistinthosetwocolumns?Wait!Don'tlookatthearray.Fromthedefinitionofanorthogonalarray,whatistheanswer?Yes,thatpairexistsalongwitheveryotherpairof1,2,and3.Thepair{1,1}isinrow1.Notethat{1,1}alsoappearsinrow6.Returningtotheoriginaldescriptionoforthogonalarrays,

Anorthogonalarrayisatwo-dimensionalarrayofnumbersthathasthisinterestingproperty—chooseanytwocolumnsinthearray.Allthepairwisecombinationsofitsvalueswilloccurineverycolumnpair.

Thisdefinitionisnottotallycomplete.Notonlywillallthepaircombinationsoccurinthearray,butifanypairoccursmultipletimes,allpairswilloccurthatsamenumberoftimes.Thisisbecauseorthogonalarraysare"balanced."Examinecolumns3and5—lookfor{3,2}.Thatcombinationappearsinrows6and17.


1 2 3 4 5

1 1 1 1 1 1

2 1 2 3 3 1

3 1 3 2 3 2

4 1 2 2 1 3

5 1 3 1 2 3

6 1 1 3 2 2

7 2 2 2 2 2

8 2 3 1 1 2

9 2 1 3 1 3

10 2 3 3 2 1

11 2 1 2 3 1

12 2 2 1 3 3

13 3 3 3 3 3

14 3 1 2 2 3

15 3 2 1 2 1

16 3 1 1 3 2

17 3 2 3 1 2

18 3 3 2 1 1

Inorthogonalarraysnotallofthecolumnsmusthavethesamerangeofvalues(1..2,1..3,1..5,etc.).Someorthogonalarraysaremixed.ThefollowingisanL18(2137)orthogonalarray.Ithasonecolumnof1sand2s,andsevencolumnsof1s,2s,and3s.


1 2 3 4 5 6 7 8

1 1 1 1 1 1 1 1 1

2 1 1 2 2 2 2 2 2

3 1 1 3 3 3 3 3 3

4 1 2 1 1 2 2 3 3

5 1 2 2 2 3 3 1 1

6 1 2 3 3 1 1 2 2

7 1 3 1 2 1 3 2 3

8 1 3 2 3 2 1 3 1

9 1 3 3 1 3 2 1 2

10 2 1 1 3 3 2 2 1

11 2 1 2 1 1 3 3 2

12 2 1 3 2 2 1 1 3

13 2 2 1 2 3 1 3 2

14 2 2 2 3 1 2 1 3

15 2 2 3 1 2 3 2 1

16 2 3 1 3 2 3 1 2

17 2 3 2 1 3 1 2 3

18 2 3 3 2 1 2 3 1

Reference

NeilJ.A.Sloanemaintainsaverycomprehensivecatalogoforthogonalarraysathttp://www.research.att.com/~njas/oadir/index.html

UsingOrthogonalArrays

Theprocessofusingorthogonalarraystoselectpairwisesubsetsfortestingis:1. Identifythevariables.

http://www.research.att.com/~njas/oadir/index.html

2. Determinethenumberofchoicesforeachvariable.

3. Locateanorthogonalarraywhichhasacolumnforeachvariableandvalueswithinthecolumnsthatcorrespondtothechoicesforeachvariable.

4. Mapthetestproblemontotheorthogonalarray.

5. Constructthetestcases.

Ifthisseemsrathervagueatthispointit'stimeforanexample.

Web-basedsystemssuchasBrown&DonaldsonandtheStatelessUniversityRegistrationSystemmustoperateinanumberofenvironments.Let'sexecutetheprocessstep-by-stepusinganorthogonalarraytochoosetestcases.ConsiderthefirstexampleintheintroductiondescribingthesoftwarecombinationsaWebsitemustoperatewith.

1. Identifythevariables.

ThevariablesareBrowser,Plug-in,Clientoperatingsystem,Server,andServeroperatingsystem.

2. Determinethenumberofchoicesforeachvariable.

Browser-InternetExplorer5.0,5.5,and6.0,Netscape6.0,6.1,and7.0,Mozilla1.1,andOpera7(8choices).

Plug-in-None,RealPlayer,andMediaPlayer(3choices).

Clientoperatingsystem-Windows95,98,ME,NT,2000,andXP(6choices).

Server-IIS,Apache,andWebLogic(3choices).

Serveroperatingsystem-WindowsNT,2000,andLinux(3choices).

Multiplying8x3x6x3x3wefindthereare1,296combinations.For"complete"testcoverage,eachofthesecombinationsshouldbetested.

3. Locateanorthogonalarraythathasacolumnforeachvariableandvalueswithinthecolumnsthatcorrespondtothechoicesofeachvariable.

Whatsizearrayisneeded?First,itmusthavefivecolumns,oneforeachvariableinthisexample.Thefirstcolumnmustsupporteightdifferentlevels(1through8).Thesecondcolumnmustsupportthreelevels(1through3).Thethirdrequiressixlevels.Thefourthandthefiftheachrequirethreelevels.Theperfectsizeorthogonalarraywouldbe816133(onecolumnof1through8,onecolumnof1through6,andthreecolumnsof1through3).Unfortunately,oneofthisexactsizedoesnotexist.Whenthisoccurs,wesimplypickthenextlargerarray.

ImportantAsatesteryoudonothavetocreateorthogonalarrays.Allyou

Note mustdoislocateoneofthepropersizeandthenperformthemappingofthetestproblemontothearray.

Thefollowingorthogonalarraymeetsourrequirements.It'sanL64(8243)array.OrthogonalarrayscanbefoundinanumberofbooksandontheWeb.AfavoritebookisQualityEngineeringUsingRobustDesignbyMadhavS.Phadke.Inaddition,anexcellentcatalogismaintainedontheWebbyNeilJ.A.SloaneofAT&T.Seehttp://www.research.att.com/~njas/oadir/index.html.

Therequirementof8161(onecolumnof1through8and1columnof1through6)ismetby82(twocolumnsof1through8).Therequirementof33(threecolumnsof1through3)ismetby43(threecolumnsof1through4).

Thenumberofcombinationsofallthevaluesofallthevariablesis1,296andthus1,296testcasesshouldbecreatedandrunforcompletecoverage.Usingthisorthogonalarray,allpairsofallthevaluesofallthevariablescanbecoveredinonlysixty-fourtests,a95percentreductioninthenumberoftestcases.


1 2 3 4 5

1 1 1 1 1 1

2 1 4 3 4 4

3 1 4 2 4 4

4 1 1 4 1 1

5 1 3 5 3 3

6 1 2 7 2 2

7 1 2 6 2 2

8 1 3 8 3 3

9 3 4 1 3 3

10 3 1 3 2 2

11 3 1 2 2 2

12 3 4 4 3 3

13 3 2 5 1 1

14 3 3 7 4 4

15 3 3 6 4 4

16 3 2 8 1 1

http://www.research.att.com/~njas/oadir/index.html

17 2 3 1 2 1

18 2 2 3 3 4

19 2 2 2 3 4

20 2 3 4 2 1

21 2 1 5 4 3

22 2 4 7 1 2

23 2 4 6 1 2

24 2 1 8 4 3

25 4 2 1 4 3

26 4 3 3 1 2

27 4 3 2 1 2

28 4 2 4 4 3

29 4 4 5 2 1

30 4 1 7 3 4

31 4 1 6 3 4

32 4 4 8 2 1

33 5 2 1 4 2

34 5 3 3 1 3

35 5 3 2 1 3

36 5 2 4 4 2

37 5 4 5 2 4

38 5 1 7 3 1

39 5 1 6 3 1

40 5 4 8 2 4

41 7 3 1 2 4

42 7 2 3 3 1

43 7 2 2 3 1

44 7 3 4 2 4

45 7 1 5 4 2

46 7 4 7 1 3

47 7 4 6 1 3

48 7 1 8 4 2

49 6 4 1 3 2

50 6 1 3 2 3

51 6 1 2 2 3

52 6 4 4 3 2

53 6 2 5 1 4

54 6 3 7 4 1

55 6 3 6 4 1

56 6 2 8 1 4

57 8 1 1 1 4

58 8 4 3 4 1

59 8 4 2 4 1

60 8 1 4 1 4

61 8 3 5 3 2

62 8 2 7 2 3

63 8 2 6 2 3

64 8 3 8 3 2

4. Mapthetestproblemontotheorthogonalarray.

TheBrowserchoiceswillbemappedontocolumn1oftheorthogonalarray.Cellscontaininga1willrepresentIE5.0;cellswitha2willrepresentIE5.5;cellswitha3willrepresentIE6.0;etc.Themappingis:

1↔IE5.0

2↔IE5.5

3↔IE6.0

4↔Netscape6.0

5↔Netscape6.1

6↔Netscape7.0

7↔Mozilla1.1

8↔Opera7

Partiallyfillinginthefirstcolumngives:

Table6-6:L64(8243)withapartialmappingofitsfirstcolumn.

Browser 2 3 4 5

1 IE5.0 1 1 1 1

2 1 4 3 4 4

3 1 4 2 4 4

4 1 1 4 1 1

5 1 3 5 3 3

6 1 2 7 2 2

7 1 2 6 2 2

8 1 3 8 3 3

9 IE6.0 4 1 3 3

10 3 1 3 2 2

11 3 1 2 2 2

12 3 4 4 3 3

13 3 2 5 1 1

14 3 3 7 4 4

15 3 3 6 4 4

16 3 2 8 1 1

17 IE5.5 3 1 2 1

18 2 2 3 3 4

19 2 2 2 3 4

20 2 3 4 2 1

21 2 1 5 4 3

22 2 4 7 1 2

23 2 4 6 1 2

24 2 1 8 4 3

25 Net6.0 2 1 4 3

26 4 3 3 1 2

27 4 3 2 1 2

28 4 2 4 4 3

29 4 4 5 2 1

30 4 1 7 3 4

31 4 1 6 3 4

32 4 4 8 2 1

33 Net6.1 2 1 4 2

34 5 3 3 1 3

35 5 3 2 1 3

36 5 2 4 4 2

37 5 4 5 2 4

38 5 1 7 3 1

39 5 1 6 3 1

40 5 4 8 2 4

41 Moz1.1 3 1 2 4

42 7 2 3 3 1

43 7 2 2 3 1

44 7 3 4 2 4

45 7 1 5 4 2

46 7 4 7 1 3

47 7 4 6 1 3

48 7 1 8 4 2

49 Net7.0 4 1 3 2

50 6 1 3 2 3

51 6 1 2 2 3

52 6 4 4 3 2

53 6 2 5 1 4

54 6 3 7 4 1

55 6 3 6 4 1

56 6 2 8 1 4

57 Opera7 1 1 1 4

58 8 4 3 4 1

59 8 4 2 4 1

60 8 1 4 1 4

61 8 3 5 3 2

62 8 2 7 2 3

63 8 2 6 2 3

64 8 3 8 3 2

Isitclearwhatishappening?Incolumn1(whichwehavechosentorepresenttheBrowser)everycellcontaininga1isbeingreplacedwith"IE5.0."Everycellcontaininga2isbeingreplacedwith"IE5.5."Everycellcontainingan8isbeingreplacedwith"Opera7,"etc.

We'llcontinuebycompletingthemapping(replacement)ofallthecellsincolumn1.Notethatthemappingbetweenthevariablevaluesandthe1s,2s,and3sistotallyarbitrary.Thereisnologicalconnectionbetween"1"andIE5.0or"7"andMozilla1.1.But,althoughtheinitialassignmentisarbitrary,oncechosen,theassignmentsandusemustremainconsistentwithineachcolumn.

Table6-7:L64(8243)withafullmappingofitsfirstcolumn.

Browser 2 3 4 5

1 IE5.0 1 1 1 1

2 IE5.0 4 3 4 4

3 IE5.0 4 2 4 4

4 IE5.0 1 4 1 1

5 IE5.0 3 5 3 3

6 IE5.0 2 7 2 2

7 IE5.0 2 6 2 2

8 IE5.0 3 8 3 3

9 IE6.0 4 1 3 3

10 IE6.0 1 3 2 2

11 IE6.0 1 2 2 2

12 IE6.0 4 4 3 3

13 IE6.0 2 5 1 1

14 IE6.0 3 7 4 4

15 IE6.0 3 6 4 4

16 IE6.0 2 8 1 1

17 IE5.5 3 1 2 1

18 IE5.5 2 3 3 4

19 IE5.5 2 2 3 4

20 IE5.5 3 4 2 1

21 IE5.5 1 5 4 3

22 IE5.5 4 7 1 2

23 IE5.5 4 6 1 2

24 IE5.5 1 8 4 3

25 Net6.0 2 1 4 3

26 Net6.0 3 3 1 2

27 Net6.0 3 2 1 2

28 Net6.0 2 4 4 3

29 Net6.0 4 5 2 1

30 Net6.0 1 7 3 4

31 Net6.0 1 6 3 4

32 Net6.0 4 8 2 1

33 Net6.1 2 1 4 2

34 Net6.1 3 3 1 3

35 Net6.1 3 2 1 3

36 Net6.1 2 4 4 2

37 Net6.1 4 5 2 4

38 Net6.1 1 7 3 1

39 Net6.1 1 6 3 1

40 Net6.1 4 8 2 4

41 Moz1.1 3 1 2 4

42 Moz1.1 2 3 3 1

43 Moz1.1 2 2 3 1

44 Moz1.1 3 4 2 4

45 Moz1.1 1 5 4 2

46 Moz1.1 4 7 1 3

47 Moz1.1 4 6 1 3

48 Moz1.1 1 8 4 2

49 Net7.0 4 1 3 2

50 Net7.0 1 3 2 3

51 Net7.0 1 2 2 3

52 Net7.0 4 4 3 2

53 Net7.0 2 5 1 4

54 Net7.0 3 7 4 1

55 Net7.0 3 6 4 1

56 Net7.0 2 8 1 4

57 Opera7 1 1 1 4

58 Opera7 4 3 4 1

59 Opera7 4 2 4 1

60 Opera7 1 4 1 4

61 Opera7 3 5 3 2

62 Opera7 2 7 2 3

63 Opera7 2 6 2 3

64 Opera7 3 8 3 2

Nowthatthefirstcolumnhasbeenmapped,let'sproceedtothenextone.ThePlug-inchoiceswillbemappedontocolumn2ofthearray.Cellscontaininga1willrepresentNone(Noplug-in);cellswitha2willrepresentRealPlayer;cellswitha3willrepresentMediaPlayer;cellswitha4willnotbemappedatthepresenttime.Themappingis:

1↔None

2↔RealPlayer

3↔MediaPlayer

4↔Notused(atthistime)

Fillinginthesecondcolumngives:

Table6-8:L64(8243)withafullmappingofitsfirstandsecondcolumns.

Browser Plug-In 3 4 5

1 IE5.0 None 1 1 1

2 IE5.0 4 3 4 4

3 IE5.0 4 2 4 4

4 IE5.0 None 4 1 1

5 IE5.0 MediaPlayer 5 3 3

6 IE5.0 RealPlayer 7 2 2



9 IE6.0 4 1 3 3

10 IE6.0 None 3 2 2

11 IE6.0 None 2 2 2

12 IE6.0 4 4 3 3









21 IE5.5 None 5 4 3

22 IE5.5 4 7 1 2

23 IE5.5 4 6 1 2

24 IE5.5 None 8 4 3

25 Net6.0 RealPlayer 1 4 3

26 Net6.0 MediaPlayer 3 1 2



29 Net6.0 4 5 2 1

30 Net6.0 None 7 3 4

31 Net6.0 None 6 3 4

32 Net6.0 4 8 2 1





37 Net6.1 4 5 2 4

38 Net6.1 None 7 3 1

39 Net6.1 None 6 3 1

40 Net6.1 4 8 2 4

41 Moz1.1 MediaPlayer 1 2 4

42 Moz1.1 RealPlayer 3 3 1

43 Moz1.1 RealPlayer 2 3 1

44 Moz1.1 MediaPlayer 4 2 4

45 Moz1.1 None 5 4 2

46 Moz1.1 4 7 1 3

47 Moz1.1 4 6 1 3

48 Moz1.1 None 8 4 2

49 Net7.0 4 1 3 2

50 Net7.0 None 3 2 3

51 Net7.0 None 2 2 3

52 Net7.0 4 4 3 2





57 Opera7 None 1 1 4

58 Opera7 4 3 4 1

59 Opera7 4 2 4 1

60 Opera7 None 4 1 4

61 Opera7 MediaPlayer 5 3 2

62 Opera7 RealPlayer 7 2 3

63 Opera7 RealPlayer 6 2 3

64 Opera7 MediaPlayer 8 3 2

Nowthatthefirstandsecondcolumnshavebeenmapped,let'sproceedtomapthenextthreecolumnssimultaneously.

ThemappingforClientoperatingsystemis:1↔Windows95

2↔Windows98

3↔WindowsME

4↔WindowsNT

5↔Windows2000

6↔WindowsXP



ThemappingforServersis:1↔IIS

2↔Apache

3↔WebLogic


ThemappingforServeroperatingsystemis:1↔WindowsNT

2↔Windows2000

3↔Linux


Fillingintheremainderofthecolumnsgives:

Table6-9:L64(8243)withafullmappingofallitscolumns.

Browser Plug-in ClientOS Server ServerOS

1 IE5.0 None Win95 IIS WinNT

2 IE5.0 4 WinME 4 4

3 IE5.0 4 Win98 4 4

4 IE5.0 None WinNT IIS WinNT

5 IE5.0 MediaPlayer Win2000 WebLogic Linux

6 IE5.0 RealPlayer 7 Apache Win2000

7 IE5.0 RealPlayer WinXP Apache Win2000

8 IE5.0 MediaPlayer 8 WebLogic Linux

9 IE6.0 4 Win95 WebLogic Linux

10 IE6.0 None WinME Apache Win2000

11 IE6.0 None Win98 Apache Win2000

12 IE6.0 4 WinNT WebLogic Linux

13 IE6.0 RealPlayer Win2000 IIS WinNT


15 IE6.0 MediaPlayer WinXP 4 4

16 IE6.0 RealPlayer 8 US WinNT

17 IE5.5 MediaPlayer Win95 Apache WinNT

18 IE5.5 RealPlayer WinME WebLogic 4

19 IE5.5 RealPlayer Win98 WebLogic 4

20 IE5.5 MediaPlayer WinNT Apache WinNT

21 IE5.5 None Win2000 4 Linux

22 IE5.5 4 7 IIS Win2000

23 IE5.5 4 WinXP IIS Win2000

24 IE5.5 None 8 4 Linux

25 Net6.0 RealPlayer Win95 4 Linux

26 Net6.0 MediaPlayer WinME IIS Win2000

27 Net6.0 MediaPlayer Win98 IIS Win2000

28 Net6.0 RealPlayer WinNT 4 Linux

29 Net6.0 4 Win2000 Apache WinNT

30 Net6.0 None 7 WebLogic 4

31 Net6.0 None WinXP WebLogic 4

32 Net6.0 4 8 Apache WinNT

33 Net6.1 RealPlayer Win95 4 Win2000

34 Net6.1 MediaPlayer WinME IIS Linux

35 Net6.1 MediaPlayer Win98 IIS Linux

36 Net6.1 RealPlayer WinNT 4 Win2000

37 Net6.1 4 Win2000 Apache 4

38 Net6.1 None 7 WebLogic WinNT

39 Net6.1 None WinXP WebLogic 1WinNT

40 Net6.1 4 8 Apache 4

41 Moz1.1 MediaPlayer Win95 Apache 4

42 Moz1.1 RealPlayer WinME WebLogic WinNT

43 Moz1.1 RealPlayer Win98 WebLogic WinNT

44 Moz1.1 MediaPlayer WinNT Apache 4

45 Moz1.1 None Win2000 4 Win2000

46 Moz1.1 4 7 IIS Linux

47 Moz1.1 4 WinXP IIS Linux

48 Moz1.1 None 8 4 Win2000

49 Net7.0 4 Win95 WebLogic Win2000

50 Net7.0 None WinME Apache Linux

51 Net7.0 None Win98 Apache Linux

52 Net7.0 4 WinNT WebLogic Win2000

53 Net7.0 RealPlayer Win2000 IIS 4

54 Net7.0 MediaPlayer 7 4 WinNT

55 Net7.0 MediaPlayer WinXP 4 WinNT

56 Net7.0 RealPlayer 8 IIS 4

57 Opera7 None Win95 IIS 4

58 Opera7 4 WinME 4 WinNT

59 Opera7 4 Win98 4 WinNT

60 Opera7 None WinNT IIS 4

61 Opera7 MediaPlayer Win2000 WebLogic Win2000

62 Opera7 RealPlayer 7 Apache Linux

63 Opera7 RealPlayer WinXP Apache Linux

64 Opera7 MediaPlayer 8 WebLogic Win2000

Wereitnotforthefewcellsthatremainunassigned,themappingoftheorthogonalarray,andthustheselectionofthetestcases,wouldbecompleted.Whatabouttheunassignedcells—first,whydotheyexist?;second,whatshouldbedonewiththem?

Theunassignedcellsexistbecausetheorthogonalarraychosenwas"toobig."Theperfectsizewouldbean816133array;thatis,onecolumnthatvariesfrom1to8;onecolumnthatvariesfrom1to6;andthreecolumnsthatvaryfrom1to3.Unfortunately,thatspecificsizeorthogonalarraydoesnotexist.Orthogonalarrayscannotbeconstructedforanyarbitrarysizeparameters.Theycomeinfixed,"quantum"sizes.Youcanconstructone"thisbig";youcanconstructone"thatbig";butyoucannotnecessarilyconstructonein-between.FamousSoftwareTesterMickJaggergivesexcellentadviceregardingthis,"Youcan'talwaysgetwhatyouwant,Butifyoutrysometimes,Youjustmightfind,yougetwhatyouneed."

FamousSoftwareTester

Iftheperfectsizearraydoesnotexist,chooseonethatisslightlybiggerandapplythesetworulestodealwiththe"excess."Thefirstruledealswithextracolumns.Iftheorthogonalarraychosenhasmorecolumnsthanneededforaparticulartestscenario,simplydeletethem.Thearraywillremainorthogonal.Thesecondruledealswithextravaluesforavariable.Inthecurrentexample,column3runsfrom1to8butonly1through6isneeded.ItistemptingtodeletetherowsthatcontainthesecellsbutDON'T.The"orthogonalness"maybelost.Eachrowinthearrayexiststoprovideatleastonepaircombinationthatappearsnowhereelseinthearray.Ifyoudeletearow,youlosethattestcase.Insteadofdeletingthem,simplyconverttheextracellstovalidvalues.Someautomatedtoolsrandomlychoosefromthesetofvalidvaluesforeachcellwhileotherschooseonevalidvalueanduseitineverycellwithinacolumn.Eitherapproachisacceptable.Usingthissecondapproach,we'llcompletetheorthogonalarray.Notethatitmaybedifficulttomaintainthe"balanced"aspectofthearraywhenassigningvaluestotheseextracells.

Table6-10:L64(8243)withafullmappingofallitscolumnsincludingthe"extra"cells.

Browser Plug-in ClientOS Server ServerOS


2 IE5.0 None WinME IIS WinNT


4 IE5.0 None WinNT IIS WinNT


6 IE5.0 RealPlayer Win95 Apache Win2000

7 IE5.0 RealPlayer WinXP Apache Win2000


9 IE6.0 None Win95 WebLogic Linux

10 IE6.0 None WinME Apache Win2000

11 IE6.0 None Win98 Apache Win2000

12 IE6.0 None WinNT WebLogic Linux


14 IE6.0 MediaPlayer Win95 IIS WinNT

15 IE6.0 MediaPlayer WinXP IIS WinNT


17 IE5.5 MediaPlayer Win95 Apache WinNT

18 IE5.5 RealPlayer WinME WebLogic WinNT

19 IE5.5 RealPlayer Win98 WebLogic WinNT

20 IE5.5 MediaPlayer WinNT Apache WinNT

21 IE5.5 None Win2000 IIS Linux

22 IE5.5 None Win95 IIS Win2000

23 IE5.5 None WinXP IIS Win2000

24 IE5.5 None Win98 IIS Linux

25 Net6.0 RealPlayer Win95 IIS Linux

26 Net6.0 MediaPlayer WinME IIS Win2000

27 Net6.0 MediaPlayer Win98 IIS Win2000

28 Net6.0 RealPlayer WinNT IIS Linux

29 Net6.0 None Win2000 Apache WinNT

30 Net6.0 None Win95 WebLogic WinNT

31 Net6.0 None WinXP WebLogic WinNT


33 Net6.1 RealPlayer Win95 IIS Win2000

34 Net6.1 MediaPlayer WinME IIS Linux

35 Net6.1 MediaPlayer Win98 IIS Linux

36 Net6.1 RealPlayer WinNT IIS Win2000


38 Net6.1 None Win95 WebLogic WinNT

39 Net6.1 None WinXP WebLogic WinNT


41 Moz1.1 MediaPlayer Win95 Apache WinNT

42 Moz1.1 RealPlayer WinME WebLogic WinNT

43 Moz1.1 RealPlayer Win98 WebLogic WinNT

44 Moz1.1 MediaPlayer WinNT Apache WinNT

45 Moz1.1 None Win2000 IIS Win2000

46 Moz1.1 None Win95 IIS Linux

47 Moz1.1 None WinXP IIS Linux

48 Moz1.1 None Win98 IIS Win2000

49 Net7.0 None Win95 WebLogic Win2000

50 Net7.0 None WinME Apache Linux

51 Net7.0 None Win98 Apache Linux

52 Net7.0 None WinNT WebLogic Win2000

53 Net7.0 RealPlayer Win2000 IIS WinNT

54 Net7.0 MediaPlayer Win95 IIS WinNT

55 Net7.0 MediaPlayer WinXP IIS WinNT

56 Net7.0 RealPlayer Win98 IIS WinNT

57 Opera7 None Win95 IIS WinNT

58 Opera7 None WinME IIS WinNT

59 Opera7 None Win98 IIS WinNT

60 Opera7 None WinNT IIS WinNT


62 Opera7 RealPlayer Win95 Apache Linux

63 Opera7 RealPlayer WinXP Apache Linux


5. Constructthetestcases.

Now,allthatremainsistoconstructatestcaseforeachrowintheorthogonalarray.Notethatthearrayspecifiesonlytheinputconditions.Anoracle(usuallythetester)isrequiredtodeterminetheexpectedresultforeachtest.

AllpairsAlgorithm

Usingorthogonalarraysisonewaytoidentifyallthepairs.Asecondwayistouseanalgorithmthatgeneratesthepairsdirectlywithoutresortingtoan"external"devicelikeanorthogonalarray.

Reference

JamesBachprovidesatooltogenerateallpairscombinationsathttp://www.satisfice.com.ClickonTestMethodologyandlookforAllpairs.

WardCunninghamprovidesfurtherdiscussionandthesourcecodeforaJavaprogramtogenerateallpairscombinationsathttp://fit.c2.com/wiki.cgi?AllPairs.

JamesBachpresentsanalgorithmtogenerateallpairsinLessonsLearnedinSoftwareTesting.Inaddition,heprovidesaprogramcalled"Allpairs"thatwillgeneratetheallpairscombinations.Itisavailableathttp://www.satisfice.com.Clickon"TestMethodology"andlookforAllpairs.Let'sapplytheAllpairsalgorithmtothepreviousWebsitetestingproblem.

Afterdownloadingandunzipping,touseAllpairscreateatab-delimitedtableofthevariablesandtheirvalues.IfyouareaWindowsuser,theeasiestwayistolaunchExcel,enterthedataintothespreadsheet,andthenSaveAsa.txtfile.Thefollowingtablewascreatedandsavedasinput.txt.

Table6-11:InputtotheAllpairsprogram.

Browser ClientOS Plug-in Server ServerOS

IE5.0 Win95 None IIS WinNTIE5.5 Win98 RealPlayer Apache Win2000IE6.0 WinME MediaPlayer WebLogic LinuxNetscape6.0 WinNTNetscape6.1 Win2000Netscape7.0 WinXPMozilla1.1Opera7

http://www.satisfice.com

http://fit.c2.com/wiki.cgi?AllPairs

http://www.satisfice.com

ThenruntheAllpairsprogrambytyping:allpairsinput.txt>output.txtwhereoutput.txtwillcontainthelistofallpairstestcases.Thefollowingtablewascreated:

Table6-12:OutputfromtheAllpairsprogram.

Browser ClientOS Plug-in Server ServerOS

1 IE5.0 Win95 None IIS WinNT2 IE5.0 Win98 RealPlayer Apache Win20003 IE5.0 WinME MediaPlayer WebLogic Linux4 IE5.5 Win95 RealPlayer WebLogic WinNT5 IE5.5 Win98 None IIS Linux6 IE5.5 WinME None Apache Win20007 IE6.0 Win95 MediaPlayer Apache Linux8 IE6.0 Win98 RealPlayer IIS WinNT9 IE6.0 WinME None WebLogic Win200010 Netscape6.0 WinME RealPlayer IIS Linux11 Netscape6.0 WinNT MediaPlayer IIS Win200012 Netscape6.0 Win2000 None Apache WinNT13 Netscape6.1 WinNT None WebLogic Linux14 Netscape6.1 Win2000 MediaPlayer IIS Win200015 Netscape6.1 WinXP RealPlayer Apache WinNT16 Netscape7.0 WinNT RealPlayer Apache WinNT17 Netscape7.0 Win2000 MediaPlayer WebLogic Linux18 Netscape7.0 WinXP MediaPlayer IIS Win200019 Mozilla1.1 WinXP MediaPlayer WebLogic WinNT20 Mozilla1.1 Win98 MediaPlayer Apache Linux21 Mozilla1.1 Win95 RealPlayer IIS Win200022 Opera7 WinXP None WebLogic Linux23 Opera7 Win98 RealPlayer WebLogic Win200024 Opera7 WinME MediaPlayer Apache WinNT25 IE5.5 Win2000 RealPlayer ~WebLogic ~Linux26 IE5.5 WinNT MediaPlayer ~IIS ~WinNT27 Netscape6.0 Win95 ~None WebLogic ~Win2000

28 Netscape7.0 Win95 None ~Apache ~Linux29 Mozilla1.1 WinME None ~IIS ~WinNT30 Opera7 WinNT ~RealPlayer IIS ~Linux31 IE5.0 WinNT ~None ~Apache ~Win200032 IE5.0 Win2000 ~RealPlayer ~IIS ~WinNT33 IE5.0 WinXP ~None ~WebLogic ~Linux34 IE5.5 WinXP ~RealPlayer ~Apache ~Win200035 IE6.0 Win2000 ~None ~Apache ~Win200036 IE6.0 WinNT ~RealPlayer ~WebLogic ~WinNT37 IE6,0 WinXP ~MediaPlayer ~IIS ~Linux38 Netscape6.0 Win98 ~MediaPlayer ~WebLogic ~WinNT39 Netscape6.0 WinXP ~RealPlayer ~Apache ~Linux40 Netscape6.1 Win95 ~MediaPlayer ~Apache ~Win200041 Netscape6.1 Win98 ~None ~IIS ~WinNT42 Netscape6.1 WinME ~RealPlayer ~WebLogic ~Linux43 Netscape7.0 Win98 ~None ~WebLogic ~Win200044 Netscape7.0 WinME ~RealPlayer -US ~WinNT45 Mozilla1.1 WinNT ~None ~Apache ~Linux46 Mozilla1.1 Win2000 ~RealPlayer ~WebLogic ~Win200047 Opera7 Win95 ~MediaPlayer ~IIS ~WinNT48 Opera7 Win2000 ~None ~Apache ~Win2000

Whenaparticularvalueinthetestcasedoesn'tmatter,becauseallofitspairingshavealreadybeenselected,itismarkedwitha~.Bach'salgorithmchoosesthevaluethathasbeenpairedthefewesttimesrelativetotheothersinthetestcase.Anyothervaluecouldbesubstitutedforoneprefixedwitha~andallpairscoveragewouldstillbemaintained.Thismightbedonetotestmorecommonlyusedormorecriticalcombinationsmoreoften.Inaddition,Bach'sprogramdisplaysinformationonhowthepairingsweredone.Itlistseachpair,showshowmanytimesthatpairoccursinthetable,andindicateseachtestcasethatcontainsthatpair.

Becauseofthe"balanced"natureoforthogonalarrays,thatapproachrequiredsixty-fourtestcases.The"unbalanced"natureoftheallpairsselectionalgorithmrequiresonlyforty-eighttestcases,asavingsof25percent.

NotethatthecombinationschosenbytheOrthogonalArraymethodmaynotbethesameasthosechosenbyAllpairs.Itdoesnotmatter.Whatdoesmatteristhatallofthepair

combinationsofparametersarechosen.Thosearethecombinationswewanttotest.

ProponentsoftheAllpairsalgorithmpointoutthatgivenaproblemwith100parameters,eachcapableoftakingononeoftwovalues,101testcaseswouldberequiredusinga(balanced)orthogonalarraywhiletheun-balancedallpairsapproachrequiresonlytentests.Sincemanyapplicationshavelargenumbersofinputsthattakeononlyafewvalueseach,theyarguetheallpairsapproachissuperior.

Tool

TheAETGtoolfromTelcordiaimplementstheall-pairstestingapproach.Seehttp://aetgweb.argreenhouse.com.

FinalComments

Insomesituations,constraintsexistbetweencertainchoicesofsomeofthevariables.Forexample,Microsoft'sIISandApple'sMacOSarenotcompatible.Itiscertainthatthepairwisetechniqueswillchoosethatcombinationfortest.(Remember,itdoesselectallthepairs.)Whencreatingpairwisesubsetsbyhand,honoringthesevariousconstraintscanbedifficult.BoththerdExpertandAETGtoolshavethisability.Youdefinetheconstraintsandthetoolselectspairsmeetingthoseconstraints.

Giventhetwoapproachestopairwisetesting,orthogonalarraysandtheAllpairsalgorithm,whichismoreeffective?Oneexpert,whofavorsorthogonalarrays,believesthatthecoverageprovidedbyAllpairsissubstantiallyinferior.Henotesthattheuniformdistributionoftestpointsinthedomainofferssomecoverageagainstfaultsthataremorecomplexthandouble-modefaults.Anotherexpert,whofavorstheAllpairsapproach,notesthatAllpairsdoes,infact,testallthepairs,whichisthegoal.Heclaimsthereisnoevidencethattheorthogonalarrayapproachdetectsmoredefects.HealsonotesthattheAllpairstoolisavailablefreeontheWeb.Whatbothexpertsacknowledgeisthatnodocumentedstudiesexistcomparingtheefficacyofoneapproachovertheother.

Theexcitinghopeofpairwisetestingisthatbycreatingandrunningbetween1percentto20percentofthetestsyouwillfindbetween70percentand85percentofthetotaldefects.Thereisnopromisehere,onlyahope.Manyothershaveexperiencedthissignificantresult.Trythistechnique.Discoverwhetheritworksforyou.

Cohenreportedthatinadditiontoreducingthenumberoftestcasesandincreasingthedefectfindrate,testcasescreatedbytheAllpairsalgorithmalsoprovidedbettercodecoverage.Asetof300randomlyselectedtestsachieved67percentstatementcoverageand58percentdecisioncoveragewhilethe200allpairstestcasesachieved92percentblockcoverageand85percentdecisioncoverage,asignificantincreaseincoveragewithfewertestcases.

Onefinalcomment—itispossiblethatcertainimportantcombinationsmaybemissedbybothpairwiseapproaches.The80:20ruletellsusthatcombinationsarenotuniformlyimportant.Useyourjudgmenttodetermineifcertainadditionaltestsshouldbecreatedforthosecombinations.

http://aetgweb.argreenhouse.com

Inthepreviousexamplewecanbeassuredthatthedistributionofbrowsersisnotidentical.Itwouldbetrulyamazingif12.5percentofourusershadIE5.0,12.5percenthadIE5.5,12.5percenthadIE6.0,etc.Certaincombinationsoccurmorefrequentlythanothers.Inaddition,somecombinationsexistthat,whileusedinfrequently,absolutelypositivelymustworkproperly—"shutdownthenuclearreactor"isagoodexample.Incasepairwisemissesanimportantcombination,pleaseaddthatcombinationtoyourtestcases.

ApplicabilityandLimitationsLikeothertestdesignapproachespreviouslypresented,pairwisetestingcansignificantlyreducethenumberoftestcasesthatmustbecreatedandexecuted.Itisequallyapplicableattheunit,integration,system,andacceptancetestlevels.Allitrequiresarecombinationsofinputs,eachtakingonvariousvalues,thatresultinacombinatorialexplosion,toomanycombinationstotest.

Remember,thereisnounderlying"softwaredefectphysics"thatguaranteespairwisetestingwillbeofbenefit.Thereisonlyonewaytoknow—tryit.

SummaryWhenthenumberofcombinationstotestisverylarge,donottoattempttotestallcombinationsforallthevaluesforallthevariables,buttestallpairsofvariables.Thissignificantlyreducesthenumberofteststhatmustbecreatedandrun.

Studiessuggestthatmostdefectsareeithersingle-modedefects(thefunctionundertestsimplydoesnotwork)ordouble-modedefects(thepairingofthisfunction/modulewiththatfunction/modulefails).Pairwisetestingdefinesaminimalsubsetthatguidesustotestforallsingle-modeanddouble-modedefects.Thesuccessofthistechniqueonmanyprojects,bothdocumentedandundocumented,isagreatmotivationforitsuse.

Anorthogonalarrayisatwo-dimensionalarrayofnumbersthathasthisinterestingproperty—chooseanytwocolumnsinthearray,allthecombinationswilloccurineverycolumnpair.

Thereisnounderlying"softwaredefectphysics"thatguaranteespairwisetestingwillbeofbenefit.Thereisonlyonewaytoknow—tryit.

Practice1. NeithertheBrown&DonaldsonnortheStatelessUniversityRegistrationSystemcase

studiescontainhugenumbersofcombinationssuitableforthepairwisetestingapproach.Asexercises,usetheorthogonalarrayand/orallpairstechniqueontheothertwoexamplesinthischapter.Determinethesetofpairwisetestcasesusingthechosentechnique.

1. Abankhascreatedanewdataprocessingsystemthatisreadyfortesting.Thisbankhasdifferentkindsofcustomers—consumers,veryimportantconsumers,businesses,andnon-profits;differentkindsofaccounts—checking,savings,mortgages,consumerloans,andcommercialloans;theyoperateindifferentstates,eachwithdifferentregulations—California,Nevada,Utah,Idaho,Arizona,andNewMexico.

2. Inanobject-orientedsystem,anobjectofclassAcansendamessagecontainingaparameterPtoanobjectofclassX.ClassesB,C,andDinheritfromAsotheytoocansendthemessage.ClassesQ,R,S,andTinheritfromPsotheytoocanbepassedastheparameter.ClassesYandZinheritfromXsotheytoocanreceivethemessage.

References

Brownlie,Robert,etal."RobustTestingofAT&TPMX/StarMAILUsingOATS,"AT&TTechnicalJournal,Vol.71,No.3,May/June1992,pp.41–47.

Cohen,D.M.,etal."TheAETGSystem:AnApproachtoTestingBasedonCombinatorialDesign."IEEETransactionsonSoftwareEngineering,Vol.23,No.7,July,1997.

Kaner,Cem,JamesBach,andBretPettichord(2002).LessonsLearnedinSoftwareTesting:AContext-DrivenApproach.JohnWiley&Sons.

Kuhn,D.RichardandMichaelJ.Reilly."AnInvestigationoftheApplicabilityofDesignofExperimentstoSoftwareTesting,"27thNASA/IEEESoftwareEngineeringWorkshop,NASAGoddardSpaceFlightCenter,4–6December,2002.http://csrc.nist.gov/staff/kuhn/kuhn-reilly-02.pdf

Mandl,Robert."OrthogonalLatinSquares:AnApplicationofExperimentDesigntoCompilerTesting,"CommunicationsoftheACM,Vol.128,No.10,October1985,pp.1054–1058.

Phadke,MadhavS.(1989).QualityEngineeringUsingRobustDesign.Prentice-Hall.

Wallace,DeloresR.andD.RichardKuhn."FailureModesInMedicalDeviceSoftware:AnAnalysisOf15YearsOfRecallData,"InternationalJournalofReliability,Quality,andSafetyEngineering,Vol.8,No.4,2001.

http://csrc.nist.gov/staff/kuhn/kuhn-reilly-02.pdf

Chapter7:State-TransitionTestingColonelCleatusYorbvillehadbeenoneseriouslyboredastronautforthefirstfewmonthsofhisdiplomaticmissiononthethirdplanetoftheFrangelicusXIVsystem,butallthathadchangedonthedayhe'ddiscoveredthathistiny,multipedalandinfinitelyhospitablealienhostswerenotonlyediblebuttastedremarkablylikethatstuffthat'sleftonthepanafteryou'vemadecinnamonbunsandburnedthemalittle.

—MarkSilcox

IntroductionState-Transitiondiagrams,likedecisiontables,areanotherexcellenttooltocapturecertaintypesofsystemrequirementsandtodocumentinternalsystemdesign.Thesediagramsdocumenttheeventsthatcomeintoandareprocessedbyasystemaswellasthesystem'sresponses.Unlikedecisiontables,theyspecifyverylittleintermsofprocessingrules.Whenasystemmustremembersomethingaboutwhathashappenedbeforeorwhenvalidandinvalidordersofoperationsexist,state-transitiondiagramsareexcellenttoolstorecordthisinformation.

Thesediagramsarealsovitaltoolsinthetester'spersonaltoolbox.Unfortunately,manyanalysts,designers,programmers,andtestersarenotfamiliarwiththistechnique.

Technique

State-TransitionDiagrams

Itiseasiertointroducestate-transitiondiagramsbyexampleratherthanbyformaldefinition.SinceneitherBrown&DonaldsonnortheStatelessUniversityRegistrationSystemhassubstantialstate-transitionbasedrequirementslet'sconsideradifferentexample.TogettoStatelessU,weneedanairlinereservation.Let'scallourfavoritecarrier(GraceL.FergusonAirline&StormDoorCompany)tomakeareservation.Weprovidesomeinformationincludingdepartureanddestinationcities,dates,andtimes.Areservationagent,actingasourinterfacetotheairline'sreservationsystem,usesthatinformationtomakeareservation.Atthatpoint,theReservationisintheMadestate.Inaddition,thesystemcreatesandstartsatimer.Eachreservationhascertainrulesaboutwhenthereservationmustbepaidfor.Theserulesarebasedondestination,classofservice,dates,etc.Ifthistimerexpiresbeforethereservationispaidfor,thereservationiscancelledbythesystem.Instate-transitionnotationthisinformation

isrecordedas: Figure7-1:TheReservationisMade.

ThecirclerepresentsonestateoftheReservation—inthiscasetheMadestate.ThearrowshowsthetransitionintotheMadestate.Thedescriptiononthearrow,giveInfo,isaneventthatcomesintothesystemfromtheoutsideworld.Thecommandafterthe"/"denotesanactionofthesystem;inthiscasestartPayTimer.Theblackdotindicatesthestartingpointofthediagram.

SometimeaftertheReservationismade,but(hopefully)beforethePayTimerexpires,theReservationispaidfor.ThisisrepresentedbythearrowlabeledPayMoney.WhentheReservationispaidittransitionsfromtheMadestatetothePaidstate.

Figure7-2:TheReservationtransitionstothePaidstate.

Beforeweproceedlet'sdefinethetermsmoreformally:

State(representedbyacircle)—Astateisaconditioninwhichasystemiswaitingforoneormoreevents.States"remember"inputsthesystemhasreceivedinthepastanddefinehowthesystemshouldrespondtosubsequenteventswhentheyoccur.Theseeventsmaycausestate-transitionsand/orinitiateactions.Thestateisgenerallyrepresentedbythevaluesofoneormorevariableswithinasystem.

Transition(representedbyanarrow)—Atransitionrepresentsachangefromonestatetoanothercausedbyanevent.

Event(representedbyalabelonatransition)—Aneventissomethingthatcausesthesystemtochangestate.Generally,itisaneventintheoutsideworldthatentersthesystemthroughitsinterface.SometimesitisgeneratedwithinthesystemsuchasTimerexpiresorQuantityonHandgoesbelowReorderPoint.Eventsareconsideredtobeinstantaneous.Eventscanbeindependentorcausallyrelated(eventBcannottakeplacebeforeeventA).Whenaneventoccurs,thesystemcanchangestateorremaininthesamestateand/orexecuteanaction.Eventsmayhaveparametersassociatedwiththem.Forexample,PayMoneymayindicateCash,Check,DebitCard,orCreditCard.

Action(representedbyacommandfollowinga"/")—Anactionisanoperationinitiatedbecauseofastatechange.ItcouldbeprintaTicket,displayaScreen,turnonaMotor,etc.Oftentheseactionscausesomethingtobecreatedthatareoutputsofthesystem.Notethatactionsoccurontransitionsbetweenstates.Thestatesthemselvesarepassive.

Theentrypointonthediagramisshownbyablackdotwhiletheexitpointisshownbyabulls-eyesymbol.

ThisnotationwascreatedbyMealy.AnalternatenotationhasbeendefinedbyMoorebutislessfrequentlyused.Foramuchmorein-depthdiscussionofstate-transitiondiagramsseeFowlerandScott'sbook,UMLDistilled:ABriefGuideToTheStandardObjectModelingLanguage.Itdiscussesmorecomplexissuessuchaspartitionedandnestedstate-transitiondiagrams.

Notethatthestate-transitiondiagramrepresentsonespecificentity(inthiscaseaReservation).Itdescribesthestatesofareservation,theeventsthataffectthereservation,thetransitionsofthereservationfromonestatetoanother,andactionsthatareinitiatedbythereservation.Acommonmistakeistomixdifferententitiesintoonestate-transitiondiagram.AnexamplemightbemixingReservationandPassengerwitheventsandactionscorrespondingtoeach.

FromthePaidstatetheReservationtransitionstotheTicketedstatewhentheprintcommand(anevent)isissued.NotethatinadditiontoenteringtheTicketedstate,aTicketisoutputbythesystem.

Figure7-3:TheReservationtransitionstotheTicketedstate.

FromtheTicketedstatewegiveTickettothegateagenttoboardtheplane.

Figure7-4:TheReservationtransitionstotheUsedstate.

Aftersomeotheractionorperiodoftime,notindicatedonthisdiagram,thestate-transitionpathendsatthebulls-eyesymbol.

Figure7-5:Thepathends.

Doesthisdiagramshowallthepossiblestates,events,andtransitionsinthelifeofaReservation?No.IftheReservationisnotpaidforinthetimeallotted(thePayTimerexpires),itiscancelledfornon-payment.

Figure7-6:ThePayTimerexpiresandtheReservationiscancelledfornonpayment.

Finishedyet?No.Customerssometimescanceltheirreservations.FromtheMadestatethecustomer(throughthereservationagent)askstocanceltheReservation.Anewstate,CancelledByCustomer,isrequired.

Figure7-7:CanceltheReservationfromtheMadestate.

Inaddition,aReservationcanbecancelledfromthePaidstate.InthiscaseaRefundshouldbegeneratedandleavethesystem.TheresultingstateagainisCancelledByCustomer.

Figure7-8:CancellationfromthePaidstate.

Onefinaladdition.FromtheTicketedstatethecustomercancanceltheReservation.InthatcaseaRefundshouldbegeneratedandthenextstateshouldbeCancelledbyCustomer.Butthisisnotsufficient.TheairlinewillgeneratearefundbutonlywhenitreceivestheprintedTicketfromthecustomer.Thisintroducesonenewnotationalelement—squarebrackets[]thatcontainaconditionalthatcanbeevaluatedeitherTrueorFalse.Thisconditionalactsasaguardallowingthetransitiononlyiftheconditionistrue.

Figure7-9:CancellationfromtheTicketedstate.

Notethatthediagramisstillincomplete.Noarrowsandbulls-eyesemergefromtheCancelledstates.PerhapswecouldreinstateareservationfromtheCancelledNonPaystate.Wecouldcontinueexpandingthediagramtoincludeseatselection,flightcancellation,andothersignificanteventsaffectingthereservationbutthisissufficienttoillustratethetechnique.

Asdescribed,state-transitiondiagramsexpresscomplexsystemrulesandinteractionsinaverycompactnotation.Hopefully,whenthiscomplexityexists,analystsanddesignerswillhavecreatedstate-transitiondiagramstodocumentsystemrequirementsandtoguidetheirdesign.

State-TransitionTables

Astate-transitiondiagramisnottheonlywaytodocumentsystembehavior.Thediagramsmaybeeasiertocomprehend,butstate-transitiontablesmaybeeasiertouseinacompleteandsystematicmanner.State-transitiontablesconsistoffourcolumns—CurrentState,Event,Action,andNextState.

Table7-1:State-TransitiontableforReservation.

CurrentState Event Action NextState

null giveInfo startPayTimer Made

null payMoney -- null

null print -- null

null giveTicket -- null

null cancel -- null

null PayTimerExpires -- null

Made giveInfo -- Made

Made payMoney -- Paid

Made print -- Made

Made giveTicket -- Made

Made cancel -- Can-Cust

Made PayTimerExpires -- Can-NonPay

Paid giveInfo -- Paid

Paid payMoney -- Paid

Paid print Ticket Ticketed

Paid giveTicket -- Paid

Paid cancel Refund Can-Cust

Paid PayTimerExpires -- Paid

Ticketed giveInfo -- Ticketed

Ticketed payMoney -- Ticketed

Ticketed print -- Ticketed

Ticketed giveTicket -- Used

Ticketed cancel Refund Can-Cust

Ticketed PayTimerExpires -- Ticketed

Used giveInfo -- Used

Used payMoney -- Used

Used print -- Used

Used giveTicket -- Used

Used cancel -- Used

Used PayTimerExpires -- Used

Can-NonPay giveInfo -- Can-NonPay

Can-NonPay payMoney -- Can-NonPay

Can-NonPay print -- Can-NonPay

Can-NonPay giveTicket -- Can-NonPay

Can-NonPay cancel -- Can-NonPay

Can-NonPay PayTimerExpires -- Can-NonPay

Can-Cust givelnfo -- Can-Cust

Can-Cust payMoney -- Can-Cust

Can-Cust print -- Can-Cust

Can-Cust giveTicket -- Can-Cust

Can-Cust cancel -- Can-Cust

Can-Cust PayTimerExpires -- Can-Cust

Theadvantageofastate-transitiontableisthatitlistsallpossiblestate-transitioncombinations,notjustthevalidones.Whentestingcritical,high-risksystemssuchasavionicsormedicaldevices,testingeverystate-transitionpairmayberequired,includingthosethatarenotvalid.Inaddition,creatingastate-transitiontableoftenunearthscombinationsthatwerenotidentified,documented,ordealtwithintherequirements.Itishighlybeneficialtodiscoverthesedefectsbeforecodingbegins.

KeyPoint

Theadvantageofastate-transitiontableisthatitlistsallpossiblestate-transitioncombinations,notjustthevalidones.

Usingastate-transitiontablecanhelpdetectdefectsinimplementationthatenableinvalidpathsfromonestatetoanother.Thedisadvantageofsuchtablesisthattheybecomeverylargeveryquicklyasthenumberofstatesandeventsincreases.Inaddition,thetablesaregenerallysparse;thatis,mostofthecellsareempty.

CreatingTestCases

Informationinthestate-transitiondiagramscaneasilybeusedtocreatetestcases.Fourdifferentlevelsofcoveragecanbedefined:

1. Createasetoftestcasessuchthatallstatesare"visited"atleastonceundertest.Thesetofthreetestcasesshownbelowmeetsthisrequirement.Generallythisisaweakleveloftestcoverage.

Figure7-10:Asetoftestcasesthat"visit"eachstate.

2. Createasetoftestcasessuchthatalleventsaretriggeredatleastonceundertest.Notethatthetestcasesthatcovereacheventcanbethesameasthosethatcovereachstate.Again,thisisaweaklevelofcoverage.

Figure7-11:Asetoftestcasesthattriggeralleventsatleastonce.

3. Createasetoftestcasessuchthatallpathsareexecutedatleastonceundertest.Whilethislevelisthemostpreferredbecauseofitslevelofcoverage,itmaynotbefeasible.Ifthestate-transitiondiagramhasloops,thenthenumberofpossiblepathsmaybeinfinite.Forexample,givenasystemwithtwostates,AandB,whereAtransitionstoBandBtransitionstoA.Afewofthepossiblepathsare:A→B

A→B→A

A→B→A→B→A→B

A→B→A→B→A→B→AA→B→A→B→A→B→A→B→A→B

...

andsoonforever.Testingofloopssuchasthiscanbeimportantiftheymayresultinaccumulatingcomputationalerrorsorresourceloss(lockswithoutcorrespondingreleases,memoryleaks,etc.).

KeyPoint

Testingeverytransitionisusuallytherecommendedlevelofcoverageforastate-transitiondiagram.

4. Createasetoftestcasessuchthatalltransitionsareexercisedatleastonceundertest.Thisleveloftestingprovidesagoodlevelofcoveragewithoutgeneratinglargenumbersoftests.Thislevelisgenerallytheonerecommended.

Figure7-12:Asetoftestcasesthattriggeralltransitionsatleastonce.

Testcasescanalsobereaddirectlyfromthestate-transitiontable.Thegrayrowsinthefollowingtableshowallthevalidtransitions.

Table7-2:TestingallvalidtransitionsfromaState-transitiontable.

CurrentState Event Action NextState

null giveInfo startPayTimer Made

null payMoney -- null

null print -- null

null giveTicket -- null

null cancel -- null

null PayTimerExpires -- null

Made giveInfo -- Made

Made payMoney -- Paid

Made print -- Made

Made giveTicket -- Made

Made cancel -- Can-Cust

Made PayTimerExpires -- Can-NonPay

Paid giveInfo -- Paid

Paid payMoney -- Paid

Paid print Ticket Ticketed

Paid giveTicket -- Paid

Paid cancel Refund Can-Cust

Paid PayTimerExpires -- Paid

Ticketed giveInfo -- Ticketed

Ticketed payMoney -- Ticketed

Ticketed print -- Ticketed

Ticketed giveTicket -- Used

Ticketed cancel Refund Can-Cust

Ticketed PayTimerExpires -- Ticketed

Used giveInfo -- Used

Used payMoney -- Used

Used print -- Used

Used giveTicket -- Used

Used cancel -- Used

Used PayTimerExpires -- Used

Can-NonPay giveInfo -- Can-NonPay

Can-NonPay payMoney -- Can-NonPay

Can-NonPay print -- Can-NonPay

Can-NonPay giveTicket -- Can-NonPay

Can-NonPay cancel -- Can-NonPay

Can-NonPay PayTimerExpires -- Can-NonPay

Can-Cust givelnfo -- Can-Cust

Can-Cust payMoney -- Can-Cust

Can-Cust print -- Can-Cust

Can-Cust giveTicket -- Can-Cust

Can-Cust cancel -- Can-Cust

Can-Cust PayTimerExpires -- Can-Cust

Inaddition,dependingonthesystemrisk,youmaywanttocreatetestcasesforsomeoralloftheinvalidstate/eventpairstomakesurethesystemhasnotimplementedinvalidpaths.

ApplicabilityandLimitationsState-Transitiondiagramsareexcellenttoolstocapturecertainsystemrequirements,namelythosethatdescribestatesandtheirassociatedtransitions.Thesediagramsthencanbeusedtodirectourtestingeffortsbyidentifyingthestates,events,andtransitionsthatshouldbetested.

State-Transitiondiagramsarenotapplicablewhenthesystemhasnostateordoesnotneedtorespondtoreal-timeeventsfromoutsideofthesystem.Anexampleisapayrollprogramthatreadsanemployee'stimerecord,computespay,subtractsdeductions,savestherecord,printsapaycheck,andrepeatstheprocess.

SummaryState-Transitiondiagramsdirectourtestingeffortsbyidentifyingthestates,events,actions,andtransitionsthatshouldbetested.Together,thesedefinehowasysteminteractswiththeoutsideworld,theeventsitprocesses,andthevalidandinvalidorderoftheseevents.

Astate-transitiondiagramisnottheonlywaytodocumentsystembehavior.Theymaybeeasiertocomprehend,butstate-transitiontablesmaybeeasiertouseinacompleteandsystematicmanner.

Thegenerallyrecommendedleveloftestingusingstate-transitiondiagramsistocreateasetoftestcasessuchthatalltransitionsareexercisedatleastonceundertest.Inhigh-risksystems,youmaywanttocreateevenmoretestcases,approachingallpathsifpossible.

Practice1. ThisexercisereferstotheStatelessUniversityRegistrationSystemWebsite

describedinAppendixB.Belowisastate-transitiondiagramforthe"enrollinacourse"and"dropacourse"process.Determineasetoftestcasesthatyoufeeladequatelycovertheenrollanddropprocess.

Thefollowingtermsareusedinthediagram:Eventscreate-Createanewcourse.

enroll-Addastudenttothecourse.

drop-Dropastudentfromthecourse.

AttributesID-Thestudentidentificationnumber.

max-Themaximumnumberofstudentsacoursecanhold.

#enrolled-Thenumberofstudentscurrentlyenrolledinthecourse.

#waiting-ThenumberofstudentscurrentlyontheWaitListforthiscourse.

TestsisEnrolled-Answers"isthestudentenrolled(ontheSectionList)?"

onWaitList-Answers"isthestudentontheWaitList?"

ListsSectionList-Alistofstudentsenrolledintheclass.

WaitList-Alistofstudentswaitingtobeenrolledinafullclass.

Symbols++Incrementby1.

--Decrementby1.

Figure7-13:State-transitiondiagramforenrollanddropacourseatStatelessU.

References


Fowler,MartinandKendallScott(1999).UMLDistilled:ABriefGuidetotheStandardObjectModelingLanguage(2ndEdition).Addison-Wesley.

Harel,David."Statecharts:avisualformalismforcomplexsystems."ScienceofComputerProgramming8,1987,pp231–274.

Mealy,G.H."Amethodforsynthesizingsequentialcircuits."BellSystemTechnicalJournal,34(5):1045–1079,1955.

Moore,E.F."Gedanken-experimentsonsequentialmachines,"AutomataStudies(C.E.ShannonandJ.McCarthy,eds.),pp.129–153,Princeton,NewJersey:PrincetonUniversityPress,1956.

Rumbaugh,James,etal.(1991).Object-OrientedModelingandDesign.Prentice-Hall.

Chapter8:DomainAnalysisTestingStandingintheconcessionscaroftheOrientExpressasithissedandlurchedawayfromthestation,SpecialAgentChucouldfeelenemyeyeswatchinghimfromtheinkyshadowsandknewthathewasbeingtested,foralthoughhehadnevertastedaplugoftobaccoinhislife,hewasimpersonatinganarmsdealerknowntobeaconnoisseur,soheknewthathe,theChosenOne,ChowChu,hadnochoicebuttochoosethechoicestchewonthechoo-choo.

—LorenHaarsma

IntroductionInthechaptersonEquivalenceClassandBoundaryValuetesting,weconsideredthetestingofindividualvariablesthattookonvalueswithinspecifiedranges.Inthischapterwewillconsiderthetestingofmultiplevariablessimultaneously.Therearetworeasonstoconsiderthis:

Werarelywillhavetimetocreatetestcasesforeveryvariableinoursystems.Therearesimplytoomany.

Oftenvariablesinteract.Thevalueofonevariableconstrainstheacceptablevaluesofanother.Inthiscase,certaindefectscannotbediscoveredbytestingthemindividually.

Domainanalysisisatechniquethatcanbeusedtoidentifyefficientandeffectivetestcaseswhenmultiplevariablescanorshouldbetestedtogether.Itbuildsonandgeneralizesequivalenceclassandboundaryvaluetestingtonsimultaneousdimensions.Likethosetechniques,wearesearchingforsituationswheretheboundaryhasbeendefinedorimplementedincorrectly.

KeyPoint

Domainanalysisisatechniquethatcanbeusedtoidentifyefficientandeffectivetestcaseswhenmultiplevariablesshouldbetestedtogether.

Intwodimensions(withtwointeractingparameters)thefollowingdefectscanoccur:

Ashiftedboundaryinwhichtheboundaryisdisplacedverticallyorhorizontally

Atiltedboundaryinwhichtheboundaryisrotatedatanincorrectangle

Amissingboundary

Anextraboundary

Figure8-1isadaptedfromBinder.Itillustratesthesefourtypesofdefectsgraphically.

Figure8-1:Twodimensionalboundarydefects.

Certainlytherecanbeinteractionsbetweenthreeormorevariables,butthediagramsaremoredifficulttovisualize.

TechniqueThedomainanalysisprocessguidesusinchoosingefficientandeffectivetestcases.First,anumberofdefinitions:

Anonpointisavaluethatliesonaboundary.

Anoffpointisavaluethatdoesnotlieonaboundary.

Aninpointisavaluethatsatisfiesalltheboundaryconditionsbutdoesnotlieonaboundary.

Anoutpointisavaluethatdoesnotsatisfyanyboundarycondition.

Choosingonandoffpointsismorecomplicatedthatitmayappear.

Whentheboundaryisclosed(definedbyanoperatorcontaininganequality,i.e.,≤,≥or=)sothatpointsontheboundaryareincludedinthedomain,thenanonpointliesontheboundaryandisincludedwithinthedomain.Anoffpointliesoutsidethedomain.

Whentheboundaryisopen(definedbyaninequalityoperator<or>)sothatpointsontheboundaryarenotincludedinthedomain,thenanonpointliesontheboundarybutisnotincludedwithinthedomain.Anoffpointliesinsidethedomain.

Confused?Atthispointexamplesarecertainlyinorder.

Figure8-2:Examplesofon,off,in,andoutpointsforbothclosedandopenboundaries.

Ontheleftisanexampleofaclosedboundary.Theregiondefinedconsistsofallthepointsgreaterthanorequalto10.Theonpointhasthevalue10.Theoffpointisslightlyofftheboundaryandoutsidethedomain.Theinpointiswithinthedomain.Theoutpointisoutsidethedomain.

Ontherightisanexampleofanopenboundary.Theregiondefinedconsistsofallthepointsgreaterthan(butnotequalto)10.Again,theonpointhasavalueof10.Theoffpointisslightlyofftheboundaryandinsidethedomain.Theinpointiswithinthedomain.Theoutpointisoutsidethedomain.

Havingdefinedthesepoints,the1x1("one-by-one")domainanalysistechniqueinstructsustochoosethesetestcases:

Foreachrelationalcondition(≥,>,≤,or<)chooseoneonpointandoneoffpoint.

Foreachstrictequalitycondition(=)chooseoneonpointandtwooffpoints,oneslightlylessthantheconditionalvalueandoneslightlygreaterthanthevalue.

Notethatthereisnoreasontorepeatidenticaltestsforadjacentdomains.Ifanoffpointforonedomainistheinpointforanother,donotduplicatethesetests.

Bindersuggestsaveryusefultablefordocumenting1x1domainanalysistestcasescalledtheDomainTestMatrix.

Table8-1:ExampleDomainTestMatrix.

Notethattestcases1through8testtheonpointsandoffpointsforeachconditionofthefirstvariable(X1)whileholdingthevalueofthesecondvariable(X2)atatypicalinpoint.Testcases9through16holdthefirstvariableatatypicalinpointwhiletestingtheonandoffpointsforeachconditionofthesecondvariable.Additionalvariablesandconditionswouldfollowthesamepattern.

ExampleAdmissiontoStatelessUniversityismadebyconsideringacombinationofhighschoolgradesandACTtestscores.Theshadedcellsinthefollowingtableindicatethecombinationsthatwouldguaranteeacceptance.GradePointAverages(GPAs)areshownacrossthetopwhileACTscoresareshowndowntheleftside.StatelessUniversityisafairlyexclusiveschoolintermsofitsadmissionpolicy.

Explanation

TheACTAssessmentisanexaminationdesignedtoassesshighschoolstudents'generaleducationaldevelopmentandtheirabilitytocompletecollege-levelwork.

TheGradePointAverageisbasedonconvertinglettergradestonumericvaluesA=4.0(Best)

B=3.0

C=2.0(Average)

D=1.0

Table8-2:StatelessUniversityAdmissionsMatrix.

Thistablecanberepresentedasthesolutionsetofthesethreelinearequations:ACT≤36(thehighestscorepossible)GPA≤4.0(thehighestvaluepossible)10*GPA+ACT≥71

(Thethirdequationcanbefoundbyusingthegoodoldy=mx+bformulafromelementaryalgebra.Usepoints{ACT=36,GPA=3.5}and{ACT=31,GPA=4.0}andcrank—that'smathslangforsolvethepairofsimultaneousequationsobtainedbysubstitutingeachofthesetwopointsintothey=mx+bequation.)

Figure8-3:StatelessUniversityAdmissionsMatrixingraphicalform.

Thefollowingtestcasescoverthesethreeboundariesusingthe1x1domainanalysisprocess.

Table8-3:1x1DomainAnalysistestcasesforStatelessUniversityadmissions.

Testcases1and2verifytheGPA≤4.0constraint.Case1checksontheGPA=4.0boundarywhilecase2checksjustoutsidetheboundarywithGPA=4.1.BothofthesecasesusetypicalvaluesfortheACTandGPA/ACTconstraints.

Testcases3and4verifytheACT≤36constraint.Case3checksontheACT=36boundarywhilecase4checksjustoutsidetheboundarywithACT=37.BothofthesecasesusetypicalvaluesfortheGPAandGPA/ACTconstraints.

Testcases5and6verifythe10*GPA+ACT≥71constraint.Case5checksontheGPA=3.7andACT=34boundarywhilecase6checksjustoutsidetheboundarywithGPA=3.8andACT=32.BothofthesecasesusetypicalvaluesfortheGPAandACTconstraints.

ApplicabilityandLimitationsDomainanalysisisapplicablewhenmultiplevariables(suchasinputfields)shouldbetestedtogethereitherforefficiencyorbecauseofalogicalinteraction.Whilethistechniqueisbestsuitedtonumericvalues,itcanbegeneralizedtoBooleans,strings,enumerations,etc.

SummaryDomainanalysisfacilitatesthetestingofmultiplevariablessimultaneously.Itisusefulbecausewerarelywillhavetimetocreatetestcasesforeveryvariableinoursystems.Therearesimplytoomany.Inaddition,oftenvariablesinteract.Whenthevalueofonevariableconstrainstheacceptablevaluesofanother,certaindefectscannotbediscoveredbytestingthemindividually.

Itbuildsonandgeneralizesequivalenceclassandboundaryvaluetestingtonsimultaneousdimensions.Likethosetechniques,wearesearchingforsituationswheretheboundaryhasbeenimplementedincorrectly.

Inusingthe1x1domainanalysistechniqueforeachrelationalcondition(≥,>,≤,or<)wechooseoneonpointandoneoffpoint.Foreachstrictequalitycondition(=)wechooseoneonpointandtwooffpoints,oneslightlylessthantheconditionalvalueandoneslightlygreaterthanthevalue.

Practice1. StatelessUniversitypridesitselfinpreparingnotjusteducatedstudentsbutgood

citizensoftheirnation.(That'swhattheiradvertisingbrochuresays.)Inadditiontotheirmajorandminorcoursework,StatelessU.requireseachstudenttotake(andpass)anumberofGeneralEducationclasses.Theseare:

CollegeAlgebra(thestudentmayeithertakethecourseorshowcompetencythroughtesting).

OurNation'sInstitutions—asurveycourseofournation'shistory,government,andplaceintheworld.

FromfourtosixteenhoursofSocialSciencecourses(numbers100–299).

FromfourtosixteenhoursofPhysicalSciencecourses(numbers100–299)

Nomorethantwenty-fourcombinedhoursofSocialScienceandPhysicalSciencecoursesmaybecountedtowardgraduation.

Apply1x1domainanalysistotheserequirements,derivethetestcases,anduseBinder'sDomainTestMatrixtodocumentthem.



Chapter9:UseCaseTestingTheInsectKeeperGeneral,sittingastridehisgianthoveringaphid,surveyedthebattlefieldwhichreekedwiththestenchofdecayandresonatedwiththelowdroneofthetatteredanddyingmutantswarmsastheirlegskickedforlornlyattheskybeforeturningtohismasterandsaying,'MyLord,yourfliesareundone.'

—AndrewVincent

IntroductionUpuntilnowwehaveexaminedtestcasedesigntechniquesforpartsofasystem—inputvariableswiththeirrangesandboundaries,businessrulesasrepresentedindecisiontables,andsystembehaviorsasrepresentedinstate-transitiondiagrams.Nowitistimetoconsidertestcasesthatexerciseasystem'sfunctionalitiesfromstarttofinishbytestingeachofitsindividualtransactions.

Definingthetransactionsthatasystemprocessesisavitalpartoftherequirementsdefinitionprocess.Variousapproachestodocumentingthesetransactionshavebeenusedinthepast.Examplesincludeflowcharts,HIPOdiagrams,andtext.Today,themostpopularapproachistheusecasediagram.Likedecisiontablesandstate-transitiondiagrams,usecasesareusuallycreatedbydevelopersfordevelopers.But,liketheseothertechniques,usecasesholdawealthofinformationusefultotesters.

UsecaseswerecreatedbyIvarJacobsenandpopularizedinhisbookObject-OrientedSoftwareEngineering:AUseCaseDrivenApproach.Jacobsendefinesa"usecase"asascenariothatdescribestheuseofasystembyanactortoaccomplishaspecificgoal.By"actor"wemeanauser,playingarolewithrespecttothesystem,seekingtousethesystemtoaccomplishsomethingworthwhilewithinaparticularcontext.Actorsaregenerallypeoplealthoughothersystemsmayalsobeactors.A"scenario"isasequenceofstepsthatdescribetheinteractionsbetweentheactorandthesystem.Notethattheusecaseisdefinedfromtheperspectiveoftheuser,notthesystem.Notealsothattheinternalworkingsofthesystem,whilevital,arenotpartoftheusecasedefinition.Thesetofusecasesmakesupthefunctionalrequirementsofasystem.

TheUnifiedModelingLanguagenotionforusecasesis:

Figure9-1:SomeStatelessUniversityusecases.

Thestickfiguresrepresenttheactors,theellipsesrepresenttheusecases,andthearrowsshowwhichactorsinitiatewhichusecases.

Itisimportanttonotethatwhileusecaseswerecreatedinthecontextofobject-orientedsystemsdevelopment,theyareequallyusefulindefiningfunctionalrequirementsinotherdevelopmentparadigmsaswell.

Thevalueofusecasesisthatthey:

Capturethesystem'sfunctionalrequirementsfromtheuser'sperspective;notfromatechnicalperspective,andirrespectiveofthedevelopmentparadigmtobeused.

Canbeusedtoactivelyinvolveusersintherequirementsgatheringanddefinitionprocess.

Providethebasisforidentifyingasystem'skeyinternalcomponents,structures,databases,andrelationships.

Serveasthefoundationfordevelopingtestcasesatthesystemandacceptancelevel.

TechniqueUnfortunately,thelevelofdetailspecifiedintheusecasesisnotsufficient,eitherfordevelopersortesters.InhisbookWritingEffectiveUseCases,AlistairCockburnhasproposedadetailedtemplatefordescribingusecases.Thefollowingisadaptedfromhiswork:

Table9-1:Usecasetemplate.

UseCaseComponent Description

UseCaseNumberorIdentifier Auniqueidentifierforthisusecase

UseCaseName Thenameshouldbethegoalstatedasashortactiveverbphrase

GoalinContext Amoredetailedstatementofthegoalifnecessary

Scope Corporate|System|Subsystem

Level Summary|Primarytask|Subfunction

PrimaryActor Rolenameordescriptionoftheprimaryactor

Preconditions Therequiredstateofthesystembeforetheusecaseistriggered

SuccessEndConditions Thestateofthesystemuponsuccessfulcompletionofthisusecase

FailedEndConditions Thestateofthesystemiftheusecasecannotexecutetocompletion

Trigger Theactionthatinitiatestheexecutionoftheusecase

MainSuccessScenario

Step Action

1

2

...

Extensions Conditionsunderwhichthemainsuccessscenariowillvaryandadescriptionofthosevariations

Sub-Variations Variationsthatdonotaffectthemainflowbutthatmustbeconsidered

Priority Criticality

ResponseTime Timeavailabletoexecutethisusecase

Frequency Howoftenthisusecaseisexecuted

Channelsto Interactive|File|Database|...

PrimaryActor

SecondaryActors Otheractorsneededtoaccomplishthisusecase

ChannelstoSecondaryActors Interactive|File|Database|...

DateDue Scheduleinformation

CompletenessLevel

UseCaseidentified(0.1)|Mainscenariodefined(0.5)|Allextensionsdefined(0.8)|Allfieldscomplete(1.0)

OpenIssues Unresolvedissuesawaitingdecisions

ExampleConsiderthefollowingexamplefromtheStatelessUniversityRegistrationSystem.AstudentwantstoregisterforacourseusingSU'sonlineregistrationsystem,SURS.

Table9-2:Exampleusecase.

UseCaseComponent Description

UseCaseNumberorIdentifier SURS1138

UseCaseName Registerforacourse(aclasstaughtbyafacultymember)

GoalinContext

Scope System

Level Primarytask

PrimaryActor Student

Preconditions None

SuccessEndConditions Thestudentisregisteredforthecourse—thecoursehasbeenaddedtothestudent'scourselist

FailedEndConditions Thestudent'scourselistisunchanged

Trigger Studentselectsacourseand"Registers"

MainSuccessScenarioA:ActorS:System

StepAction

1 A:Selects"Registerforacourse"

2 A:Selectscourse(e.g.Math1060)

3 S:Displayscoursedescription

4 A:Selectssection(Mon&Wed9:00am)

5 S:Displayssectiondaysandtimes

6 A:Accepts

7 S:Addscourse/sectiontostudent'scourselist

Extensions

2aCoursedoesnotexistS:Displaymessageandexit

4aSectiondoesnotexistS:Displaymessageandexit

4bSectionisfull

S:Displaymessageandexit

6a StudentdoesnotacceptS:Displaymessageandexit

Sub-Variations

Studentmayuse

Web

Phone

Priority Critical

ResponseTime 10secondsorless

Frequency Approximately5coursesx10,000studentsovera4-weekperiod

ChannelstoPrimaryActor Interactive

SecondaryActors None

ChannelstoSecondaryActors N/A

DateDue 1Feb

CompletenessLevel 0.5

OpenIssues None

Hopefullyeachusecasehasbeenthroughaninspectionprocessbeforeitwasimplemented.Totesttheimplementation,thebasicruleistocreateatleastonetestcaseforthemainsuccessscenarioandatleastonetestcaseforeachextension.

Becauseusecasesdonotspecifyinputdata,thetestermustselectit.Typicallyweusetheequivalenceclassandboundaryvaluetechniquesdescribedearlier.AlsoaDomainTestMatrix(seetheDomainAnalysisTestingchapterforanexample)maybeausefulwayofdocumentingthetestcases.

Itisimportanttoconsidertheriskofthetransactionanditsvariantsundertest.Lessriskytransactionsmeritlesstesting.Moreriskytransactionsshouldreceivemoretesting.Forthemconsiderthefollowingapproach.

KeyPoint

Alwaysremembertoevaluatetheriskofeachusecaseandextensionandcreatetestcasesaccordingly.

Tocreatetestcases,startwithnormaldataforthemostoftenusedtransactions.Thenmovetoboundaryvaluesandinvaliddata.Next,choosetransactionsthat,whilenotusedoften,arevitaltothesuccessofthesystem(i.e.,ShutDownTheNuclearReactor).MakesureyouhaveatleastonetestcaseforeveryExtensionintheusecase.Trytransactionsinstrangeorders.Violatethepreconditions(ifthatcanhappeninactualuse).Ifatransactionhasloops,don'tjustloopthroughonceortwice—bediabolical.Lookforthelongest,mostconvolutedpaththrough

thetransactionandtryit.Iftransactionsshouldbeexecutedinsomelogicalorder,tryadifferentorder.Insteadofenteringdatatop-down,trybottom-up.Create"goofy"testcases.Ifyoudon'ttrystrangethings,youknowtheuserswill.

FreeStuffDownloadHolodeckfromhttp://www.sisecure.com/holodeck/holodeck-trial.aspx.

Mostpathsthroughatransactionareeasytocreate.Theycorrespondtovalidandinvaliddatabeingentered.Moredifficultarethosepathsduetosomekindofexceptionalcondition—lowmemory,diskfull,connectionlost,drivernotloaded,etc.Itcanbeverytimeconsumingforthetestertocreateorsimulatetheseconditions.Fortunately,atoolisavailabletohelpthetestersimulatetheseproblems—Holodeck,createdbyJamesWhittakerandhisassociatesatFloridaInstituteofTechnology.Holodeckmonitorstheinteractionsbetweenanapplicationanditsoperatingsystem.Itlogseachsystemcallandenablesthetestertosimulateafailureofanycallatwill.Inthisway,thediskcanbe"madefull,"networkconnectionscan"becomedisconnected,"datatransmissioncan"begarbled,"andahostofotherproblemscanbesimulated.

Amajorcomponentoftransactiontestingistestdata.BorisBeizersuggeststhat30percentto40percentoftheeffortintransactiontestingisgenerating,capturing,orextractingtestdata.Don'tforgettoincluderesources(timeandpeople)forthisworkinyourproject'sbudget.

NoteOnetestinggroupdesignatesa"dataczar"whosesoleresponsibilityistoprovidetestdata.

http://www.sisecure.com/holodeck/holodeck-trial.aspx

ApplicabilityandLimitationsTransactiontestingisgenerallythecornerstoneofsystemandacceptancetesting.Itshouldbeusedwheneversystemtransactionsarewelldefined.Ifsystemtransactionsarenotwelldefined,youmightconsiderpolishingupyourresumeorC.V.

Whilecreatingatleastonetestcaseforthemainsuccessscenarioandatleastoneforeachextensionprovidessomeleveloftestcoverage,itisclearthat,nomatterhowmuchwetry,mostinputcombinationswillremainuntested.Donotbeoverconfidentaboutthequalityofthesystematthispoint.

SummaryAusecaseisascenariothatdescribestheuseofasystembyanactortoaccomplishaspecificgoal.An"actor"isauser,playingarolewithrespecttothesystem,seekingtousethesystemtoaccomplishsomethingworthwhilewithinaparticularcontext.Ascenarioisasequenceofstepsthatdescribetheinteractionsbetweentheactorandthesystem.

Amajorcomponentoftransactiontestingistestdata.BorisBeizersuggeststhat30percentto40percentoftheeffortintransactiontestingisgenerating,capturing,orextractingtestdata.Don'tforgettoincluderesources(timeandpeople)forthisworkinyourproject'sbudget.

Whilecreatingatleastonetestcaseforthemainsuccessscenarioandatleastoneforeachextensionprovidessomeleveloftestcoverage,itisclearthat,nomatterhowmuchwetry,mostinputcombinationswillremainuntested.Donotbeoverconfidentaboutthequalityofthesystematthispoint.

Practice1. Giventhe"RegisterForACourse"usecasefortheStatelessUniversityRegistration

Systemdescribedpreviously,createasetoftestcasessothatthemainsuccessscenarioandeachoftheextensionsaretestedatleastonce.Choose"interesting"testdatausingtheequivalenceclassandboundaryvaluetechniques.

References


Beizer,Boris(1995).Black-BoxTesting:TechniquesforFunctionalTestingofSoftwareandSystems.JohnWiley&Sons.

Cockburn,Alistair(2000).WritingEffectiveUseCases.Addison-Wesley.

Fowler,MartinandKendallScott(1999).UMLDistilled:ABriefGuidetotheStandardObjectModelingLanguage(2ndEdition).Addison-Wesley.

Jacobsen,Ivar,etal.(1992).Object-OrientedSystemsEngineering:AUseCaseDrivenApproach.Addison-Wesley.

SectionII:WhiteBoxTestingTechniques

ChapterList

Chapter10:ControlFlowTesting

Chapter11:DataFlowTesting

PartOverview

DefinitionWhiteboxtestingisastrategyinwhichtestingisbasedontheinternalpaths,structure,andimplementationofthesoftwareundertest(SUT).Unlikeitscomplement,blackboxtesting,whiteboxtestinggenerallyrequiresdetailedprogrammingskills.

Thegeneralwhiteboxtestingprocessis:

TheSUT'simplementationisanalyzed.

PathsthroughtheSUTareidentified.

InputsarechosentocausetheSUTtoexecuteselectedpaths.Thisiscalledpathsensitization.Expectedresultsforthoseinputsaredetermined.

Thetestsarerun.

Actualoutputsarecomparedwiththeexpectedoutputs.

AdeterminationismadeastotheproperfunctioningoftheSUT.

ApplicabilityWhiteboxtestingcanbeappliedatalllevelsofsystemdevelopment—unit,integration,andsystem.Generallywhiteboxtestingisequatedwithunittestingperformedbydevelopers.Whilethisiscorrect,itisanarrowviewofwhiteboxtesting.

Whiteboxtestingismorethancodetesting—itispathtesting.Generally,thepathsthataretestedarewithinamodule(unittesting).Butwecanapplythesametechniquestotestpathsbetweenmoduleswithinsubsystems,betweensubsystemswithinsystems,andevenbetweenentiresystems.

DisadvantagesWhiteboxtestinghasfourdistinctdisadvantages.First,thenumberofexecutionpathsmaybesolargethantheycannotallbetested.Attemptingtotestallexecutionpathsthroughwhiteboxtestingisgenerallyasinfeasibleastestingallinputdatacombinationsthroughblackboxtesting.

Second,thetestcaseschosenmaynotdetectdatasensitivityerrors.Forexample:p=q/r;

mayexecutecorrectlyexceptwhenr=0.

y=2*x//shouldready=x2

willpassfortestcasesx=0,y=0andx=2,y=4

Third,whiteboxtestingassumesthecontrolflowiscorrect(orveryclosetocorrect).Sincethetestsarebasedontheexistingpaths,nonexistentpathscannotbediscoveredthroughwhiteboxtesting.

Fourth,thetestermusthavetheprogrammingskillstounderstandandevaluatethesoftwareundertest.Unfortunately,manytesterstodaydonothavethisbackground.

AdvantagesWhenusingwhiteboxtesting,thetestercanbesurethateverypaththroughthesoftwareundertesthasbeenidentifiedandtested.

Chapter10:ControlFlowTestingItwasfromtheprimevalwellspringofanantediluvianpassionthatmystoryariseswhich,liketheroundearthflattenedonamap,isbutalinearprojectionofanotherwiseperiphrasticandpolyphiloprogenitive,non-planar,non-didactic,self-invertingconstructionwhoseobscurantistgeotropicliminalityisbeyondreasonabledoubt.

—MilindaBanerjee

IntroductionControlflowtestingisoneoftwowhiteboxtestingtechniques.Thistestingapproachidentifiestheexecutionpathsthroughamoduleofprogramcodeandthencreatesandexecutestestcasestocoverthosepaths.Thesecondtechnique,discussedinthenextchapter,focusesondataflow.

KeyPoint

Path:Asequenceofstatementexecutionthatbeginsatanentryandendsatanexit.

Unfortunately,inanyreasonablyinterestingmodule,attemptingexhaustivetestingofallcontrolflowpathshasanumberofsignificantdrawbacks.

Thenumberofpathscouldbehugeandthusuntestablewithinareasonableamountoftime.Everydecisiondoublesthenumberofpathsandeveryloopmultipliesthepathsbythenumberofiterationsthroughtheloop.Forexample:for(i=1;i<=1000;i++)for(j=1;j<=1000;j++)for(k=1;k<=1000;k++)doSomethingWith(i,j,k);

executesdoSomethingWith()onebilliontimes(1000x1000x1000).Eachuniquepathdeservestobetested.

Pathscalledforinthespecificationmaysimplybemissingfromthemodule.Anytestingapproachbasedonimplementedpathswillneverfindpathsthatwerenotimplemented.if(a>0)doIsGreater();if(a==0)dolsEqual();//missingstatement-if(a<0)dolsLess();

Defectsmayexistinprocessingstatementswithinthemoduleeventhroughthecontrolflowitselfiscorrect.//actual(butincorrect)codea=a+1;//correctcodea=a-1;

Themodulemayexecutecorrectlyforalmostalldatavaluesbutfailforafew.intblech(inta,intb){returna/b;}

failsifbhasthevalue0butexecutescorrectlyifbisnot0.

Eventhoughcontrolflowtestinghasanumberofdrawbacks,itisstillavitaltoolinthetester's

toolbox.

Technique

ControlFlowGraphs

Controlflowgraphsarethefoundationofcontrolflowtesting.Thesegraphsdocumentthemodule'scontrolstructure.Modulesofcodeareconvertedtographs,thepathsthroughthegraphsareanalyzed,andtestcasesarecreatedfromthatanalysis.Controlflowgraphsconsistofanumberofelements:

KeyPoint Controlflowgraphsarethefoundationofcontrolflowtesting.

ProcessBlocks

Aprocessblockisasequenceofprogramstatementsthatexecutesequentiallyfrombeginningtoend.Noentryintotheblockispermittedexceptatthebeginning.Noexitfromtheblockispermittedexceptattheend.Oncetheblockisinitiated,everystatementwithinitwillbeexecutedsequentially.Processblocksarerepresentedincontrolflowgraphsbyabubblewithoneentryandoneexit.

DecisionPoint

Adecisionpointisapointinthemoduleatwhichthecontrolflowcanchange.Mostdecisionpointsarebinaryandareimplementedbyif-then-elsestatements.Multi-waydecisionpointsareimplementedbycasestatements.Theyarerepresentedbyabubblewithoneentryandmultipleexits.

JunctionPoint

Ajunctionpointisapointatwhichcontrolflowsjointogether.

Thefollowingcodeexampleisrepresentedbyitsassociatedflowgraph:

Figure10-1:Flowgraphequivalentofprogramcode.

LevelsofCoverage

Incontrolflowtesting,differentlevelsoftestcoveragearedefined.By"coverage"wemeanthepercentageofthecodethathasbeentestedvs.thatwhichistheretotest.Incontrolflowtestingwedefinecoverageatanumberofdifferentlevels.(Notethatthesecoveragelevelsarenotpresentedinorder.Thisisbecause,insomecases,itiseasiertodefineahighercoveragelevelandthendefinealowercoveragelevelintermsofthehigher.)

Level1Thelowestcoveragelevelis"100%statementcoverage"(sometimesthe"100%"isdroppedandisreferredtoas"statementcoverage").Thismeansthateverystatementwithinthemoduleisexecuted,undertest,atleastonce.Whilethismayseemlikeareasonablegoal,manydefectsmaybemissedwiththislevelofcoverage.Considerthefollowingcodesnippet:if(a>0){x=x+1;}

if(b==3){y=0;}

Thiscodecanberepresentedingraphicalformas:

Figure10-2:Graphicalrepresentationofthetwo-linecodesnippet.

Thesetwolinesofcodeimplementfourdifferentpathsofexecution:

Figure10-3:Fourexecutionpaths.

Whileasingletestcaseissufficienttotesteverylineofcodeinthismodule(forexample,usea=6andb=3asinput),itisapparentthatthislevelofcoveragewillmisstestingmanypaths.Thus,statementcoverage,whileabeginning,isgenerallynotanacceptableleveloftesting.

Eventhoughstatementcoverageisthelowestlevelofcoverage,eventhatmaybedifficulttoachieveinpractice.Oftenmoduleshavecodethatisexecutedonlyinexceptionalcircumstances—lowmemory,fulldisk,unreadablefiles,lostconnections,etc.Testersmayfinditdifficultorevenimpossibletosimulatethesecircumstancesandthuscodethatdealswiththeseproblemswillremainuntested.

Holodeckisatoolthatcansimulatemanyoftheseexceptionalsituations.AccordingtoHolodeck'sspecificationit"willallowyou,thetester,totestsoftwarebyobservingthesystemcallsthatitmakesandcreatetestcasesthatyoumayuseduringsoftwareexecutiontomodifythebehavioroftheapplication.Modificationsmightincludemanipulatingtheparameterssenttofunctionsorchangingthereturnvaluesoffunctionswithinyoursoftware.Inaddition,youmayalsoseterror-codesandothersystemevents.Thissetofpossibilitiesallowsyoutoemulateenvironmentsthatyoursoftwaremightencounter-hencethename'Holodeck.'Insteadofneedingtounplugyournetworkconnection,createadiskwithbadsectors,corruptpacketsonthenetwork,orperformanyoutsideorspecialmanipulationofyourmachine,youcanuseHolodecktoemulatetheseproblems.FaultscaneasilybeplacedintoanysoftwaretestingprojectthatyouareusingwithHolodeck."

Holodeck

TodownloadHolodeckvisithttp://www.sisecure.com/holodeck/holodeck-trial.aspx.

Level0Actually,thereisalevelofcoveragebelow"100%statementcoverage."Thatlevelisdefinedas"testwhateveryoutest;lettheuserstesttherest."Thecorporatelandscapeisstrewnwiththesun-bleachedbonesoforganizationswhohaveusedthistestingapproach.Regardingthislevelofcoverage,BorisBeizerwrote"testinglessthanthis[100%statementcoverage]fornewsoftwareisunconscionableandshouldbecriminalized....IncaseIhaven'tmademyselfclear,...untestedcodeinasystemisstupid,shortsighted,andirresponsible."

Level2Thenextlevelofcontrolflowcoverageis"100%decisioncoverage."Thisisalsocalled"branchcoverage."AtthislevelenoughtestcasesarewrittensothateachdecisionthathasaTRUEandFALSEoutcomeisevaluatedatleastonce.Inthepreviousexamplethiscanbeachievedwithtwotestcases(a=2,b=2anda=4,b=3).

http://www.sisecure.com/holodeck/holodeck-trial.aspx

Figure10-4:Twotestcasesthatyield100%decisioncoverage.

Casestatementswithmultipleexitswouldhavetestsforeachexit.Notethatdecisioncoveragedoesnotnecessarilyguaranteepathcoveragebutitdoesguaranteestatementcoverage.

Level3Notallconditionalstatementsareassimpleastheonespreviouslyshown.Considerthesemorecomplicatedstatements:if(a>0&&c==1){x=x+1;}if(b==3||d<0){y=0;}

TobeTRUE,thefirststatementrequiresagreaterthan0andcequal1.Thesecondrequiresbequal3ordlessthan0.

Inthefirststatementifthevalueofaweresetto0fortestingpurposesthenthec==1partoftheconditionwouldnotbetested.(Inmostprogramminglanguagesthesecondexpressionwouldnotevenbeevaluated.)Thenextlevelofcontrolflowcoverageis"100%conditioncoverage."AtthislevelenoughtestcasesarewrittensothateachconditionthathasaTRUEandFALSEoutcomethatmakesupadecisionisevaluatedatleastonce.Thislevelofcoveragecanbeachievedwithtwotestcases(a>0,c=1,b=3,d<0anda≤0,c≠1,b≠3,d≥0).Conditioncoverageisusuallybetterthandecisioncoveragebecauseeveryindividualconditionistestedatleastoncewhiledecisioncoveragecanbeachievedwithouttestingeverycondition.

Level4Considerthissituation:if(x&&y){conditionedStatement;}//note:&&indicateslogicalAND

Wecanachieveconditioncoveragewithtwotestcases(x=TRUE,y=FALSEandx=FALSE,y=TRUE)butnotethatwiththesechoicesofdatavaluestheconditionedStatementwillneverbeexecuted.Giventhepossiblecombinationofconditionssuchasthese,tobemorecomplete"100%decision/condition"coveragecanbeselected.Atthisleveltestcasesarecreatedforeveryconditionandeverydecision.

Level5Tobeevenmorethorough,considerhowtheprogramminglanguagecompileractuallyevaluatesthemultipleconditionsinadecision.Usethatknowledgetocreatetestcasesyielding"100%multipleconditioncoverage."if(a>0&&c==1){x=x+1;}if(b==3||d<0){y=0;}//note:||meanslogicalOR

willbeevaluatedas:

Figure10-5:Compilerevaluationofcomplexconditions.

Thislevelofcoveragecanbeachievedwithfourtestcases:a>0,c=1,b=3,d<0a≤0,c=1,b=3,d≥0a>0,c≠1,b≠3,d<0a≤0,c≠1,b≠3,d≥0

Achieving100%multipleconditioncoveragealsoachievesdecisioncoverage,conditioncoverage,anddecision/conditioncoverage.Notethatmultipleconditioncoveragedoesnotguaranteepathcoverage.

Level7Finallywereachthehighestlevel,whichis"100%pathcoverage."Forcodemoduleswithoutloopsthenumberofpathsisgenerallysmallenoughthatatestcasecanactuallybeconstructedforeachpath.Formoduleswithloops,thenumberofpathscanbeenormousandthusposeanintractabletestingproblem.

Figure10-6:Aninterestingflowdiagramwithmany,manypaths.

Level6Whenamodulehasloopsinthecodepathssuchthatthenumberofpathsisinfinite,asignificantbutmeaningfulreductioncanbemadebylimitingloopexecutiontoasmallnumberofcases.Thefirstcaseistoexecutetheloopzerotimes;thesecondistoexecutethelooponetime,thethirdistoexecutetheloopntimeswherenisasmallnumberrepresentingatypicalloopvalue;thefourthistoexecutetheloopitsmaximumnumberoftimesm.Inadditionyoumighttrym-1andm+1.

Beforebeginningcontrolflowtesting,anappropriatelevelofcoverageshouldbechosen.

StructuredTesting/BasisPathTesting

Nodiscussiononcontrolflowtestingwouldbecompletewithoutapresentationofstructuredtesting,alsoknownasbasispathtesting.StructuredtestingisbasedonthepioneeringworkofTomMcCabe.Itusesananalysisofthetopologyofthecontrolflowgraphtoidentifytestcases.

Thestructuredtestingprocessconsistsofthefollowingsteps:

Derivethecontrolflowgraphfromthesoftwaremodule.

Computethegraph'sCyclomaticComplexity(C).

SelectasetofCbasispaths.

Createatestcaseforeachbasispath.

Executethesetests.

Considerthefollowingcontrolflowgraph:

Figure10-7:Anexamplecontrolflowgraph.

McCabedefinestheCyclomaticComplexity(C)ofagraphas

C=edges-nodes+2

Edgesarethearrows,andnodesarethebubblesonthegraph.Theprecedinggraphhas24edgesand19nodesforaCyclomaticComplexityof24-19+2=7.

Insomecasesthiscomputationcanbesimplified.Ifalldecisionsinthegrapharebinary(theyhaveexactlytwoedgesflowingout),andtherearepbinarydecisions,then

C=p+1

CyclomaticComplexityisexactlytheminimumnumberofindependent,nonloopingpaths(calledbasispaths)thatcan,inlinearcombination,generateallpossiblepathsthroughthemodule.Intermsofaflowgraph,eachbasispathtraversesatleastoneedgethatnootherpathdoes.

McCabe'sstructuredtestingtechniquecallsforcreatingCtestcases,oneforeachbasispath.

IMPORTANT!

CreatingandexecutingCtestcases,basedonthebasispaths,guaranteesbothbranchandstatementcoverage.

Becausethesetofbasispathscoversalltheedgesandnodesofthecontrolflowgraph,satisfyingthisstructuredtestingcriteriaautomaticallyguaranteesbothbranchandstatementcoverage.

AprocessforcreatingasetofbasispathsisgivenbyMcCabe:1. Picka"baseline"path.Thispathshouldbeareasonably"typical"pathofexecution

ratherthananexceptionprocessingpath.Thebestchoicewouldbethemostimportantpathfromthetester'sview.

Figure10-8:ThechosenbaselinebasispathABDEGKMQS

2. Tochoosethenextpath,changetheoutcomeofthefirstdecisionalongthebaselinepathwhilekeepingthemaximumnumberofotherdecisionsthesameasthebaselinepath.

Figure10-9:ThesecondbasispathACDEGKMQS

3. Togeneratethethirdpath,beginagainwiththebaselinebutvarytheseconddecisionratherthanthefirst.

Figure10-10:ThethirdbasispathABDFILORS

4. Togeneratethefourthpath,beginagainwiththebaselinebutvarythethirddecisionratherthanthesecond.Continuevaryingeachdecision,onebyone,untilthebottomofthegraphisreached.

Figure10-11:ThefourthbasispathABDEHKMQS

Figure10-12:ThefifthbasispathABDEGKNQS

5. Nowthatalldecisionsalongthebaselinepathhavebeenflipped,weproceedtothesecondpath,flippingitsdecisions,onebyone.Thispatterniscontinueduntilthebasispathsetiscomplete.

Figure10-13:ThesixthbasispathACDFJLORS

Figure10-14:TheseventhbasispathACDFILPRS

Thus,asetofbasispathsforthisgraphare:ABDEGKMQS

ACDEGKMQS

ABDFILORS

ABDEHKMQS

ABDEGKNQS

ACDFJLORS

ACDFILPRS

Structuredtestingcallsforthecreationofatestcaseforeachofthesepaths.Thissetoftestcaseswillguaranteebothstatementandbranchcoverage.

Notethatmultiplesetsofbasispathscanbecreatedthatarenotnecessarilyunique.Eachset,however,hasthepropertythatasetoftestcasesbasedonitwillexecuteeverystatementandeverybranch.

ExampleConsiderthefollowingexamplefromBrown&Donaldson.ItisthecodethatdetermineswhetherB&Dshouldbuyorsellaparticularstock.Unfortunately,theinnerworkingsareahighlyclassifiedtradesecretsotheactualprocessingcodehasbeenremovedandgenericstatementslikes1;s2;etc.havesubstitutedforthem.Thecontrolflowstatementshavebeenleftintactbuttheiractualconditionshavebeenremovedandgenericconditionslikec1andc2havebeenputintheirplace.(Youdidn'tthinkwe'dreallyshowyouhowtoknowwhethertobuyorsellstocks,didyou?)

Note s1,s2,...representJavastatementswhilec1,c2,...representconditions.

booleanevaluateBuySell(TickerSymbolts){s1;s2;s3;if(c1){s4;s5;s6;}else{s7;s8;}while(c2){s9;s10;switch(c3){case-A:s20;s21;s22;break;//EndofCase-Acase-B:s30;s31;if(c4){s32;s33;s34;}else{s35;}break;//EndofCase-Bcase-C:s40;s41;break;//EndofCase-Ccase-D:

s50;break;//EndofCase-D}//EndSwitchs60;s61;s62;if(c5){s70;s71;}s80;s81;}//EndWhiles90;s91;s92;returnresult;

Figure10-15:JavacodeforBrown&Donaldson'sevaluateBuySellmodule.

ThefollowingflowdiagramcorrespondstothisJavacode: Figure10-16:ControlflowgraphforBrown&Donaldson'sevaluateBuySellmodule.

Thecyclomaticcomplexityofthisdiagramiscomputedbyedges-nodes+2

or

22-16+2=8

Let'sremovethecodeandlabeleachnodeforsimplicityindescribingthepaths.

Figure10-17:ControlflowgraphforBrown&Donaldson'sevaluateBuySellmodule.

Asetofeightbasispathsis:1. ABDP

2. ACDP

3. ABDEFGMODP

4. ABDEFHKMODP

5. ABDEFIMODP

6. ABDEFJMODP

7. ABDEFHLMODP

8. ABDEFIMNODP

Rememberthatbasispathsetsarenotunique;therecanbemultiplesetsofbasispathsforagraph.

Thisbasispathsetisnowimplementedastestcases.Choosevaluesfortheconditionsthat

wouldsensitizeeachpathandexecutethetests.

Table10-1:Datavaluestosensitizethedifferentcontrolflowpaths.

TestCase C1 C2 C3 C4 C5

1 False False N/A N/A N/A

2 True False N/A N/A N/A

3 False True A N/A False

4 False True B False False

5 False True C N/A False

6 False True D N/A False

7 False True B True False

8 False True C N/A True

ApplicabilityandLimitationsControlflowtestingisthecornerstoneofunittesting.Itshouldbeusedforallmodulesofcodethatcannotbetestedsufficientlythroughreviewsandinspections.Itslimitationsarethatthetestermusthavesufficientprogrammingskilltounderstandthecodeanditscontrolflow.Inaddition,controlflowtestingcanbeverytimeconsumingbecauseofallthemodulesandbasispathsthatcompriseasystem.

SummaryControlflowtestingidentifiestheexecutionpathsthroughamoduleofprogramcodeandthencreatesandexecutestestcasestocoverthosepaths.

Controlflowgraphsarethefoundationofcontrolflowtesting.Modulesofcodeareconvertedtographs,thepathsthroughthegraphsareanalyzed,andtestcasesarecreatedfromthatanalysis.

CyclomaticComplexityisexactlytheminimumnumberofindependent,nonloopingpaths(calledbasispaths)thatcan,inlinearcombination,generateallpossiblepathsthroughthemodule.

Becausethesetofbasispathscoversalltheedgesandnodesofthecontrolflowgraph,satisfyingthisstructuredtestingcriteriaautomaticallyguaranteesbothbranchandstatementcoverage.

Practice1. Belowisabriefprogramlisting.Createthecontrolflowdiagram,determineits

CyclomaticComplexity,chooseasetofbasispaths,anddeterminethenecessaryvaluesfortheconditionstosensitizeeachpath.

if(c1){while(c2){if(c3){s1;s2;if(c5)s5;elses6;break;//Skiptoendofwhileelseif(c4){}else{s3;s4;break;}}//Endofwhile}//Endofifs7;if(c6)s8;s9;s10;

References



Pressman,RogerS.(1982).SoftwareEngineering:APractitioner'sApproach(FourthEdition).McGraw-Hill.

Watson,ArthurH.andThomasJ.McCabe.StructuredTesting:ATestingMethodologyUsingtheCyclomaticComplexityMetric.NISTSpecialPublication500-235availableathttp://www.mccabe.com/nist/nist_pub.php

http://www.mccabe.com/nist/nist_pub.php

Chapter11:DataFlowTestingHollyhadreachedtheageandlevelofmaturitytocomprehendtheemotionalnuancesofThomasWolfe'sassertion"youcan'tgohomeagain,"butinhercaseitwasevenmorepoignantbecausetherewasnohometoreturnto:herparentshadseparated,soldthehouse,euthanizedBowser,anddisownedHollyfordroppingoutofhighschooltomarrythat43-year-oldmanagerofTrailerTowninIdaho—andeventheirtrailerwasn'taplaceshecouldcallhomebecauseitwasonlyasummersublet.

—EileenOstrowFeldman

IntroductionAlmosteveryprogrammerhasmadethistypeofmistake:main(){intx;if(x==42){...}}

Themistakeisreferencingthevalueofavariablewithoutfirstassigningavaluetoit.Naivedevelopersunconsciouslyassumethatthelanguagecompilerorrun-timesystemwillsetallvariablestozero,blanks,TRUE,42,orwhatevertheyrequirelaterintheprogram.AsimpleCprogramthatillustratesthisdefectis:#include<stdio.h>main(){intx;printf("%d",x);}

Thevalueprintedwillbewhatevervaluewas"leftover"inthememorylocationtowhichxhasbeenassigned,notnecessarilywhattheprogrammerwantedorexpected.

Dataflowtestingisapowerfultooltodetectjustsucherrors.RappsandWeyuker,popularizersofthisapproach,wrote,"Itisourbeliefthat,justasonewouldnotfeelconfidentaboutaprogramwithoutexecutingeverystatementinitaspartofsometest,oneshouldnotfeelconfidentaboutaprogramwithouthavingseentheeffectofusingthevalueproducedbyeachandeverycomputation."

KeyPoint

Dataflowtestingisapowerfultooltodetectimproperuseofdatavaluesduetocodingerrors.

TechniqueVariablesthatcontaindatavalueshaveadefinedlifecycle.Theyarecreated,theyareused,andtheyarekilled(destroyed).Insomeprogramminglanguages(FORTRANandBASIC,forexample)creationanddestructionareautomatic.Avariableiscreatedthefirsttimeitisassignedavalueanddestroyedwhentheprogramexits.

Inotherlanguages(likeC,C++,andJava)thecreationisformal.Variablesaredeclaredbystatementssuchas:intx;//xiscreatedasanintegerstringy;//yiscreatedasastring

Thesedeclarationsgenerallyoccurwithinablockofcodebeginningwithanopeningbrace{andendingwithaclosingbrace}.Variablesdefinedwithinablockarecreatedwhentheirdefinitionsareexecutedandareautomaticallydestroyedattheendofablock.Thisiscalledthe"scope"ofthevariable.Forexample:{//beginouterblockintx;//xisdefinedasanintegerwithinthisouterblock...;//xcanbeaccessedhere{//begininnerblockinty;//yisdefinedwithinthisinnerblock...;//bothxandycanbeaccessedhere}//yisautomaticallydestroyedattheendof//thisblock...;//xcanstillbeaccessed,butyisgone}//xisautomaticallydestroyed

Variablescanbeusedincomputation(a=b+1).Theycanalsobeusedinconditionals(if(a>42)).Inbothusesitisequallyimportantthatthevariablehasbeenassignedavaluebeforeitisused.

Threepossibilitiesexistforthefirstoccurrenceofavariablethroughaprogrampath:

1.~dthevariabledoesnotexist(indicatedbythe~),thenitisdefined(d)

2.~uthevariabledoesnotexist,thenitisused(u)

3.~k thevariabledoesnotexist,thenitiskilledordestroyed(k)

Thefirstiscorrect.Thevariabledoesnotexistandthenitisdefined.Thesecondisincorrect.Avariablemustnotbeusedbeforeitisdefined.Thethirdisprobablyincorrect.Destroyingavariablebeforeitiscreatedisindicativeofaprogrammingerror.

Nowconsiderthefollowingtime-sequencedpairsofdefined(d),used(u),andkilled(k):

ddDefinedanddefinedagain—notinvalidbutsuspicious.Probablyaprogrammingerror.

duDefinedandused—perfectlycorrect.Thenormalcase.

dkDefinedandthenkilled—notinvalidbutprobablyaprogrammingerror.

udUsedanddefined—acceptable.

uuUsedandusedagain—acceptable.

uk Usedandkilled—acceptable.

kdKilledanddefined—acceptable.Avariableiskilledandthenredefined.

kuKilledandused—aseriousdefect.Usingavariablethatdoesnotexistorisundefinedisalwaysanerror.

kk Killedandkilled—probablyaprogrammingerror.

KeyPoint Examinetime-sequencedpairsofdefined,used,andkilledvariablereferences.

Adataflowgraphissimilartoacontrolflowgraphinthatitshowstheprocessingflowthroughamodule.Inaddition,itdetailsthedefinition,use,anddestructionofeachofthemodule'svariables.Wewillconstructthesediagramsandverifythatthedefine-use-killpatternsareappropriate.First,wewillperformastatictestofthediagram.By"static"wemeanweexaminethediagram(formallythroughinspectionsorinformallythroughlook-sees).Second,weperformdynamictestsonthemodule.By"dynamic"wemeanweconstructandexecutetestcases.Let'sbeginwiththestatictesting.

StaticDataFlowTesting

Thefollowingcontrolflowdiagramhasbeenannotatedwithdefine-use-killinformationforeachofthevariablesusedinthemodule.

Figure11-1:Thecontrolflowdiagramannotatedwithdefine-use-killinformationforeachofthemodule'svariables.

Foreachvariablewithinthemodulewewillexaminedefine-use-killpatternsalongthecontrolflowpaths.Considervariablexaswetraversetheleftandthentherightpath:

Figure11-2:Thecontrolflowdiagramannotatedwithdefine-use-killinformationforthexvariable.

Thedefine-use-killpatternsforx(takeninpairsaswefollowthepaths)are:

~define correct,thenormalcase

define-definesuspicious,perhapsaprogrammingerror

define-use correct,thenormalcase

Nowforvariabley.Notethatthefirstbranchinthemodulehasnoimpactontheyvariable.

Figure11-3:Thecontrolflowdiagramannotatedwithdefine-use-killinformationfortheyvariable.

Thedefine-use-killpatternsfory(takeninpairsaswefollowthepaths)are:

~use majorblunder

use-defineacceptable

define-usecorrect,thenormalcase

use-kill acceptable

define-kill probableprogrammingerror

Nowforvariablez.

Figure11-4:Thecontrolflowdiagramannotatedwithdefine-use-killinformationforthezvariable.

Thedefine-use-killpatterns(takeninpairsaswefollowthepaths)are:

~kill programmingerror

kill-use majorblunder

use-use correct,thenormalcase

use-defineacceptable

kill-kill probablyaprogrammingerror

kill-define acceptable

define-usecorrect,thenormalcase

Inperformingastaticanalysisonthisdataflowmodelthefollowingproblemshavebeendiscovered:x:define-definey:~usey:define-killz:~killz:kill-usez:kill-kill

Unfortunately,whilestatictestingcandetectmanydataflowerrors,itcannotfindthemall.Considerthefollowingsituations:

1. Arraysarecollectionsofdataelementsthatsharethesamenameandtype.Forexampleintstuff[100];

definesanarraynamedstuffconsistingof100integerelements.InC,C++,andJava

theindividualelementsarenamedstuff[0],stuff[1],stuff[2],etc.Arraysaredefinedanddestroyedasaunitbutspecificelementsofthearrayareusedindividually.Oftenprogrammersrefertostuff[j]wherejchangesdynamicallyastheprogramexecutes.Inthegeneralcase,staticanalysiscannotdeterminewhetherthedefine-use-killruleshavebeenfollowedproperlyunlesseachelementisconsideredindividually.

2. Incomplexcontrolflowsitispossiblethatacertainpathcanneverbeexecuted.Inthiscaseanimproperdefine-use-killcombinationmightexistbutwillneverbeexecutedandsoisnottrulyimproper.

3. Insystemsthatprocessinterrupts,someofthedefine-use-killactionsmayoccurattheinterruptlevelwhileotherdefine-use-killactionsoccuratthemainprocessinglevel.Inaddition,ifthesystemusesmultiplelevelsofexecutionpriorities,staticanalysisofthemyriadofpossibleinteractionsissimplytoodifficulttoperformmanually.

Forthisreason,wenowturntodynamicdataflowtesting.

DynamicDataFlowTesting

Becausedataflowtestingisbasedonamodule'scontrolflow,itassumesthatthecontrolflowisbasicallycorrect.Thedataflowtestingprocessistochooseenoughtestcasessothat:

Every"define"istracedtoeachofits"uses"

Every"use"istracedfromitscorresponding"define"

Todothis,enumeratethepathsthroughthemodule.Thisisdoneusingthesameapproachasincontrolflowtesting:Beginatthemodule'sentrypoint,taketheleftmostpaththroughthemoduletoitsexit.Returntothebeginningandvarythefirstbranchingcondition.Followthatpathtotheexit.Returntothebeginningandvarythesecondbranchingcondition,thenthethird,andsoonuntilallthepathsarelisted.Then,foreveryvariable,createatleastonetestcasetocovereverydefine-usepair.

ApplicabilityandLimitationsDataflowtestingbuildsonandexpandscontrolflowtestingtechniques.Aswithcontrolflowtesting,itshouldbeusedforallmodulesofcodethatcannotbetestedsufficientlythroughreviewsandinspections.Itslimitationsarethatthetestermusthavesufficientprogrammingskilltounderstandthecode,itscontrolflow,anditsvariables.Likecontrolflowtesting,dataflowtestingcanbeverytimeconsumingbecauseofallthemodules,paths,andvariablesthatcompriseasystem.

SummaryAcommonprogrammingmistakeisreferencingthevalueofavariablewithoutfirstassigningavaluetoit.

Adataflowgraphissimilartoacontrolflowgraphinthatitshowstheprocessingflowthroughamodule.Inaddition,itdetailsthedefinition,use,anddestructionofeachofthemodule'svariables.Wewillusethesediagramstoverifythatthedefine-use-killpatternsareappropriate.

Enumeratethepathsthroughthemodule.Then,foreveryvariable,createatleastonetestcasetocovereverydefine-usepair.

Practice1. Thefollowingmoduleofcodecomputesn!(nfactorial)givenavalueforn.Create

dataflowtestcasescoveringeachvariableinthismodule.Remember,asingletestcasecancoveranumberofvariables.intfactorial(intn){intanswer,counter;answer=1;counter=1;

loop:if(counter>n)returnanswer;answer=answer*counter;counter=counter+1;gotoloop;}

2. Diagramthecontrolflowpathsandderivethedataflowtestcasesforthefollowingmodule:intmodule(intselector){intfoo,bar;switchselector{caseSELECT-1:foo=calc_foo_method_1();break;caseSELECT-2:foo=calc_foo_method_2();break;caseSELECT-3:foo=calc_foo_method_3();break;}switchfoo{caseFOO-1:bar=calc_bar_method_1();break;caseFOO-2:bar=calc_bar_method_2();break;}returnfoo/bar;}

Doyouhaveanyconcernswiththiscode?Howwouldyoudealwiththem?

References

Beizer,Boris(1990).SoftwareTestingTechniques.VanNostrandReinhold.


Marick,Brian(1995).TheCraftofSoftwareTesting:SubsystemTestingIncludingObject-BasedandObject-OrientedTesting.Prentice-Hall.

Rapps,SandraandElaineJ.Weyuker."DataFlowAnalysisTechniquesforTestDataSelection."SixthInternationalConferenceonSoftwareEngineering,Tokyo,Japan,September13–16,1982.

SectionIII:TestingParadigms

ChapterList

Chapter12:ScriptedTesting

Chapter13:ExploratoryTesting

Chapter14:TestPlanning

PartOverview

ParadigmsInhisbook,Paradigms:TheBusinessofDiscoveringtheFuture,JoelBarkerdefinesaparadigmas"asetofrulesandregulations(writtenorunwritten)thatdoestwothings:(1)itestablishesordefinesboundaries,and(2)ittellsyouhowtobehaveinsidetheboundariesinordertobesuccessful."FuturistMarilynFergusonseesaparadigmas"aframeworkofthought...aschemeforunderstandingandexplainingcertainaspectsofreality."

Paradigmsareusefulbecausetheyhelpusmakesenseofthecomplexitiesoftheworldaroundus.Inthisway,paradigmssharpenourvision.Butparadigmscanblindustorealities.Paradigmsactaspsychologicalfilters.Datathatdoesnotmatchourparadigmsisblocked.Inthisway,paradigmscloudourvision.

Insoftwaretestingtoday,twoverydifferentparadigmsarebattlingforadherents—scriptedtestingandexploratorytesting.

Scriptedtestingisbasedonthesequentialexaminationofrequirements,followedbythedesignanddocumentationoftestcases,followedbytheexecutionofthosetestcases.Thescriptedtester'smottois,"Planyourwork,workyourplan."

Exploratorytestingisaverydifferentparadigm.Ratherthanasequentialapproach,exploratorytestingemphasizessimultaneouslearning,testdesign,andtestexecution.Thetesterdesignsandexecutestestswhileexploringtheproduct.

WordOfWarning!

Inthefollowingchaptersthescriptedandexploratoryparadigmsaredefinedattheextremeendsofthespectrum.Rarelywilleitherbeusedasinflexiblyasdescribed.

Thenexttwochaptersdescribetheseparadigms.Awordofwarningthough—eachparadigmisdescribedattheextremeendoftheprocessspectrum.Rarelywilleitherparadigmbeusedasinflexiblyasdescribed.Moreoften,scriptedtestingmaybesomewhatexploratoryandexploratorytestingmaybesomewhatscripted.

TestPlanningPlanninghasbeendefinedassimply"figuringoutwhattodonext."Tobemosteffectiveandefficient,planningisimportant.Butwhenandhowshouldthatplanningbedone?Scriptedtestingemphasizesthevalueofearlytestdesignasamethodofdetectingrequirementsanddesigndefectsbeforethecodeiswrittenandthesystemputintoproduction.Itsfocusisonaccountabilityandrepeatability.Exploratorytestingchallengestheideathattestsmustbedesignedsoveryearlyintheproject,whenourknowledgeistypicallyatitsminimum.Itsfocusisonlearningandadaptability.

References

Barker,JoelArthur(1992).Paradigms:TheBusinessofDiscoveringtheFuture.HarperCollins.

Ferguson,Marilyn(1980).TheAquarianConspiracy:PersonalandSocialTransformationinOurTime.PutnamPublishingGroup.

Chapter12:ScriptedTestingJanewastoast,andnotthelightbutterykind,nay,shewasthekindthat'sbeencharredandblackenedinthebottomofthetoasterandhastobethrownawaybecausenomatterhowmuchoftheburntpartyouscrapeoffwithaknife,there'salwaysmoreblackenedtoastbeneath,thekindthatnotevenstarvingbirdsinwinterwilleat,thatkindoftoast.

—BethKnutson

IntroductionForscriptedtestingtobeunderstood,itmustbeunderstoodinitshistoricalcontext.ScriptedtestingemergedasoneofthecomponentpartsoftheWaterfallmodelofsoftwaredevelopment.TheWaterfallmodeldefinesanumberofsequentialdevelopmentphaseswithspecificentryandexitcriteria,taskstobeperformed,anddeliverables(tangibleworkproducts)tobecreated.Itisaclassicexampleofthe"planyourwork,workyourplan"philosophy.TypicalWaterfallphasesinclude:

1. SystemRequirements-Gatheringtherequirementsforthesystem.

2. SoftwareRequirements-Gatheringtherequirementsforthesoftwareportionofthesystem.

3. RequirementsAnalysis-Analyzing,categorizing,andrefiningthesoftwarerequirements.

4. ProgramDesign-Choosingarchitectures,modules,andinterfacesthatdefinethesystem.

5. Coding-Writingtheprogrammingcodethatimplementsthedesign.

6. Testing-Evaluatingwhethertherequirementswereproperlyunderstood(Validation)andthedesignproperlyimplementedbythecode(Verification).

7. Operations-Putthesystemintoproduction.

InterestingTriviaAGooglesearchfor"planyourwork"and"workyourplan"found3,570matchesincluding:

Footballrecruiting

Businessplanning

Buildingwithconcreteblocks

Onlinemarketing

Industrialdistribution

ThePrincetonUniversity'sWomen'sWaterPoloTeam

Andthousandsmore

Thismodelwasfirstdescribedin1970inapaperentitled"ManagingtheDevelopmentofLargeScaleSystems"byDr.WinstonW.Royce.Roycedrewthefollowingdiagramshowingtherelationshipsbetweendevelopmentphases:

Figure12-1:TheWaterfalllifecyclemodel.

WhatprocesswasusedbeforeWaterfall?Itisaprocessknownas"Code&Fix."Programmerssimplycoded.Sloganslike"Requirements?Requirements?Wedon'tneednostinkin'Requirements!"hungonthewallsofprogrammers'offices.DevelopmentwaslikethesceneinthemovieRaidersoftheLostArk.Ourhero,IndianaJones,ishidingfromthebadguys.Indysays,"I'mgoingtogetthattruck."Marion,ourheroine,turnstohimandasks,"Howareyougoingtogetthattruck?"Indyreplies,"Idon'tknow.I'mmakingthisupasIgo."Ifwesubstituted"buildthatsystem"for"getthattruck"we'dhavethewayrealmenandrealwomenbuiltsoftwaresystemsinthegoodolddays.

CuriousHistoricalNoteToday,WinstonRoyceisknownasthefatheroftheWaterfallmodelofsoftwaredevelopment.Infact,inhispaperhewasactuallyproposinganiterativeandincrementalprocessthatincludedearlyprototyping-somethingmanyorganizationsarejustnowdiscovering.

Todaywetakeadifferentviewofscriptedtesting.AnydevelopmentmethodologyalongthespectrumfromWaterfalltoRapidApplicationDevelopment(RAD)mayusescriptedtesting.Wheneverrepeatability,objectivity,andauditabilityareimportant,scriptedtestingcanbeused.

Repeatabilitymeansthatthereisadefinitionofatest(fromdesignthroughtodetailedprocedure)atalevelofdetailsufficientforsomeoneotherthantheauthortoexecuteitinanidenticalway.Objectivitymeansthatthetestcreationdoesnotdependontheextrordinary(nearmagical)skillofthepersoncreatingthetestbutisbasedonwellunderstoodtestdesignprinciples.Auditabilityincludestraceabilityfromrequirements,design,andcodetothetestcasesandbackagain.Thisenablesformalmeasuresoftestingcoverage.

"Planyourwork,workyourplan."Nophrasesoepitomizesthescriptedtestingapproachasdoesthisone,andnodocumentsoepitomizesthescriptedtestingapproachasdoesIEEEStd829-1998,the"IEEEStandardforSoftwareTestDocumentation."

Thisstandarddefineseightdocumentsthatcanbeusedinsoftwaretesting.Thesedocumentsare:

Testplan

Testdesignspecification

Testcasespecification

Testprocedurespecification

Testitemtransmittalreport

Testlog

Testincidentreport

Testsummaryreport

Figure12-2showstherelationshipsbetweenthesedocuments.Notethatthefirstfourdocumentsthatdefinethetestplan,testdesigns,andtestcasesareallcreatedbeforetheproductisdevelopedandtheactualtestingisbegun.Thisisakeyideainscriptedtesting—planthetestsbasedontheformalsystemrequirements.

Figure12-2:TheIEEE829TestDocuments Curiously,theIEEE829standardstates,"Thisstandardspecifiestheformandcontentofindividualtestdocuments.Itdoesnotspecifytherequiredsetoftestdocuments."Inotherwords,thestandarddoesnotrequireyoutocreateanyofthedocumentsdescribed.Thatchoiceislefttoyouasatester,ortoyourorganization.But,thestandardrequiresthatifyouchoosetowriteatestplan,testcasespecification,etc.,thatdocumentmustfollowtheIEEE829standard.

TheIEEE829standardliststheseadvantagesforitsuse:

"Astandardizedtestdocumentcanfacilitatecommunicationbyprovidingacommonframeofreference.

Thecontentdefinitionofastandardizedtestdocumentcanserveasacompletenesschecklistfortheassociatedtestingprocess.

Astandardizedsetcanalsoprovideabaselinefortheevaluationofcurrenttestdocumentationpractices.

Theuseofthesedocumentssignificantlyincreasesthemanageabilityoftesting.Increasedmanageabilityresultsfromthegreatlyincreasedvisibilityofeachphaseofthetestingprocess."

IEEE829DocumentDescriptionsTheIEEE829standarddefineseightdifferentdocuments.Eachdocumentiscomposedofanumberofsections.

TestPlan

Thepurposeofthetestplanistodescribethescope,approach,resources,andscheduleofthetestingactivities.Itdescribestheitems(components)andfeatures(functionality,performance,security,usability,etc.)tobetested,taskstobeperformed,deliverables(tangibleworkproducts)tobecreated,testingresponsibilities,schedules,andapprovalsrequired.Testplanscanbecreatedattheprojectlevel(mastertestplan)oratsubsidiarylevels(unit,integration,system,acceptance,etc.).Thetestplaniscomposedofthefollowingsections:

1. Testplanidentifier-Auniqueidentifiersothatthisdocumentcanbedistinguishedfromallotherdocuments.

2. Introduction-Asummaryofthesoftwaretobetested.Abriefdescriptionandhistorymaybeincludedtosetthecontext.Referencestootherrelevantdocumentsusefulforunderstandingthetestplanareappropriate.Definitionsofunfamiliartermsmaybeincluded.

3. Testitems-Identifiesthesoftwareitemsthataretobetested.Theword"item"ispurposelyvague.Itisa"chunk"ofsoftwarethatistheobjectoftesting.

4. Featurestobetested-Identifiesthecharacteristicsoftheitemstobetested.Theseincludefunctionality,performance,security,portability,usability,etc.

5. Featuresnottobetested-Identifiescharacteristicsoftheitemsthatwillnotbetestedandthereasonswhy.

6. Approach-Theoverallapproachtotestingthatwillensurethatallitemsandtheirfeatureswillbeadequatelytested.

7. Itempass/failcriteria-Thecriteriausedtodeterminewhethereachtestitemhaspassedorfailedtesting.

8. Suspensioncriteriaandresumptionrequirements-Theconditionsunderwhichtestingwillbesuspendedandthesubsequentconditionsunderwhichtesting

willberesumed.

9. Testdeliverables-Identifiesthedocumentsthatwillbecreatedasapartofthetestingprocess.

10. Testingtasks-Identifiesthetasksnecessarytoperformthetesting.

11. Environmentalneeds-Specifiestheenvironmentrequiredtoperformthetestingincludinghardware,software,communications,facilities,tools,people,etc.

12. Responsibilities-Identifiesthepeople/groupsresponsibleforexecutingthetestingtasks.

13. Staffingandtrainingneeds-Specifiesthenumberandtypesofpeoplerequiredtoperformthetesting,includingtheskillsneeded.

14. Schedule-Definestheimportantkeymilestonesanddatesinthetestingprocess.

15. Risksandcontingencies-Identifieshigh-riskassumptionsofthetestingplan.Specifiespreventionandmitigationplansforeach.

16. Approvals-Specifiesthenamesandtitlesofeachpersonwhomustapprovetheplan.

TestDesignSpecification

Thepurposeofthetestdesignspecificationistoidentifyasetoffeaturestobetestedandtodescribeagroupoftestcasesthatwilladequatelytestthosefeatures.Inaddition,refinementstotheapproachlistedinthetestplanmaybespecified.Thetestdesignspecificationiscomposedofthefollowingsections:

1. Testdesignspecificationidentifier-Auniqueidentifiersothatthisdocumentcanbedistinguishedfromallotherdocuments.

2. Featurestobetested-Identifiesthetestitemsandthefeaturesthataretheobjectofthistestdesignspecification.

3. Approachrefinements-Specifiesthetesttechniquestobeusedforthistestdesign.

4. Testidentification-Liststhetestcasesassociatedwiththistestdesign.

Providesauniqueidentifierandashortdescriptionforeachtestcase.

5. Featurepass/failcriteria-Thecriteriausedtodeterminewhethereachfeaturehaspassedorfailedtesting.

TestCaseSpecification

Thepurposeofthetestcasespecificationistospecifyindetaileachtestcaselistedinthetestdesignspecification.Thetestcasespecificationiscomposedofthefollowingsections:

1. Testcasespecificationidentifier-Auniqueidentifiersothatthisdocumentcanbedistinguishedfromallotherdocuments.

2. Testitems-Identifiestheitemsandfeaturestobetestedbythistestcase.

3. Inputspecifications-Specifieseachinputrequiredbythistestcase.

4. Outputspecifications-Specifieseachoutputexpectedafterexecutingthistestcase.

5. Environmentalneeds-Anyspecialhardware,software,facilities,etc.requiredfortheexecutionofthistestcasethatwerenotlistedinitsassociatedtestdesignspecification.

6. Specialproceduralrequirements-Definesanyspecialsetup,execution,orcleanupproceduresuniquetothistestcase.

7. Intercasedependencies-Listsanytestcasesthatmustbeexecutedpriortothistestcase.

TestProcedureSpecification

Thepurposeofthetestprocedurespecificationistospecifythestepsforexecutingatestcaseandtheprocessfordeterminingwhetherthesoftwarepassedorfailedthetest.Thetestprocedurespecificationiscomposedofthefollowingsections:

1. Testprocedurespecificationidentifier-Auniqueidentifiersothatthisdocumentcanbedistinguishedfromallotherdocuments.

2. Purpose-Describesthepurposeofthetestprocedureanditscorrespondingtestcases.

3. Specialrequirements-Listsanyspecialrequirementsfortheexecutionofthistestprocedure.

4. Proceduresteps-Liststhestepsoftheprocedure.Possiblestepsinclude:Setup,Start,Proceed,Measure,ShutDown,Restart,Stop,andWrapUp.

TestItemTransmittalReport(a.k.a.ReleaseNotes)

Thepurposeofthetestitemtransmittalreportistospecifythetestitemsbeingprovidedfortesting.Thetestitemtransmittalreportiscomposedofthefollowingsections:

1. Transmittalreportidentifier-Auniqueidentifiersothatthisdocumentcanbedistinguishedfromallotherdocuments.

2. Transmitteditems-Liststheitemsbeingtransmittedfortestingincludingtheirversionorrevisionlevel.

3. Location-Identifiesthelocationofthetransmitteditems.

4. Status-Describesthestatusoftheitemsbeingtransmitted.Includeanydeviationsfromtheitem'sspecifications.

5. Approvals-Specifiesthenamesandtitlesofallpersonswhomustapprovethistransmittal.

TestLog

Thepurposeofthetestlogistoprovideachronologicalrecordaboutrelevantdetailsobservedduringthetestexecution.Thetestlogiscomposedofthefollowingsections:

1. Testlogidentifier-Auniqueidentifiersothatthisdocumentcanbedistinguishedfromallotherdocuments.

2. Description-Identifiestheitemsbeingtestedandtheenvironmentunderwhichthetestwasperformed.

3. Activityandevententries-Foreachevent,liststhebeginningandendingdateandtime,abriefdescriptionofthetestexecution,theresultsofthetest,anduniqueenvironmentalinformation,anomalouseventsobserved,andtheincidentreportidentifierifanincidentwaslogged.

TestIncidentReport(a.k.a.BugReport)

Thepurposeofthetestincidentreportistodocumentanyeventobservedduringtestingthatrequiresfurtherinvestigation.Thetestincidentreportiscomposedofthefollowingsections:

1. Testincidentreportidentifier-Auniqueidentifiersothatthisdocumentcanbedistinguishedfromallotherdocuments.

2. Summary-Summarizestheincident.

3. Incidentdescription-Describestheincidentintermsofinputs,expectedresults,actualresults,environment,attemptstorepeat,etc.

4. Impact-Describestheimpactthisincidentwillhaveonothertestplans,testdesignspecifications,testprocedures,andtestcasespecifications.Alsodescribes,ifknown,theimpactthisincidentwillhaveonfurthertesting.

TestSummaryReport

Thepurposeofthetestsummaryreportistosummarizetheresultsofthetestingactivitiesandtoprovideanevaluationbasedontheseresults.Thetestsummaryreportiscomposedofthefollowingsections:

1. Testsummaryreportidentifier-Auniqueidentifier(imaginethat!)sothatthis

documentcanbedistinguishedfromallotherdocuments.

2. Summary-Summarizestheevaluationofthetestitems.

3. Variance-Reportsanyvariancesfromtheexpectedresults.

4. Comprehensiveassessment-Evaluatestheoverallcomprehensivenessofthetestingprocessitselfagainstcriteriaspecifiedinthetestplan.

5. Summaryofresults-Summarizestheresultsofthetesting.Identifiesallunresolvedincidents.

6. Evaluation-Providesanoverallevaluationofeachtestitemincludingitslimitations.

7. Summaryofactivities-Summarizesthemajortestingactivitiesbytaskandresourceusage.

8. Approvals-Specifiesthenamesandtitlesofeachpersonwhomustapprovethereport.

AdvantagesofScriptedTesting1. Scriptedtestingprovidesadivisionoflabor—planning,testcasedesign,testcase

implementation,andtestcaseexecutionthatcanbeperformedbypeoplewithspecificskillsandatdifferenttimesduringthedevelopmentprocess.

2. Testdesigntechniquessuchasequivalenceclasspartitioning,boundaryvaluetesting,controlflowtesting,pairwisetesting,etc.canbeintegratedintoaformaltestingprocessdescriptionthatnotonlyguidesourtestingbutthatcouldalsobeusedtoauditforprocesscompliance.

3. Becausescriptedtestsarecreatedfromrequirements,design,andcode,allimportantattributesofthesystemwillbecoveredbytestsandthiscoveragecanbedemonstrated.

4. Becausethetestcasescanbetracedbacktotheirrespectiverequirements,design,andcode,coveragecanbeclearlydefinedandmeasured.

5. Becausethetestsaredocumented,theycanbeeasilyunderstoodandrepeatedwhennecessarywithoutadditionaltestanalysisordesigneffort.

6. Becausethetestsaredefinedindetail,theyaremoreeasilyautomated.

7. Becausethetestsarecreatedearlyinthedevelopmentprocess,thismayfreeupadditionaltimeduringthecriticaltestexecutionperiod.

8. Insituationswhereagoodrequirementsspecificationislacking,thetestcases,attheendoftheproject,becomethedefactorequirementsspecification,includingtheresultsthatdemonstratewhichrequirementswereactuallyfulfilledandwhichwerenot.

9. Scriptedtests,whenwrittentotheappropriatelevelofdetail,canberunbypeoplewhowouldotherwisenotbeabletotestthesystembecauseoflackofdomainknowledgeorlackoftestingknowledge.

10. Youmayhavespecialcontractualrequirementsthatcanonlybemetbyscriptedtesting.

11. Theremaybecertainteststhatmustbeexecutedinjustthesameway,everytime,inordertoserveasakindofbenchmark.

12. Bycreatingthetestsearlyintheprojectwecandiscoverwhatwedon'tknow.

13. Bycreatingthetestsearlywecanfocusonthe"bigpicture."

InhisbookSoftwareSystemTestingandQualityAssurance,BorisBeizersummarizesinthisway:

"Testingislikeplayingpool.There'srealpoolandkiddiepool.Inkiddiepool,youhitthe

ballsandwhateverpockettheyhappentofallinto,youclaimastheintendedpocket.It'snotmuchofagameandalthoughsuitabletoten-year-oldsit'shardlyachallenge.Theobjectofrealpoolistospecifythepocketinadvance.Similarlyfortesting.There'srealtestingandkiddietesting.Inkiddietesting,thetestersays,afterthefact,thattheobservedoutcomewastheintendedoutcome.Inrealtestingtheoutcomeispredictedanddocumentedbeforethetestisrun."

DisadvantagesofScriptedTesting1. Scriptedtestingisverydependentonthequalityofthesystem'srequirements.Will

therequirementsreallybecomplete,consistent,unambiguous,andstableenoughasthefoundationforscriptedtesting?Perhapsnot.

2. Scriptedtestingis,bydefinition,inflexible.Itfollowsthescript.If,whiletesting,weseesomethingcurious,wenoteitinaTestIncidentReportbutwedonotpursueit.Whynot?Becauseitisnotinthescripttodoso.Manyinterestingdefectscouldbemissedwiththisapproach.

3. Scriptedtestingisoftenusedto"de-skill"thejoboftesting.Theapproachseemstobe,"Teachatesteraskillortwoandsendthemofftodocumentmountainsoftests.Thesheerbulkofthetestswillprobablyfindmostofthedefects."

Summary"Planyourwork,workyourplan."LiketheWaterfallmodel,nophrasesoepitomizesthescriptedtestingapproachasdoesthisone,andnodocumentsoepitomizesthescriptedtestingapproachasdoesIEEEStd829-1998,the"IEEEStandardforSoftwareTestDocumentation."

TheIEEEStandard829defineseightdocumentsthatcanbeusedinsoftwaretesting.Thesedocumentsare:testplan,testdesignspecification,testcasespecification,testprocedurespecification,testitemtransmittalreport,testlog,testincidentreport,andtestsummaryreport.

Theadvantagesofscriptedtestingincludeformaldocumentation,coverage,andtraceability.

References

Beizer,Boris(1984).SoftwareSystemTestingandQualityAssurance.VanNostrandReinhold.

"IEEEStandardforSoftwareTestDocumentation,"IEEEStd829-1998.TheInstituteofElectricalandElectronicsEngineers,Inc.ISBN0-7381-1443-X

Royce,WinstonW."ManagingtheDevelopmentofLargeSoftwareSystems,"Proceedingsofthe9thInternationalConferenceonSoftwareEngineering,Monterey,CA,IEEEComputerSocietyPress,LosAlamitos,CA,1987.http://www.ipd.bth.se/uodds/pd&agile/royce.pdf

http://www.ipd.bth.se/uodds/pd&agile/royce.pdf

Chapter13:ExploratoryTestingAsshecontemplatedthesettingsun,itsdyingrayscastingthelastoftheirbrilliantpurplelightonthered-goldwatersofthelake,Debbierealizedthatsheshouldneveragainbuyhersunglassesfromaguyparkedbythesideoftheroad.

—MalindaLingwall

IntroductionTheterm"exploratorytesting,"coinedbyCemKanerinhisbookTestingComputerSoftware,referstoanapproachtotestingthatisverydifferentfromscriptedtesting.Ratherthanasequentialexaminationofrequirements,followedbythedesignanddocumentationoftestcases,followedbytheexecutionofthosetestcases,exploratorytesting,asdefinedbyJamesBach,is"simultaneouslearning,testdesign,andtestexecution."Thetesterdesignsandexecutestestswhileexploringtheproduct.

InanarticleforStickyMinds.comentitled"ExploratoryTestingandthePlanningMyth,"Bachwrote,"ExploratoryTesting,asIpracticeit,usuallyproceedsaccordingtoaconsciousplan.Butnotarigorousplan...it'snotscriptedindetail."Jamesadds,"Rigorrequirescertaintyandimpliescompleteness,butIperformexploratorytestingpreciselybecausethere'ssomuchIdon'tknowabouttheproductandIknowmytestingcanneverbefullycomplete."Jamescontinues,"Totheextentthatthenexttestwedoisinfluencedbytheresultofthelasttestwedid,wearedoingexploratorytesting.Webecomemoreexploratorywhenwecan'ttellwhattestsshouldberun,inadvanceofthetestcycle."

ExploratoryTesting

Totheextentthatthenexttestwedoisinfluencedbytheresultofthelasttestwedid,wearedoingexploratorytesting.Webecomemoreexploratorywhenwecan'ttellwhattestsshouldberun,inadvanceofthetestcycle.

Inexploratorytesting,thetestercontrolsthedesignoftestcasesastheyareperformedratherthandays,weeks,orevenmonthsbefore.Inaddition,theinformationthetestergainsfromexecutingasetofteststhenguidesthetesterindesigningandexecutingthenextsetoftests.

Notethisprocessiscalledexploratorytestingtodistinguishitfromadhoctestingwhich(bymydefinition,althoughothersmaydisagree)oftendenotessloppy,careless,unfocused,random,andunskilledtesting.Anyone,nomatterwhattheirexperienceorskilllevel,candoadhoctesting.Thatkindoftestingisineffectiveagainstallbutthemostdefect-riddensystems,andeventhenmaynotfindasubstantialportionofthedefects.

Bachsuggeststhatintoday'stopsy-turvyworldofincomplete,rapidlychangingrequirementsandminimaltimefortesting,theclassicalsequentialapproachofTestAnalysisfollowedbyTestDesignfollowedbyTestCreationfollowedbyTestExecutionislikeplayingthegameof"TwentyQuestions"bywritingoutallthequestionsinadvance.Considerthefollowingdiscussionfromatestingseminardiscussingexploratorytesting:

Instructor:Let'splayagamecalled"TwentyQuestions."Iamthinkingaboutsomethingintheuniverse.I'mgivingyou,theclass,twentyquestionstoidentifywhatI'mthinkingabout.Eachquestionmustbephrasedinawaythatitcanbeanswered"Yes"or"No."(IfIletyouphrasethequestioninanyformyoucouldask"Whatareyouthinkingabout"andwewouldthencallthisgame"OneQuestion.")Ready?Brian,let'sbeginwithyou.

TwentyQuestions:TheGame

Agameinwhichonepersonthinksofsomethingandothersaskupto20questionstodeterminewhathasbeenselected.Thequestionsmustbeanswerable"Yes"or"No."

Whenplayedwell,eachquestionisbasedonthepreviousquestionsandtheiranswers.Writingthequestionsoutinadvancepreventsusingtheknowledgeacquiredfromeachanswer.

Brian:Doesithaveanythingtodowithsoftwaretesting?

Instructor:No,thatwouldbetooeasy.

Michael:Isitlarge?

Instructor:No,it'snotlarge.

Rebecca:Isitananimal?

Instructor:No.

Rayanne:Isitaplant?

Instructor:Yes,itisaplant.

Henry:Isitatree?

Instructor:No,itisnotatree.

Sree:Isitbig?

Instructor:No,I'vealreadysaiditisnotlarge.

Eric:Isitgreen?

Instructor:Yes,itisgreen.

Cheryl:Doesithaveleaves?

Instructor:Yes,ithasleaves.

Galina:Isitanoutdoorplant?

Instructor:Yes,itgenerallygrowsoutdoors.

Jae:Isitafloweringplant?

Instructor:No,Idon'tbelievesobutI'mnotabotanist.

Melanie:Isitashrub?

Instructor:No.

Patrick:Isitacactus?

Instructor:No,itisnotacactus.

Angel:Isitacucumber?

Instructor:No,perhapsratherthanguessingindividualplantsitwouldbemoreeffectivetoidentifycategories.

Sundari:Isitaweed?

Instructor:No,goodtrythough.

Lynn:Isitaperennial?

Instructor:No,Idon'tbelieveso.Ithinkitmustbereplantedeachyear.

Julie:Doesitgrowfrombulbs?

Instructor:No.

Michelle:Isitineveryone'syard?

Instructor:No,atleastit'snotinmine.

Kristie:Isitillegal?(Laughterintheclass)

Instructor:No,it'squitelegal.Well,we'vegonethroughtheclassonce.Brian,let'sgobacktoyou.

Brian:Isitpoisonous?

Instructor:No,althoughmychildrenthinkso.

Michael:Isiteaten?

Instructor:Yes,itiseaten.

Rebecca:Isitlettuce?

Instructor:No,notlettuce.

Rayanne:Isitspinach?

Instructor:Yes,itisspinach.Verygood.

Howsuccessfulwouldwebeatthisgameifwehadtowriteoutallthequestionsinadvance?Whenweplaythisgamewell,eachquestiondependsonthepreviousquestionsandtheir

answers.Soitisinexploratorytesting.Eachtestprovidesuswithinformationabouttheproduct.Wemayseeevidenceoftheproduct'scorrectness;wemayseeevidenceofitsdefects.Wemayseethingsthatarecurious;we'renotsurewhattheymean,thingsthatwewonderaboutandwanttoexplorefurther.So,aswepracticeexploratorytesting,weconcurrentlylearntheproduct,designthetests,andexecutethesetests.

DescriptionInhisclassictimemanagementbook,HowtoGetControlofYourTimeandYourLife,AlanLakeinsuggestsweshouldconstantlyaskourselves:WhatisthemostimportantthingIcandowithmytimerightnow?Exploratorytestersaskanequivalentquestion:WhatisthemostimportanttestIcanperformrightnow?

KeyQuestion

WhatisthemostimportanttestIcanperformrightnow?

Apossibleexploratorytestingprocessis:

Creatingaconjecture(amentalmodel)oftheproperfunctioningofthesystem

Designingoneormoreteststhatwoulddisprovetheconjecture

Executingthesetestsandobservingtheoutcomes

Evaluatingtheoutcomesagainsttheconjecture

Repeatingthisprocessuntiltheconjectureisprovedordisproved

Anotherprocessmightbesimplytoexploreandlearnbeforeformingconjecturesofproperbehavior.

Exploratorytestingcanbedonewithina"timebox,"anuninterruptedblockoftimedevotedtotesting.Thesearetypicallybetweensixtyand120minutesinlength.Thisislongenoughtoperformsolidtestingbutshortenoughsothatthetesterdoesnotmentallywander.Inaddition,atimeboxofthislengthistypicallyeasiertoschedule,easiertocontrol,andeasiertoreport.

Whenperforming"charteredexploratorytesting,"acharterisfirstcreatedtoguidethetesterwithinthetimebox.Thischarterdefinesaclearmissionforthetestingsession.Thechartermaydefine:

Whattotest

Whatdocuments(requirements,design,usermanual,etc.)areavailabletothetester

Whattacticstouse

Whatkindsofdefectstolookfor

Whatrisksareinvolved

Thischarterisaguidelinetobeused,notascripttobefollowed.Becauseofthisapproach,exploratorytestingmakesfulluseoftheskillsoftesters.Bachwrites,"Themorewecanmaketestingintellectuallyrichandfluid,themorelikelywewillhitupontherighttestsattherighttime."

KeyPoint Thecharterisaguidelinetobeused,notascripttobefollowed.

Chartersfocustheexploratorytester'seffortswithinthetimebox.Possiblechartersinclude:

Thoroughlyinvestigateaspecificsystemfunction

Defineandthenexaminethesystem'sworkflows

Identifyandverifyalltheclaimsmadeintheusermanual

Understandtheperformancecharacteristicsofthesoftware

Ensurethatallinputfieldsareproperlyvalidated

Forceallerrorconditionstoverifyeacherrormessage

Checkthedesignagainstuserinterfacestandards

Itispossibletoperformexploratorytestingwithoutacharter.Thisiscalled"freestyleexploratorytesting."Inthisprocesstestersusetheirskillstotheutmostastheyconcurrentlylearntheproductanddesignandexecutetests.

Exploratorytestersareskilledtesters.(Ofcourse,wewanttesterstobeskillednomatterwhattestingprocessweareusing!)Theexploratorytestingapproachrespectsthoseskillsand,infact,dependsonthem.Goodexploratorytestersare:

Goodmodelers,abletocreatementalmodelsofthesystemanditsproperbehavior.

Carefulobservers,abletosee,hear,read,andcomprehend.

Skilledtestdesigners,abletochooseappropriatetestdesigntechniquesineachsituation.Bachemphasizes,"Anexploratorytesterisfirstandforemostatestdesigner."

Abletoevaluateriskandletitguidetheirtesting.

Criticalthinkers,abletogeneratediverseideas,integratetheirobservations,skills,andexperiencestoconcurrentlyexploretheproduct,designthetests,andexecutethetests.

Carefulreporters,abletorigorouslyandeffectivelyreporttootherswhattheyhaveobserved.

Selfmanaged,abletotaketheleadintestingratherthanexecuteaplandevisedbyothers.

Notdistractedbytrivialmatters.

Testerswithouttheseskillscanstillperformusefulexploratorytestingiftheyareproperlysupervisedandcoached.

Ingeneral,processesthathaveweak,slow,ornonexistentfeedbackmechanismsoftendonotperformwell.Scriptedtestingisaprimeexampleofaslowfeedbackloop.Exploratorytestingprovidesatightfeedbackloopbetweenbothtestdesignandtestexecution.Inaddition,itprovidestightfeedbackbetweentestersanddevelopersregardingthequalityoftheproductbeingtested.

AdvantagesofExploratoryTesting1. Exploratorytestingisvaluableinsituationswherechoosingthenexttestcasetobe

runcannotbedeterminedinadvance,butshouldbebasedonprevioustestsandtheirresults.

2. Exploratorytestingisusefulwhenyouareaskedtoproviderapidfeedbackonaproduct'squalityonshortnotice,withlittletime,offthetopofyourhead,whenrequirementsarevagueorevennonexistent,orearlyinthedevelopmentprocesswhenthesystemmaybeunstable.

3. Exploratorytestingisusefulwhen,onceadefectisdetected,wewanttoexplorethesize,scope,andvariationsofthatdefecttoprovidebetterfeedbacktoourdevelopers.

4. Exploratorytestingisausefuladditiontoscriptedtestingwhenthescriptedtestsbecome"tired,"thatis,theyarenotdetectingmanyerrors.

DisadvantagesofExploratoryTesting1. Exploratorytestinghasnoabilitytopreventdefects.Becausethedesignofscripted

testcasesbeginsduringtherequirementsgatheringanddesignphases,defectscanbeidentifiedandcorrectedearlier.

2. Ifyouarealreadysureexactlywhichtestsmustbeexecuted,andinwhichorder,thereisnoneedtoexplore.Writeandthenexecutescriptedtests.

3. Ifyouarerequiredbycontract,rule,orregulationtousescriptedtestingthendoso.Consideraddingexploratorytestsasacomplementarytechnique.

SummaryExploratorytestingisdefinedas"simultaneouslearning,testdesign,andtestexecution."Thetesterdesignsandexecutestestswhileexploringtheproduct.

Inexploratorytesting,thetestercontrolsthedesignoftestcasesastheyareperformedratherthandays,weeks,orevenmonthsbefore.Inaddition,theinformationthetestergainsfromexecutingasetofteststhenguidesthetesterindesigningandexecutingthenextsetoftests.

Exploratorytestingisvitalwheneverchoosingthenexttestcasetoberuncannotbedeterminedinadvancebutshouldbechosenbasedonprevioustestsandtheirresults.

References

Bach,James."ExploratoryTestingandthePlanningMyth."http://www.stickyminds.com/r.asp?F=DART_2359,19March2001.

Bach,James."ExploratoryTestingExplained."v.1.316April2003.http://www.satisfice.com/articles/et-article.pdf

Kaner,Cem,JackFalk,andHungQ.Nguyen(1999).TestingComputerSoftware.JohnWiley&Sons.


Weinberg,GeraldM.(1975).AnIntroductiontoGeneralSystemsThinking.JohnWiley&Sons.

http://www.stickyminds.com/r.asp?F=DART_2359

http://www.satisfice.com/articles/et-article.pdf

Chapter14:TestPlanningJohnStevensonlivesinVancouverwithhiswifeCindyandtheirtwokidsShawnandCassie,whoarethesecondcousinsofMaryShaw,whoismarriedtoRichardShaw,whosegrandmotherwasStewartWerthington'shousekeeper,whosekidsDamienandCharliewenttotheMansfieldChristianSchoolforBoyswithDannyRobinson,whosesisterBertaRobinsonranoffwithChrisTanner,whoridesamotorcycleandgreaseshishairandtheirkidChristausedtogooutwithmypalTomSlipper,whoisthemaincharacterofthisstory,butnotthenarrator'causeIam(Tommycouldn'twritetosavehislife).

—EmmaDolan

IntroductionMortSahl,thebrilliantsocialcommentatorofthe1960s,oftenbeganhisactbydividingtheworldintothe"rightwing,"the"leftwing,"andthe"socialdemocrats."Theprevioustwochaptershavedescribedtherightandleftwings.Nowit'stimeforthesocialdemocrats.

Scriptedtestingisbasedonthesequentialexaminationofrequirements,followedbythedesignanddocumentationoftestcases,followedbytheexecutionofthosetestcases.Thescriptedtester'smottois,"Planyourwork,workyourplan."Exploratorytestingisaverydifferentparadigm.Ratherthanasequentialapproach,exploratorytestingemphasizesconcurrentproductlearning,testdesign,andtestexecution.Thetesterdesignsandexecutestestswhileexploringtheproduct.

TechniquePlanninghasbeendefinedsimplyas"figuringoutwhattodonext."Mostofuswouldadmitthattobeeffectiveandefficient,planningisimportant.Butwhenandhowshouldthatplanningbedone?Scriptedtestingemphasizesthevalueofearlytestplanninganddesignasamethodofdetectingrequirementsanddesigndefectsbeforethecodeiswrittenandthesystemputintoproduction.Exploratorytestingchallengestheideathattestsmustbedesignedsoveryearlyintheproject,whenourknowledgeistypicallyatitsminimum.Inhisarticle,"ExploratoryTestingandthePlanningMyth,"publishedonStickyMinds.com,JamesBachdiscussestheplanningofplaysthatareruninafootballgame.Heexamineswhentheplayscanorshouldbeplanned.Let'sconsiderthissporttolearnmoreaboutplanning.

Butfirst,anapologyorexplanation.Inthischaptertheterm"football"referstothegameofthesamenameasplayedintheUnitedStatesandCanadaandexported,withonlymarginalsuccess,totherestoftheworld."Football"doesnotrefertothatmarvelousgameplayedworld-widethatNorthAmericanscall"soccer."

ForMoreInformation

TolearnmoreaboutthegameoffootballasplayedinNorthAmericaseeww2.nfl.com/basics/history_basics.html

Whenarefootballplaysplanned?Ourfirstthoughtmightbeinthehuddlejustbeforetheplaybegins,butthefollowinglistshowsmorepossibilities:

PlannedFootballPlay

Beforethegamebegins-thefirstnplaysarechosenandexecutedwithoutregardtotheirsuccessorfailuretoevaluatebothteams'abilities

Beforeeachplay-inthehuddle,basedonanoverallgameplan,fieldposition,teams'strengthsandweaknesses,andplayerskillsandexperience

Atthelineofscrimmage-dependingonthedefensivelineup

Atthestartofaplay-playaction-runorpassdependingonthedefense

http://www.ww2.nfl.com/basics/history_basics.html

Duringtheplay-runforyourlifewhenallelsehasfailed

AdaptivePlanning

Adaptiveplanningisnotanindustrystandardterm.Otherpossibletermsare:

Dynamic

Flexible

Just-In-Time

Responsive

Pliable

Progressive

Purposefulplanning

Wecoulddefinetheterms"classicalplanning"and"adaptiveplanning"toindicatethesedifferentapproaches.Therelationshipbetweenclassicalplanningandadaptiveplanninginfootballis:

Table14-1:Classicalplanningvs.Adaptiveplanning.

ClassicalPlanning Beforethegamebegins(thefirsttenplaysarescripted)

AdaptivePlanning

Beforeeachplay(inthehuddle)

Atthelineofscrimmage(dependingonthedefensivesetup)

Atthestartofaplay(playaction-runorpass)

Duringtheplay(scramblewhenallelsehasfailed)

Let'snowleavefootballandconsidersoftwaretestplanning.(Whilewe'dratherstayandwatchthegame,we'vegotsoftwaretotest.)

Table14-2:Classicaltestplanningvs.Exploratorytestplanning.

ClassicalTestPlanning

Asrequirements,analysis,design,andcodingarebeingdone—longbeforesystemisbuiltandthetestingcanbegin

AdaptiveTestPlanning

Chooseastrategy(dependingonourcurrentknowledge)

Beforeeachscreen/function/flowistobetested

Atthestartofanindividualtest(choosedifferentstrategies)

Duringthetest(asweobservethingswedon'texpectorunderstand)

Areasonableplanningheuristicwouldbe:

Weplanasmuchaswecan(basedontheknowledgeavailable),

Whenwecan(basedonthetimeandresourcesavailable),

Butnotbefore.

Asidefromthesenewlabels,haven'tgoodplannersalwaysdonethis?Isthisconceptreallynew?

AremarkablelittlebooksimplytitledPlanning,publishedbytheUnitedStatesMarineCorpsin1997,describestheconceptsofadaptiveplanningindetail.

TheMarineCorpsdefinesplanningasencompassingtwobasicfunctions—"envisioningadesiredfutureandarrangingaconfigurationofpotentialactionsintimeandspacethatwillallowustorealizethatfuture."But,totheMarines,planningisnotsomethingdoneearlywhichthenbecomescastinconcrete."Weshouldthinkofplanningasalearningprocess—asmentalpreparationwhichimprovesourunderstandingofasituation."Plansarenotconcreteeither."Sinceplanningisanongoingprocess,itisbettertothinkofaplanasaninterimproductbasedontheinformationandunderstandingknownatthemomentandalwayssubjecttorevisionasnewinformationandunderstandingemerge."

TheauthorsofPlanninglisttheseplanningpitfallstoavoid:

Attemptingtoforecasteventstoofarintothefuture.Byplanningwemayfoolourselvesintothinkingwearecontrolling.Thereisadifference.

Tryingtoplanintoomuchdetail.HelmuthvonMoltke,GermanArmyChiefofStaffduringWorldWarIsaid,"Noplansurvivescontactwiththeenemy."Inexactlythatsameway,notestplansurvivescontactwiththedefectsinthesystemundertest.

Institutionalizingplanningmethodsthatleadustoinflexibleorlockstepthinkinginwhichbothplanningandplansbecomerigid.Ratherthan"Planyourworkandworkyourplan"asourmantra,weshouldconstantly"Planourwork,workourplan,re-evaluateourwork,re-evaluateourplan."

Thinkingofaplanasanunalterablesolutiontoaproblem.Rather,itshouldbeviewedasanopenarchitecturethatallowsustopursuemanyalternatives."Wewillrarely,ifever,conductanevolutionexactlythewayitwasoriginallydeveloped."

Ignoringtheneedforafeedbackmechanismtoidentifyshortcomingsintheplanandmakenecessaryadjustments.Thisisacomponentofplanningwhichoftendoesnotreceiveadequateemphasis."Manyplansstopshortofidentifyingthesignals,conditions,andfeedbackmechanismsthatwillindicatesuccessfulordysfunctionalexecution."

Adaptiveplanning,asdescribedabove,acknowledgesanddealswiththesepitfalls.

ThefollowingexcerptfromPlanningsummarizestheseconceptswell:"Planningisacontinuousprocessinvolvingtheongoingadjustmentofmeansandends.Weshouldalsoviewplanningasanevolutionaryprocessinvolvingcontinuousadjustmentandimprovement.Wecanthinkofplanningassolution-by-evolutionratherthansolution-by-engineering.Weshouldgenerallynotviewplanningastryingtosolveaprobleminoneiterationbecausemost...problemsaretoocomplextobesolvedthatway.Inmanycases,itismoreadvisabletofindaworkablesolutionquicklyandimprovethesolutionastimepermits.Whatmattersmostisnotgeneratingthebestpossibleplanbutachievingthebestpossibleresult.Likewise,weshouldseeeachplanasanevolvingratherthanastaticdocument.Likeplanning,plansshouldbedynamic;astaticplanisofnovaluetoanadaptiveorganizationinafluidsituation."

SummaryJamesBachasks,"Whatifit[theplan]comesintoexistenceonlymomentsbeforethetesting?"Whymusttheplanbecreatedsoveryearlyintheproject,whenourknowledgeistypicallyatitsminimum?

Inadaptiveplanningweplanasmuchaswecan(basedontheknowledgeavailable),whenwecan(basedonthetimeandresourcesavailable),butnotbefore.

KeyPoint

Theuseofscriptedtestingdoesnotprecludetheuseofexploratorytesting.Theuseofexploratorytestingdoesnotprecludetheuseofscriptedtesting.Smarttestersusewhatevertoolintheirtoolboxisrequired.

Sinceplanningisanongoingprocess,itisbettertothinkofaplanasaninterimproductbasedontheinformationandunderstandingknownatthemomentandalwayssubjecttorevisionasnewinformationandunderstandingemerge.

Theuseofscriptedtestingdoesnotprecludetheuseofexploratorytesting.Theuseofexploratorytestingdoesnotprecludetheuseofscriptedtesting.AsRexBlackwrote,"Smarttestersusewhatevertoolintheirtoolboxisrequired.Noparadigmshere.Noworldviewshere.Noscrewdriversvs.hammers.Let'sdowhatevermakessensegiventheproblemathand."

Practice1. Inwhatareascouldyouuseadaptiveplanningwhereyounowuseclassicalplanning?

Withwhatbenefit?Whatwouldthechallengesbe?Whowouldsupportyouinthisnewprocess?Whowouldopposeyourefforts?Why?

2. InwhatmoviesabouttheMarineCorpsweretheprocessofplanningandthevalueofplansemphasizedoveraction?Canyouexplainwhy?

ReferencesBach,James."ExploratoryTestingandthePlanningMyth."19March2001.http://www.stickyminds.com/r.asp?F=DART_2359

Copeland,Lee."ExploratoryPlanning."3September2001.http://www.stickyminds.com/r.asp?F=DART_2805

"IEEEStandardforSoftwareTestDocumentation,"IEEEStd829-1998.TheInstituteofElectricalandElectronicsEngineers,Inc.ISBN0-7381-1443-X

Planning.MCDP5.UnitedStatesMarineCorps.https://www.doctrine.usmc.mil/mcdp/view/mpdpub5.pdf



http://www.https://www.doctrine.usmc.mil/mcdp/view/mpdpub5.pdf

SectionIV:SupportingTechnologies

ChapterList

Chapter15:DefectTaxonomies

Chapter16:WhentoStopTesting

PartOverview

TheBookendsTwoquestions,likebookends,frameoursoftwaretesting:

Wheredowestart?

Whendowestop?

Wheredowestarttesting?Ofalltheplacestolookfordefects,whereshouldwebegin?Oneansweriswithadefecttaxonomy.Ataxonomyisaclassificationofthingsintoorderedgroupsorcategoriesthatindicatenatural,hierarchicalrelationships.Taxonomieshelpidentifythekindsofdefectsthatoftenoccurinsystems,guideyourtestingbygeneratingideas,andaudityourtestplanstodeterminethecoverageyouareobtainingwithyourtestcases.Intime,theycanhelpyouimproveyourdevelopmentprocess.

Andstopping.Howdowelogicallydecidewhenwehavetestedenoughandthesoftwareisreadyfordeliveryandinstallation?BorisBeizerhaswritten,"Thereisnosingle,valid,rationalcriterionforstopping."Ifheiscorrect,howdowemakethatdecision?

Thenexttwochaptersaddresstheseimportantissues.

Chapter15:DefectTaxonomies'Failure'wassimplynotawordthatwouldevercrossthelipsofMissEvelynDuberry,mainlybecauseEvelyn,ahaughtysocialitewithfire-redhairandacoltishgate,couldpronounceneithertheletters'f'nor'r'asaresultofanunfortunatekissinggesturemademanyyearsearliertowardherbelovedchildhoodparrot,Snippy.

—DavidKenyon

IntroductionWhatisataxonomy?Ataxonomyisaclassificationofthingsintoorderedgroupsorcategoriesthatindicatenatural,hierarchicalrelationships.ThewordtaxonomyisderivedfromtwoGreekroots:"taxis"meaningarrangementand"onoma"meaningname.Taxonomiesnotonlyfacilitatetheorderlystorageofinformation,theyfacilitateitsretrievalandthediscoveryofnewideas.Taxonomieshelpyou:

Guideyourtestingbygeneratingideasfortestdesign

Audityourtestplanstodeterminethecoverageyourtestcasesareproviding

Understandyourdefects,theirtypesandseverities

Understandtheprocessyoucurrentlyusetoproducethosedefects(Alwaysremember,yourcurrentprocessisfinelytunedtocreatethedefectsyou'recreating)

Improveyourdevelopmentprocess

Improveyourtestingprocess

Trainnewtestersregardingimportantareasthatdeservetesting

Explaintomanagementthecomplexitiesofsoftwaretesting

KeyPoint

Ataxonomyisaclassificationofthingsintoorderedgroupsorcategoriesthatindicatenatural,hierarchicalrelationships.

InhisbookTestingObject-OrientedSystems,RobertBinderdescribesa"faultmodel"asalistoftypicaldefectsthatoccurinsystems.Anotherphrasetodescribesuchalistisadefecttaxonomy.Binderthendescribestwoapproachestotesting.Thefirstusesa"non-specificfaultmodel."Inotherwords,nodefecttaxonomyisused.Usingthisapproach,therequirementsandspecificationsguidethecreationofallofourtestcases.Thesecondapproachusesa"specificfaultmodel."Inthisapproach,ataxonomyofdefectsguidesthecreationoftestcases.Inotherwords,wecreatetestcasestodiscoverfaultsliketheoneswehaveexperiencedbefore.Wewillconsidertwolevelsoftaxonomies—projectlevelandsoftwaredefectlevel.Ofmostimportanceintestdesignarethesoftwaredefecttaxonomies.Butitwouldbefoolishtobegintestdesignbeforeevaluatingtherisksassociatedwithboththeproductanditsdevelopmentprocess.

Notethatnoneofthetaxonomiespresentedbelowarecomplete.Eachcouldbeexpanded.Eachissubjectivebasedontheexperienceofthosewhocreatedthetaxonomies.

ProjectLevelTaxonomies

SEIRiskIdentificationTaxonomy

TheSoftwareEngineeringInstitutehaspublisheda"Taxonomy-BasedRiskIdentification"thatcanbeusedtoidentify,classify,andevaluatedifferentriskfactorsfoundinthedevelopmentofsoftwaresystems.

Table15-1:TheSEITaxonomy-BasedRiskIdentificationtaxonomy.

Class Element Attribute

ProductEngineering

Requirements

Stability

Completeness

Clarity

Validity

Feasibility

Precedent

Scale

Design

Functionality

Difficulty

Interfaces

Performance

Testability

CodeandUnitTest

Feasibility

Testing

Coding/Implementation

IntegrationandTest

Environment

Product

System

EngineeringSpecialties

Maintainability

Reliability

Safety

Security

HumanFactors

Specifications

DevelopmentEnvironment

DevelopmentProcess

Formality

Suitability

ProcessControl

Familiarity

ProductControl

DevelopmentSystem

Capacity

Suitability

Usability

Familiarity

Reliability

SystemSupport

Deliverability

ManagementProcess

Planning

ProjectOrganization

ManagementExperience

ProgramInterfaces

ManagementMethods

Monitoring

PersonnelManagement

QualityAssurance

ConfigurationManagement

WorkEnvironment

QualityAttitude

Cooperation

Communication

Morale

Resources

Schedule

Staff

Budget

Facilities

ProgramConstraints

Contract

TypesofContract

Restrictions

Dependencies

ProgramInterfaces

Customer

AssociateContractors

Subcontractors

PrimeContractor

CorporateManagement

Vendors

Politics

If,asatester,youhadconcernswithsomeoftheseelementsandattributes,youwouldwanttostresscertaintypesoftesting.Forexample:

Ifyouareconcernedabout: Youmightwanttoemphasize:

Thestabilityoftherequirements Formaltraceability

Incompleterequirements Exploratorytesting

Impreciselywrittenrequirements Decisiontablesand/orstate-transitiondiagrams

Difficultyinrealizingthedesign Controlflowtesting

Systemperformance Performancetesting

Lackofunittesting Additionaltestingresources

Usabilityproblems Usabilitytesting

ISO9126QualityCharacteristicsTaxonomy

TheISO9126Standard"SoftwareProductEvaluation—QualityCharacteristicsandGuidelines"focusesonmeasuringthequalityofsoftwaresystems.Thisinternationalstandarddefinessoftwareproductqualityintermsofsixmajorcharacteristicsandtwenty-onesubcharacteristicsanddefinesaprocesstoevaluateeachofthese.Thistaxonomyofqualityattributesis:

Table15-2:TheISO9126QualityCharacteristicstaxonomy.

QualityCharacteristic Subcharacteristic

Suitability

Functionality(Aretherequiredfunctionsavailableinthesoftware?)

Accuracy

Interoperability

Security

Reliability(Howreliableisthesoftware?)

Maturity

Faulttolerance

Recoverability

Usability(Isthesoftwareeasytouse?)

Understandability

Learnability

Operability

Attractiveness

Efficiency(Howefficientisthesoftware?)

Timebehavior

Resourcebehavior

Maintainability(Howeasyisittomodifythesoftware?)

Analyzability

Changeability

Stability

Testability

Portability(Howeasyisittotransferthesoftwaretoanotheroperatingenvironment?)

Adaptability

Installability

Coexistence

Replaceability

Eachofthesecharacteristicsandsubcharacteristicssuggestareasofriskandthusareasforwhichtestsmightbecreated.Anevaluationoftheimportanceofthesecharacteristicsshouldbeundertakenfirstsothattheappropriateleveloftestingisperformed.Asimilar"ifyouareconcernedabout/youmightwanttoemphasize"processcouldbeusedbasedontheISO9126taxonomy.

Theseprojectleveltaxonomiescanbeusedtoguideourtestingatastrategiclevel.Forhelpinsoftwaretestdesignweusesoftwaredefecttaxonomies.

SoftwareDefectTaxonomiesInsoftwaretestdesignweareprimarilyconcernedwithtaxonomiesofdefects,orderedlistsofcommondefectsweexpecttoencounterinourtesting.

Beizer'sTaxonomy

OneofthefirstdefecttaxonomieswasdefinedbyBorisBeizerinSoftwareTestingTechniques.Itdefinesafour-levelclassificationofsoftwaredefects.Thetoptwolevelsareshownhere.

Table15-3:AportionofBeizer'sBugTaxonomy.

1xxx Requirements

11xx Requirementsincorrect

12xx Requirementslogic

13xx Requirements,completeness

14xx Verifiability

15xx Presentation,documentation

16xx Requirementschanges

2xxx FeaturesAndFunctionality

21xx Feature/functioncorrectness

22xx Featurecompleteness

23xx Functionalcasecompleteness

24xx Domainbugs

25xx Usermessagesanddiagnostics

26xx Exceptionconditionsmishandled

3xxx StructuralBugs

31xx Controlflowandsequencing

32xx Processing

4xxx Data

41xx Datadefinitionandstructure

42xx Dataaccessandhandling

5xxx ImplementationAndCoding

51xx Codingandtypographical

52xx Styleandstandardsviolations

53xx Documentation

6xxx Integration

61xx Internalinterfaces

62XX Externalinterfaces,timing,throughput

7XXX SystemAndSoftwareArchitecture

71XX O/Scallanduse

72XX Softwarearchitecture

73XX Recoveryandaccountability

74XX Performance

75XX Incorrectdiagnostics,exceptions

76XX Partitions,overlays

77XX Sysgen,environment

8XXX TestDefinitionAndExecution

81XX Testdesignbugs

82XX Testexecutionbugs

83XX Testdocumentation

84XX Testcasecompleteness

Evenconsideringonlythetoptwolevels,itisquiteextensive.Allfourlevelsofthetaxonomyconstituteafine-grainedframeworkwithwhichtocategorizedefects.

Attheoutset,adefecttaxonomyactsasachecklist,remindingthetestersothatnodefecttypesareforgotten.Later,thetaxonomycanbeusedasaframeworktorecorddefectdata.Subsequentanalysisofthisdatacanhelpanorganizationunderstandthetypesofdefectsitcreates,howmany(intermsofrawnumbersandpercentages),andhowandwhythesedefectsoccur.Then,whenfacedwithtoomanythingstotestandnotenoughtime,youwillhavedatathatenablesyoutomakerisk-based,ratherthanrandom,testdesigndecisions.Inadditiontotaxonomiesthatsuggestthetypesofdefectsthatmayoccur,alwaysevaluatetheimpactonthecustomerandultimatelyonyourorganizationiftheydooccur.Defectsthathavelowimpactmaynotbeworthtrackingdownandrepairing.

Kaner,Falk,andNguyen'sTaxonomy

ThebookTestingComputerSoftwarecontainsadetailedtaxonomyconsistingofover400typesofdefects.Onlyafewexcerptsfromthistaxonomyarelistedhere.

Table15-4:AportionofthedefecttaxonomyfromTestingComputerSoftware.

UserInterfaceErrors

Functionality

Communication

Commandstructure

Missingcommands

Performance

Output

ErrorHandling

Errorprevention

Errordetection

Errorrecovery

Boundary-RelatedErrors

Numericboundaries

Boundariesinspace,time

Boundariesinloops

CalculationErrors

Outdatedconstants

Calculationerrors

Wrongoperationorder

Overflowandunderflow

InitialAndLaterStates

Failuretosetadataitemto0

Failuretoinitializealoopcontrolvariable

Failuretoclearastring

Failuretoreinitialize

ControlFlowErrors

Programrunsamok

Programstops

Loops

IF,THEN,ELSEormaybenot

ErrorsInHandlingOrInterpretingData

Datatypeerrors

Parameterlistvariablesoutoforderormissing

Outdatedcopiesofdata

Wrongvaluefromatable

Wrongmaskinbitfield

RaceConditions

Assumingoneeventalwaysfinishesbeforeanother

Assumingthatinputwillnotoccurinaspecificinterval

Taskstartsbeforeitsprerequisitesaremet

LoadConditionsRequiredresourcenotavailable

Doesn'treturnunusedmemory

HardwareDeviceunavailable

Unexpectedendoffile

SourceAndVersionControlOldbugsmysteriouslyreappear

Sourcedoesn'tmatchbinary

Documentation None

TestingErrors

Failuretonoticeaproblem

Failuretoexecuteaplannedtest

Failuretousethemostpromisingtestcases

Failuretofileadefectreport

Binder'sObject-OrientedTaxonomy

RobertBindernotesthatmanydefectsintheobject-oriented(OO)paradigmareproblemsusingencapsulation,inheritance,polymorphism,messagesequencing,andstate-transitions.Thisistobeexpectedfortworeasons.First,thesearecornerstoneconceptsinOO.Theyformthebasisoftheparadigmandthuswillbeusedextensively.Second,thesebasicconceptsareverydifferentfromtheproceduralparadigm.DesignersandprogrammersnewtoOOwouldbeexpectedtofindthemforeignideas.AsmallportionofBinder'sOOtaxonomyisgivenheretogiveyouasenseofitscontents:

Table15-5:AportionofBinder'sMethodScopeFaultTaxonomy.

MethodScope Fault

Requirements Requirementomission

Design

Abstraction LowCohesion

RefinementFeatureoverridemissing

Featuredeletemissing

EncapsulationNakedaccess

Overuseoffriend

ResponsibilitiesIncorrectalgorithm

Invariantviolation

Exceptions Exceptionnotcaught

Table15-6:AportionofBinder'sClassScopeFaultTaxonomy.

ClassScope Fault

Design

AbstractionAssociationmissingorincorrect

Inheritanceloops

RefinementWrongfeatureinherited

Incorrectmultipleinheritance

EncapsulationPublicinterfacenotviaclassmethods

Implicitclass-to-classcommunication

ModularityObjectnotused

Excessivelylargenumberofmethods

Implementation Incorrectconstructor

Notehowthistaxonomycouldbeusedtoguidebothinspectionsandtestcasedesign.BinderalsoreferencesspecificdefecttaxonomiesforC++,Java,andSmalltalk.

Whittaker's"HowtoBreakSoftware"Taxonomy

JamesWhittaker'sbookHowtoBreakSoftwareisatester'sdelight.Proponentsofexploratorytestingexhortusto"explore."Whittakertellsusspecifically"wheretoexplore."Notonlydoesheidentifyareasinwhichfaultstendtooccur,hedefinesspecifictestingattackstolocatethesefaults.Onlyasmallportionofhistaxonomyispresented:

Table15-7:AportionofWhittaker'sFaultTaxonomy.

FaultType Attack

Inputsandoutputs

Forceallerrormessagestooccur

Forcetheestablishingofdefaultvalues

Overflowinputbuffers

DataandcomputationForcethedatastructuretostoretoofewortoomanyvalues

Forcecomputationresultstobetoolargeortoosmall

FilesysteminterfaceFillthefilesystemtoitscapacity

Damagethemedia

SoftwareinterfacesCauseallerrorhandlingcodetoexecute

Causeallexceptionstofire

Vijayaraghavan'seCommerceTaxonomy

Beizer's,Kaner's,andWhittaker'staxonomiescatalogdefectsthatcanoccurinanysystem.Binder'sfocusesoncommondefectsinobject-orientedsystems.GiriVijayaraghavanhaschosenamuchnarrowerfocus—theeCommerceshoppingcart.Usingthisfamiliarmetaphor,aneCommerceWebsitekeepstrackofthestateofauserwhileshopping.Vijayaraghavanhasinvestigatedthemanywaysshoppingcartscanfail.Hewrites,"Wedevelopedthelistofshoppingcartfailurestostudytheuseoftheoutlineasatestideagenerator."Thisisoneoftheprimeusesofanydefecttaxonomy.Histaxonomylistsoversixtyhigh-leveldefectcategories,someofwhicharelistedhere:

Performance

Reliability

Softwareupgrades

Userinterfaceusability

Maintainability

Conformance

Stability

Operability

Faulttolerance

Accuracy

Internationalization

Recoverability

Capacityplanning

Third-partysoftwarefailure

Memoryleaks

Browserproblems

Systemsecurity

Clientprivacy

Aftergeneratingthelistheconcludes,"Wethinkthelistisasufficientlybroadandwell-researchedcollectionthatitcanbeusedasastartingpointfortestingotherapplications."Hisassertioniscertainlycorrect.

AFinalObservation

Notethateachofthesetaxonomiesisalistofpossibledefectswithoutanyguidanceregardingtheprobabilitythatthesewilloccurinyoursystemsandwithoutanysuggestionofthelossyourorganizationwouldincurifthesedefectsdidoccur.Taxonomiesareusefulstartingpointsforourtestingbuttheyarecertainlynotacompleteanswertothequestionofwheretostarttesting.

YourTaxonomyNowthatwehaveexaminedanumberofdifferentdefecttaxonomies,thequestionarises—whichisthecorrectoneforyou?Thetaxonomythatismostusefulisyourtaxonomy,theoneyoucreatefromyourexperiencewithinyourorganization.Oftentheplacetostartiswithanexistingtaxonomy.Thenmodifyittomoreaccuratelyreflectyourparticularsituationintermsofdefects,theirfrequencyofoccurrence,andthelossyouwouldincurifthesedefectswerenotdetectedandrepaired.

KeyPoint Thetaxonomythatismostusefulisyourtaxonomy,theoneyoucreate.

Justasinotherdisciplineslikebiology,psychology,andmedicine,thereisnoone,single,rightwaytocategorize,thereisnoonerightsoftwaredefecttaxonomy.Categoriesmaybefuzzyandoverlap.Defectsmaynotcorrespondtojustonecategory.Ourlistmaynotbecomplete,correct,orconsistent.Thatmattersverylittle.Whatmattersisthatwearecollecting,analyzing,andcategorizingourpastexperienceandfeedingitforwardtoimproveourabilitytodetectdefects.Taxonomiesaremerelymodelsand,asGeorgeBox,thefamousstatistician,remindsus,"Allmodelsarewrong;somemodelsareuseful."

Tocreateyourowntaxonomy,firststartwithalistofkeyconcepts.Don'tworryifyourlistbecomeslong.Thatmaybejustfine.Makesuretheitemsinyourtaxonomyareshort,descriptivephrases.Keepyourusers(that'syouandothertestersinyourorganization)inmind.Usetermsthatarecommonforthem.Later,lookfornaturalhierarchicalrelationshipsbetweenitemsinthetaxonomy.Combinetheseintoamajorcategorywithsubcategoriesunderneath.Trynottoduplicateoroverlapcategoriesandsubcategories.Continuetoaddnewcategoriesastheyarediscovered.Revisethecategoriesandsubcategorieswhennewitemsdon'tseemtofitwell.Shareyourtaxonomywithothersandsolicittheirfeedback.Youareonyourwaytoataxonomythatwillcontributetoyourtestingsuccess.

SummaryTaxonomieshelpyou:

Guideyourtestingbygeneratingideasfortestcasedesign

Audityourtestplanstodeterminethecoverageyourtestcasesareproviding

Understandyourdefects,theirtypesandseverities

Understandtheprocessyoucurrentlyusetoproducethosedefects(Alwaysremember,yourcurrentprocessisfinelytunedtocreatethedefectsyou'recreating)

Improveyourdevelopmentprocess

Improveyourtestingprocess

Trainnewtestersregardingimportantareasthatdeservetesting

Explaintomanagementthecomplexitiesofsoftwaretesting

Testingcanbedonewithouttheuseoftaxonomies(nonspecificfaultmodel)orwithataxonomy(specificfaultmodel)toguidethedesignoftestcases.

Taxonomiescanbecreatedatanumberoflevels:genericsoftwaresystem,developmentparadigm,typeofapplication,anduserinterfacemetaphor.

References



Carr,MarvinJ.,etal.(1993)"Taxonomy-BasedRiskIdentification."TechnicalReportCMU/SEI-93-TR-6,ESC-TR-93-183,June1993.http://www.sei.cmu.edu/pub/documents/93.reports/pdf/tr06.93.pdf

ISO(1991).ISO/IECStandard9126-1.SoftwareEngineering-ProductQuality-Part1:QualityModel,ISOCopyrightOffice,Geneva,June2001.

Kaner,Cem,JackFalkandHungQuocNguyen(1999).TestingComputerSoftware(SecondEdition).JohnWiley&Sons.

Whittaker,JamesA.(2003).HowtoBreakSoftware:APracticalGuidetoTesting.AddisonWesley.

Vijayaraghavan,GiriandCemKaner."Bugsinyourshoppingcart:ATaxonomy."http://www.testingeducation.org/articles/BISC_Final.pdf

http://www.sei.cmu.edu/pub/documents/93.reports/pdf/tr06.93.pdf

http://www.testingeducation.org/articles/BISC_Final.pdf

Chapter16:WhentoStopTestingTheballerinastoodonpoint,hertoescurledlikeshrimp,notdeep-friedshrimpbecause,asbrittleastheyare,theywouldhavecrackedunderthepressure,buttenderebi-kind-of-shrimp,pinkandlusciousasaTokyosunset,wonderingifherloverwasintheGinza,wooingthegeishawitheyesreminiscentofroe,whichshelikedbetterthanebianyway.

—BrianTacang

TheBananaPrincipleInhisclassicbookAnIntroductiontoGeneralSystemsThinking,GeraldWeinbergintroducesustothe"BananaPrinciple."Alittleboycomeshomefromschoolandhismotherasks,"Whatdidyoulearninschooltoday?"Theboyresponds,"Todaywelearnedhowtospell'banana'butwedidn'tlearnwhentostop."Inthisbookwehavelearnedhowtodesigneffectiveandefficienttestcases,buthowdoweknowwhentostop?Howdoweknowwehavedoneenoughtesting?

WhentoStopInTheCompleteGuidetoSoftwareTesting,BillHetzelwroteregardingsystemtesting,"Testingendswhenwehavemeasuredsystemcapabilitiesandcorrectedenoughoftheproblemstohaveconfidencethatwearereadytoruntheacceptancetest."Thephrases"correctedenough"and"haveconfidence,"whilecertainlycorrect,arevague.

Regardingstopping,BorisBeizerhaswritten,"Thereisnosingle,valid,rationalcriterionforstopping.Furthermore,givenanysetofapplicablecriteria,howexactlyeachisweighteddependsverymuchupontheproduct,theenvironment,thecultureandtheattitudetorisk."Again,notmuchhelpinknowingwhentostoptesting.

EventhoughBeizersaysthereisnosinglecriterionforstopping,manyorganizationshavechosenoneanyway.Thefivebasiccriteriaoftenusedtodecidewhentostoptestingare:

Youhavemetpreviouslydefinedcoveragegoals

Thedefectdiscoveryratehasdroppedbelowapreviouslydefinedthreshold

Themarginalcostoffindingthe"next"defectexceedstheexpectedlossfromthatdefect

Theprojectteamreachesconsensusthatitisappropriatetoreleasetheproduct

Thebosssays,"Shipit!"

CoverageGoals

Coverageisameasureofhowmuchhasbeentestedcomparedwithhowmuchisavailabletotest.Coveragecanbedefinedatthecodelevelwithmetricssuchasstatementcoverage,branchcoverage,andpathcoverage.Attheintegrationlevel,coveragecanbedefinedintermsofAPIstestedorAPI/parametercombinationstested.Atthesystemlevel,coveragecanbedefinedintermsoffunctionstested,usecases(oruserstories)tested,orusecasescenarios(mainpathplusalltheexceptionpaths)tested.Onceenoughtestcaseshavebeenexecutedtomeetthepreviouslydefinedcoveragegoals,weare,bydefinition,finishedtesting.Forexample,wecoulddefineaproject'sstoppingcriteriaas:

100%statementcoverage

90%usecasescenariocoverage

Whenthisnumberoftestspass,wearefinishedtesting.(Ofcourse,therearemanyothercombinationsoffactorsthatcouldbeusedasstoppingcriteria.)Notalltestersapproveofthisapproach.GlenfordMyersbelievesthatthismethodishighlycounterproductive.Hebelievesthatbecausehumanbeingsareverygoaloriented,thiscriterioncouldsubconsciouslyencouragetesterstowritetestcasesthathavealowprobabilityofdetectingdefectsbutdomeetthecoveragecriteria.Hebelievesthatmorespecificcriteriasuchasasetofteststhatcoverallboundaryvalues,state-transitionevents,decisiontablerules,etc.aresuperior.

DefectDiscoveryRate

Anotherapproachistousethedefectdiscoveryrateasthecriteriaforstopping.Eachweek(orothershortperiodoftime)wecountthenumberofdefectsdiscovered.Typically,thenumberofdefectsfoundeachweekresemblesthecurveinFigure16-1.Oncethediscoveryrateislessthanacertainpreviouslyselectedthreshold,wearefinishedtesting.Forexample,ifwehadsetthethresholdatthreedefects/week,wewouldstoptestingafterweek18.

Figure16-1:DefectDiscoveryRate

Whilethisapproachappealstoourintuition,weshouldconsiderwhatothersituationswouldproduceacurvelikethis—creationofadditional,butlesseffectivetestcases;testersonvacation;"killer"defectsthatstillexistinthesoftwarebutthathideverywell.ThisisonereasonwhyBeizersuggestsnotdependingononlyonestoppingcriterion.

MarginalCost

Inmanufacturing,wedefine"marginalcost"asthecostassociatedwithoneadditionalunitofproduction.Ifwe'remaking1,000donuts,whatistheadditionalcostofmakingonemore?Notverymuch.Inmanufacturing,themarginalcosttypicallydecreasesasthenumberofunitsmadeincreases.Insoftwaretesting,however,justtheoppositeoccurs.Findingthefirstfewdefectsisrelativelysimpleandinexpensive.Findingeachadditionaldefectbecomesmoreandmoretimeconsumingandcostlybecausethesedefectsareveryadeptathidingfromourtestcases.Thusthecostoffindingthe"next"defectincreases.Atsomepointthecostoffindingthatdefectexceedsthelossourorganizationwouldincurifweshippedtheproductwiththatdefect.Clearly,itis(past)timetostoptesting.

Noteverysystemshouldusethiscriterion.Systemsthatrequirehighreliabilitysuchas

weaponssystems,medicaldevices,industrialcontrols,andothersafety-criticalsystemsmayrequireadditionaltestingbecauseoftheirriskandsubsequentlossshouldafailureoccur.

TeamConsensus

Basedonvariousfactorsincludingtechnical,financial,political,andjust"gutfeelings,"theprojectteam(managers,developers,testers,marketing,sales,qualityassurance,etc.)decidethatthebenefitsofdeliveringthesoftwarenowoutweighthepotentialliabilitiesandreachconsensusthattheproductshouldbereleased.

ShipIt!

Formanyofus,thiswillbetheonlystrategywewilleverpersonallyexperience.It'softenverydishearteningfortesters,especiallyaftermanyarduoushoursoftesting,andwithasureknowledgethatmanydefectsarestillhidinginthesoftware,tobetold"Shipit!"Whattestersmustrememberisthattheremaybeveryreasonableandlogicalreasonsforshippingtheproductbeforewe,astesters,thinkitisready.Intoday'sfast-pacedmarketeconomy,oftenthe"firsttomarket"winsasubstantialmarketshare.Eveniftheproductislessthanperfect,itmaystillsatisfytheneedsofmanyusersandbringsignificantprofitstoourorganization;profitsthatmightbelostifwedelayedshipment.

Someofthecriteriathatshouldbeconsideredinmakingthisdecisionarethecomplexityoftheproductitself,thecomplexityofthetechnologiesusedtoimplementitandourskillsandexperienceinusingthosetechnologies,theorganization'scultureandtheimportanceofriskaversioninourorganization,andtheenvironmentwithinwhichthesystemwilloperateincludingthefinancialandlegalexposurewehaveifthesystemfails.

Asatester,youmaybefrustratedbythe"ShipIt"decision.Remember,ourroleastestersistoinformmanagementoftherisksofshippingtheproduct.Theroleofyourorganization'smarketingandsalesgroupsshouldbetoinformmanagementofthebenefitsofshippingtheproduct.Withthisinformation,bothpositiveandnegative,projectmanagerscanmakeinformed,rationaldecisions.

SomeConcludingAdviceLesson185inLessonsLearnedinSoftwareTestingstates:

"Becausetestingisaninformationgatheringprocess,youcanstopwhenyou'vegatheredenoughinformation.Youcouldstopafteryou'vefoundeverybug,butitwouldtakeinfinitetestingtoknowthatyou'vefoundeverybug,sothatwon'twork.Instead,youshouldstopwhenyoureasonablybelievethattheprobabilityislowthattheproductstillhasimportantundiscoveredproblems.

"Severalfactorsareinvolvedindecidingthattestingisgoodenough(lowenoughchanceofundiscoveredsignificantbugs):

Youareawareofthekindsofproblemsthatwouldbeimportanttofind,iftheyexisted.

Youareawareofhowdifferentpartsoftheproductcouldexhibitimportantproblems.

Youhaveexaminedtheproducttoadegreeandinamannercommensuratewiththeserisks.

Yourteststrategywasreasonablydiversifiedtoguardagainsttunnelvision.

Youusedeveryresourceavailablefortesting.

Youmeteverytestingprocessstandardthatyourclientswouldexpectyoutomeet.

Youexpressedyourteststrategy,testresults,andqualityassessmentsasclearlyasyoucould."

SummaryRegardingstopping,BorisBeizerhaswritten,"Thereisnosingle,valid,rationalcriterionforstopping.Furthermore,givenanysetofapplicablecriteria,howexactlyeachisweighteddependsverymuchupontheproduct,theenvironment,thecultureandtheattitudetorisk."

Thefivebasiccriteriaoftenusedtodecidewhentostoptestingare:

Youhavemetpreviouslydefinedcoveragegoals

Thedefectdiscoveryratehasdroppedbelowapreviouslydefinedthreshold

Themarginalcostoffindingthe"next"defectexceedstheexpectedlossfromthatdefect

Theprojectteamreachesconsensusthatitisappropriatetoreleasetheproduct

Thebosssays,"Shipit!"

References

Hetzel,Bill(1998).TheCompleteGuidetoSoftwareTesting(SecondEdition).JohnWiley&Sons.



Weinberg,GeraldM.(1975).AnIntroductiontoGeneralSystemsThinking.JohnWiley&Sons.

SectionV:SomeFinalThoughts

AppendixList

AppendixA:Brown&DonaldsonCaseStudy

AppendixB:StatelessUniversityRegistrationSystemCaseStudy

PartOverview

YourTestingToolboxMyoldestsonShawnisaglazier—heinstallsglass,mirrors,showerdoors,etc.Heisanartistinglass.Asafather,Idecideditwouldbegoodtoknowwhatmysondoesforaliving,soIrodewithhiminhistruckforafewhourswatchinghimwork.

Atthefirstjobsitehepulledoutaclipboardwithaworkorderthattoldhimwhatwasneeded.Hehoppedoutandwalkedaroundtothebackofthetruck.There,hegrabbedhistoolbucket(anoldfive-gallonpaintbucket)androotedaroundthroughit.Hepulledoutsometools,walkeduptothehouse,didhismagic,camebacktothetruck,putthetoolsinthebucket,andawaywewent.Atthesecondjobsiteherepeatedtheprocess.Onceagain,hepulledouttheclipboard,hoppedout,walkedaroundtothebackofthetruck,grabbedhistoolbucket,androotedaroundthroughit.Hepulledoutsometools,butdifferenttoolsthistime,walkeduptothehouse,didhismagic,camebacktothetruck,putthetoolsinthebucket,andawaywewent.Aswewentfromjobtojobitoccurredtomethatallgoodcraftspeople,includingsoftwaretesters,needabucketoftools.Inaddition,goodcraftspeopleknowwhichtooltouseinwhichsituation.Myintentinwritingthisbookwastohelpputmoretoolsinyourpersonaltestingtoolbucketandtohelpyouknowwhichtooltouseinwhichsituation.Remember,noteverytoolneedstobeusedeverytime.

Now,it'suptoyou.Thenextlevelofskillcomeswithpractice.FamouseducatorBenjaminBloomcreatedataxonomyforcategorizinglevelsofcompetencyinschoolsettings.Thefirstthreelevelsare:

Knowledge

Comprehension

Application

Thisbookhasfocusedonknowledgeandcomprehension.The"application"isuptoyou.

Bestwishesinyourtesting...

References

Bloom,BenjaminS.(1969).TaxonomyofEducationalObjectives:TheClassificationofEducationalGoals.LongmanGroup.

AppendixA:Brown&DonaldsonCaseStudy

IntroductionBrown&Donaldson(B&D)isafictitiousonlinebrokeragefirmthatyoucanusetopracticethetestdesigntechniquespresentedinthisbook.B&DwasoriginallycreatedforSoftwareQualityEngineering'sWeb/eBusinessTestingcourse(seehttp://www.sqe.com).TheactualB&DWebsiteisfoundathttp://bdonline.sqe.com.AnyresemblancetoanyactualonlinebrokerageWebsiteispurelycoincidental.

http://www.sqe.com

http://bdonline.sqe.com

LoginTheLoginpageisthegatewayintotheB&Dsite.Itrequiresalegitimateusernameandpassword.

MarketNewsTheMarketNewspageisthemainpageoftheB&Dsite.Itcontainsnavigationbuttonsontheleftsideofthepage,stockperformancechartsatthetop,andnewsstoriesofinteresttoB&D'sinvestors.

TradeTheTradepageallowsaB&Dclienttobuyandsellstocks.Itcontainsabuy/sellbutton,atextboxforthestocktickersymbol,atextboxforthenumberofsharestobeboughtorsold(quantity),andboxesindicatingthetypeoftrade.

SymbolLookupTheSymbolLookuppageisreachedfromtheTradepage.ItisusedwhentheB&Dclientisunsureofthestocktickersymbolandmustlookitup.Itcontainsonefieldwherethefirstfewcharactersoftheorganization'snameareentered.

LookupResultsTheLookupResultspageistheresultofthepreviousSymbolLookuppage.Itdisplaysthestocksymbolsthatmatchedtheprevioussearch.

HoldingsPerhapsthemostimportantpageontheB&Dsite,theHoldingspagedisplaysthestockscurrentlyownedbythisclient.

GlossaryTheGlossarypagecanbeusedtolookuptermsthatunfamiliartotheB&Dclient.

AppendixB:StatelessUniversityRegistrationSystemCaseStudy

SystemDocumentationStatelessUniversityRegistrationSystem(SURS)UserInterfaceSpecification

May1,2002

Version2.3

PreparedBy:

OtisKribblekoblisSuperDuperSoftwareCompany(SDSC)

422South5thAvenueAnytown,USA

Introduction

ThepurposeofthisdocumentistodescribetheplanneduserinterfacefortheStatelessUniversityRegistrationSystem.Itwillberevisedtoreflectthat"as-built"softwareaftersystemtestinghasbegun.ItisacustomizedversionoftheregistrationsystemdeliveredtoUniversalOnlineUniversity(UOU)lastyear.StatelessUhasrequestedsomemajormodificationstotheUOUversion,sothatitisessentiallyarewriteofthesoftware.Someofthemodulesfordatabasecreationandbackuphavebeenreused,butthatisnotapparentfromtheuserinterface,whichisallnew.

Thismanualhastheuserinterfacescreensdefinedintheorderinwhichtheyarecustomarilyused.Itstartswiththeloginscreen.Thenitprovidesthedatabaseset-upfields:theaddition/change/deletionofstudents,theaddition/change/deletionofcourses,andtheaddition/change/deletionofclasssections.Thefinaldataentryscreenprovidestheselectionofspecificcoursesectionsforeachstudent.Thereisalsoanadministrativefunctionthatisaccessibletoonlythesupervisor.Itprovidesaccesstotheadministrativefunctionsofbackupandrestoreofthedatabases.Eachscreenisdefinedinaseparatesectionprovidingthefollowinginformation:

Functionalitysupported

Formattingrequirementsforeachdataentryfield

Asamplescreenlayout(thefinalimplementedsoftwaremaydiffer)

Thefigurebelowsummarizesthescreensandtheirnavigationoptions.

2.1Log-inandSelectFunctionScreen

2.1.1Functions

EachuserisrequiredtoenteraUserIDandaPassword.Theidentificationofthestatusoftheuser(supervisor:yesorno)ismandatoryatthetimeoflog-in.OnlyYesorNomaybeselectedbyclickingontheappropriatebox(notboth).Afterasuccessfullog-inhasbeencompleted,thenthenextfunctiontobeexecutedcanbeselected.OnlyasupervisormayaccesstheAdministrativescreen.TheExitbuttonisactiveatalltimes.

2.1.2DataEntryFormats

Theformatsforthefieldsonthisscreenare:UserID:eightcharactersatleasttwoofwhicharenotalphabetic(canbenumericorspecialcharacters).

Password:eightcharactersatleasttwoofwhicharenotalphabetic(canbenumericorspecialcharacters).

2.1.3ScreenFormat

3.1StudentDatabaseMaintenanceScreen

3.1.1Functions

Thisscreenallowstheentryoftheidentifyinginformationforanewstudentandtheassignmentofhis/herstudentIDnumber.AllfieldsarerequiredtobefilledinbeforetheEnterbuttonisselected.Thefieldsmaybeenteredinanyorder.Thebackspacekeywillworkatanytimeinanyfield,buttheResetbuttonwillclearallofthefieldswhenitispressed.

IftheStudentIDisenteredfirst,thentheDelete(allowsastudenttoberemovedfromthedatabase)andModify(allowsthemodificationofthestudent'scontactinformation—thedatacurrentlyinthedatabasewillbedisplayed)buttonsbecomeactive.TheEnterbuttonwillcausetheDeleteortheModifytobeexecutedandthefieldsonthescreentobecleared.


Theformatsforthefields(allmandatory)onthisscreenare:Firstname:onetotencharactersMiddlename:onetotencharactersorNMNfornomiddlenameLastname:onetofifteencharacters(alpha,period,hyphen,apostrophe,space,numbers)Streetaddress:fourtotwentyalphanumericcharactersCity:threetotenalphacharactersState:twoalphacharactersZip:thestandardfivenumericshyphenfournumericsPhone:telephonenumberinthefollowingformat703.555.1212

StudentID:twocharactersrepresentingthehomecampusandasix-digitnumberwhichisuniqueforeachstudent.Thehomecampusdesignationsare:

ANforAnnandale

LOforLoudoun

MAforManassas

WOforWoodbridge

ARforArlington

Thesix-digitnumberisgeneratedbythesystemwhentheEnterbuttonisselected.ItremainsdisplayeduntiltheResetbuttonisdepressed.Atthattime,allfieldsareclearedforthenextsetofentries.

3.1.3ScreenLayout

3.2CourseDatabaseMaintenanceScreen

3.2.1Functions

ThisscreenallowstheentryoftheidentifyinginformationforanewcourseandtheassignmentofthecourseIDnumber.AllfieldsarerequiredtobefilledinbeforetheEnterbuttonispressed.Thefieldsmaybeenteredinanyorder.Thebackspacekeywillworkatanytimeinanyfield,buttheResetbuttonwillclearallofthefieldswhenitisentered.TheBackbuttoncausesareturntothepreviousscreen.TheExitbuttoncausesanexitfromthisapplication.

IftheCourseIDisenteredfirst,thentheDelete(allowsacoursetoberemovedfromthedatabase)andModify(allowsthemodificationofanexistingcourse'sinformation—thedatacurrentlyinthedatabasewillbedisplayed)buttonsbecomeactive.TheEnterbuttonwillcausetheDeleteortheModifytobeexecutedandthefieldsonthescreentobecleared.


Theformatsforthefields(allaremandatory)onthisscreenare:CourseID:threealphacharactersrepresentingthedepartmentfollowedbyasix-digitintegerwhichistheuniquecourseidentificationnumber.Thepossibledepartmentsare:

PHY-Physics

EGR-Engineering

ENG-English

LAN-Foreignlanguages

CHM-Chemistry

MAT-Mathematics

PED-Physicaleducation

SOC-Sociology

LIB-Libraryscience

HEC-Homeeconomics

Coursename:afreeformatalphanumericfieldofuptofortycharactersCoursedescription:afreeformatalphanumericfieldofupto250characters

3.2.3ScreenLayout

3.3ClassSectionDatabaseMaintenanceScreen

3.3.1Functions

Thisscreenallowstheentryoftheidentifyinginformationforanewcoursesection.AllfieldsarerequiredtobefilledinbeforetheEnterbuttonispressed.Thefieldsmaybeenteredinanyorder.Thebackspacekeywillworkatanytimeinanyfield,buttheResetbuttonwillclearallofthefieldswhenitisentered.TheBackbuttoncausesareturntothepreviousscreen.TheExitbuttoncausesanexitfromthisapplication.

TheCourseIDisrequiredtobeenteredfirst(allexistingsectionswillbedisplayedassoonasitisentered),followedbythenewSection#,DatesandTimefields.TheDelete(allowsasectiontoberemovedfromthedatabase)andModify(allowsthemodificationofanexistingsection'sinformation)buttonsbecomeactiveaftertheSection#isentered.Ifthesectionisalreadyinthedatabase,thecurrentinformationwillbedisplayedassoonastheSection#fieldisfilledin.TheEnterbuttonwillcausetheDeleteortheModifytobeexecutedandthefieldsonthescreentobecleared.


Theformatsforthefields(allmandatory)onthisscreenare:CourseID:threealphacharactersrepresentingthedepartmentfollowedbyasix-digitintegerSection#:athree-digitinteger(leadingzerosarerequired)assignedbytheuserDates:thedaysoftheweektheclassmeets(uptothreewithhyphensinbetween);theweekdaydesignationsare:

Sun

Mon

Tue

Wed

Thr

Fri

Sat

Time:thestartingandendingtimesofthesection(usingmilitarytime)withahypheninbetween,e.g.,12:00–13:30.

3.3.3ScreenLayout

3.4SectionSelectionEntryScreen

3.4.1Functions

Thisscreenallowstheentryoftheselectionofspecificcoursesectionsforanindividualstudent.AllfieldsarerequiredtobefilledinbeforetheEnterbuttonispressed.Thefieldsmaybeenteredinanyorder.Thebackspacekeywillworkatanytimeinanyfield,buttheResetbuttonwillclearallofthefieldswhenitisentered.TheBackbuttoncausesareturntothepreviousscreen.TheExitbuttoncausesanexitfromthisapplication.

TheStudentIDisrequiredtobeenteredfirst,followedbytheCourseID(allavailablesectionswillbedisplayedassoonasitisentered).Sectionsareselectedbyclickingonthesectiontobeassigned.TheEnterbuttonwillcausethestudenttobeaddedtotheselectedsection.EnteringanewCourseIDwillcauseanewlistofavailablesectionstobedisplayed,allowinganothercoursesectiontobeselectedforthesamestudent.


Theformatsforthefields(allmandatory)onthisscreenare:

CourseID:threealphacharactersrepresentingthedepartmentfollowedbyasix-digitinteger

StudentID:twocharactersrepresentingthehomecampusandasix-digitnumberthatisuniqueforeachstudent

Availablesections:alistofallofthesectionsthatarenotfull

3.4.3ScreenLayout

3.5AdministrativeScreen

3.5.1Functions

Onlythesupervisormayaccesstheadministrativescreen.Itpermitsoneofthefollowingthreeactivitiesatatime:

Creationofabackupofanyorallofthedatabases

Restoreofabackupofanyorallofthedatabases

Printingofareportofanyorallofthedatabases

Aftertheactivity(createorrestore)andthedatabaseshavebeenselected,thenameofthebackupistobeentered.

TheBackandExitbuttonsareactiveatalltimes.


Theformatsforthefieldsonthisscreenare:

BackupID:aannnn(requiredonlyforbackups,notreports)Commentary:afreeformatcharacterfield200charactersinlength(requiredonlyforbackups,notreports)

3.5.3ScreenLayout

Bibliography

WorksCitedBach,James."ExploratoryTestingandthePlanningMyth."19March2001.http://www.stickyminds.com/r.asp?F=DART_2359

Bach,James."ExploratoryTestingExplained."v.1.3,16April2003.http://www.satisfice.com/articles/et-article.pdf

Beizer,Boris(1990).SoftwareTestingTechniques(SecondEdition).VanNostrandReinhold.ISBN0-442-20672-0.

Beizer,Boris(1995).Black-BoxTesting:TechniquesforFunctionalTestingofSoftwareandSystems.JohnWiley&Sons.ISBN0-471-12094-4.

Binder,RobertV.(2000).TestingObject-OrientedSystems:Models,Patterns,andTools.Addison-Wesley.ISBN0-201-80938-9.

Brownlie,Robert,etal."RobustTestingofAT&TPMX/StarMAILUsingOATS,"AT&TTechnicalJournal,Vol.71,No.3,May/June1992.

Carr,MarvinJ.,etal.(1993)Taxonomy-BasedRiskIdentification.TechnicalReportCMU/SEI-93-TR-6,ESC-TR-93-183,June1993.http://www.sei.cmu.edu/pub/documents/93.reports/pdf/tr06.93.pdf

Cockburn,Alistair(2000).WritingEffectiveUseCases.Addison-Wesley.ISBN0-201-70225-8.

Cohen,D.M.,etal."TheAETGSystem:AnApproachtoTestingBasedonCombinatorialDesign."IEEETransactionsonSoftwareEngineering,Vol.23,No.7,July1997.

Copeland,Lee."ExploratoryPlanning."3September2001.http://www.stickyminds.com/r.asp?F=DART_2805

Craig,RickD.andStefanP.Jaskiel(2002).SystematicSoftwareTesting.ArtechHousePublishers.ISBN1-58053-508-9.

Fowler,MartinandKendallScott(1999).UMLDistilled:ABriefGuidetotheStandardObjectModelingLanguage(2ndEdition).Addison-Wesley.ISBN0-201-65783X.

Gilb,TomandDorothyGraham(1993).SoftwareInspection.Addison-Wesley.ISBN0-201-63181-4.

Harel,David."Statecharts:avisualformalismforcomplexsystems."ScienceofComputerProgramming8,1987.


http://www.satisfice.com/articles/et-article.pdf

http://www.sei.cmu.edu/pub/documents/93.reports/pdf/tr06.93.pdf


Hetzel,Bill(1998).TheCompleteGuidetoSoftwareTesting(SecondEdition).JohnWiley&Sons.ISBN0-471-56567-9.

IEEEStandardforSoftwareTestDocumentation:IEEEStandard829-1998.ISBN0-7381-1443-X.

IEEEStandardGlossaryofSoftwareEngineeringTerminology:IEEEStandard610.12-1990.ISBN1-55937-067-X.

ISO(1991).ISO/IECStandard9126-1.SoftwareEngineering-ProductQuality-Part1:QualityModel.ISOCopyrightOffice,Geneva,June2001.

Jacobsen,Ivar,etal(1992).Object-OrientedSystemsEngineering:AUseCaseDrivenApproach.Addison-Wesley.ISBN0-201-54435-0.

Kaner,Cem,JackFalkandHungQuocNguyen(1999).TestingComputerSoftware(SecondEdition).JohnWiley&Sons.ISBN0-471-35846-0.

Kaner,Cem,JamesBach,andBretPettichord(2002).LessonsLearnedinSoftwareTesting:AContext-DrivenApproach.JohnWiley&Sons.ISBN0-471-08112-4.

Kuhn,D.RichardandMichaelJ.Reilly."AnInvestigationoftheApplicabilityofDesignofExperimentstoSoftwareTesting,"27thNASA/IEEESoftwareEngineeringWorkshop,NASAGoddardSpaceFlightCenter,4–6December2002.http://csrc.nist.gov/staff/kuhn/kuhn-reilly-02.pdf

Marick,Brian(1995).TheCraftofSoftwareTesting:SubsystemTestingIncludingObject-BasedandObject-OrientedTesting.Prentice-Hall.ISBN0-131-77411-5.

Mandl,Robert."OrthogonalLatinSquares:AnApplicationofExperimentDesigntoCompilerTesting,"CommunicationsoftheACM,Vol.128,No.10,October1985.

Mealy,G.H."Amethodforsynthesizingsequentialcircuits."BellSystemTechnicalJournal,34(5):1955.

Myers,Glenford(1979).TheArtofSoftwareTesting.JohnWiley&Sons.ISBN0-471-04328-1.

Moore,E.F."Gedanken-experimentsonsequentialmachines,"AutomataStudies(C.E.ShannonandJ.McCarthy,eds.),Princeton,NewJersey:PrincetonUniversityPress,1956.

Phadke,MadhavS.(1989).QualityEngineeringUsingRobustDesign.Prentice-Hall.ISBN0-13-745167-9.

Planning,MCDP5.UnitedStatesMarineCorps.https://www.doctrine.usmc.mil/mcdp/view/mpdpub5.pdf

http://csrc.nist.gov/staff/kuhn/kuhn-reilly-02.pdf

http://www.https://www.doctrine.usmc.mil/mcdp/view/mpdpub5.pdf

Pressman,RogerS.(1982).SoftwareEngineering:APractitioner'sApproach(FourthEdition).McGraw-Hill.ISBN0-07-052182-4.

Rapps,SandraandElaineJ.Weyuker."DataFlowAnalysisTechniquesForTestDataSelection."SixthInternationalConferenceonSoftwareEngineering,Tokyo,Japan,September13–16,1982.

Rumbaugh,James,etal.(1991).Object-OrientedModelingandDesign.Prentice-Hall.ISBN0-13-629841-9.

Watson,ArthurH.andThomasJ.McCabe.StructuredTesting:ATestingMethodologyUsingtheCyclomaticComplexityMetric.NISTSpecialPublication500-235.http://www.mccabe.com/nist/nist_pub.php

Wallace,DeloresR.andD.RichardKuhn."FailureModesinMedicalDeviceSoftware:AnAnalysisof15YearsofRecallData,"InternationalJournalofReliability,Quality,andSafetyEngineering,Vol.8,No.4,2001.http://csrc.nist.gov/staff/kuhn/final-rqse.pdf

Weinberg,GeraldM.(1975).AnIntroductiontoGeneralSystemsThinking.JohnWiley&Sons.ISBN0-471-92563-2.

Whittaker,JamesA.(2003).HowtoBreakSoftware:APracticalGuidetoTesting.AddisonWesley.ISBN0-201-79619-8.

http://www.mccabe.com/nist/nist_pub.php

http://csrc.nist.gov/staff/kuhn/final-rqse.pdf

OtherUsefulPublicationsBeizer,Boris(1984).SoftwareSystemTestingandQualityAssurance.VanNostrandReinhold.ISBN0-442-21306-9.

Black,Rex(1999).ManagingtheTestingProcess.MicrosoftPress.ISBN0-7356-0584-X.

BritishComputerSociety.StandardonSoftwareComponentTesting.BS7925-2.http://www.testingstandards.co.ukhttp://www.testingstandards.com/BS7925_3_4.zip

Kit,Edward(1995).SoftwareTestingintheRealWorld:ImprovingtheProcess.Addison-Wesley.ISBN0-201-87756-2.

McGregor,JohnD.andDavidA.Sykes(2001).APracticalGuidetoTestingObject-OrientedSoftware.Addison-Wesley.ISBN0-201-32564-0.

Meyer,Bertrand(2000).Object-OrientedSoftwareConstruction(2ndEdition).Prentice-Hall.ISBN0-136-29155-4.

Roper,Marc(1994).SoftwareTesting.McGraw-Hill.ISBN0-07-707466-1.

Tamres,Louise(2002).IntroducingSoftwareTesting.Addison-Wesley.ISBN0-201-71974–6.

http://www.testingstandards.co.uk

http://www.testingstandards.com/BS7925_3_4.zip

Index

Aacceptancetesting,10,135ACTtestscores,121action,51–52,95,97actor,128adhoctesting,202adaptiveplanning,213–214,216alleventscoverage,106allpairstesting,64allpairstoolsAETG,88Allpairs,85rdExpert,68

allpathscoverage,107allstatescoverage,105alltransitionscoverage,108Allison,Chuck,xviAllpairsalgorithm,66,85–88comparisonwithorthogonalarrays,88–89unbalanced,87–88

AnIntroductiontoGeneralSystemsThinking,236automaticvariables,169

Index

BBach,James,xvi,85,202,206,212bananaprinciple,236Barker,Joel,182baselinepath,155basispathsets,164basispathtesting,154–159baselinepath,155basispathsets,164example,160–164multiplesets,159pathcreation,155

basispaths,154–155Beizer,Boris,2,6,135,198,220,226,236Beizer'staxonomy,226binarydecisions,155Binder,Robert,11,222,229Binder'sobject-orientedtaxonomy,229blackboxtesting,8,20–22,140blackdotsymbol,95,97Black,Rex,xviBloom,Benjamin,246bookends,220boundaries,40,134boundaryvaluetesting,40,197boundaries,40,134examples,45

boundaryvalues,134Box,George,232Brown&Donaldson,16,54,71,91,94,160,250authorizationcode,17

bulls-eyesymbol,97Bulwer-LyttonFictionContest,xvi

Bulwer-Lytton,EdwardGeorge,xvibusinessrules,50,58

Index

CcasestudiesBrown&Donaldson,16,54,71,91,94,160,250StatelessUniversityRegistrationSystem,17,36,47,56,71,91,94,111,131,136,260

charteredexploratorytesting,206choosingoffpoints,118choosingonpoints,118choosingtestcases,119classicalplanning,213closedboundary,118Cockburn,Alistair,130codequalitytesting,11collapsedrules,55combinationtesting,Seepairwisetestingcombinations,63combinedrules,55competency,246complexbusinessrules,50,53–54,58conditioncoverage,151conditional,102conditions,50,51–53,55,58connectionlost,134contract,27controlflow,141controlflowgraphs,145–147,154–159,171–176controlflowpaths,172controlflowtestingconditioncoverage,151controlflowgraphs,145–147,154–159,171–176decisionpoint,146decision/conditioncoverage,151executionpaths,144exhaustivetesting,144junctionpoint,146

levelsoftestcoverage,147–153limitingloopexecution,153missingpaths,144multipleconditioncoverage,152pathcoverage,153processblock,146statementcoverage,147,150testcoverage,147–153

coverage,147,153Craig,Rick,xiii,xvi,2createdvariables,169creatingtestcases,109–110,134cyclomaticcomplexity,154–155binarydecisions,155edges,154–155nodes,154–155

Index

Ddataflowgraphs,171–174dataflowtesting,168~define,170~kill,170~use,170createdvariables,169dataflowgraph,171–174define-define,170define-kill,170define-use,170define-use-kill,172–174destroyedvariables,169dynamicdataflowtesting,176kill-define,170killedvariables,169kill-kill,170kill-use,170lifecycle,169processingflow,171static,171–176usedvariables,169use-define,170use-kill,170use-use,170variablesincomputation,169variablesinconditionals,169

datasensitivityerrors,141debugging,3decisioncoverage,150decisionpoint,146decisiontablesactions,51–52collapsedrules,55combinedrules,55conditions,50–53,55,58derivingtestcases,52–53examples,54–58expectedresults,54

"firing",51decisiontabletesting,50–54decision/conditioncoverage,151defecttaxonomies,222–232defensive-design,28defensive-testing,28define-define,170define-kill,170define-use,170define-use-kill,172–174design-by-contract,27destroyedvariables,169developmentmanagers,xvdevelopmentparadigm,129diskfull,134documentingtransactionsflowcharts,128HIPOdiagrams,128text,128usecases,128

domainanalysis,116domainanalysistestingchoosingoffpoints,118choosingonpoints,118choosingtestcases,119closedboundary,118DomainTestMatrix,120example,121extraboundary,116–117inpoint,118,120interactionsbetweenvariables,116missingboundary,116–117offpoint,118,120onpoint,118,120openboundary,118outpoint,118shiftedboundary,116–117tiltedboundary,116–117

DomainTestMatrix,120,134

double-modedefects,65drivernotloaded,134dynamicdataflowtesting,176

Index

Eedges,154–155equivalenceclasspartitioning,197equivalenceclasstesting,24–33examples,33–34inputequivalenceclasses,29outputequivalentclasses,33

equivalenceclasstypescontinuousrangeofvalues,29discretevalueswithinarange,29–30multipleselection,30–31singleselection,30

equivalenceclasses,25,28events,94–96executionpaths,141,144exhaustivetesting,21,144existingpaths,141exploratoryplanningrigorousplan,202exploratorytesting,182,202–208,212chartered,206charters,206consciousplan,202freestyle,207process,205

extraboundary,116–117

Index

FFAFSA,59Faught,Danny,xvifaultmodel,222non-specific,222specific,222

Ferguson,Marilyn,182figuresdecisioncoverage,150annotatedcontrolflowdiagram,171–173B&Dcontrolflowgraph,162–163B&DJavacode,160baselinebasispath,156boundaryvalues,43canceltheReservation,101cancellationfromPaidstate,102cancellationfromTicketedstate,103continuousequivalenceclasses,29dataontheboundaries,44defectdiscoveryrate,238discreteequivalenceclasses,29enrollanddropacourse,113evaluationofcomplexconditions,152examplecontrolflowgraph,154fifthbasispath,158flowgraphequivalent,147fourexecutionpaths,148fourthbasispath,157graphicalrepresentation,148IEEE829testdocumentation,188interestingflowdiagram,153multipleselectionequivalenceclasses,30on,off,in,andoutpoints,119pathterminates,99PayTimerexpires,100ReservationinMadestate,95ReservationinPaidstate,95ReservationinTicketedstate,97ReservationinUsedstate,98secondbasispath,156

seventhbasispath,159singleselectionequivalenceclasses,30sixthbasispath,158StatelessUniversityAdmissionsMatrix,121StatelessUniversityusecases,129SURSmaintenancescreen,56testcasesthattriggerallevents,106testcasesthatvisiteachstate,106thirdbasispath,157twodimensionalboundarydefects,117Waterfallmodel,187

fire,51firing,51FloridaInstituteofTechnology,135flowcharts,128football,212–213FreeApplicationforFederalStudentAid,59freestyleexploratorytesting,207functionalitytesting,11

Index

GGerrard,Paul,xviGraceL.FergusonAirline&StormDoorCompany,94GradePointAverage,121Graham,Dorothy,xvigrayboxtesting,8guard,102

Index

HHagar,Jon,xviHetzel,Bill,236highschoolgrades,121HIPOdiagrams,128Holodeck,135,149HowToBreakSoftware,230HowtoGetControlofYourTimeandYourLife,205

Index

IIEEE829advantagescompletenesschecklist,190evaluationoftestpractices,190facilitatecommunication,190increasedmanageability,190

IEEE829standard,189–190bugreport,195releasenotes,194testcasespecification,188,193testdesignspecification,188,192testincidentreport,188,195testitemtransmittalreport,188,194testlog,188,194testplan,188,190testprocedurespecification,188,193testsummaryreport,188,196

IEEE829testcasespecificationenvironmentalneeds,193inputspecifications,193intercasedependencies,193outputspecifications,193specialproceduralrequirements,193testcasespecificationidentifier,193testitems,193

IEEE829testdesignspecificationapproachrefinements,192featurepass/failcriteria,193featurestobetested,192testdesignspecificationidentifier,192testidentification,192

IEEE829testincidentreportimpact,195incidentdescription,195summary,195testincidentreportidentifier,195

IEEE829testitemtransmittalreportapprovals,194location,194status,194

transmittalreportidentifier,194transmitteditems,194

IEEE829testlogactivityandevententries,195description,195testlogidentifier,195

IEEE829testplanapproach,191approvals,192environmentalneeds,192featuresnottobetested,191featurestobetested,191introduction,191itempass/failcriteria,191responsibilities,192risksandcontingencies,192schedule,192staffingandtrainingneeds,192suspensionandresumptioncriteria,191testdeliverables,191testitems,191testplanidentifier,191testingtasks,191

IEEE829testprocedurespecificationproceduresteps,194purpose,194specialrequirements,194testprocedurespecificationidentifier,194

IEEE829testsummaryreportapprovals,196comprehensiveassessment,196evaluation,196summary,196summaryofactivities,196summaryofresults,196testsummaryreportidentifier,196variance,196

IEEEStandard610.12–1990,2IEEEStandardforSoftwareTestDocumentation,188–196IEEEStandardGlossaryofSoftwareEngineeringTerminology,2IEEEStd829–1998,188–196inpoint,118,120

inspection,4,41,134,164,171,176,230integration,140integrationtesting,9interactionsbetweenvariables,116ISO9126StandardSoftwareProductEvaluation—QualityCharacteristicsandGuidelines,225

Index

JJacobsen,Ivar,128Jagger,Mick,83Jaskiel,Stefan,xiii,2junctionpoint,146

Index

KKaner,Cem,202Kaner,Falk,andNguyen'staxonomy,228keypointsadaptiveplanning,213advantageofstate-transitiontable,105bankcombinations,62blackboxtestinghelpsefficiencyandeffectiveness,21boundariesarewheredefectshide,40choosingcombinations,65comparingwhatiswithwhatoughttobe,2controlflowgraphs,145createtestcases,53cultivatetheskillofchoosingpoorly,63dataczar,135dataflowtesting,168domainanalysis,116doubleintegralsign,xivequivalenceclasses,26evaluatetherisk,134executingCtestcases,155exploratorycharter,206exploratorytesting,202expressyourappreciation,33Holodeck,149importanceoftestdesign,xiiilocateorthogonalarrays,67Methuselah,27mostimportanttest,205objectorientedcombinations,62pathdefinition,144postinstallationtestplanning,63randomselection,64rarelywillwehavetime,31rdExperttool,68scriptedandexploratorytesting,217scriptedandexploratoryparadigms,182taxonomy,222testcasesareinputs,outputs,andorder,6testcasesateachboundary,42testcasesmustbedesigned,5

testingeverytransition,108testinglevels-unit,integration,system,acceptance,9time-sequencedpairs,170triggeralltransitions,108TwentyQuestions,203useGUIdesigntools,34warm,fuzzyfeelings,29wecanneverbesureofcoverage,21webcombinations,62whogetstheblame,28yourtaxonomy,232

kiddiepool,198kill-define,170killedvariables,169kill-kill,170kill-use,170

Index

LL18(2137)orthogonalarray,69

L18(35)orthogonalarray,68




Lakein,Alan,205largenumberofcombinations,63LessonsLearnedinSoftwareTesting,85,241levelsofcompetency,246lifecycle,169limitingloopexecution,153lowmemory,134

Index

MMcCabe,Tom,154Mealy,G.H.,97Meilof,Anne,xviMyers,Glenford,3,21,237Middleton,Wayne,xvimissingboundary,116–117missingpaths,144Moore,E.F.,97multipleconditioncoverage,152multiplebasispathsets,159

Index

Nnestedstate-transitiondiagrams,97nodes,154–155nonexistentpaths,141non-specificfaultmodel,222

Index

OO'Loughlin,Martin,xviiObject-OrientedSoftwareEngineering:AUseCaseDrivenApproach,128offpoints,118,120onpoints,118,120openboundary,118oraclesexistingprograms,7kiddie,6purchasedtestsuites,7regressiontestsuites,7validateddata,7

orderofexecution,6,7orthogonalarrays,66–70balanced,87definition,68L18(2137),69L18(35),68L4(23),66L64(8243),72L9(34),67mappingonto,74–76comparisonwithAllpairsalgorithm,88–89dealingwithextracolumns,83dealingwithextravaluesinarow,83notation,67unassignedcells,82using70–85

outpoint,118

Index

Ppairwisetesting,64–65,197additionaltests,89allpairstesting,64Allpairsalgorithm,66,85,88–89constraints,88documentedstudies,64–65effectiveness,64–65,89

paradigms,182cloudourvision,182sharpenourvision,182

Paradigms:TheBusinessofDiscoveringtheFuture,182partitionedstate-transitiondiagram,97pathcoverage,153pathcreation,155pathtesting,140performancetesting,11Perry,Dale,xviPhadke,MadhavS.,66,72planning,183,212,216adaptive,213classical,213

Planning,215,216planningfunctions,215planningheuristic,214planningpitfalls,215post-conditions,27–28pre-conditions,27–28Pressman,Roger,5processblock,146processingflow,171projectleveltaxonomies,223purchasedtestsuites,7

Index

Qqualityassuranceengineers,xvQualityEngineeringUsingRobustDesign,66,72Quentin,Geoff,xvi

Index

RRAD,187rapidapplicationdevelopment,187Rapps,Sandra,168regressiontestsuites,7reviews,4Rice,Dr.Scott,xviirisk,3,134Rose-Coutré,Robert,xviRoyce,WinstonW.,186

Index

Sscenario,128scriptedtesting,182,186,208,212auditability,187objectivity,187repeatability,187

securitytesting,11shiftedboundary,116–117single-modedefects,65Sloane,NeilJ.A.,72Snook,Sid,xvisoftwaredefecttaxonomies,223–232Beizer'staxonomy,226Binder'sobject-orientedtaxonomy,229Kaner,Falk,andNguyen'staxonomy,228Vijayaraghavan'seCommercetaxonomy,231Whittaker'sHowToBreakSoftwaretaxonomy,230

softwaredevelopers,xvSoftwareEngineeringInstitute,223softwareinspection,4,41,134,164,171,176,230SoftwareInspection,41SoftwareQualityEngineering,16SoftwareSystemTestingandQualityAssurance,198softwaretestengineers,xvSoftwareTestingTechniques,226specificfaultmodel,222state,51,95–96StatelessUniversity,59,94,121,124admisions,121

StatelessUniversityRegistrationSystem,17,36,47,56,71,91,94,111,131,136,260statementcoverage,147state-transitiondiagrams,94–95,128action,95,97alleventscoverage,106

allpathscoverage,107allstatescoverage,105alltransitionscoverage,108blackdotsymbol,95,97bulls-eyesymbol,97conditional,102creatingtestcases,105–110event,94–96guard,102mixingdifferententities,97nestedstate-transitiondiagrams,97partitionedstate-transitiondiagrams,97state,95–96transition,95–96

state-transitiontablesaction,104advantage,105creatingtestcases,105–110currentstate,104disadvantage,105event,104nextstate,104

staticdataflowtesting,171StickyMinds.com,202,212stoppingcriteria,236–240defectdiscoveryrate,238marginalcost,239metcoveragegoals,237teamconsensus,239thebosssays"Shipit!",240

structuredtesting,154–159system,140systemstate,51systemtesting,10,135systemtransactions,135SystematicSoftwareTesting,xiii,2

Index

TtablesAllpairsprograminput,86Allpairsprogramoutput,86Beizer'sBugtaxonomy,226Binder'sClassScopetaxonomy,230Binder'sMethodScopetaxonomy,229classicalvs.adaptiveplanning,214classicalvs.exploratoryplanning,214collapseddecisiontable,55decisiontable,51–52,54DomainAnalysistestcases,122DomainTestMatrix,120exampleusecase,132furthercollapseddecisiontable,55genericdecisiontable,50invaliddatavalues,32ISO9126QualityCharacteristics,225Kaner'staxonomy,228L18(2137)orthogonalarray,70L18(35)orthogonalarray,69L4(23)orthogonalarray,66L64(8243)orthogonalarray,73,75,77,79,81,84L9(34)orthogonalarray,67sampletestcases,53SEItaxonomy,223sensitizingcontrolflowpaths,164setoftestcases,44StatelessUniversityAdmissionsMatrix,121State-Transitiontable,104SURSdecisiontable,57testcasetable,54testingallvalidtransitions,109usecasetemplate,130validdatavalues,31varyingvalidandinvalidvalues,32,34Whittaker'sFaulttaxonomy,230

taxonomies,222–223creatingyourown,232projectlevel,223–226

softwaredefect,226–232taxonomy-basedriskidentification,223templateforusecases,130testanalysis,2testcasecomponentsinputs,6–7orderofexecution,7–8outputs,6–7

testcasecreation,134testcasedefinition,5testcasedesign,2testcasedesignstylescascading,7independent,8

testcasespecification,SeeIEEE829testcasespecificationtestcasesubsets,22testcoverage,147–153testdesign,Seetestcasedesigntestdesignspecification,SeeIEEE829testdesignspecificationtestincidentreport,SeeIEEE829testincidentreporttestitemtransmittalreport,SeeIEEE829testitemtransmittalreporttestlog,SeeIEEE829testlogtestmanagers,xvtestmultiplevariablessimultaneously,116testoracles,6–7testplan,SeeIEEE829testplantestplanning,2,212–217testprocedurespecification,SeeIEEE829testprocedurespecificationtestsuitespurchased,7regression,7

testsummaryreport,SeeIEEE829testsummaryreporttesterskillscarefulobservers,207carefulreporters,207criticalthinkers,207evaluaterisk,207

goodmodelers,207notdistracted,207selfmanaged,207testdesigners,207

testingmaturitylevels,2–4blackbox,8,20–21exhaustive,11,21graybox,8whitebox,8,20

testingchallenges,4TestingComputerSoftware,202,228testinginteractingvariables,116testinglevelsacceptancetesting,10integrationtesting,9systemtesting,10unittesting,9

testingmaturity,2–4testingnsimultaneousdimensions,116TestingObject-OrientedSystems,222testingtechniquesAllpairsalgorithm,85allpairstesting,64boundaryvaluetesting,40dataflowtesting,168decisiontables,50domainanalysis,116equivalenceclasstesting,24pairwisetesting,64

testingtoolbox,246testing,definitionaconcurrentlifecycleprocess,2comparison,2

testing-by-contract,27TheCompleteGuideToSoftwareTesting,236tiltedboundary,116–117timebox,206toolsAETG,88

Allpairs,85rdExpert,68

transactiontesting,135transactions,128transitions,95–96TwentyQuestions,203typesoftestingdefensive-testing,28testing-by-contract,27

Index

UUMLDistilled:ABriefGuideToTheStandardObjectModelingLanguage,97UnifiedModelingLanguage,129unit,140unittesting,9,140UnitedStatesMarineCorps,215usabilitytesting,11usecases,128–129actor,128example,131–133functionalrequirements,128–129scenario,128template,130–131value,129

usecasetemplateactors,130–131channelstoprimaryactor,131channelstosecondaryactors,131completenesslevel,131datedue,131extensions,131failedendconditions,130frequency,131goalincontext,130level,130mainsuccessscenario,130name,130numberoridentifier,130openissues,131preconditions,130primaryactor,130priority,131responsetime,131scope,130secondaryactors,131sub-variations,131successendcondition,130trigger,130

usedvariables,169

use-define,170use-kill,170use-use,170

Index

Vvariablesincomputation,169variablesinconditionals,169validation,186verification,186Vijayaraghavan'seCommercetaxonomy,231vonMoltke,Helmuth,215

Index

Wwaterfalldevelopmentmodel,186–187waterfalldevelopmentmodelphasescoding,186operations,186programdesign,186requirementsanalysis,186softwarerequirements,186systemrequirements,186

waterfalldevelopmentmodeltesting,186Webtestinglevels,11codequality,11functionality,11performance,11security,11usability,11

Weinberg,Gerald,236Weyuker,Elaine,168whiteboxtesting,8,20,140–142existingpaths,141nonexistentpaths,141

whiteboxtestingtechniquescontrolflowtesting,144–159dataflowtesting,168–176

Whittaker,James,135,230Whittaker'sHowToBreakSoftwaretaxonomy,230WritingEffectiveUseCases,130

Index

Yy=mx+b,121

ListofFigures

Chapter3:EquivalenceClassTestingFigure3-1:Continuousequivalenceclasses

Figure3-2:Discreteequivalenceclasses

Figure3-3:Singleselectionequivalenceclasses

Figure3-4:Multipleselectionequivalenceclass

Chapter4:BoundaryValueTestingFigure4-1:Boundaryvaluesforacontinuousrangeofinputs.

Figure4-2:Boundaryvaluesforadiscreterangeofinputs.

Figure4-3:Datapointsontheboundariesanddatapointsjustoutsidetheboundaries.

Chapter5:DecisionTableTestingFigure5-1:SURSStudentDatabaseMaintenanceScreen.

Chapter6:PairwiseTestingFigure6-1:Orthogonalarraynotation

Chapter7:State-TransitionTestingFigure7-1:TheReservationisMade.

Figure7-2:TheReservationtransitionstothePaidstate.

Figure7-3:TheReservationtransitionstotheTicketedstate.

Figure7-4:TheReservationtransitionstotheUsedstate.

Figure7-5:Thepathends.

Figure7-6:ThePayTimerexpiresandtheReservationiscancelledfornonpayment.

Figure7-7:CanceltheReservationfromtheMadestate.

Figure7-8:CancellationfromthePaidstate.

Figure7-9:CancellationfromtheTicketedstate.

Figure7-10:Asetoftestcasesthat"visit"eachstate.

Figure7-11:Asetoftestcasesthattriggeralleventsatleastonce.

Figure7-12:Asetoftestcasesthattriggeralltransitionsatleastonce.

Figure7-13:State-transitiondiagramforenrollanddropacourseatStatelessU.

Chapter8:DomainAnalysisTestingFigure8-1:Twodimensionalboundarydefects.

Figure8-2:Examplesofon,off,in,andoutpointsforbothclosedandopenboundaries.

Figure8-3:StatelessUniversityAdmissionsMatrixingraphicalform.

Chapter9:UseCaseTestingFigure9-1:SomeStatelessUniversityusecases.

Chapter10:ControlFlowTestingFigure10-1:Flowgraphequivalentofprogramcode.

Figure10-2:Graphicalrepresentationofthetwo-linecodesnippet.

Figure10-3:Fourexecutionpaths.

Figure10-4:Twotestcasesthatyield100%decisioncoverage.

Figure10-5:Compilerevaluationofcomplexconditions.

Figure10-6:Aninterestingflowdiagramwithmany,manypaths.

Figure10-7:Anexamplecontrolflowgraph.

Figure10-8:ThechosenbaselinebasispathABDEGKMQS

Figure10-9:ThesecondbasispathACDEGKMQS

Figure10-10:ThethirdbasispathABDFILORS

Figure10-11:ThefourthbasispathABDEHKMQS

Figure10-12:ThefifthbasispathABDEGKNQS

Figure10-13:ThesixthbasispathACDFJLORS

Figure10-14:TheseventhbasispathACDFILPRS

Figure10-15:JavacodeforBrown&Donaldson'sevaluateBuySellmodule.



Chapter11:DataFlowTestingFigure11-1:Thecontrolflowdiagramannotatedwithdefine-use-killinformationforeachofthemodule'svariables.

Figure11-2:Thecontrolflowdiagramannotatedwithdefine-use-killinformationforthexvariable.

Figure11-3:Thecontrolflowdiagramannotatedwithdefine-use-killinformationfortheyvariable.

Figure11-4:Thecontrolflowdiagramannotatedwithdefine-use-killinformationforthezvariable.

Chapter12:ScriptedTestingFigure12-1:TheWaterfalllifecyclemodel.

Figure12-2:TheIEEE829TestDocuments

Chapter16:WhentoStopTestingFigure16-1:DefectDiscoveryRate

ListofTables

Chapter3:EquivalenceClassTestingTable3-1:Atestcaseofvaliddatavalues.

Table3-2:Atestcaseofallinvaliddatavalues.Thisisnotagoodapproach.


Table3-4:Asetoftestcasesvaryinginvalidvaluesonebyonebutalsovaryingthevalidvalues.


Chapter4:BoundaryValueTestingTable4-1:Asetoftestcasescontainingcombinationsofvalid(ontheboundary)valuesandinvalid(offtheboundary)points.

Chapter5:DecisionTableTestingTable5-1:Thegeneralformofadecisiontable.

Table5-2:Adecisiontablewithtwobinaryconditions.

Table5-3:Addingasingleactiontoadecisiontable.

Table5-4:Adecisiontablewithmultipleactions.

Table5-5:Adecisiontablewithnon-binaryconditions.

Table5-6:Sampletestcases.

Table5-7:Adecisiontableconvertedtoatestcasetable.

Table5-8:AdecisiontablefortheBrown&DonaldsonBuyorder.

Table5-9:Acollapseddecisiontablereflecting"Don'tCare"conditions.

Table5-10:Afurthercollapseddecisiontablereflecting"Don'tCare"conditions.


Chapter6:PairwiseTestingTable6-1:L4(23)OrthogonalArray





Table6-6:L64(8243)withapartialmappingofitsfirstcolumn.

Table6-7:L64(8243)withafullmappingofitsfirstcolumn.

Table6-8:L64(8243)withafullmappingofitsfirstandsecondcolumns.

Table6-9:L64(8243)withafullmappingofallitscolumns.

Table6-10:L64(8243)withafullmappingofallitscolumnsincludingthe"extra"cells.

Table6-11:InputtotheAllpairsprogram.

Table6-12:OutputfromtheAllpairsprogram.

Chapter7:State-TransitionTestingTable7-1:State-TransitiontableforReservation.

Table7-2:TestingallvalidtransitionsfromaState-transitiontable.

Chapter8:DomainAnalysisTestingTable8-1:ExampleDomainTestMatrix.

Table8-2:StatelessUniversityAdmissionsMatrix.

Table8-3:1x1DomainAnalysistestcasesforStatelessUniversityadmissions.

Chapter9:UseCaseTestingTable9-1:Usecasetemplate.

Table9-2:Exampleusecase.

Chapter10:ControlFlowTestingTable10-1:Datavaluestosensitizethedifferentcontrolflowpaths.

Chapter14:TestPlanningTable14-1:Classicalplanningvs.Adaptiveplanning.

Table14-2:Classicaltestplanningvs.Exploratorytestplanning.

Chapter15:DefectTaxonomiesTable15-1:TheSEITaxonomy-BasedRiskIdentificationtaxonomy.

Table15-2:TheISO9126QualityCharacteristicstaxonomy.

Table15-3:AportionofBeizer'sBugTaxonomy.

Table15-4:AportionofthedefecttaxonomyfromTestingComputerSoftware.

Table15-5:AportionofBinder'sMethodScopeFaultTaxonomy.

Table15-6:AportionofBinder'sClassScopeFaultTaxonomy.

Table15-7:AportionofWhittaker'sFaultTaxonomy.

ListofExamples

Chapter3:EquivalenceClassTestingExample1

Example2

Example3

Example4

Chapter4:BoundaryValueTestingExample1

Example2

Chapter5:DecisionTableTestingExample1

Example2