Testing1Software Testing1Testing2Testing ConceptsTesting3BackgroundMain objectives of a project: High Quality & High Productivity (Q&P)Quality has many dimensionsreliability, maintainability, interoperability etc.Reliability is perhaps the most importantReliability: The chances of software failingMore defects => more chances of failure => lesser reliabilityHence Q goal: Have as few defects as possible in the delivered software3Testing4Faults & FailureFailure: A software failure occurs if the behavior of the s/w is different from expected/specified.Fault: cause of software failureFault = bug = defectFailure implies presence of defectsA defect has the potential to cause failure.Definition of a defect is environment, project specific4Testing5Role of TestingReviews are human processes - can not catch all defectsHence there will be requirement defects, design defects and coding defects in codeThese defects have to be identified by testingTherefore testing plays a critical role in ensuring quality.All defects remaining from before as well as new ones introduced have to be identified by testing.5Testing6Detecting defects in TestingDuring testing, software under test (SUT) executed with set of test casesFailure during testing => defects are presentNo failure => confidence grows, but can not say defects are absentTo detect defects, must cause failures during testing6Testing7Test OracleTo check if a failure has occurred when executed with a test case, we need to know the correct behaviorI.e. need a test oracle, which is often a humanHuman oracle makes each test case expensive as someone has to check the correctness of its outputTesting8Test case and test suiteTest case a set of test inputs and execution conditions designed to exercise SUT in a particular mannerTest case should also specify the expected output oracle uses this to detect failureTest suite - group of related test cases generally executed togetherTesting9Test harnessDuring testing, for each test case in a test suite, conditions have to be set, SUT called with inputs, output checked against expected to declare fail/passMany test frameworks (or test harness) exist that automate the testing processEach test case is often a function/methodA test case sets up the conditions, calls the SUT with the required inputsTests the results through assert statementsIf any assert fails declares failureTesting10Levels of TestingThe code contains requirement defects, design defects, and coding defectsNature of defects is different for different injection stagesOne type of testing will be unable to detect the different types of defectsDifferent levels of testing are used to uncover these defects10Testing11User needsAcceptance testingRequirementspecificationSystem testingDesigncodeIntegration testingUnit testing11Testing12Unit TestingDifferent modules tested separatelyFocus: defects injected during codingEssentially a code verification technique, covered in previous chapterUT is closely associated with codingFrequently the programmer does UT; coding phase sometimes called coding and unit testing12Testing13Integration TestingFocuses on interaction of modules in a subsystemUnit tested modules combined to form subsystemsTest cases to exercise the interaction of modules in different waysMay be skipped if the system is not too large
13Testing14System TestingEntire software system is testedFocus: does the software implement the requirements?Validation exercise for the system with respect to the requirementsGenerally the final testing stage before the software is deliveredMay be done by independent peopleDefects removed by developersMost time consuming test phase14Testing15Acceptance TestingFocus: Does the software satisfy user needs?Generally done by end users/customer in customer environment, with real dataOnly after successful AT software is deployedAny defects found,are removed by developersAcceptance test plan is based on the acceptance test criteria in the SRS15Testing16Other forms of testingPerformance testingtools needed to measure performance Stress testingload the system to peak, load generation tools neededRegression testingtest that previous functionality works alrightimportant when changes are madePrevious test records are needed for comparisons Prioritization of testcases needed when complete test suite cannot be executed for a change16Testing17Testing ProcessTesting18TestingTesting only reveals the presence of defectsDoes not identify nature and location of defectsIdentifying & removing the defect => role of debugging and reworkPreparing test cases, performing testing, defects identification & removal all consume effortOverall testing becomes very expensive : 30-50% development cost18Testing19TestingMultiple levels of testing are done in a projectAt each level, for each SUT, test cases have to be designed and then executedOverall, testing is very complex in a project and has to be done wellTesting process at a high level has: test planning, test case design, and test execution19Testing20Test PlanTesting usually starts with test plan and ends with acceptance testingTest plan is a general document that defines the scope and approach for testing for the whole projectInputs are SRS, project plan, designTest plan identifies what levels of testing will be done, what units will be tested, etc in the projectTesting21Test PlanTest plan usually containsTest unit specs: what units need to be tested separatelyFeatures to be tested: these may include functionality, performance, usability,Approach: criteria to be used, when to stop, how to evaluate, etcTest deliverablesSchedule and task allocationTesting22Test case DesignTest plan focuses on testing a project; does not focus on details of testing a SUTTest case design has to be done separately for each SUTBased on the plan (approach, features,..) test cases are determined for a unitExpected outcome also needs to be specified for each test case22Testing23Test case designTogether the set of test cases should detect most of the defectsWould like the set of test cases to detect any defects, if it existsWould also like set of test cases to be small - each test case consumes effortDetermining a reasonable set of test case is the most challenging task of testing23Testing24Test case designThe effectiveness and cost of testing depends on the set of test casesQ: How to determine if a set of test cases is good? I.e. the set will detect most of the defects, and a smaller set cannot catch these defectsNo easy way to determine goodness; usually the set of test cases is reviewed by expertsThis requires test cases be specified before testing a key reason for having test case specsTest case specs are essentially a table24Testing25Test case specifications Seq.NoCondition to be testedTest DataExpected resultsuccessful25Testing26Test case specificationsSo for each testing, test case specs are developed, reviewed, and executedPreparing test case specifications is challenging and time consumingTest case criteria can be usedSpecial cases and scenarios may be usedOnce specified, the execution and checking of outputs may be automated through scriptsDesired if repeated testing is neededRegularly done in large projects26Testing27Test case executionExecuting test cases may require drivers or stubs to be written; some tests can be auto, others manualA separate test procedure document may be preparedTest summary report is often an output gives a summary of test cases executed, effort, defects found, etcMonitoring of testing effort is important to ensure that sufficient time is spentComputer time also is an indicator of how testing is proceedingTesting28Defect logging and trackingA large software may have thousands of defects, found by many different peopleOften person who fixes (usually the coder) is different from who findsDue to large scope, reporting and fixing of defects cannot be done informallyDefects found are usually logged in a defect tracking system and then tracked to closureDefect logging and tracking is one of the best practices in industryTesting29Defect loggingA defect in a software project has a life cycle of its own, likeFound by someone, sometime and logged along with info about it (submitted)Job of fixing is assigned; person debugs and then fixes (fixed)The manager or the submitter verifies that the defect is indeed fixed (closed)More elaborate life cycles possibleTesting30Defect logging
Testing31Defect loggingDuring the life cycle, info about defect is logged at diff stages to help debug as well as analysisDefects generally categorized into a few types, and type of defects is recordedODC is one classificationSome std categories: Logic, standards, UI, interface, performance, documentation,.. Testing32Defect loggingSeverity of defects in terms of its impact on sw is also recordedSeverity useful for prioritization of fixingOne categorizationCritical: Show stopperMajor: Has a large impactMinor: An isolated defectCosmetic: No impact on functionalityTesting33Defect loggingIdeally, all defects should be closedSometimes, organizations release software with known defects (hopefully of lower severity only)Organizations have standards for when a product may be releasedDefect log may be used to track the trend of how defect arrival and fixing is happeningTesting34Black Box TestingTesting35Role of Test casesIdeally would like the following for test casesNo failure implies no defects or high qualityIf defects present, then some test case causes a failureRole of test cases is clearly very criticalOnly if test cases are good, the confidence increases after testing35Testing36Test case designDuring test planning, have to design a set of test cases that will detect defects presentSome criteria needed to guide test case selectionTwo approaches to design test casesfunctional or black boxstructural or white boxBoth are complimentary; we discuss a few approaches/criteria for both36Testing37Black Box testingSoftware tested to be treated as a block boxSpecification for the black box is givenThe expected behavior of the system is used to design test casesi.e test cases are determined solely from specification.Internal structure of code not used for test case design37Testing38Black box TestingPremise: Expected behavior is specified.Hence just test for specified expected behavior How it is implemented is not an issue.For modules,specification produced in design specify expected behaviorFor system testing, SRS specifies expected behavior38Testing39Black Box TestingMost thorough functional testing - exhaustive testingSoftware is designed to work for an input spaceTest the software with all elements in the input spaceInfeasible - too high a costNeed better method for selecting test casesDifferent approaches have been proposed39Testing40Equivalence Class partitioningDivide the input space into equivalent classesIf the software works for a test case from a class the it is likely to work for allCan reduce the set of test cases if such equivalent classes can be identifiedGetting ideal equivalent classes is impossibleApproximate it by identifying classes for which different behavior is specified 40Testing41Equivalence class partitioning Rationale: specification requires same behavior for elements in a classSoftware likely to be constructed such that it either fails for all or for none.E.g. if a function was not designed for negative numbers then it will fail for all the negative numbersFor robustness, should form equivalent classes for invalid inputs also
41Testing42Equivalent class partitioning..Every condition specified as input is an equivalent classDefine invalid equivalent classes alsoE.g. range 0< value max is an invalid classWhenever that entire range may not be treated uniformly - split into classes42Testing43Equivalent class partitioning..Should consider eq. classes in outputs also and then give test cases for different classesE.g.: Compute rate of interest given loan amount, monthly installment, and number of monthsEquivalent classes in output: + rate, rate = 0 ,-ve rateHave test cases to get these outputs43Testing44Equivalence classOnce eq classes selected for each of the inputs, test cases have to be selectedSelect each test case covering as many valid eq classes as possibleOr, have a test case that covers at most one valid class for each inputPlus a separate test case for each invalid classTesting45ExampleConsider a program that takes 2 inputs a string s and an integer nProgram determines n most frequent charactersTester believes that programmer may deal with diff types of chars separatelyA set of valid and invalid equivalence classes is givenTesting46Example..InputValid Eq ClassInvalid Eq classS1: Contains numbers2: Lower case letters3: upper case letters4: special chars5: str len between 0-N(max)1: non-ascii char2: str len > NN6: Int in valid range3: Int out of rangeTesting47ExampleTest cases (i.e. s , n) with first methods : str of len < N with lower case, upper case, numbers, and special chars, and n=5Plus test cases for each of the invalid eq classesTotal test cases: 1+3= 4With the second approachA separate str for each type of char (i.e. a str of numbers, one of lower case, ) + invalid casesTotal test cases will be 5 + 2 = 7 Testing48Boundary value analysisPrograms often fail on special valuesThese values often lie on boundary of equivalence classesTest cases that have boundary values have high yieldThese are also called extreme casesA BV test case is a set of input data that lies on the edge of a eq class of input/output48Testing49BVA...For each equivalence class choose values on the edges of the classchoose values just outside the edgesE.g. if 0 21 -> 22 -> 11 -> 33 -> 33 -> 44 -> 55 -> 2Req()Req(); req(); req(); req();req(); req()Seq for 2; req()Req(); fail()Req(); fail(); req()Req(); fail(); req(); req(); req();req(); req()Seq for 6; req()Seq for 6; req(); recover()Testing72State-based testingSB testing focuses on testing the states and transitions to/from themDifferent system scenarios get tested; some easy to overlook otherwiseState model is often done after design information is availableHence it is sometimes called grey box testing (as it not pure black box)Testing73White Box TestingTesting74White box testingBlack box testing focuses only on functionalityWhat the program does; not how it is implementedWhite box testing focuses on implementationAim is to exercise different program structures with the intent of uncovering errorsIs also called structural testingVarious criteria exist for test case designTest cases have to be selected to satisfy coverage criteria74Testing75Types of structural testingControl flow based criterialooks at the coverage of the control flow graphData flow based testinglooks at the coverage in the definition-use graphMutation testinglooks at various mutants of the programWe will discuss only control flow based criteria these are most commonly used 75Testing76Control flow based criteri...