MSc Software Maintenance - Háskólinn á Akureyristaff.unak.is/andy/MScTesting0708/Lectures/STLec1and2.pdf · MSc Software Testing ... – The user enters a search term in the GUI

29/08/2007 Dr Andy Brooks 1

MSc Software TestingMSc Prófun hugbúnaðar

Fyrirlestrar 1 & 2The SWEBOK Chapter on

Software Testing © IEEE

http://www.swebok.org/


Repeat after Andy:

“software contains faults, software contains faults, ...”


• Static techniques (no execution) are beginning tomature.

• Exhaustive testing on even simple programs can takemonths or years.

• Selecting test techniques often relies on riskmanagement and engineering expertise.

• Observed behavior can be checked against userexpectations (validation), against a specification(verification), or against implicit requirements orreasonable expectations.

“Software testing consists of the dynamicverification of the behavior of a program on afinite set of test cases, suitably selected fromthe usually infinite executions domain, againstthe expected behavior.”


• Planning for testing should begin whenrequirements are being specified.

• Testing should be done throughoutdevelopment and maintenance.

• Corrective maintenance fixes faults afterdelivery.– Even after extensive testing, software always

contains faults.

“Testing is no longer seen as an activity whichstarts after the coding phase is complete, withthe limited purpose of detecting failures.”


Software Testing

1. SoftwareTesting

Fundamentals2. Test Levels 3. Test

Techniques4. Test Related

Measures

Testing-RelatedTerminology

Key Issues

Relationshipsof Testing to

Other Activities

The Target ofthe Test

Objectives ofTesting

Intuition andExperience

Specification-based

Evaluation ofthe ProgramUnder Test

Evaluation ofthe TestsPerformedCode-based

Fault-based

Usage-based5. Test Process


1.1.2. Faults vs. Failures• A fault (defect) is the cause of a

malfunction.• A failure is an undesired effect observed

during system operation.

1. Software Testing Fundamentals

“Testing can reveal failures, but it is the faults thatcan and must be removed.... the cause of a failurecannot always be unequivocally identified.”

- requirements may have been poorly defined.


1.2.1. Test selection criteria/Test adequacy criteria

• What is a suitable set of test cases ?• Is a set of test cases adequate ?

– Can the testing be stopped ?

1. Software Testing Fundamentals 1.2 Key Issues

1.2.3. Testing for defect identification

• A succesful test case is one which causesa failure.


1.2.4. The oracle problem

• A human or mechanical agent is neededto calculate the correct outcomes of tests.


Can you rely on:i. the values of constants in books,ii. your pocket calculator, oriii hand calculations ?

véfrétt



1.2.5. Theoretical and practical limitations of testing

• Complete testing for real programs is notpossible.

• Dijkstra: “program testing can be used to showthe presence of bugs, but never to show theirabsence”

1.2.6. The problem of infeasible paths

• There may be control flow paths that cannot beexercised by any input data.



1.2.7. Testability

• How easy is it to test the software?– Is technical documentation detailed, accurate,

organised, and accessible?– Is source code accessible?– Is code modular and are coding standards used?– Is software changed frequently?

• Frequent changes imply frequent disruptions to testing.

– Can tests be automated?– Does the software suffer feature creep?

• Testing unncessary features is a waste of resources.


2. Test Levels 2.1 The target of the test

“the target of the test can vary: a single module,a group of such modules..., or a whole system.

“Three big test stages can be conceptuallydistinguished, namely Unit, Integration, and System.”



2.1.1. Unit testing

• “Unit testing verifies the functioning in isolationof software pieces which are separatelytestable.”– Involves executing code.– Testing can be performed by the developers

themselves.

http://junit.org/



2.1.2. Integration testing

• “Integration testing is the process of verifying theinteraction between software components.”

• Traditional top-down or bottom-up strategies are used fortraditional, hierarchically structured software.

• Modern integration testing involves identifying functionalthreads.– The user enters a search term in the GUI dialogue box, which is

transformed to an SQL query in the middle layer, and then thedatabase in the bottom layer is queried...

– Mock objects may be needed if functionality is not currentlyimplemented.

• “program stubs”


a

b c

d fe g

h

j

i

Procedures b and c could bestubbed in a top-down approach.

Procedures e and g could bestubbed in a bottom-up approach.

n

m

l

l, m, or n could be mock objects.



2.1.3. System testing• “System testing is concerned with the behavior of a

whole system.”• Most faults should have already been identified during

unit and integration testing.• System testing often involves testing the non-functional

requirements:– Security– Speed– Accuracy– Reliability

• System testing often also involves testing externalinterfaces to other software and hardware.


2. Test Levels 2.2 Objectives of Testing

2.2.1. Acceptance/qualification testing“Acceptance testing checks the system behavior againstthe customer´s requirements, however these may havebeen expressed.”

– Customers or developers may undertake typical tasks.

2.2.2. Installation testingInstallation testing is much like system testing except thesoftware has now been installed in the target environment.

2.2.3. Alpha and beta testingThis testing involves trial usage in-house (alpha) or byexternal parties (beta).


2.2.4. Conformance/Functional/Correctness testingDoes the observed behavior of the software conform to thespecification?

The program should read in four integers, either in a single line or on separatelines. The first integer represents a year (y), the second integer represents a month(m), the third integer represents a day (d), and the fourth integer represents a windspeed in mph (w).

The program should output historical statements such as:On that Monday it was an F5 tornado.

The program must calculate the tornado strengths according to the following table:Force description speed (in mph)F0 light 40-72F1 moderate 73-112F2 considerable 113-157F3 severe 158-206F4 devastating 207-260F5 incredible 261-318


2.1 What is the Julian calendar?The Julian calendar was introduced by Julius Caesar in45 BC. It was in common use until the late 1500s, whencountries started changing to the Gregorian calendar(section 2.2). However, some countries (for example,Greece and Russia) used it into the early 1900s, and theOrthodox church in Russia still uses it, as do some otherOrthodox churches.

Frequently Asked Questions about CalendarsVersion 2.8

Claus Tøndering15 December 2005

http://www.tondering.dk/claus/cal/calendar28.html



2.2.5. Reliability achievement and evaluationStatistical measures of reliability can be achieved bygenerating random test cases according to the operationalprofile.

Frequency of logical tasks in internet bankingLogin and password ................................33%Display account information ....................33%Pay bill .....................................................25%Transfer money between accounts ...........5%O.s.f.

operational profile1 failure every 5000 logical tasks.1 failure every 50000 logical tasks.

1 failure every 500000 logical tasks.?



2.2.6. Regression testingRegression testing is “the selective retesting of a system orcomponent to verify that modifications have not causedunintended effects...”

(Retesting using all the tests is very expensive in terms oftime and resources, but it is the safest thing to do.)

2.2.7. Performance testingDoes the software meet performance requirements such ascapacity and response time?



2.2.8. Stress testingThe software is tested beyond the maximum design load.

– number of simultaneous users– number of simultaneous applications– very large file transfers– osf

2.2.9. Back-to-back testing

The same set of tests is applied to two versions of thesystem and the results compared.



2.2.10. Recovery testingRecovery testing checks restart capability after a crash.

2.2.11. Configuration testingConfiguration testing analyses the software under differentmemory, network, server, database, browser,...configurations.



2.2.12. Usability testingHow easy is it for users to learn to use the software?How long do users take to recover from mistakes?

– ATMs have a “walk-up-and-use” useability requirement.

2.2.13. Test-driven development (TDD)

Tests cases are written before the code is executed.– Under TDD, sometimes, less attention is paid to producing

software requirement specification documents.


3. Test Techniques 3.1 Intuition and experience

3.1.1. Ad hoc testing

Tests are based on skill, intuition, and experience withsimilar programs. A widely practiced technique.

– things that are empty– division by 0– overflow– characters in a numeric field

3.1.2. Exploratory testing

Tests are dynamically designed, executed and modified.– Not defined in advanced.

í sérstöku augnamiði


3. Test Techniques 3.2. Specification-based

3.2.1. Equivalence partitionThe input domain is divided into subsets of equivalentclasses and an input is taken from each subset as beingrepresentative of the class.

– Age<0 ( -5 ), 0<=Age<=135 ( 45 ), Age >135 ( 142 )

Equivalence partitioning may also involve classes such as:– ‘is a number’ and ‘is not a number’– ‘number of inputs’

The size of the set of test cases (test suite) can grow veryquickly.

world´s oldest woman is 125?



3.2.2. Boundary-value analysisTest cases are chosen near boundaries of the input domainsince faults tend to be found near extreme values.

Robustness testing is an extension of this technique andincludes values outside the input domain.

– Age ( -1, 0, 1 ) ( 134, 135, 136)

world´s oldest woman is 125?



3.2.3. Decision Table (Cause-Effect Graphing)Decision tables represent relationships between conditions(inputs) and actions (outputs). Test cases are derived byconsidering all possible combination of conditions andactions.

If the customer is billed using a fixed rate method, a fixedmonthly charge applies if electricity used is less than 100kwh. If electricity usage is greater, then the customer ischarged according to Schedule A.

If the customer is billed using a variable rate method,Schedule A charging applies to the first 100 kwh ofelectricity consumption and Schedule B charging toadditional consumption.

example


Decision Table example

xSchedule B chargingxxxSchedule A charging

xfixed monthly chargeactions

TFTF>= 100kwhFTFT< 100 kwhTTFFvariable rate accountFFTTfixed rate account4321conditions

4 rules 4 tests



3.2.4. Finite-state machine-basedThe program is modelled as a finite-state machine and testsderived to cover the states and transitions.

3.2.5 Testing from formal specificationsTests can be derived from specifications written in a formallanguage.

3.2.6 Random testingTests are generated at random. The input domain issampled at random.


3. Test Techniques 3.3. Code-based

3.3.1 Control-flow based criteria

The strongest criteria is path testing: all entry-to-exit controlflow paths in the flowgraph are excuted.

Other criteria are statement testing, branch testing,condition/decision testing.

Adequacy is measure by the coverage achieved by the set oftests cases (test suite) expressed as a percentage:

– e.g. 100% statement coverage

If


// Pay overtime at "time and a half"if (hours > STANDARD)

pay = STANDARD*RATE+(hours-STANDARD)*(RATE*1.5);else

pay = hours*RATE;

A single test may involve executing the if-else statement,so the statement is covered.

But a single test will mean the condition only takes one ofits two possible values (TRUE,FALSE) and only one ofthe two possible branches are executed. The conditionand branches are not fully covered.

If


3. Test Techniques 3.3. Code-based

3.3.2 Data-flow based criteria

How are variables defined, used, and killed (undefined)?

The strongest criteria, all definition-use paths, requires thatfor each variable, “every control flow path segment from adefinition of that variable to a use of that definition isexecuted”.

A weaker criteria is all-uses.

double pay = 0.0;

double pay = 0.0;System.out.print ("Enter the number of hours worked: ");int hours = scan.nextInt();System.out.println ();// Pay overtime at "time and a half"if (hours > STANDARD)

pay = STANDARD*RATE+(hours-STANDARD)*(RATE*1.5);else

pay = hours*RATE;NumberFormat fmt = NumberFormat.getCurrencyInstance();System.out.println ("Gross earnings: " + fmt.format(pay));


3. Test Techniques 3.4. Fault-based

3.4.1. Error guessing

Test cases are designed with the aim of identifying likelyfaults in a program given a history of faults in earlier projects,as well as the experience of the software engineer.

3.4.2. Mutation testing

A mutant is a slightly modified version of a program (‘negatedecision’ or ‘replace operator’,...). The test suite is modifiedso that a test case “kills” the mutant.


3. Test Techniques 3.5. Usage-based

3.5.1. Operational profile

The test environment reproduces the operationalenvironment of the software.

Future reliability of the software can be estimatedfrom the results.

Test inputs are assigned a probability distributionaccording to their occurrence in actual operation.


3. Test Techniques 3.6. Nature of application

Specialized testing fields:

Object-oriented testing.Component-based testing.Web-based testing.GUI testing.Concurrent programs.Protocol conformance.Real-time systems.Safety-critical systems.


3. Test Techniques 3.7. Select and combine

3.7.1. Functional and structuralSpecification (functional) and code-based(structural) techniques should not be seen asalternative but as complementary.

3.7.2. Deterministic and random

Select and combine techniques.

Documents

MSc Software Maintenance - Háskólinn á Akureyristaff.unak.is/andy/MScTesting0708/Lectures/STLec1and2.pdf · MSc Software Testing ... – The user enters a search term in the GUI