Some Testing Techniques(enhanced 6 + 4)

Course Software Testing & Verification2014/15

Wishnu Prasetya

Plan

• Unit testing tool• Mocking (Sec. 6.2.1) enhanced• Black box testing Input Partitioning (Ch 4).• Regression (6.1) enhanced

2

Unit Testing

• Making sure that the units are correct.• Invest in unit testing! Debugging an error at

the system-test level is much more costly than at the unit level.

• Note: so-called “unit testing tool” can often also be used to facilitate integration and system testing.

3

But what is a “unit” ?

• In principle, you decide. “Units” are just something you can compose to build something bigger. Possibilities: function/method , or class as unit.

• However, different types of units may have types of interactions and complexity, thus requiring different approaches– a function’s behavior depends only on its parameters; does

not do any side effect.– procedure depends-on and affects params– method: params, instance vars, static vars– class: is a collection of interacting methods

4

Unit testing in C#

• You need at least Visual Studio Professional; and for code coverage feedback you need at least Premium.

• Check these tutorials/docs (Visual Studio 2010):– Walkthrough: Creating and Running Unit Tests– Walkthrough: Run Tests and View Code Coverage– Walkthrough: Using the Command-line Test Utility– API Reference for Testing Tools for Visual Studio, in

particular Microsoft.VisualStudio.TestTools.UnitTesting, containg classes like

• Assert, CollectionAssert, ...

• In this lecture we will just go through the concepts5

The structure of a solution with “test projects”

6

class Thermometer private double valprivate double scaleprivate double offsetpublic Thermometer(double s, double o)public double value()public double warmUp(double v) public double coolDown(double v)

A test project is a just a project in your solution that contains your test-classes.

The structure of a “test project”

• A solution may contain multiple projects; it may thus contain multiple test projects.

• A test project is used to group related test classes.• You decide what “related” means; e.g. you may want

to put all test-cases for the class Thermometer in its own test project.

• A test class is used to group related test method.• A test method does the actual testing work. It may

encode a single test-case, or multiple test-cases. You decide.

7

Test Class and Test Method

8

[TestClass()]public class ThermometerTest { private TestContext testContextInstance; //[ClassInitialize()] //public static void MyClassInitialize(...) ... //[ClassCleanup()] //public static void MyClassCleanup() ... //[TestInitialize()] //public void MyTestInitialize() ... //[TestCleanup()] //public void MyTestCleanup() ...

[TestMethod()] public void valueTest1() ... [TestMethod()] public void valueTest2() ....}

public void valueTest1() { target = new Thermometer(1,0); double expected = - 273.15 ; double actual = target.value(); Assert.AreEqual(expected, actual);}

Be careful when comparing floating numbers, you may have to take imprecision into account, e.g. use this instead:

AreEqual(expected,actual,delta,”...”)

Positive and Negative Test

• Positive test: test the program on its normal parameters’ range.

• But can we afford to assume that the program is always called in its normal range? Else do Negative test: test that the program beyond its normal range.

• E.g. when unit testing a method, to test if it throws the right kind of exceptions.

9

Inspecting Test Result

10

Inspecting Coverage

11

Finding the source of an error: use a debugger!

12

• Add break points; execution is stopped at every BP.

• You can proceed to the next BP, or execute one step at a time: step-into, step-over, step-out.

• VisualStudio uses IntelliTrace logging you can even inspect previous BPs.

The Debug class

• Debug.Print(“therm. created”)• Debug.Assert(t.scale() > 0) to check for

properties that should hold in your program.• Will be removed if you build with debug off

(for release).• Check the doc of System.Diagnostics.Debug

13

Test Oraclesome patterns will return later

• Full: Assert. IsTrue( t == -273.15)• Partial: Assert.IsTrue( t -274) • Property-based : Assert.IsTrue(value_postcond(T))

14

public void valueTest1() { T = new Thermometer(0) Assert.IsEqual(T.value(), - 273.15)}

An oracle specifies your expectation on the program’s responses.

More costly to maintain, e.g. if you change the intended behavior of the program.

Discussion: propose test casesin particular the oracles...

15

reverse(a) { N = a.length if (N 1) return for (int i=0; i< N/2 ; i++) swap(a,i, N-1-i)}

incomeTax(i) { if (i18218) return 0.023 * i t = 419 if (i32738) return t + 0.138 * (i – 18218) t += 1568 if (i54367) return t + 0.42 * (i – 32738)}

Property-based testing fits nicely for reverse, but not for incomeTax; for the latter we’ll have to fall back to conrete-value oracles, which unfortunately tend to be more costly to maintain.

Discussion: the Oracle Problem (6.5)

• Every test-case needs an oracle; how to construct it!? always a big problem!

• Using concrete values as oracles is often powerful, but potentially expensive to maintain.

• Using “properties” on the other hand has the problem that it can be hard to write a complete yet simple property capturing correctness.

• A pragmatic middle way could be redundancy-based testing.

16

(Unit) Testing a class

• Many classes have methods that interact with each other (e.g. as in Stack). How to test these interactions?

• How to specify (and test) these interactions? Options:– class invariant– Abstract Data Type (ADT)– Finite State Machine (FSM)

17

Specifying with class invariant

• Regardless the interaction with other methods, each method of a class C has to keep the state of its target object consistent.

• Express this with a class invariant, e.g. – this.T >= -273.15– this.saldo >= 0– forall x in persons, typeOf(x) is Employee

• Class invariant cannot express a constraint over the interaction itself.

18

Formulate and test the class invariant

19

Stack<T>private Object[] contentprivate int toppublic Stack()push(T x)T pop()bool classinv() { return 0top && top<content.length } }

Example test-cases, check class-inv after :1. call to the constructor2. constructor ; push(x)3. constructor ; push(x) ; push()4. constructor ; push(x) ; pop()5. some random sequence of

push and pop

Specifying a class as an ADT

• An Abstract Data Type (ADT) is a model of a (stateful) data structure. The data structure is modeled abstractly by only describing a set of operations (without exposing the actual state).

• The semantic is described in terms of “logical properties” (also called the ADT’s axioms) over those operations.

20

Example : stack

21

Stack<T>bool isEmpty()push(T x)T pop()

Stack axioms :

• For all x,s : s.push(x) ; y = s.pop() ; assert (y==x )

• For all x and s : s.push(x) ; assert ( s.isEmpty())

• For all x and s.isEmpty() : s.push(x) ; s.pop() assert (s.isEmpty())

Depending of the offered operations, it may be hard/not

possible to get a complete axiomatization.

Test each ADT’s axiom

22

For all x,s : s.push(x) ; y = s.pop ; assert (y==x )

For example, three test cases :

1. empty s2. non-empty s3. s already contains x

Specifying a class with a finite state machine (FSM) (2.5.1, 2.5.2)

23

open()

close()

write()

File:

• Specifies which sequences of operations are valid• Can be combined with class-inv: test that after every valid

sequence the class-inv hold. • You can re-apply graph-based coverage concepts we had before

FSM can also come from your UML models.

In a larger project....

• You want to test your class Heater ; it uses Thermometer which is not ready yet!

• We can opt to use a mock Thermometer. A mock of a program P:– has the same interface as P– only implement a very small subset of P’s behavior– fully under your control

• Analogously we have the concept of mock object. • Make mocks yourself e.g. exploiting inheritance, or use

a mocking tool.

24

Mocking with Moq

25

test1() { Heater heater = new Heater() var mock = new Mock<IThermometer>() mock.Setup(t => t.value()).Returns(-275.15) heater.thermometer = mock.object heater.limit = 303.0 heater.check() Assert.IsFalse(heater.active)}

interface IThermometer double value()double warmUp(double v)

class Heaterdouble limitbool activepublic check() { if (thermometer.value() >= limit) active = false}

thermometer

Mocking with Moq(google it for more info!)

26

var mock = new Mock<IThermometer>() mock.Setup(t => t.value()).Returns(303.00001)

mock.Setup(t => t.warmUp(0)).Returns(0)

mock.Setup(t => t.warmUp(It.IsInRange <double>(-10, 10, Range.Inclusive)) .Returns(0)

mock.Setup(t => t.warmUp (It.IsAny<double>())) .Returns((double s) => s + 273.15)

Many more mock-functionalities in Moq.But well, mocking can be tedious. E.g. what to do when your Heater wants to call warmUp in an iteration?

Beyond unit testing, what if we want to abstract away from the source code ?

• Or you simply don’t have access to it• (Def 1.26) White box testing : common at the

unit-testing level• (Def 1.25) Black box testing: common at the

system-testing level. Approaches :– Partition-based testing– Model-based testing

27

Partitioning the inputs

• Based on “your best understanding” of save’s semantic.• Terminology: characteristic, block. The domain of a

characteristic is divided into disjoint blocks; the union of these blocks must cover the entire domain of the characteristic.

• Assumption : values of the same block are “equivalent”28

save(String fname, Object o)

fname : (A) existing file (B) non-existing file

o : (P) null (Q) non-null serializable (R) non-null non-serializable

So, what input values to choose?

• (C4.23, ALL) All combinations must be tested. |T| = (i: 0i<k: Bi) ; does not scale up.

• (C4.24, EACH CHOICE) Each block must be tested. |T| = (max i: 0i<k: Bi) ; usually too weak.

29

save(String fname, Object o)

fname : (A) existing file (B) non-existing file

o : (P) null (Q) non-null serializable (R) non-null non-serializable

t-wise coverage

• (C4.25, pair-wise coverage). Each pair of blocks (from different characteristics) must be tested.

• (C4.26, t-wise coverage). Generalization of pair-wise.• Obviously stronger than EACH CHOICE, and still

scalable. • Problem: we just blindly combine; no semantical

awareness. 30

AB

PQ

XY

T : (A,P,X) , (A,Q,Y) , ... more?

Adding a bit of semantic

• (C4.27, Base Choice Coverage, BCC) Decide a single base test t0. Make more tests by each time removing one block from t0, and forming combinations with all remaining blocks (of the same characteristics).

|T| = 1 + (i : 0i<k : Bi - 1) 31

AB

PQ

XYZ

Example: t0 = (A,P,X), generates these additional test requirements : (B,P,X) (A,Q,X) (A,P,Y) (A,P,Z)

Or, more bits of semantics

• (C4.28, Multiple Base Choices). For each characteristic we decide at least one base block. Then decide a set of base tests; each only include base blocks. For each base test, generate more tests by each time removing one base block, and forming combinations with remaining non-base blocks.|T| at most M + (i : 0i<k : M*(Bi - mi)) 32

AB

PQ

XYZ

Example, base tests = (A,P,X), (A,P,Y)(A,P,X) generates (B,P,X) (A,Q,X) (A,P,Z)

(A,P,Y) generates (B,P,Y) (A,Q,Y) (A,P,Z)

Example-2, MBCC

• (C4.28, Multiple Base Choices). For each characteristic we decide at least one base block. Then decide a set of base tests; each only include base blocks. For each base test, generate more tests by each time removing one base block, and forming combinations with remaining non-base blocks. 33

ABC

PQR

XYZ

Bold : base blocksChosen base tests = (A,P,X), (A,Q,Y)These produce these additional test requirements:

(B,P,X)(C,P,X)

(B,Q,Y)(C,Q,Y)

(A,R,X)

(A,R,Y)

(A,P,Z)

(A,Q,Z)• base-blocks are not cross-combined

except as in the base tests.• non-base blocks are not cross-

combined with each other.

Constraints, to exclude non-sensical cases

• Example:– combo (A,P,Y) is not allowed.– if P is selected, then X must also be selected.

• Solvable: pair-wise coverage + (A,P,Y) is not allowed.• Can be unsolvable, e.g. pair-wise coverage + (A,P) is

not allowed.• General problem: given a coverage criterion C and a

set of constraints, find a test set T satisfying both. • In general the problem is not trivial to solve.

34

Overview of partition-based coverage

35

EACH CHOICE

ALL

t-Wise Multiple Base Choice Coverage

Pair-Wise Base Choice Coverage

Model-based testing

• We already seen that use an FSM as a model of the system you test

• Such an FSM specifies which sequences of interactions are valid, you can use this information to guide your testing.

36

open()

close()

write()

File:

Model-based testing

• We can add further details, e.g. system-invariants, or by decorating the states in the FSM with predicates that are supposed to hold there; provided the system allows such predicates to be inspected (in the black-box setting the system under test will have limited observability)

37

open()

close()

write()

File:

locked() locked()

Regression Test

• To test that a new modification in your program does not break old functionalities. To be efficient, people typically reuse existing test sets.

• Usually applied for system-testing, where the problem is considered as more urgent. Challenge: very time consuming (hours/days!).

38

Some concepts first...• Test Selection Problem: suppose P has been modified

to P’. Let T be the test set used on P. Choose a subset T’T to test P’.

• Obviously, exclude obsolete test cases: those that can’t execute or whose oracles no longer reflect P’ semantic. Let’s assume: we can identify them.

• You want the selection to be safe : T’ includes all test-cases in T that will execute differently on P’.

• Only attractive if the cost of calculating T’ + executing T’ is less than simply re-executing the whole T.

39

Idea: select those that pass through modified code

• If m is the only method in P that changes, the obvious strategy is to select only test-cases that pass through m.

• Better: only select test-cases that pass m’s “modified” branch.

40

m(x) { if (d) y = x+1 else y=0 }

m(x) { if (d) y = x-1 else y=0 }

(orginal)

(modified)

Corner case

• Both branches of the first if are modified. Using the previous strategy means that we have to select all test-cases that pass m. Yet we see that the paths [d, e, stmt] and [ d, e] present in both old and new m; so there is actually no need to select them.

41

m(x) { if (d) y = x+1 ; if (e) stmt }

m(x) { if (d) { y = x+1 ; if (e) stmt ; u = 0 } else if (e) stmt }

Looking at it abstractly with CFG

42

m(x) { if (d) y = x+1 ; if (e) stmt }

m(x) { if (d) { y = x+1 ; if (e) stmt ; u = 0 } else if (e) stmt }

dy=x+1

e stmt

u=0

end

estmt

dy=x+1

e stmt

end

Notice that [d, e, stmt, end] and [d, e, end] appear in both, and equivalent.

Some concepts• We assume: P is deterministic. each test-case

always generate the same test path.• Let p and p’ be the test-paths of a test-case t when

executed on P and P’; t is modification traversing if not(p p’). let’s select modification traversing test-cases.

• p p’ if they have the same length, and for each i, pi p’i the latter means they contain the same sequence of instructions.

• So far this is not helpful, because such a selection strategy requires us to first execute t on P’. Then it is not attractive anymore!

43

“Intersection” Graph

• First, extend CFG so that branches are labelled by the corresponding decision value (e.g. T/F for if-branch). Label non-branch edge with some constant value.

• Each node of G’’ is a pair (u,u’). Then G’’ is defined like this :– The pair of initial nodes (s0,s0’) G’’. – If (u,u’)G’’, and uu’, and uv is an edge in G, and u’v’ and edge in G’

both with the same label, then (u,u’) (v,v’) should be an edge in G’’.44

a

c

end

b

G : a

c

end

b

G’ :

c

d

G’’ = G G’: a,a

c,c

end,end

b,bc,c

end,d

“Intersection” Graph

• Each path p in G’’ describes how a path in G would be executed on G’ if the same decisions are taken along the way. Note that this is calculated without re-executing any test-case on P’.

• Any path in G’’ ends either in a proper exit node (green), or in a pair (u,u’) where not uv (red). This would be the first time a test-case would hit a modified code when re-executed on P’.

• The old test-cases are assumed to have been instrumented, so that we know which nodes/edges in G it traversed. 45

a

c

end

b

G : a

c

end

b

G’ :

c

d

G’’ = G G’: a,a

c,c

end,end

b,bc,c

end,d

Selection Algorithm

• Select test-cases that pass [a,b,c,end] in G a bit expensive• (Safe but not minimalistic) Select test-cases that pass a node u in G that is

part of a red-node in G’’. same problem as before, it will select also select [a,c,end] which is not modification traversal.

• (Rothermel-Harold, 1997) Select test-cases that pass an edge e in G that in G’’ leads to a red-node in G’’. actually the same problem.

46

a

c

end

b

G : a

c

end

b

G’ :

c

d

G’’ = G G’: a,a

c,c

end,end

b,bc,c

end,d

Selection Algorithm

• (Ball algorithm,1998) Partition G’’ nodes to those can can reach a green-node (G partition), and those that cannot (NG partition). Look at edges in G’’ that cross these partitions (so, from G to NG).

A test path p is modification traversing if and only if it passes through a crossing edge (as meant above). use this as the selection criterion.

47

a

c

end

b

G : a

c

end

b

G’ :

c

d

G’’ = G G’: a,a

c,cend,end

b,b c,c

end,d

NG-partition

G-partition