1

State-based testing

These slides are used with kind permission of Robert Hierons, Brunel

University.

Professor Hierons is an internationally leading expert in state-based

testing

2

State-based systems

• Many real systems have some internal state.

• These systems might be specified using e.g.Statecharts (now part of the UML) or SDL.

• Due to their criticality, there is particularinterest in state-based testing for:

– embedded control systems

– communications protocols

• Relevant to most object-oriented systems.

3

States and transitions

• A system may be modelled by:

– a set of logical states.

– transitions between these states.

• Then:

– each state will normally represent some set of

values for the state variables

– each transition will represent the use of some

operation in the state.

4

State diagram

• A state-based system can be represented by a state

diagram.

• Each state is represented by a node.

• The transitions are represented by arcs between

nodes:

– transition t with label op (operation op), goes from state

s to state s’ if the use of op in state s can lead to state s’.

– Transition t is represented by (s,s’,op).

5

State diagram: a simple example

• The following represents a light.

– There are two states: on and off

– There are two operations: turn_on and turn_off

On Off

turn_off turn_off

turn_on

turn_on

6

State diagrams and behaviour

• There is an initial state.

• If input is received, one of the transitions istriggered, possibly producing output andchanging the state.

• The state diagram tell us which state isreached when an operation is used.

• It thus tells us which sequences ofoperations are allowed.

7

Behaviour of the light

• It starts in state Off.

• If we use operation turn_off we don’tchange the state.

• If we use turn_on the state becomes On.

• We might now apply turn_on, failing tochange the state.

• Note: for these states to be useful there mustbe output from some operations.

8

Sequences allowed by light

• The model specifies the sequences of

actions allowed.

• These include:

– turn_on, turn_off, turn_on, turn_off

– turn_on, turn_on, turn_on, turn_off, turn_on,

turn_off

– And many more …

9

Dependencies in testing

• We have the following situation:

– in order to test a transition t we need to use

other transitions to:

• set up the initial state of t

• check the final state of t

– How do we know these are correct?

• We will produce a number of test sequences

from the state machine.

10

Finite State Machines

• A (deterministic) finite state machine is defined bytuple (S,s1,X,Y,!,") in which:

– S is a finite set of states and s1 is the initial state

– X is the finite input alphabet/set

– Y is the finite output alphabet/set

– function ! is the state transfer function

– function " is the output function

• We can extend ! and " to take sequences giving !#

and "#.

11

Behaviour of an FSM

• If we input a sequence x when M is in itsinitial state we get output sequence "#(s1,x)

and M moves to state !#(s1,x).

• If we input a sequence x when M is in states we get output sequence "#(s,x) and M

moves to state !#(s,x).

12

FSMs and directed graphs

• FSM M can be represented by a directed

graph (digraph) G=(V,E) in which:

– a state si is represented by a vertex vi

– if input x can move M from state si to state sj

with output y we add an edge (si,sj,x/y): an edge

from si to sj with label x/y.

• Then the paths (from v1) in G represent the

input/output sequences of M.

13

Example: traffic lights

• We will consider the following system:

– there are three colours for the lights: red, amber,

and green

– the control system receives a message ch

indicating when it should change the colour.

– it changes state and outputs a value to the lights

telling them what the colour should be.

14

State diagram: traffic lights

• State diagram for FSM MT.

• Note: we need two copies of ‘Amber’:

Green Red

Amber1

Amber2

ch/amber

ch/amber

ch/red

ch/green

15

The FSM

• The FSM MT is defined by:• State set {Red,Green,Amber1,Amber2}

• Initial state: Green

• Input alphabet {ch}

• Output alphabet {green, red, amber}

• State transfer function: !(Green,ch)=Amber1 ,

!(Amber1,ch)=Red , !(Red,ch)=Amber2 ,

!(Amber2,ch)=Green.

• Output function: "(Green,ch)=amber , "(Amber1,ch)=red ,

"(Red,ch)=amber , "(Amber2,ch)=green.

16

Actions in MT

• Suppose we input sequence ch,ch when MT is in

state Amber2.

• The first input of ch moves MT to state Green and

produces output green.

• The second input of ch move MT from state Green

to state Amber1 and leads to output amber.

– We have that "#(Amber2,chch) = green,amber

– and !#(Amber2,chch) = Amber1.

17

Deterministic state machines

• An FSM M is deterministic if:

– for each state s and input x such that there is a

transition from s with x, there is only one

transition from s with input x.

• This means: there is only one allowed state

and output after applying x in s.

• Note: if we want non-determinism we need

to change our representation.

18

Initially and Strongly connected

FSMs

• M is initially connected if:

– every state can be reached from the initial state– i.e. if for each state s there is some sequenceof edges from the initial state to s.

• M is strongly connected if:

– for every ordered pair of states (s,s’) there issome input sequence that takes M from s to s’ –i.e. if for each s, s’ there is some sequence ofedges from s to s’.

19

Distinguishing states of an FSM

• Two states s and s’ are distinguished byinput sequence x if:

– the response of M to x is different in states sand s’.

• If there is such an input sequence x then sand s’ are said to be distinguishable.

• States s and s’ are said to be equivalent ifthey are not distinguishable.

20

Distinguishing states: an example

s1

s3

s2

a/0

b/1

a/1

a/0

b/0

b/1

a distinguishes states

s2 and s3 since:

–The response

from receiving a

in s2 is 1 and the

response from

receiving a in s3

is 0.

In this case any two

states are

distinguishable by a

single input. This need

not always be the case.

21

Example 2

– States s2 and s3 are distinguished by aa but no

shorter sequence

s1

s3

s2

a/0

b/0

a/1

a/1

b/1

b/0

22

Equivalent states: an example

– States s2 and s3 are equivalent

s1

s3

s2

a/0

a/1

b/0

b/1

b/0

a/1

23

FSM equivalence

• Two FSMs M and M’ with the same input

alphabets are equivalent if, for each input

sequence they produce the same output

sequence.

24

Minimal FSMs

• An FSM is minimal if there is no equivalent

FSM with fewer states.

• If M is not minimal, it can be rewritten to

form an equivalent minimal FSM.

25

Reset operations

• A reset operation is one that always takes the FSMto the initial state.

• Sometimes we assume that there is a reliable resetoperation: there is some reset operation that weknow is correct.

• This helps in testing: we can use it to separate testsequences

• It may involve switching the machine off and thenon again.

26

Further assumptions

• It is also normal to assume that:

– M is minimal, strongly connected andcompletely specified.

• Often we assume that:

– there is some reset operation (called a reliablereset) that has been correctly implemented in I.This might simply involves switching thesystem off and then on again.

These simplify test generation.

27

Faults and FSMs

• There are two main classes of fault:

– Output faults - a transition has the wrong output

– State transfer faults - a transition goes to the

wrong state

• Note: state transfer faults may lead to MI

having more states that M.

28

Finding output faults

• To find output faults we just need to executetransitions.

• We might generate a single sequence (atransition tour) that covers each transition.

• We might produce a minimal sequence.

• Note: the presence of state transfer faultsmight mean the test no longer covers everytransition.

29

Transition Tour Method

• In the transition tour method we:

– Find some path/walk, from the initial state, that

covers every edge/transition.

– Our test is the input sequence defined by

following this sequence.

30

Transition tour example

• Consider:

• We could follow the path with edges:

– a/0, b/1, a/1, a/0, b/0, b/1

• This gives test sequence:

– abaabb

s1

s3

s2

a/0b/1

a/1

a/0

b/0

b/1

31

Generating a Transition Tour

• We can simply follow a path, at each step

extending it by:

– Choosing an edge we have yet to take

– Adding a path from where we are to the initial

state of this edge

– Adding the edge

• Note: there are also algorithms that produce

minimal length transition tours.

32

Observation

A transition tour need not find state transfer

faults

To find state transfer faults we need to be able

to check states.

33

The fundamental problem

• We want to check whether a correct

transition is followed

• To do this we need to do three things

– Get to the start state of the transition

– Execute the transition

– Check the end state is the right one.

34

35

36

37

38

Checking states

• We can devise sequences that distinguish

between states of M.

• We will learn about:-

– A distinguishing sequence

– A unique input/output sequence

– A characterising set

39

Distinguishing sequences

• An input sequence D is a distinguishing sequence if:

– for every pair of states s, s’ of M such that s$s’ we have

that "*(s,D) $ "*(s’,D).

• That is: the output produced in response to D

identifies the state.

• To check the final state of transition t it is sufficient

to follow t by D.

40

Using a distinguishing sequence:

example

• Here aa is a distinguishing sequence since

the output produced identifies the state

s1

s2

s3

a/0

a/0a/1

b/1b/1

b/0

10s3

01s2

00s1

Response

to aa

State

41

Testing a transition

• In order to test a transition we execute it and thenenter the distinguishing sequence.

• For example, to test (s3,s3,b/1) we might enter thefollowing test sequence:

– a/0, b/0, b/1, a/1, a/0.

– corresponding input sequence: abbaa.

• Here a, b takes us from state s1 to state s3 while a/1,a/0 is checking the transition takes us to thecorrect state.

• Note: this test is checking that the correspondingtransition in the implementation is correct.

42

Finding distinguishing sequences

• We can simply check different input

sequences, producing a column in a table

for each.

• Normally we start with short sequences and

extend these.

43

Unique input/output sequences

• A sequence x/y is a unique input/output

sequence (UIO) for state s if:

– y="*(s,x) and for every state s’ of M such that

s$s’ we have that "*(s,x) $ "*(s’,x).

• This means that input x identifies the state

since: if y is produced in response to x we

must have been in state s, otherwise we

must have been in a different state.

44

More on UIOs

• Thus, x is capable of verifying s in M but

not necessarily any other state of M.

• Thus: if the state is not s, the output tells us

this but need not tell us which state we were

in.

• To check transition t=(s,s’,x/y) we can

follow it by a UIO for state s’.

45

UIO Example

s1

s2

s3

a/0

a/0

a/1

b/1

b/1

b/0

a/1s3

b/0s2

b/1,a/1s1

UIOState

46

Why b/0 is a UIO for s2

• It is sufficient to consider the following table:

• The entry for s2 is different from the others: if we

input b and get output 0 we must have been in

state s2.

1s3

0s2

1s1

Response to bState

47

Using UIOs to test transitions

• In order to test the transition (s2,s2,b/0) wecan use the following test sequence:

– a/0, b/0, b/0

– Corresponding test input sequence: abb

• Note: here a/0 reaches the initial state of thetransition, the first b/0 is the transition, andthe second b/0 checks the state after thetransition.

48

Characterising sets

• A set W of input sequences is a

characterising set for M if:

– for every pair of states s, s’ of M such that s$s’

we have some w%W such that "*(s,w) $

"*(s’,w).

• This means that: for each pair of states there

is some input sequence from W that

distinguishes them.

49

More on characterising sets

• If we know the output triggered by each input

sequence from W we can identify the state.

• To check transition t=(si,sj,x/y) we can follow it

separately by each element of W.

• Thus, using a characterising set leads to multiple

tests for a transition.

• Note: there is always a characterising set (since we

assume our FSM is minimal).

50

Example

• Consider the following FSM

s1

s3

s2

a/0

a/0b/1

b/1a/1

b/0

51

The Characterising set

• The set {a,b} is a characterising set.

• To see this, observe that in the following

table each row is unique (the response to a

and b identifies a state)

11s3

00s2

10s1

Response

to b

Response

to a

State

52

Consequence

• If we are checking the final state of a

transition t we separately follow it by a and

b (we use two tests for the transition).

• If the response to a after t is 0 and the

response to b after t is 0 the transition must

have taken the implementation to a state

corresponding to s2.

53

Chow’s method

• This is based on using:

– a characterising set

– a reliable reset.

• We also have a state cover: a set V of

sequences such that each state of M is

reached by some (unique) sequence from V.

54

Revision - Characterising sets

• A set W of input sequences is a

characterising set for M if:

– for every pair of states s, s’ of M such that s$s’

we have some w%W such that "*(s,w) $

"*(s’,w).

• This means that: for each pair of states there

is some input sequence from W that

distinguishes them.

55

Generating a state cover

• We might just apply a breadth-first search in M.

• We start with the set containing the emptysequence & - this reaches s1.

• At each step we consider the set of states reached.

• If a state that has yet to be reached can be reached

from a current state by one input we add a

corresponding sequence.

56

State cover in the example

• Here we have:

– The empty sequence reaches s1.

– Sequence a reaches s2.

– Sequence b reaches s3.

• We have state cover V={&,a,b}.

s1

s3

s2

a/0

a/0b/1

b/1a/1

b/0

57

Chow’s method - ‘no extra states’

• If we assume that the implementation has no more

states than M we use two sets of sequences:

– The set VW: each element of the state cover V followed

by each element of the characterising set W

– The set VXW: {vxw| v%V, x is an input, and w % W}

• We use every sequence from these sets, separating

each sequence by a reset.

58

Applied to the example• Here we have W={a,b} and V={&,a,b}.

• We need VW ' VXW

• We thus get the following test sequences:{&,a,b}{a,b} '{&,a,b}{a,b} {a,b}

• Expanded out this is:

{a,b,aa,ab,ba,bb,aaa,aab,aba,abb,baa,

bab,bba,bbb}

• Note we can remove some tests: those that areprefixes of others.

• Nonetheless, for large FSMs, there are a lot of testcases!

59

The roles of the two sets

– The set VW essentially checks the state cover.

– The set VXW uses this to check the transitions:

the transition involving input x in state s is

checked by using a sequence from the state

cover (V) to get to s, inputting x (from X) and

following this separately by each element of W.

60

Summary

• Testing from an FSM may be seen as

testing the transitions of the

implementation.

• To do this we might use tests that

distinguish states

• Where we use a characterising set we might

apply Chow’s method.