Model-Based Testing UsingSpec Explorer

Aditya MathurPurdue University

CS 49000 Software TestingSpring 2011

Material extracted mostly from:“Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer”, Margus Veanes, Colin Campbell, Wolfgang Grieskamp, Wolfram Schulte, Nikolai Tillmann, and Lev Nachmanson, Published by: Springer Verlag, Lecture Notes in Computer Science, Volume 4949, Pages 39-76, 2007.

Spec Explorer 2

Objective

• The purpose of this presentation is to introduce modeling using the Spec Explorer tool from Microsoft.

• The example presented here is from a paper cited in the title slide.

• Familiarity with Chapter 3 of the textbook is assumed.

Spec Explorer 3

Model Based Conformance Testing

Requirements Implementation

Test harness(send inputs, receive outputs, and compare)

Model

Generate(manual)

Tests

Generate(automated)

Test outcome

Spec Explorer 4

Example: Chat Room

Chat Room

Client

Client

Client Client

Client

Spec Explorer 5

Chat Room: Operation

• Each client may post text messages.• Each message is delivered to all clients logged

into the chat room.• Pending messages from a client are delivered

in the order sent.• Messages from multiple senders are

interleaved arbitrarily.

Spec Explorer 6

Client status

// Client entered the chat room or notbool entered;

// Queue of messages sent by other clients but not received by this client

Map<Client,Seq<string>> unreceivedMsgs;

Spec Explorer 7

Client Actions: Creator

• Create a new instance of a client.• State changes so that

Empty message queues between the new client and the previously created clients.

Spec Explorer 8

Client Actions: Creator

// Create a client[Action] Client() { this.unreceivedMsgs = Map;foreach (Client c in enumof(Client), c != this){

c.unreceivedMsgs[this] = Seq{}; // Empty sequencethis.unreceivedMsgs[c] = Seq{};}

entered = false;}

enumof(T): set of instances of type T in the current state.

Denotes an action in the abstract state machine.

Spec Explorer 9

Client Actions: Enter

• Changes the state of a client to indicate that it has entered the chat room.

Spec Explorer 10

Client Actions: Enter

// Model client entry into the chat room[Action] void Enter()requires !entered; {entered = true;}

Method pre-condition

Spec Explorer 11

Client Actions: Send

• Appends a new message to the queue of unreceived messages in all clients in the chat room.

Spec Explorer 12

Client Actions: Send message

// Send a message[Action] void Send(string message)requires entered; {

foreach (Client c in enumof(Client), c != this, c.entered)

c.unreceivedMsgs[this] += Seq{message};}

Spec Explorer 13

Client Actions: Receive

• Extracts a message sent from a given sender from the sender’s queue in the client.

Spec Explorer 14

Client Actions: Receive

[Action(Kind=ActionAttributeKind.Observable)]void Receive(Client sender, string message)requires sender != this &&unreceivedMsgs[sender].Length > 0 &&unreceivedMsgs[sender].Head == message; { unreceivedMsgs[sender] = unreceivedMsgs[sender].Tail;}

Spec Explorer 15

Client Model Program

class Client { bool entered;Map<Client,Seq<string>> unreceivedMsgs;[Action] Client() [Action] void Enter()[Action] void Send(string message) [Action(Kind=ActionAttributeKind.Observable)]

}

Spec Explorer 16

Action types

Controllable: Input by the user client, send, enter

Observable: Output from the systemreceive

Spec Explorer 17

Client Model Scenario Start

Passive state:

Active state:

Timeout

Spec Explorer 18

Model Programs in Spec Explorer

Finite set of actions or update rules (Acts); e.g. client, send , enter, receive

Vocabulary S: function symbols

State: values, or interpretations, of state vocabulary symbols

Execution of an action method in a given state leads to the next state where some state variables may have changed.

State variables: V in S; (e.g., entered, unreceivedMsgs)

Each action is associated with a pre- and a post-condition.

Spec Explorer 19

FSM and Model Automaton

FSM=(X, Y, Q, q0, δ, O),

where X is a set of input symbols, Y a set of output symbols,q0 in Q, δ is transition function and O the output function

Model automaton=(Q, Q0, Qf, δ, A ),

where Q is a set of states, Q0 is a set of initial states, Qf is a set of final states, and A is a set of actions, A=Ctrl U Obs,

Ctrl is a set of control actions and Obs is a set of observable actions, Ctrl Obs = empty

U

Spec Explorer uses the notion of Model Automata:

Spec Explorer 20

Model program and model automaton

A model automaton is a complete unwinding of the model program.

Exploration: Unlike an FSM with a given sets of nodes and arcs, the

states and transitions of a model program must be deduced by executing

sequences of atomic actions starting in the initial state.

Model program Model automatonunwind

Spec Explorer 21

Exploration for the Chat Example

In state s0, the precondition is true and hence Client constructor is invoked.

The dynamic universe Client is updated by the addition of client c0. This is

denoted by enumof(Client).

The new state is denoted as s1.

The transition explored is δ(s0, Client/c0)=s1

Spec Explorer 22

Accepting state

Needed particularly in testing distributed and multithreaded programs. Why?

A state is considered an accepting state if the accepting condition is true in

that state.

A test is allowed to terminate in an accepting state.

An action the execution of which takes the implementation to a state where

no actions are enabled is known as a succeed action. It forces the system

into an accepting state.

Spec Explorer 23

Accepting condition example

enumof(Client).Size > 0 && // Exclude the initial state, and

Forall{ c in enumof(Client), s in c.unreceivedMsgs.Keys;

c.unreceivedMsgs[s].Length == 0

// states where pending messages have not been

received.

}

A state that satisfies the above condition has no observable actions enabled.

Spec Explorer 24

Scenario Control

A model program may correspond to a large, or infinite state, automaton.

Techniques:

Parameter selection, method restriction, state filtering, directed

search, state grouping

Techniques are available to control the size of a model automata for a

specific test purpose.

Spec Explorer 25

Scenario Control: Parameter selection

Select (s, m, v) for each state s, action m, and v sets of tuples.

In Chat example:

• send has an implicit parameter this and explicit parameter message.

• These can be restricted using the pair: Set {(c in enumof(Client)); <c, “hi”>}

Restrictions by triples may lead to reduction in the number of transitions

and hence a smaller automaton.

Spec Explorer 26

Scenario Control: Method restriction

An action m is enabled in state s if all pre-conditions associated with m are

satisfied.

In Chat example: restriction: Clients send messages only after all configured

clients are created and have entered the system

enum Mode { Creating, Entering, Sending };Mode CurrentMode {

get {if (enumof(Client).Size < 2) return Mode.Creating;if(Set{cin enumof(Client),!c.entered;c}.Size<2) return

Mode.Entering;return Mode.Sending}

}

Strengthening the pre-conditions can be used to limit the scenarios.

Spec Explorer 27

Scenario Control: Method restriction

In Chat example: restriction: Clients send messages only after all configured

clients are created and have entered the system

enum Mode { Creating, Entering, Sending };Mode CurrentMode {

get {if (enumof(Client).Size < 2)

return Mode.Creating;if(Set{c in enumof(Client),!c.entered;c}.Size<2)

return Mode.Entering;return Mode.Sending}

}Enabling of actions can now be restricted using expressions such ascurrentMode==Mode.Creating;

Spec Explorer 28

Scenario Control: State filteringA state filter is a set Sf, where Sinit is in Sf. [The subscript f stands for filter, and

not for final.]

In Chat example: Using state filter avoid states in which the same message is posted more than once before it is received.

Forall{c in enumof(Client), s in c.unreceivedMsgs.Keys, m1 in c.unreceivedMsgs[s], m2 in c.unreceivedMsgs[s]; m1 != m2}

A transition from state s to state t is included in the automaton if t is in Sf.

Sf is specified using a state based expression.

Spec Explorer 29

Test GenerationOffline:

Generate tests in advance from the model.

Online:

Generate tests on the fly as testing progresses.

Spec Explorer 30

Test SuiteTest suite T: Is another automaton generated from an automaton M.

• States ST in T may use new state variables (test variables).

• Test variables make it possible to record test history; e.g., which states have

been traversed so far.

• It contains two new methods called test actions: Observe and Timeout.

• Transitions corresponding to test actions are called test transitions.

• The Observe action encodes a decision to wait for an observable action.

• The Timeout action indicates that no other observable action happened. An accepting state is reachable from every state in ST .

Spec Explorer 31

Test Suite: ExampleConsider the following model program P:

enum Mode = {A,B,C} Mode mode = A; void F() requires mode == A {mode = B;} void G() requires mode == B {mode = C;} void H() requires mode == B {mode = C;}void I() requires mode == C {mode = A;} // Added to P to create P’.

Accepting state: Where mode is C.

Exploration: M generated from P and M’ from P’.

Spec Explorer 32

Test Suite: Generate Test automaton

Add a test variable n to indicate test number.

T: (F, G) and (F, H)

Spec Explorer 33

Automaton Traversal algorithms

Algorithm(s) used in Spec Explorer:

• T covers all states in M• T covers all transitions in M• Each action is associated with a weight and cost using a state-based

expression. Tests are generated to optimize the expected cost of a test.

Spec Explorer 34

Summary

Spec Explorer :

• Allows the creation of a model program P that captures the

expected behavior(s) of the implementation under test (IUT).

• Generates one or more model automaton (M) from P using

exploration subjected to scenario restrictions.

• Generates a test suite T from M either offline or online.