Testing Transition Systems with Input and Output Testers

Alexandre Petrenko Nina Yevtushenko

Jia Le Huo

TestCom’03, May 27th, 2003

2

Outline

• Motivation

• Basic idea

• Testing frameworks– Queued-quiescence– Queued-suspension

• Conclusions

3

Conformance Testing

Spec Imp

Tests Test execution

Does Imp conform to Spec?

Fault model

4

Input/Output Transition Systems

• Non-deterministic IOTS: model of Spec and Imp• Output of IOTS cannot be blocked: basic assumption• Spec and Imp are input-enabled

coin

coffee,milk

?coin

!coffee !milk?coin

?coin

?coin

?coinSpec

5

?coin

Motivating Example?coin

!coffee

!milk?coin

?coin

?coin?coin

!coffee !milk?coin

?coin

?coin!coffee

Spec Imp

Quiescent trace coin coin coffee milk yields the ioco test case:

fail

coin coin coffee

milk, coffee

pass

milk,

fail

?coin

Is the test case still sound if output of Spec cannot be blocked?

6

Contradiction• The ioco test case blocks coffee after coin• Output of IOTS cannot be blocked: basic assumption

How can I block a coffee on its way out?

7

Problem Statement• A tester not blocking output of Imp must be input-

enabled• An input-enabled tester usually

– has unbounded test runs– makes choice between input and output

• Both features are not desirable in practice• We have to define a tester that

– is input-enabled– reaches verdict in finite steps– never chooses between input and output

8

Basic Idea

Tester

Imp

Output queue

finite capacity for practical reasons

Input testprocess

executes a given input sequence

Output testprocess

produces pass when an expected output sequence followed by quiescence is read from the queue and fail otherwise

Input-enabled, and no choice between input and output

Input queue

9

Queued-Quiescent Traces

• For a quiescent trace of Spec, I O Qqtraces(Spec)• For input sequence , Qqtraceso(Spec, ) includes all the output sequences in response to : {' O* | ' Qqtraces(Spec)}• Qqtraces(Spec) = Qqtraces(Imp) (qq-trace equivalent)

?coin

!coffee !milk?coin

?coin

?coin

Spec ?coin ?coin

?coin

!coffee

!milk?coin

?coin

?coin

!coffee

Imp

!coffee

!milk

Qqtraceso(Spec, )

!coffee

!milk

Qqtraceso(Imp, )

10

Queued-Quiescence Relations

• For non-oscillating Spec and Imp, if input sequence exists such that :

– Qqtraceso(Imp, ) Qqtraceso(Spec, ) = (no pass)

then Imp is qq-separable from Spec

– Qqtraceso(Imp, ) Qqtraceso(Spec, ) (fail, pass)

then Imp is qq-distinguishable from Spec

– Qqtraceso(Imp, ) Qqtraceso(Spec, ) (no fail)

then Imp is qq-weakly-distinguishable from Spec

• qq-separable qq-distinguishable qq-weakly-distinguishable

11

Test Derivation with Fault Model

• Given Spec and Imp, check input sequences of length 0, 1, 2, … until a sequence is found such that– qq-separates, or– qq-distinguishes, or

– qq-weakly-distinguishes Imp from Spec, or

– the input buffer capacity k is reached

• Repeat for all Imp • k = ?

12

A Finer Tester

After executing the qq-test case with input coin on Spec and Imp

!coin

?coin

?coin

!coffee

!milk?coin

?coin

?coin?coin

!coffee !milk?coin

?coin

?coin!coffee

Spec Imp

fail

?milk pass

fail

?milk, ?coffee

?milk, ?coffee, ?milk, ?coffee,

The resulting IOTS can be distinguished by a qq-test case:

?coin

13

Queued-Suspension Testing

1

2

11

qq-test case of Spec-after-(1, 11)

qs traces are sequences of qq traces executed by the qq testers: (11i)(22j)...(nnk)

2

12

qq-test case of Spec-after-(1, 12)

... ...

3

21 22 23 2425

33 3

3

qq-test case of Spec

14

Conclusions

• Outputs cannot be blocked: basic assumption• Testing frameworks: qq and qs• Comparison with the ioco framework:

– information of Interleaving of inputs and ouputs are lost

– so the conformance relations are coarser than ioco

– but ioco testing is not applicable here

• Open questions: oscillating Spec, k = ?, etc

15

Merci beaucoup!

16

An Example?coin

!coffee !milk?coin

?coin

?coin

?coinSpec

coin coin

Output queue(capacity 2)

Input queue(capacity 2)

?coin

!coffee

!milk?coin

!milk

?coin

!coffee

17

An example (Cntd.)

We havecoin coin coffee

milk

pass

fail

When the ioco test case below is applied to Spec with queues

fail

coin coin coffee

milk, coffee

pass

milk,

fail

Is this test case still sound?