Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

Test Generation based on Abstraction and Test

purposes to complement Structural tests

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson

A-MOST 2010 - April 6, 2010 - Paris

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 1/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

Context

Model-Based Testing

MBT from static selection criterion

MBT from dynamic selection criterion

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 2/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

Context

Model-Based Testing

MBT from static selection criterion

MBT from dynamic selection criterion

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 2/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

Problem and Motivations

Motivations

Complete the tests from static criteria with tests fromdynamic criteria

Evaluate the coverage of the generated tests

Problem

Combinatorial explosion of the size of an LTS

Solution

Abstraction to reduce the size of the state space

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 3/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

Outline

1 Problem and Motivations

2 Process overview

3 Process applicationThe robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

4 Experimental resultsAbstraction computationTest generation

5 Conclusion and Future works

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 4/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

Our process

Differences

Introduction of an abstraction computation

Replace the model by an abstraction

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 5/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

A conveying system : Robot

Specifications

A gripper moves parts in two dimensions

3 sort of parts : T1, T2, T3

Operations : Arrival, Load, Unload, Up, Down, Left, Right,LeftEvac, RightEvac

Unloading rules : T1 left, T2 right, T3 regardless

T3 parts : Left unloading priority

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 6/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

A test purpose for Robot

What is a test purpose ?

Description of a test intention

Combine operation calls and states to reach

Formalization : state machine or regular expression

Left unloading priority

true G = T3 ∧ LE 6= free true true G = T3 trueLeftEvac $op Unload

$op $op

$op Right

q0 q1 q2 q3 q4 qf

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 7/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

Definition of an abstraction

What is an abstraction ?

Gives a thumbnail of the model

Simulates executions of the model

Abstraction computation with GeneSyst (Bert,Potet,Stouls)

Disjunction of state predicates defined by user

One predicate ⇐⇒ one symbolic state of the abstraction

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 8/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

The GeneSyst tool

Principles

; How does GeneSyst compute an abstraction ?

Predicate abstractionCompuation of weakest precondition and resolution ofsatisfiability problems with prover technologyProof failure for a transition ⇒ Transition is kept in theabstractionFilter transitions with CLP

; How to define state predicates ?

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 9/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

Defining a set of abstraction predicates

Predicates which reveal changes in test purposeWhich ones do we propose ?

Present used to define target states in test purposeInvolving on variables modified by all operations of test purposeDefining a partition of the state space

Left unloading priority

($op)+; (LE ∈ {T1,T3} ∧ G = T3)

. Right

. LeftEvac

. ($op)+; (G = T3)

. Unload

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 10/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

Defining a set of abstraction predicates

Predicates which reveal changes in test purposeWhich ones do we propose ?

Present used to define target states in test purposeInvolving on variables modified by all operations of test purposeDefining a partition of the state space

LE ∈ {T1,T3}G ∈ T3

Left unloading priority

($op)+; (LE ∈ {T1,T3} ∧ G = T3)

. Right

. LeftEvac

. ($op)+; (G = T3)

. Unload

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 10/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

Defining a set of abstraction predicates

Predicates which reveal changes in test purposeWhich ones do we propose ?

Present used to define target states in test purposeInvolving on variables modified by all operations of test purposeDefining a partition of the state space

LE = free

G = free

Left unloading priority

($op)+; (LE ∈ {T1,T3} ∧ G = T3)

. Right

. LeftEvac

. ($op)+; (G = T3)

. Unload

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 10/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

Defining a set of abstraction predicates

Predicates which reveal changes in test purposeWhich ones do we propose ?

Present used to define target states in test purposeInvolving on variables modified by all operations of test purposeDefining a partition of the state space

LE {free},{T2},{T1,T3}G {free},{T1,T2},{T3}

Left unloading priority

($op)+; (LE ∈ {T1,T3} ∧ G = T3)

. Right

. LeftEvac

. ($op)+; (G = T3)

. Unload

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 10/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

Abstraction computation

LE {free},{T2},{T1,T3}G {free},{T1,T2},{T3}

QInit

(Deg = l & Dt = l)

Init

ApEdD

Rg

(Deg = l & Dt = T3)

C

(Deg = l & Dt : {T1, T2})

C

ApEdMRg

(Deg : {T1, T3} & Dt = l)

Dc_g

Dc_d

ApEdMRd

Dc_g

Eg

ApEdD

Rg

(Deg : {T1, T3} & Dt = T3)

C

(Deg : {T1, T3} & Dt : {T1, T2})

C

Eg

Dc_d

ApEdMRd

Eg

Dc_d

ApEdMRd

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 11/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

Synchronization between the abstraction and the TP

Abstraction

QInit

(Deg = l & Dt = l)

Init

ApEdD

Rg

(Deg = l & Dt = T3)

C

(Deg = l & Dt : {T1, T2})

C

ApEdMRg

(Deg : {T1, T3} & Dt = l)

Dc_g

Dc_d

ApEdMRd

Dc_g

Eg

ApEdD

Rg

(Deg : {T1, T3} & Dt = T3)

C

(Deg : {T1, T3} & Dt : {T1, T2})

C

Eg

Dc_d

ApEdMRd

Eg

Dc_d

ApEdMRd

Test purpose

true

G = T3 ∧ LE 6= free

true

true

G = T3

true

LeftEvac

$op

Unload

$op

Right

$op

$op

q0

q1

q2

q3

q4

qf

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 12/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

Synchronization between the abstraction and the TP

Synchronization

Instantiation of ($op)+

($op)+ ; (LE ∈ {T1,T3} ∧ G = T3)

. Right

. . .

(Deg = l & Dt = l), q0

ApEdDRg

(Deg = l & Dt = T3), q0

C

(Deg = l & Dt : {T1, T2}), q0

C

ApEdMRg

(Deg : {T1, T3} & Dt = l), q0

Dc_g

Dc_d

ApEdMRd

Dc_g

Eg

ApEdDRg

(Deg : {T1, T3} & Dt = T3), q0

C

(Deg : {T1, T3} & Dt = T3), q1

C(Deg : {T1, T3} & Dt : {T1, T2}), q0

C

Eg

Dc_d

ApEdMRd

ApEdMRd

(Deg : {T1, T3} & Dt = T3), q2

Rd

Eg

Dc_d

ApEdMRd

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 13/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

Generation and Instantiation of tests

Generation of a set of symbolic tests

All-edges criterion on synchronized product

Application of the Chinese Postman Algorithm

Instantiation of a set of symbolic tests

Why isn’t instantiation complete ?

; Tests use over-approximated transitions in abstraction

How do we deal with that ?

; Interleaving of reflexive operations calls in tests

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 14/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

Instantiation of symbolic tests

G = free G = T3. . . . . .

Arrival, RightEvac,Down, Left . . .

Load

Symbolic test

. . . ; (G = free)

. Load ; (G = T3)

Instantiated test

. . . ; (G = free)

. Arrival ; (G = free)

. Load ; (G = T3). . . . . .

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 15/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

Instantiation of symbolic tests

G = free G = T3. . . . . .

Arrival, RightEvac,Down, Left . . .

Load

Symbolic test

. . . ; (G = free)

. Load ; (G = T3)

Instantiated test

. . . ; (G = free)

. Arrival ; (G = free)

. Load ; (G = T3). . . . . .

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 15/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests

Instantiation of symbolic tests

G = free G = T3. . . . . .

Arrival, RightEvac,Down, Left . . .

Load

. . . . . .

Symbolic test

. . . ; (G = free)

. Load ; (G = T3)

Instantiated test

. . . ; (G = free)

. Arrival ; (G = free)

. Load ; (G = T3). . . . . .

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 15/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

Abstraction computationTest generation

Experimental results

Abstraction computation - Synchronization

, Reduction of the size of the models

,/ 1 abstraction per 1 test purpose

/ Abstraction computation : time consuming

B Model ♯StatesAbstraction computation Synchronization

♯States ♯Trans. Time ♯States ♯Trans.

Qui-Donc 135 24 2 min. 9 365 18 1.5 min. 12 54

Robot 3726 37 5 min. 17 1068 50 8 min. 23 214

DeMoney 1030 3 148 35 min. 7 3327 228 1.5 h. 18 422

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 16/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

Abstraction computationTest generation

Experimental results

Test generation - Instantiation

, Number of instantiated tests increase with filtering

/ Instantiation and filtering are time consuming with largeexample

B ModelWithout filtering With filtering

♯Instant. tests / Time of ♯Filter. Filter ♯Instant. tests / Time of♯Symbolic tests instantiation trans. time ♯Symbolic tests instantiation

Qui-Donc4/10 (40%)

≤ 1 s.2

≤ 1 s.

6/10 (60%)≤ 1 s.

5/15 (33%) 2 6/11 (55%)

Robot7/12 (58%) 3 min. 1 7/11 (64%) 2 min.8/23 (35%) 5 min. 0 8/23 (35%) 5 min.

DeMoney0/32 (0%)

2 h.49

2 h.0/19 (0%) 2 h.

0/32 (0%) 56 17/18 (94%) 1 h.

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 17/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

Conclusion and Future works

Conclusion

, Automatic extraction of abstraction predicates

, Abstraction related to test purpose

,/ 1 test purpose per 1 abstraction

/ Computation time of abstraction

/ Too much over-approximation for efficient test instantiation

Future works

Common abstraction (1 abstraction for N tests purposes)

Under-approximations rather than over-approximations

Tri-modal Transitions System generation (Godefroid, Ball)

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 18/19

Problem and MotivationsProcess overview

Process applicationExperimental results

Conclusion and Future works

Thanks for your attention

Questions ?

F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 19/19