Model&Based*Tes,ng:* An*Introduc,on*ceres.hh.se/mediawiki/images/5/58/Mbt_intro.pdf ·...

Model-­‐Based  Tes,ng:  An  Introduc,on  Mohammad  Mousavi    

Halmstad  University,  Sweden  

Halmstad  Summer  School  on  Tes,ng,    June  2013  

Based on joint work with: H.R. Asaadi, R. Khosravi, N. Noroozi and T.A.C. Willemse

Outline  

•  Overview  •  Mo,va,on:  Tes,ng  a  Funds  Transfer  Switch  •  Theory  •  Open  Problems  

2  

Program:  June  3  

3  

09:15-09:30 Welcome, opening remarks (Walid Taha, Halmstad) 09:30-10:30 Introduction to Model-Based Testing (Mohammad Mousavi, Halmstad) 10:30-11:00 Coffee Break 11:00-12:30 Industrial-Strength Model-Based Testing and its Methodology, Part I (Jan Peleska, Verified and University of Bremen) 12:30-14:00 Lunch Break 14:00-15:30 Industrial-Strength Model-Based Testing and its Methodology, Part II (Wen-Ling Huang, University of Bremen) 15:30-16:00 Coffee Break 16:00-17:00 Closing the V - by going from V to DEL (Tony Larsson, Halmstad)

Program:  June  4  

4  

09:00-10:30 Testing Concurrent Java Programs, Part I (Corky Cartwright, Rice) 10:30-11:00 Coffee Break 11:00-12:30 Testing Concurrent Java Programs, Part II (Corky Cartwright, Rice) 12:30-14:00 Lunch Break 14:00-15:30 Testing and Verifying Software Properties with ACL2 and ProofPad, Part I (Rex Page, Oklahoma) 15:30-16:00 Coffee Break 16:00-17:30 Testing and Verifying Software Properties with ACL2 and ProofPad, Part II (Rex Page, Oklahoma)

Program:  June  5  

5  

09:00-10:30 Accurate Programming Using ScalaCheck (Walid Taha, Halmstad) 10:30-11:00 Coffee Break 11:00-12:30 Hands-on ScalaCheck (Rickard Nilsson, Lund) 12:30-14:00 Lunch Break 14:00-15:30 Property-based testing with QuickCheck, Part I (John Hughes, QuviQ and Chalmers) 15:30-16:00 Coffee Break 16:00-17:30 Property-based testing with QuickCheck, Part II (John Hughes, QuviQ and Chalmers)

System  development:  A  planned  experiment  

Activity and Tools

Reasoning Techniques

Hypotheses (Models)

Unit Programming

Unit Integration

System Integration

Programming languages

Frameworks, Middleware

Protocols,

Deductive Reasoning Assertions

Contracts

Behavioral models

ConcJUnit,  Inductive Reasoning ScalaCheck,    

Qcheck    

ProofPad    

RT-­‐Tester  

Outline  

•  Overview  •  Mo,va,on:  Tes,ng  a  Funds  Transfer  Switch  •  Theory  •  Open  Problems  

7  

Tes,ng  a    Funds  Transfer  Switch  

8  

Complex  Architectures  

9  

Thread  Pool   Integra,on  Services  

Session  Management  

Object  Cache  

Logging  

Database  Domain  Logic  

Security  

Persistence  Frameworks  

Transac,on  Management  

10  

Database  

Thread  Pool   Integra,on  Services  

Session  Management  

Object  Cache  

Logging  

Domain  Logic  

Security  

Persistence  Frameworks  

Transac,on  Management  

Thread  Pool   Integra,on  Services  

Session  Management  

Object  Cache  

Logging  

Domain  Logic  

Security  

Persistence  Frameworks  

Transac,on  Management  

Thread  Pool   Integra,on  Services  

Session  Management  

Object  Cache  

Logging  

Domain  Logic  

Security  

Persistence  Frameworks  

Transac,on  Management  

11  

The  System  Under  Test  

•  An  Opera,onal  EFT  Switch    •  Java  Applica,on  (∼100  KLOC)  •  Extensive  use  of  Java  frameworks  

•  Already  tested,  but  not    with  a  disciplined  view  on  concurrency  

12  

Netw

ork Manger

Message M

anager

Flow M

anager

Service Layer

Com

ponent Layer

ProtocolToIFxHandler

Message Binder

Authorization

Message Processor

IFXToProtocolHandler

Exception Handler

Transaction Service

Terminal Service

Cell Charge Service

Settlement Service

DAO Service

Financial Entity Service

Reversal Handler

Settlement

Electronic  Funds  Transfer  (EFT)  

EFT  Switch  

Interbank  Network  

EFT  Switch  

Core  Banking  System  

13  

Example  Scenarios  

EFT  Switch   Core  Banking  System  

POS  Terminal  

purchase  request  

purchase  request  

purchase  response  

purchase  response  

14  

Example  Scenarios  

EFT  Switch   Core  Banking  System  

POS  Terminal  

purchase  request  

purchase  request  

purchase  response  purchase  resp.  

reversal  response  reversal  request  

15  

ISO  8583  Standard  

Message  Flows  

16  

ISO  8583  Standard  

Transac:on  Flows  

17  

ISO  8583  Standard  

Business  Rules  

18  

Concurrent  Transac,ons  

Purchase  Transac,on  #1  

POS  #1  Reversal  Transac,on  #2  

Purchase  Transac,on  #2  

POS  #2  

One  LTS  instance  per  transac:on  in  the  Switch  19  

Our  Method  

•  Using  Model-­‐Based  Tes,ng  

•  Using  a  set  of  integrated  tools  – Genera,ng  and  running  test  cases  – Logging  and  priori,zing  test  cases  – Measuring  test  coverage  – Tes,ng  business  rules  

20  

Model-­‐Based  Tes,ng  •  Modeling  the  desired  behavior  (system)  /  possible  interac,ons  (environment)    

•  Deriving  test  cases  from  the  model  rcv_pos_req?  

snd_core_req!  snd_pos_resp!  

rcv_core_resp?   ,me_out?  

21  

Model-­‐Based  Tes,ng  

•  Abstrac,ons  from  reality  •  Separa,ng  different  concerns  •  Approxima,ng  system  behavior    and  /  or  its  environment  

22  

– Restric,ng  environment  interac,ons  – Simpler  than  actual  system  – Easier  to  verify  

Equivalence  By  Observa,on  

rcv_pos_req?  

snd_core_req!  snd_pos_resp!  

rcv_core_req?   ,me_out?  

snd_core_req!  snd_pos_resp!  

rcv_core_req?   ,me_out?  

rcv_pos_req?  

23  

Outline  

•  Overview  •  Mo,va,on:  Tes,ng  a  Funds  Transfer  Switch  •  Theory:  What  is  Observa,on?  •  Open  Problems  

24  

a  

b  Next  

25  

a

a  

b  Next  

26  

b

a  

b  Next  

27  

a  

b  Next  

28  

Completed  Trace  Equivalence  

I  ≈ctr  S  

traces(I)  =  traces(S)  

c_traces(I)  =  c_traces(S)    

29  

a  

c  

a  

b  c  b  

a  

30  

a  

c  

a  

b  0  1  

a   b   c  Next  

31  

a

a  

c  

a  

b  0  1  

a   b   c  Next  

32  

a

a  

c  

a  

b  0  1  

a   b   c  Next  

33  

a  

c  

a  

b  0  1  

a   b   c  Next  

34  

a  

c  

a  

b  c  b  

a  

35  

a   a  

c  

a  

Environment   System  

b   b  

What we did using switches

36  

Tes,ng  Equivalence  

I  ≈te  S  

for  every  environment  E:  

traces(E ||  I)  =  traces(E ||  S)  

c_traces(E ||  I)  =  c_traces(E ||  S)     37  

a  

c  

a  

b  0  1  

a   b   c  idle  

Next  

38  

a

a  

c  

a  

b  0  1  

a   b   c  idle  

Next  

39  

a

a  

c  

a  

b  0  1  

a   b   c  idle  

Next  

40  

a  

c  

a  

b  0  1  

a   b   c  idle  

Next  

41  

a  

c  

a  

b  0  1  

a   b   c  idle  

Next  

42  

c

a  

c  

a  

b  0  1  

a   b   c  idle  

Next  

43  

a  

c  

a  

b  0  1  

a   b   c  idle  

Next  

44  

a  

c  

a  

b  0  1  

a   b   c  idle  

Next  

45  

a  

c  

a  

b  0  1  

a   b   c  idle  

Next  

46  

a  

c  

a  

b  0  1  

a   b   c  idle  

Next  

Now, we know that we have completed a trace.

47  

a  

a  

c  

a  

Env  

S1  

b  

b  

θ  

c   c  b  

a  

S2  occurs if no other transition can

48  

a  

Env  

S1  

b  

a  

c  

a  

b  

θ  

c   c  b  

a  

S2  

c_traces(Env  ⎤⎪S1)  =  {ab,    aθc}  

c_traces(Env  ⎤⎪S2)  =  {ab}  49  

Refusal  Equivalence  

I  ≈rf  S  

for  every  environment  E:  

traces(E  ⎤⎪  I)  =  traces(E  ⎤⎪  S)  

c_traces(E  ⎤⎪  I)  =  c_traces(E  ⎤⎪  S)     50  

And  there  is  more!  

51  

You  are  here!  

The  Linear  Time  –  Branching  Time  Spectrum  Rob  van  Glabbeek  

Defining  specifica,ons  at  a  higher-­‐level  

c  b  

a  

Specifica,on  

a 0  1  

a   b   c  idle  

Next  

a 0  1  

a   b   c  idle  

Next  

Impl.  1  

Impl.  2  

52  

Tes,ng  Pre-­‐order  

I        te  S  

for  every  environment  E:  

traces(E ||  I)   ⊆    traces(E ||  S)  

c_traces(E ||  I)  ⊆  c_traces(E ||  S)     53  

Restric,on  to  Specifica,on  

Feature  #1  Par,al  Specifica,on  

a 0  1  

a   b   c  idle  

Next   Full  Implementa,on  

c  b  

a   a  

b  

a  

c  

a  

b  

Feature  #2  Par,al  Specifica,on  

Feature  #3  Par,al  Specifica,on   54  

Restric,on  to  Specifica,on  

I    conf    S  

for  every  environment  E:  

traces(E ||  I)  ∩  traces(S)  ⊆    traces(E ||  S)  

c_traces(E ||  I)  ∩  traces(S)  ⊆  c_traces(E ||  S)    

55  

I/O  Transi,on  Systems  

Dis,nguishing  between  input  and  output  ac,ons  

!error  !response  

?request  

?response  

!request  

56  

Pre-­‐orders  on  I/O  transi,on  systems  

•  The  same  no,ons  apply  here  –  I/O  test  pre-­‐order  –  I/O  refusal  pre-­‐order  

ior  for  every  environment  E:  

traces(E  ⎤⎪  I)  ⊆  traces(E  ⎤⎪  S)  

c_traces(E  ⎤⎪  I)  ⊆  c_traces(E  ⎤⎪  S)    

I                  S  

57  

I/O  Conformance  

Informally,      I/O  Conformance  =  I/O  Refusal  restricted  to  specifica,on  traces  

Feature  #1  Par,al  Specifica,on  

a

0  

1  

a   b   c  

idle  

Next  

Full  Implementa,on  

c  b  

a  

a  

b  

a  

c  

a  

b  

Feature  #2  Par,al  Specifica,on  

Feature  #3  Par,al  Specifica,on  

I    ioco    S  

58  

Black-­‐box  tes,ng  for  ioco  

c  b  

a  

Specifica,on  

a 0  1  

a   b   c  idle  

Next  

Implementa,on  

a  

b  

Test  Case  

59  

Example  

?b  

!a  

?b  

?b  

?b  

!a  

?b  

τ  

!b  

fail   pass  pass  

θ  

θ   θ  

?a  

?a   ?a  

60  

ioco  Test  Cases  

•  I/O  transi,on  systems  •  The  only  terminal  states:                      and  •  Reversed  I/O  ac,ons  •  Special  ac,on  θ  •  Finite  and  determinis,c   !b  

fail   pass  pass  

θ  

θ   θ  

?a  

?a   ?a  

pass   fail  

61  

Automa,c  Test  Case  Genera,on  

•  Init:    – Generate  an  ini,al  state  

•  Recursion:  – At  each  point  in  the  recursion  choose  non-­‐determinis,cally  between:  1.  Stopping  the  recursion  2.  Supplying  an  input  3.  Observing  an  output  (one  transi,on  per  output  ac,on)  

62  

Outline  

•  Overview  •  Mo,va,on:  Tes,ng  a  Funds  Transfer  Switch  •  Theory  •  Open  Problems  

63  

Back  to  the  real  world!  

64  

Switch  Specifica,on  

•  Modeling  in  UPPAAL  •  ≈  25  automata  

Switch  behavior  in  purchase  scenario   65  

Purchase  Use  Case  

Specifica,on  Structure  

POS  Model   Switch  Model   Core  Model  P-­‐S    

Connector  S-­‐C    

Connector  

Four  uses  cases  :ll  now  

66  

Tes,ng  Ecosystem  

c  b  

a  

Switch  Spec.  

ioco  Test-­‐Case  Generator  

(UPPAAL  TRON)  a 0  

1  a   b   c  

idle  

Next  

Switch  Impl.  

Adaptor  

Test  Result:  pass  or  fail    

(+  counterexample)  

Coverage  Metrics  (Cobertura)  

67  

Measuring  Coverage  Using  Cobertura  code  coverage  tool                What  about  model-­‐based  coverage?     68  

Tes,ng  Ecosystem  

c  b  

a  

Switch  Spec.  

ioco  Test-­‐Case  Generator  

(UPPAAL  TRON)  a 0  

1  a   b   c  

idle  

Next  

Switch  Impl.  

Adaptor  

Test  Result:  pass  or  fail    

(+  counterexample)  

Coverage  Metrics  (Cobertura)  

Offline    Test  Suite  

69  

Logging  Offline  Test  Suite  

•  The  adaptor  logs  the  messages  from  TRON  •  Priori,za,on  based  on  coverage  •  Used  for  regression  tes,ng  

ioco  Test-­‐Case  Generator  

(UPPAAL  TRON)  

Adaptor  

Offline    Test  Suite  

70  

Tes,ng  Ecosystem  

c  b  

a  

Switch  Spec.  

ioco  Test-­‐Case  Generator  

(UPPAAL  TRON)  a 0  

1  a   b   c  

idle  

Next  

Switch  Impl.  

Adaptor  

Test  Result:  pass  or  fail    

(+  counterexample)  

Coverage  Metrics  (Cobertura)  

Offline    Test  Suite  

Test  DB  

71  

Tes,ng  Business  Rules  

72  

Adaptor  TRON   Switch  

purchase  req(amnt,  …)  

purchase  req(amnt,  …)  

purchase  resp(amnt,  …)  

remember  req.  amount  

purchase  resp(amnt,  …)  

validate  resp.  amount  

Results  

•  Have  found  a  number  of  bugs  in  the  opera,onal  system  – One  traced  back  to  null-­‐pointer  dereferencing  – Poor  excep,on  handling  

•  Code  coverage  of  about  40%  

73  

Observa,ons  

•  Modeling  language  and  tool  limita,ons  – Using  asynchronous  models?    

•  Scalability  Issues  – Running  mul,ple  TRON  instances  – Reducing  buffer  lengths,  pool  sizes,  etc.  

74  

Outline  

•  Overview  •  Mo,va,on:  Tes,ng  a  Funds  Transfer  Switch  •  Theory:  •  Open  Problems  

75  

Future  Work  

•  Characterizing  of  the  conformance  induced  by  internal  choice  test  cases  

•  Formula,ng  syntac,c  condi,ons  to  sa,sfy  seman,c  constraints  for  asynchronous  tes,ng    

•  Experimental  results:  – Size  of  the  quo,ent  models  /  test-­‐cases    – Applying  decomposi,onal  tes,ng  to  actual  code  

Neda  Noroozi   PAGE  76   8/5/13  

Future  Work  

•  Applying  decomposi,onal  tes,ng  to    product  lines  

•  Addressing  data  selec,on  issues  in  actual  implementa,ons  

Neda  Noroozi   PAGE  77   8/5/13  

Thank  You!  

Mohammad  Mousavi  m.r.mousavi@hh.se  

78