Pilot Model Based Testing

using Smartesting Test DesignerTM

3/12/08 - 5/12/08

Agenda

• Background

• Objective of the pilot

• Findings

• Conclusions & Recommendations

Background (1/2)

Expectations of model based testing...• Increased speed in test case specification• Excellent first step towards automated test

execution• Transferable and reusable testware

> opportunity in combination with India route?

• Improved involvement in requirements review> Shift up in the V-model

a possible new service offering to customers and/or in testlines

Background (2/2)

• Activities> Modelling by Smartesting> Turn around time: 3 days

• FEBO> Front end application for ....> Requirements used : release 2.1 (2.2

currently being built)> .NET application, back end simulated by

stubs> Vista based

Objective of the pilot (1/2)

In general1. To gain hands on experience with

Model Based Testing (MBT)

Specifically2. To assess the learning curve of the

UML models used> Class diagram> State diagram> Object diagram> OCL

Objective of the pilot (2/2)

3. To assess maintenance effort testware

4. To assess alignment with Test Strategy decisions

> How can test design techniques be utilized;> What coverage can be achieved;> How can coverage be influenced.

5. To compare effort and outcome with ‘normal’ test specification

> Time spent on preparation > Time spent on creating test cases> Number of test cases created

Findings (1/5)

1. and 2. Hands on experience;Learning curve> Due to reduced scope and turn around time:

no real result

3. Maintenance> Requirement change incorporated in model> Quick change

Findings (2/5)

4. Alignment with Test Strategy> Only possible to tailor (automated)

coverage with test depth in PCT> Test design techniques that can be used

are: PCT, UCT, DCoT, SEM, SYN, DTT, ECT

> To combine MBT with more test design techniques the maintenance is under pressure

Two separate activities

Findings (3/5)

5. Comparison to manual test specification

Time (hours)

Test level Use Case Manual MBT

Preparation C005 3 3

Preparation B001 3 1 (part)

Specification C005 8 5

Specification B001 24 3 (part)

Change B001 28 3

Findings (4/5)

5. Comparison to manual test specification

Test cases

Use Case Manual MBT

C005 45 21

B001 120 4 (part)

Change B001 - 18

Findings (5/5)

6. Extra: Infrastructure> Windows Vista not supported by

Smartesting Test Designer cannot be installed on Windows Vista

Sogeti> Need license for IBM Rational Software

Modeler> Export to HP Quality Center: OK> Export to IBM Raitional Quality Manager:

not yet available

Conclusions & Recommendations (1/5)

Overall:• Positive introduction• to small scale pilot: no statistical

metrics possible• Good and clear interaction with

analyst/designer> Same level of knowledge> Increases results tester> Increases acceptance designer

• Find defects in requirements

• Test Design Techniques> PCT: very suitable (adjustable test depth)> DCoT/SEM/SYN: auditable with adjustments

in test data> More analyses/experience is needed to

optimize the use of test design techniques with MBT

• Manual execution of model bases test cases is easy> Recommendation to use meaningful variables> focus not only on automated test execution

but also manuallyExtra service testers Sogeti?

Conclusions & Recommendations (2/5)

• Current infrastructure not suitable for Test Designer> Windows Vista Sogeti> Support Windows Vista> No export to Rational Quality Manager

• OCL looks a lot like programming> Not every tester will have the required skills

• Extra skill required/extra role• Suitable for outsourcing/offshoring

> Create models internally> Create and execute test cases externally

Conclusions & Recommendations (3/5)

• No discussion about who creates model> In test model there are items not used in

design modelBooleansTest data

> Customizing model takes longer> Creating model filters defects in design

Fully independent

Conclusions & Recommendations (4/5)

Conclusions & Recommendations (5/5)

• Prolong the pilot by course in MBT> Creating models and test cases> Teach this knowledge

• Use this knowledge in a real testline> Small project

• Add to services Software Control> Integrate in STaaS

TestManagement

tool

Nummer Testdoelen

1 Toon aan dat de huidige functionaliteit 1-op-1 is gemigreerd naar het nieuwe systeem

1.1 Toon aan dat de huidige functies uit “Handboek Balie” goed werken 1.2 Alle gangbare betaalvormen zijn mogelijk 1.3 De bron van kaartverkoop is traceerbaar

2 Lever inzicht in het gemak waarmee de gebruikers het nieuwe systeem kunnen gebruiken

2.1 Kaartjes kunnen makkelijk en snel worden geprint 2.2 Prijzen kunnen snel worden opgezocht 2.3 De front office applicatie van het call centre werkt 3 3.1 …

TestbasisTestgevallen

Models (UML)

TestRepository

Review

Test data

Test Cases

Review

Risks

Req’sKenmerk - deelobject

RK Opmerkingen

Functionaliteit - deelsysteem 1 A … - deelsysteem 2 C … - interface B … Gebruiksvriendelijkheid - online B …

TestAutomation