Agile Testing Days 2012 - Lean Mean BDD Automation Machine

SQS Group Limited

Mike Scott & Tom Roden

@MikeAScott @TomMRoden

Agile Testing Days

20 November 2012

Potsdam, Germany

Building a lean, mean, BDD

automation machine

Caption Competition

2

Caption Competition

A story about building an executable documentation system

• Large hedge fund COTS implementation

• Crippling regression test burden

Born out of necessity

• 6-8 month cycle daily release

• 80 times defect reduction Results:

• and again

• Same patterns automation platform

Then did it again…

Safia– Archetypes and Templates, a voyage in automating acceptance

3

The problem quantified

Real data from the field

40

67.5

100

78

23 15

12

5 3 0

28 64

151

217

311 371

393

479

545

601

0 0

25

106

284

371 393

479

545

601

0

100

200

300

400

500

600

700

0

20

40

60

80

100

120

February April August November February April May June July August

# T

es

t S

uit

es

Ma

n D

ays

Eff

ort

Month

Time taken for Release Regression testing

Man Days Effort for Regression Test # Test Suites # Automated Suites

4

The confidence problem!

Where it came from

First Approach to Acceptance Testing:

UAT in a phase, after a period of development

5

6

What it felt like…

7

Becoming acceptance driven…

Building Integrity in

8

Develop

Ready User

Story

Customer

Verification

Specify tests

/ examples

Stakeholder(s) & BA

Testers

Developers

Backlog

Demo /

Showcase

Automate

acceptance

tests

Session

Based

Testing

The initial effect of automated acceptance testing

Real data from the field

40

67.5

100

78

23 15

12

5 3 0

28 64

151

217

311 371

393

479

545

601

0 0

25

106

284

371 393

479

545

601

0

100

200

300

400

500

600

700

0

20

40

60

80

100

120

February April August November February April May June July August

# T

es

t S

uit

es

Ma

n D

ays

Eff

ort

Month

Time taken for Release Regression testing

Man Days Effort for Regression Test # Test Suites # Automated Suites

9

Knowledge and experience to design a framework

• binding tests to investment apps

A set of design patterns

• from business specialists

Inject knowledge

• easy to customise

Vanilla acceptance tests

• tests as specifications

Living documentation

What we gained from this

10

Illogical Architecture Diagram

How Safia Works

House

Adaptor

System

Adaptor

Trading system

Maps SAFIA terminology to

system terminology

e.g. Calypso, Beauchamps,

Summit

SAFIA automates at the

server level via the API

Safia fixtures

AP

I Fitnesse fixtures

Maps investment house

terminology to SAFIA

terminology e.g.

Trade Capture = Trade Entry

11

e.g.

FX, Bond, Equities

IRS, FRA, CDS

e.g.

Run a P&L report

Update a legal entity

Amend SDI

Amend fees

Creating automated test cases using Safia

How Safia Works

Test Archetype

“Test Function”

Create a trade

Instrument

Template

IRS template

Test

Create a valid IRS trade

Apply Template &

Archetype

12

Let’s put it to the test

Demo

13

It brings benefits on different levels

Why use a framework like this

14

Benefits

Increased collaboration

Accelerating implementation

& learning

Anyone can interact with it

Teaching & coaching aid

Reduced Cost of Ownership

It was easy, right?

In retrospect…

15

The path to faster release

Real data from the field

40

67.5

100

78

23

15 12

5 3 0

28 64

151

217

311 371

393

479

545

601

0 0

25

106

284

371 393

479

545

601

0

100

200

300

400

500

600

700

0

20

40

60

80

100

120

February April August November February April May June July August

# T

es

t S

uit

es

Ma

n D

ays

Eff

ort

Month

Time taken for Release Regression testing

Man Days Effort for Regression Test # Test Suites # Automated Suites

16

Tests as specs, the principles of test design

How we wrote the tests

17

What makes a good acceptance

test?

Self-documenting

About business intentions

Concise and granular

A specification not a script

Anyone can understand it

Based on testable statements

~One third is system understanding

Reducing the Cost of Ownership

18

Points to consider when automating

19

Theory ‘aint practice

Start early & iterate frequently

Distil and simplify

Tests or Documentation?

DON’T PANIC !!!

“Brace yourself Daphne…”

20

Questions

Thank you for listening

21

Please evaluate our

presentation by using the

evaluation booklets

which you can find in

your conference bag.

Thank you!

SQS Group Limited

Mike Scott & Tom Roden

mike.scott@sqs.com

tom.roden@sqs.com

SQS Group Limited

7-11 Moorgate, London EC2R 6AF

Internet: www.sqs.com