Build The Right

Stuff.Workshop by Adam Scott & Arti Mathanda

ADAM SCOTT

@adamacs

ARTI MATHANDA

@artismarti

SIMPLY BUSINESS

❖ Britain’s biggest business insurance provider

❖ Selling insurance online for 10 years

❖ 300,000+ customers and counting

❖ Iterating on Agile & Lean methodologies for 4.5 years

I HAVE AN IDEA!

BUT…

WHERE DO YOU

WANT TO GO?

BUILD

BUILD

SHOULD YOU BUILD IT?

Why? ---> Why? ---> Why? ---> Why? ---> Why?

BUILD

CAN YOU BUILD IT?What? & Who? & How?

BUILD

WILL YOU BUILD IT?When?

LEAN START UP

- Build

- Measure

- Learn

BUILD

DEFINE EXPERIMENT

BUILD

HYPOTHESIS:We believe that by doing x we will get benefit y.

ex. We believe that by making the Call To Action button

more prominent, we will get a better conversion rate

from Page 1 to Page 2

BUILD

BAD HYPOTHESIS (a fact or a given)

BUILD

GOOD HYPOTHESIS (it can be true or false)

BUILD

BASELINE:

Currently 25% conversion from Page 1 to Page 2..

BUILD

EXPECTED RESULTS:

Page 1 to Page 2 conversion increases by 10% points.

BUILD

MEASURE FOR:

2 weeks.

(Don’t forget Statistical Significance).

BUILD

GOOD MEASURES

BUILD

BAD MEASURES

BUILD

ALSO KEEP AN EYE ON:

Site Performance

Overall Conversions

Seasonal effects

Bounce Rates

etc etc etc

MEASURE

ACTUAL RESULTS:

Page 1 to Page 2 conversion increased by 3% points

MEASURE

IS IT STATISTICALLY SIGNIFICANT?

Yes!

LEARN

NOW WHAT?

LEARN

WHY DID IT FAIL?

LEARN

WHAT OTHER METRICS ARE AVAILABLE

TO ANALYSE THIS?

LEARN

TRY SOMETHING DIFFERENT?

LEARN

REVERT CHANGE?

LEARN

LEAVE IT RUNNING?

TLDR;

RECORD!

LET’S PRACTISE