Download pdf - Leanconf2014 - A/B-testing - How, When, Examples & Pittfalls by Ton Wesseling

H O W , W H E N , E X A M P L E S & P I T T F A L L S

L e a n C o n f e r e n c e – N o v e m b e r 1 7 t h - M a n c h e s t e r

To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y

A/B-testing

T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4

Manchester


Should I go for City?


Should I go for United?


Manchester


A/B-test ing

A/B-testing != CRO


A/B-test ing

CRO != user knowledge


A/B-test ing

A/B-testing

==

user knowledge


A/B-test ing

HOW & WHEN


A/B-test ing

EXAMPLES


A/B-test ing

PITTFALLS


Ton Wessel ing | @ TonW

More testing

Quality Assurance

Lower pricing


Online Experiments – A/B-test ing

Winner!

Variation B

Variation A

Default


Hyperl ink junky


Digging through log f i les


Trying to improve


Over and over again


What most start -ups do


That is:

0 / 1 Testing


Why can this lead to fai lures?


0/1 Testing: weekly change?

26 out of 44 weeks: over 5% change!


Testing should be l ike:


Scienti f ic approach

A / B Testing

Live Online Research


Experiments

“If you double the number of

experiments you do per year

you’re going to double your

inventiveness”

Jeff Bezos, CEO Amazon – 2004!


Growth l ike AmazonG

erm

an

Ecom

me

rce

mark

et,

thank

you

André

Mory

s


Experiments per year

1000+(but it did not start like that)


Growth curve


Phase 1: test solut ions


Fake adwords test ing

Do they

clickon your idea?


Fake facebook test ing

Do they

likeyour idea?


Fake landing page test ing

Do they

subscribeto your idea?


Make sure i t ’s not

0 / 1 Testing


Once you bridged the gap


The growth phase!


Duplicate?


I t ’s the web, i t ’s scalable!


You need to optimize!


Over and over again


I t takes many experiments to win


Maximize your test capacity

Time

Conversion%


Speed up – more people



erm

an

Ecom

me

rce

mark

et,

thank

you

André

Mory

s


Real growth


I t takes heavy experimenting

250+ in a year


Not every experiment is successful


I t takes a ful l stadium a week for 250+


Or roughly

1000+conversion points


Too slow? Energy wil l run out!


And you wil l never get this


You need the real fast paced culture


How does Bi l l win?


Emotion vs Ratio



Why is your user here?


How can you

help your user ’s brain

to fullfi l it ’s needs?


I t ’s not about A/B-test ing

Growth is about learning

how to serve brains!


Testing costs t ime

Use it to LEARN


FA C T & A C T – 7 s t e p s t o l e a r n

&

TellConcludeAnalyze

TestCreateAnalyzeFind


N o w : f o c u s o n C TA

Analyze

Test

Create


Create = Data & Psychology

Digital Data Persuasion Psychology

Analyze

&

Experiment


Wheel-of-Persuasion.com


Persuasion on forms

+25% sales


Same technique

Significant more

submits, but..

Less registrations


A small sentence

+18%

sales


But also

+10% applications


Motivat ion

Winner

(sales)


Win your own form A/B-test

http://testing.agency/manchester


And social proof?

Many went before you

In november 1.675

people applied for

a loan at MoneYou.

Winner

(sales)


Chunk your USP’s

+14%

sales


With learning: Phase 3 becomes easier


Testing becomes user knowledge

ConversionOp miza on MarketResearch


Pitt fal ls

How NOT to test


A/B-test ing can be


Don’t t rust your A/B-test analyt ics


I t keeps you away from this fai lure


Enough traff ic & conversions?


Low signif icance levels

Big chance the winner is not a real winner


Low power levels

Big chance a real winner is not recognized


Tested percentage is a spread!


Example test-setup


Average power: 40%


So:


Another error warning:

You just go and dig unitil you find something

(you will always find something that seems true BUT is

not!)


OK, results are in


Not signif icant

It’s not a winner, stop

But, it’s does not mean it’s a loser!

Average: 30% of your tests will have winners


More?

http://ondi.me/contentverve

http://ondi.me/conversionxl


Conclude

Do nothing?or

Re-test?or

Implement?or

Scale?and

Continue or change test roadmap


Knowledge l ibrary: ABtestguide.com


I t ’s not about A/B-test ing

Growth is about learning

how to serve brains!


So you wil l get this curve



erm

an

Ecom

me

rce

mark

et,

thank

you

André

Mory

s


A/B-test ing

HOW & WHEN


A/B-test ing

EXAMPLES


A/B-test ing

PITTFALLS


Slides & more

http://testing.agency/manchester


Be l ike Bil l , and win!

H O W , W H E N , E X A M P L E S & P I T T F A L L S

L e a n C o n f e r e n c e – N o v e m b e r 1 7 t h - M a n c h e s t e r

To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y

ht tp : / /testing.agency/manchester