Conversion World 2015 Keynote by Ton Wesseling

C o n v e r s i o n W o r l d K e y n o t e , A p r i l 2 0 t h 2 0 1 5 To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y

# C R O w o r l d - @ To n W

Growth by Online Experiments

E m a i l : t o n @ t e s t i n g . a g e n c y !L i n k e d i n : t o n w e s s e l i n g !

T w i t t e r : @ t o n w !!

Ton Wessel ing !

Ton Wesseling | @ TonW Growth by Online Experiments #CROworld

Onl ine Opt imizers


Onl ine Dia logue


Test ing.agency


Conversion Conferences wor ldwide


ConversionHotel .com


The 1st Conversion World


Conversion World speakers


More exper iments

“If you double the number of experiments you do per year

you’re going to double your inventiveness”

Jeff Bezos, CEO Amazon


Like Amazon, Like Zalando G

erm

an E

com

mer

ce m

arke

t, th

ank

you

Andr

é M

orys


Exper iments per year

1000+


Why not?

Compare to the past?


Why can this lead to fa i lures?


Why can this lead to fa i lures?


A big weekly change!







26 out of 44 weeks: over 5% change!



Methods used to improve conversion


Methods companies plan to use


I t ’s just

HOT


Lack of resources


Test more


Test more


Importance of a structured approach


A structured approach


FA C T & A C T

&

Tell Conclude Analyze

Test Create Analyze Find


The negat ive effect trend

Time span!



Time span!

Money spend on!

Optim

ization worldw

ide!



Time span!

Money spend on!

Optim

ization worldw

ide!



Time span!

Aver

age

effe

ct o

f opt

imza

tion!

at c

ompa

nies

app

lyin

g op

timiz

atio

n!M

oney spend on!O

ptimization w

orldwide!



Time span!

Aver

age

effe

ct o

f opt

imza

tion!

at c

ompa

nies

app

lyin

g op

timiz

atio

n!M

oney spend on!O

ptimization w

orldwide!


I t ’s not a solut ion

It’s a method


A/B-test ing is a method


I t ’s a way of working

It’s DNA


So i t ’s a method:

When

How What


Growth curve – how i t was teached


Optimizat ion phases - ROAR

Time span!

Con

vers

ions

per

mon

th!



Time span!

Con

vers

ions

per

mon

th!



Time span!

Con

vers

ions

per

mon

th!



Time span!

Con

vers

ions

per

mon

th!

Risk!



Time span!

Con

vers

ions

per

mon

th!

Risk! + Optimization!



Time span!

Con

vers

ions

per

mon

th!

Risk! + Optimization! + Automation!



Time span!

Con

vers

ions

per

mon

th!

Risk! + Optimization! + Automation! Re-think!



Time span!

Con

vers

ions

per

mon

th!


1.000 conversions!per month!



Time span!

Con

vers

ions

per

mon

th!


10.000 conversions per month!



Phase 1: br idge the gap – Risk


Fake adwords test ing

Do they click

on your idea?


Fake facebook test ing

Do they like your idea?


Fake landing page test ing

Do they subscribe to your idea?


Phase 1: br idge the gap – Risk


Get out of the office


Risk


ROAR - Opt imizat ion

Time span!

Con

vers

ions

per

mon

th!





Low number test ing?

Why can it hurt you?


Warning: low significance levels

ü  Big chance the winner is not a real winner


Warning: low power levels

ü  Big chance a real winner is not recognized


Example test set-up


Example test set-up

ü  90% significance level


Example test set-up


ü  100 conversions per variation


Example test set-up



ü  Average conversion rate: 2%


Example test set-up




ü  20 out of 100 ideas is a winner…


Example test set-up




ü  20 out of 100 ideas is a winner…

ü  …with an average uplift of +- 10%


Average: 75% significance & 40% power


So:


So:

ü  100 tests will give 10 false positives (90% significance)


So:


ü  40% of the 20 winners are recognized (40% power)


So:



ü  8 real winners & 10 false positives


So:




ü  NOT 18 improvements of 10%+


So:




ü  NOT 18 improvements of 10%+


You need big numbers per var iat ion

Time span!

Con

vers

ions

per

mon

th!





With 1000 conversions + a month

20+ tests a year


Tests with impact, on average

1 out of 3

Simply truth: you’re just not always able to create a winner with enough impact!


So in th is case, you’ve impact

Every 6 weeks


Slower? Energy wi l l run out!


What to test? Business cases!

Delivery –test with in stock. 4 weeks

or 2 days?!


What to test? Business cases!

Service charge, do it or not? Include in price or not?!


What to test? Big design changes


ROAR – Opt imizat ion – moving up

Time span!

Con

vers

ions

per

mon

th!





Final ly you can test detai ls…

Is Multi Variatie Testing (which is also A/B-testing but with more varations) a good idea? Almost never!!


You should grow to

Customer Buying Reasons


Reasons behind the customer journey


Why is your user here?


How can you

help your user’s brain

to fullfil it’s needs?


Data & Psychology

Digital Data Persuasion Psychology

Analyze &

Experiment


FA C T & A C T – i t e n d s w i t h Te l l

&




Remember your lack of resources?


Test ing takes t ime

Use it to LEARN


Create a test schedule


A n d t h e n s t a r t : FA C T & A C T

&


Test Create Analyze Find Find

Your optimization team needs a good analyst / researcher!


Test which dia logue has impact


A/B-test ing is the ul t imate resource


But first: map your user journeys


Based on data


Based on: asking


Based on: l istening


Many sources out there to Find


FA C T & A C T

&


Test Create Analyze Find Analyze

Really a good analyst / researcher!!


Use your analyt ics data


On page behaviour


More detai ls needed?


Use event flow report ing


Create heatmaps


Create screen recordings


To find out: The cr i t ical locat ions


FA C T & A C T

&


Test Create Analyze Find Create

And yes, your team needs an economic / consumer Psychologyst and a UX designer!


Source: Psychology


Use that persuasion knowledge

Wheel-of-Persuasion.com

Resource & courses


Bart Schutz: Apr i l 21 – 16:00 CEST


Design your new dia logue!


FA C T & A C T

&


Test Create Analyze Find Test

And your team needs a good front-end developer!


But my A/B-test sof tware has drag & drop?


No, you need Front end development


Development resources


We measure in Analyt ics


They do have analyt ics – but not enough


Event tracking var iat ion behavior


Calcu late resu l ts : ABTestGuide.com/calc


So you can A/B-test wi th Tag Managers


FA C T & A C T

&



Analyze

Your analyst will analyze the results!


Not s ignificant

ü It’s not a winner, stop

ü But, it’s does not mean it’s a losing one!

ü Average: 33% of your tests will have winners If you are optimizing dialogues!


Choose what to analyze up front

ü  You just go and dig unitil you find something

(you will always find something that seems true BUT is not!)


Don’t stop the test once i t ’s s ignificant!


Don’t stop the test once i t ’s s ignificant!


You just let i t run unt i l i t ’s s ignificant


You just let i t run unt i l i t ’s s ignificant

Your test has a fixed lenght!

It’s even a fixed amount of test population, but we love to test business cycles and/or weeks!


ABTestGuide.com/calc


Your winners are spare?

Less than 25%?

Please higher your significance level to 95% or more if it drops below this 25 number!!


T ip: Your test takes too long

ü  Cookie deletion (you will loose 10% in two weeks)

They will re-enter the the and pollute your samples!


T ip: Don’t confuse cross device users

ü Only test 1 device at a time!


T ip: Maybe you’re test ing broken stuff

ü  Always check browser compatibility!

ü Dont’ break dynamic stuff!


T ip: You forgot to leave out s o m e v is i tors

ü  Only test on those who can be changed (if you test a sales promotion – leave out your current clients who just logon)


T ip: You’re not s lowing the control down

ü  Add the same code to the control


T ip: You’re not test ing on fresh people

ü  Start with control for a full purchase cycle (and then start sending traffic to your variations – setting the control to 0%)


T ip: You forgot to give them t ime to buy

ü  Stop getting fresh visitors after your test time (send new traffic to the control, but give you tested people time to finish converting)


FA C T & A C T

&

Tell Combine Analyze


Tell Conclude

Your psychologyst should conclude and your conversion lead should tell!


Conclude


Conclude

ü Do nothing?


Conclude

ü Do nothing? or

ü Re-test?


Conclude

ü Do nothing? or

ü Re-test? or

ü Implement?


Conclude

ü Do nothing? or

ü Re-test? or

ü Implement? or

ü Scale?


Conclude

ü Do nothing? or

ü Re-test? or

ü Implement? or

ü Scale? and

ü Continue or change test roadmap


Optimize and redesign


Knowledge l ibrary: ABtestguide.com


Everyone should use your knowledge!

Data Insights Action


ROAR – Opt imizat ion – moving up

Time span!

Con

vers

ions

per

mon

th!





Now you’re test ing

On your way to 4 tests a week

Max capacity with 1 teammember on each discipline: analyst, ux, developer, psychologyst!


ROAR – Automation

Time span!

Con

vers

ions

per

mon

th!





You’re already th inking of more teams

And still have test capacity

*test capacity – the number of tests you can start per week based on your audience & conversions!


Algor i thms


You tested which locat ions are cr i t ical


You know which dia logues are cr i t ical


Automation


Now you’ve moved to

Algorithm testing

Which internal and external influencers do have an impact on which segment of users?!


+ Optimization! + Automation!

ROAR – Automation

Time span!

Con

vers

ions

per

mon

th!

Risk! Re-think!




ROAR – You don’t stop opt imiz ing

Time span!

Con

vers

ions

per

mon

th!





ROAR – Maybe you’ l l never automate

Time span!

Con

vers

ions

per

mon

th!





ROAR – Re-think

Time span!

Con

vers

ions

per

mon

th!





A/B-test ing is the ul t imate resource


Optimizat ion is a way of working


CRO br ings so much user knowledge


You wi l l make big impact on Discover


Customer knowledge center

Making money f rom convers ion improvements

Creat ing more and more cus tomer ins igh ts Which can be monet ized

on even a la rger sca le


Now, that’s

HOT


So you keep on groing G

erm

an E

com

mer

ce m

arke

t, th

ank

you

Andr

é M

orys


Done test ing? No!


Done test ing? No!

TESTING NEVER STOPS

Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge!


I f y o u u s e a m e t h o d o l o g y

&



Otherwise the company will not grow enough on your efforts!


ROAR – when to start with what

Time span!

Con

vers

ions

per

mon

th!




C o n v e r s i o n W o r l d K e y n o t e , A p r i l 2 0 t h 2 0 1 5 To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y

# C R O w o r l d - @ To n W

Growth by Online Experiments

E m a i l : t o n @ t e s t i n g . a g e n c y !L i n k e d i n : t o n w e s s e l i n g !

T w i t t e r : @ t o n w !!

Ton Wessel ing !

Data & Analytics

Conversion World 2015 Keynote by Ton Wesseling