Embed Size (px)
DESCRIPTION
A/B-testing is awesome, the holy grail of online visitor knowledge and the base of revenue growth for many companies. Ton Wesseling, Testing Guru, frequently asked CRO speaker and CEO of Testing.Agency has 15 years of online optimization experience and will share the following insights in his talk: How & when to apply testing in your company 10 inspirational A/B-test results 20 pitfalls you will make in A/B-testing if you don't attend this talk
Citation preview
H O W , W H E N , E X A M P L E S & P I T T F A L L S
L e a n C o n f e r e n c e – N o v e m b e r 1 7 t h - M a n c h e s t e r
To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
A/B-testing
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Manchester
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Should I go for City?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Should I go for United?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Manchester
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
A/B-testing != CRO
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
CRO != user knowledge
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
A/B-testing
==
user knowledge
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
HOW & WHEN
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
EXAMPLES
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
PITTFALLS
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Ton Wessel ing | @ TonW
More testing
Quality Assurance
Lower pricing
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Online Experiments – A/B-test ing
Winner!
Variation B
Variation A
Default
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Hyperl ink junky
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Digging through log f i les
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Trying to improve
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Over and over again
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
What most start -ups do
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
That is:
0 / 1 Testing
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Why can this lead to fai lures?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
0/1 Testing: weekly change?
26 out of 44 weeks: over 5% change!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Testing should be l ike:
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Scienti f ic approach
A / B Testing
Live Online Research
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Experiments
“If you double the number of
experiments you do per year
you’re going to double your
inventiveness”
Jeff Bezos, CEO Amazon – 2004!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Growth l ike AmazonG
erm
an
Ecom
me
rce
mark
et,
thank
you
André
Mory
s
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Experiments per year
1000+(but it did not start like that)
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Growth curve
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Phase 1: test solut ions
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Fake adwords test ing
Do they
clickon your idea?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Fake facebook test ing
Do they
likeyour idea?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Fake landing page test ing
Do they
subscribeto your idea?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Make sure i t ’s not
0 / 1 Testing
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Once you bridged the gap
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
The growth phase!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Duplicate?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
I t ’s the web, i t ’s scalable!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
You need to optimize!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Over and over again
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
I t takes many experiments to win
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Maximize your test capacity
Time
Conversion%
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Speed up – more people
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Growth l ike AmazonG
erm
an
Ecom
me
rce
mark
et,
thank
you
André
Mory
s
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Real growth
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
I t takes heavy experimenting
250+ in a year
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Not every experiment is successful
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
I t takes a ful l stadium a week for 250+
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Or roughly
1000+conversion points
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Too slow? Energy wil l run out!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
And you wil l never get this
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
You need the real fast paced culture
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
How does Bi l l win?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Emotion vs Ratio
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Why is your user here?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
How can you
help your user ’s brain
to fullfi l it ’s needs?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
I t ’s not about A/B-test ing
Growth is about learning
how to serve brains!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Testing costs t ime
Use it to LEARN
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
FA C T & A C T – 7 s t e p s t o l e a r n
&
TellConcludeAnalyze
TestCreateAnalyzeFind
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
N o w : f o c u s o n C TA
Analyze
Test
Create
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Create = Data & Psychology
Digital Data Persuasion Psychology
Analyze
&
Experiment
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Wheel-of-Persuasion.com
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Persuasion on forms
+25% sales
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Same technique
Significant more
submits, but..
Less registrations
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A small sentence
+18%
sales
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
But also
+10% applications
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Motivat ion
Winner
(sales)
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Win your own form A/B-test
http://testing.agency/manchester
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
And social proof?
Many went before you
In november 1.675
people applied for
a loan at MoneYou.
Winner
(sales)
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Chunk your USP’s
+14%
sales
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
With learning: Phase 3 becomes easier
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Testing becomes user knowledge
ConversionOp miza on MarketResearch
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Pitt fal ls
How NOT to test
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing can be
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Don’t t rust your A/B-test analyt ics
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
I t keeps you away from this fai lure
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Enough traff ic & conversions?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Low signif icance levels
Big chance the winner is not a real winner
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Low power levels
Big chance a real winner is not recognized
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Tested percentage is a spread!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Example test-setup
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Average power: 40%
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
So:
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Another error warning:
You just go and dig unitil you find something
(you will always find something that seems true BUT is
not!)
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
OK, results are in
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Not signif icant
It’s not a winner, stop
But, it’s does not mean it’s a loser!
Average: 30% of your tests will have winners
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
More?
http://ondi.me/contentverve
http://ondi.me/conversionxl
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Conclude
Do nothing?or
Re-test?or
Implement?or
Scale?and
Continue or change test roadmap
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Knowledge l ibrary: ABtestguide.com
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
I t ’s not about A/B-test ing
Growth is about learning
how to serve brains!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
So you wil l get this curve
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Growth l ike AmazonG
erm
an
Ecom
me
rce
mark
et,
thank
you
André
Mory
s
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
HOW & WHEN
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
EXAMPLES
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
PITTFALLS
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Slides & more
http://testing.agency/manchester
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Be l ike Bil l , and win!
H O W , W H E N , E X A M P L E S & P I T T F A L L S
L e a n C o n f e r e n c e – N o v e m b e r 1 7 t h - M a n c h e s t e r
To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
ht tp : / /testing.agency/manchester
