Embed Size (px)
Citation preview
`How to Successfully Run Your First Website A/B Test
THUE MADSEN
@thuelmadsen
@Kissmetrics
The don’ts…
The dos…
1 What is A/B testing?
Do your homework
Repetition
Succeed
2 Where do you start testing?
TABLE OF CONTENTS
3 A framework for success
Tooltips and Burgers4
What is A/B testing? A SCIENTIFIC METHOD OF PROOF
WHAT IS A/B TESTING FOR MARKETING
• A scientific method adapted for marketing
• Compare two or more versions of an asset to each other
• Use data and statistics to determine which version is better
• Implement winner and increase conversions
*Image from Kikolani
HOW DOES A/B TESTING WORK?
*Image from Dynamicyield
HOW DOES A/B TESTING WORK?
SO WHY DO WE A/B TEST?
• Lowers risk
• Helps us realize the actual impact of proposed changes
• Increase learnings
• Takes the guess work out of optimization
Where do you start testing? THE DON’TS
DON’T JUST GUESS
SUB OPTIMIZE
• Button colors
• Small copy changes
• Small layout changes
• Fonts, font size, etc.
* at least not until you have found a great variation and have lots of traffic
WATCH OUT FOR BUZZ
• Don’t just test what the gurus say
• What works for others may not work for you
PREMATURE TESTING
• Testing without traffic
• Before you know what and why you are testing
Where do you start testing? THE DOS
DO YOUR HOMEWORK
You have to put in the work to get great results
• Learn as much as you can about A/B testing
• Study the great people out there
• What’s their process and insights?
A framework for success The basis of a successful testing program
Research Know what to test and see the greatest improvements
DIVE INTO YOUR DATA
PaidEntered CCAdded to CartViewed Product
FIND YOUR BIGGEST OPPORTUNITIES
PaidViewed CheckoutAdded to CartViewed Product
INVESTIGATE THE OPPORTUNITY TO UNDERSTAND IT
And/Or
FORM A HYPOTHESIS
By doing A,
B will happen,
Because of C
FORM A HYPOTHESIS
By updating branding and increasing form visibility (A),
We will get more prospects to register for webinars (B),
Because prospect attention will be drawn to the form (C).
BUILD THE TEST
LAUNCH THE TEST
VS.
Control Variant
THE TEST
VS.+24.01% increase in registrations!!
100% Significance
DOCUMENT AND SHARE YOUR FINDINGS
DOCUMENT AND SHARE YOUR FINDINGS
Documenting all your tests and results
• Increases company transparency
• Helps others know where tests are going on
• Makes it easier to call tests
• Makes learnings available for everyone in company
*Image from Optimizely
REPEAT THE PROCESS AS OFTEN AS POSSIBLE
A STORY OF OUR FAILURE
VS
Control Variant
A STORY OF OUR FAILURE
VS
A STORY OF OUR FAILURE
VS-13.68% increase in registrations!! 80% Significance
*Quote found on VWO
NOT ALL TESTS ARE WINNERS
“ “Only 1 out of 8 A/B tests have driven significant change.”Noah Kagan AppSumo*
“ “The goal of a test is to get a learning, not a lift. With enough learnings, you can get the real lift,” Dr. Flint McGlaughlin MECLABS**
**Quote found on MarketingSherpa
RULES OF THUMB FOR TESTING
Things to remember when we are testing
• Min. 2000 visitors
• Min. a few hundred conversions
• At least one full cycle > 2 week
• >99% statistical significance
• Keep a backlog of test hypothesis
• Build next tests as the current one is running
*More info at http://kiss.ly/1vF4wUc and http://kiss.ly/1Q4vTm0
Tooltips AND BURGERS
How I like my testing burger Tooltips for your testing framework
Tool Company Why do I like it?
Analytics KissmetricsShows me my actual customers and their
behavior
Surveys Qualaroo and/or SurveyMonkey
Easy to use/Deep qualitative insights
Project Management TrelloManage my time + communicate with
collaborators
Tool Company Why do I like it?
Company Wiki Confluence Document initial thoughts for everyone
Testing platform* Optimizely, Pardot, Adwords
Easily build tests and get data
Analytics Kissmetrics Let’s me confirm and verify down my funnel
Company Wiki ConfluenceDocumentation and
learnings for everyone
*Includes actual testing platform and tools that have testing capabilities
Not everyone likes their burger the same way
How do you take yours?
Trust the process Share your leanings Stack your results
A/B TESTING FOR SUCCESS
JUSTIN THORP Senior Marketing Operations Manager, AddThis
@thorpus [email protected]
THUE MADSEN Marketing Operations Manager, Kissmetrics
@ThueLMadsen [email protected]
Questions?
Appsumo reveals its A/B testing secret: only 1 out of 8 tests produce results
MORE RESOURCES…
A Beginner’s Guide To A/B Testing: An Introduction
How to Run AB Tests that Get REAL Results
Statistical Significance Does Not Equal Validity (or Why You Get Imaginary Lifts)
How Big is the Difference Between Being 90% and 99% Certain?
My 7 Rules for A/B Testing That Triple Conversion Rates
Optimizely: What is A/B Testing
How To Determine The Statistical Significance Of An A/B Test – Basic Introduction
