The 5 Stages of A/B Testing
Byron JonesProduct Manager
What is A/B Testing?
Credit: Jonathan Rosenberg's post from 21 Aug 2012: Hippos are among the most dangerous animals in Africa. Conference rooms too.
conducting experiments to optimize your user experience and making data driven decisions
A/B Testing is…
Examples of A/B Tests
• Business Objective:Teach the world to code
• Goal: Increase participation in Hour of Code campaign
• Hypothesis: Changing the hero messaging to emphasize a sense of community around the movement would result in an increase in signups.
• Business Objective: Increase revenue
• Goal: Maximize sales on new product launches
• Hypothesis: Moving the call-to-action higher on the page by changing the way the promotional offer was displayed could drive more purchases and increase revenue generated from SimCity.com.
• Business Objective:Drive engagement and traffic
• Goal: Build follower base on social media
• Experiment: Which design of social media call-to-action buttons maximizes followers?
5 Stages of A/B Testing
MEASURE
ANALYZE
HYPOTHESIZE
BUILDRUN
of
A/B TESTING
MEASURE
ANALYZE
HYPOTHESIZE
BUILDRUN
Analyze your Data
• User Tests & Research
• Surveys
• Sales Team Feedback
• Heat Maps
• Web Traffic
• Product Usage
• Order History
• Support Tickets
QualitativeQuantitative
Analyze - Best Practices
• Look for high impact areas of your business
• Use the sample size calculator to predict volume and duration
MEASURE
ANALYZE
HYPOTHESIZE
BUILDRUN
"If _____[I do this] _____, then _____[this]_____ will happen because ___[rationale]____."
Create Your Hypothesis
Hypothesize - Best Practices
• Change one variable at a time, otherwise you won’t know which change influenced the results
• Be specific and descriptive in your hypothesis
Business Goal
Hypothesis
Idea 1 Idea 2
MEASURE
ANALYZE
HYPOTHESIZE
BUILDRUN
Build Your Experiment
1. Create a variation
2. URL targeting
3. Add an Audience
4. Traffic Allocation
5. Setup your goals
6. Preview
Example - Adding Customer Quotes
A customer testimonial increases the accounts created because it increases the perceived value of the product.
Analysis & Sample Size Hypothesis
Demo - Building an Experiment in Optimizely
Best Practices
• You can use Query string parameters to target your ad spend or to QA your test live
For exampleutm_campaign=spring15
qa_testing=true
MEASURE
ANALYZE
HYPOTHESIZE
BUILDRUN
• 95% Statistical Significance
“How likely is it that my test results will say I have a winner when I actually don’t?”
• Optimizely’s Stats Engine powers our statistical significance calculations
The goal of Stats Engine is to make interpreting results as easy as possible for anyone, at any time
Optimizely Results
MEASURE
ANALYZE
HYPOTHESIZE
BUILDRUN
of
A/B TESTING