View
105
Download
0
Category
Tags:
Preview:
DESCRIPTION
Search Marketing Thursday - Januari 2014 - Martijn Geerlings (Optimizely)
Citation preview
Friday, January 31, 14
Friday, January 31, 14
Best Practices and Lessons Learned from
100,000 A/B TestsA/B TestsMartijn Geerlings
100,000
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
9:1Friday, January 31, 14
Lessons Learned
Friday, January 31, 14
Lesson #1:
Define success .
Friday, January 31, 14
Lesson #1:
Define quantifiable success metrics.
Friday, January 31, 14
Friday, January 31, 14
Visitvisitors per day
Friday, January 31, 14
Visitvisitors per day
Emailsign up
rate
Friday, January 31, 14
Visitvisitors per day
Emailsign up
rate$
$ per recipient
Friday, January 31, 14
Visit Email $sign up
rate $ per recipientvisitors per day
Friday, January 31, 14
Media
Button
Friday, January 31, 14
Button:
1.Sign Up
2.Learn More
3. Join Us Now
4.Sign Up Now
Media:
1.Get Involved Image
2.Family Image
3.Change Image
4.Barack’s Video
5.Springfield Video
6.Sam’s Video
Splash page experimentVariations:
Friday, January 31, 14
Button: “Sign Up”Friday, January 31, 14
Button: “Learn More”Friday, January 31, 14
Button: “Join Us Now”Friday, January 31, 14
Button: “Sign Up Now”Friday, January 31, 14
Button:
1.Sign Up
2.Learn More
3. Join Us Now
4.Sign Up Now
Media:
1.Get Involved Image
2.Family Image
3.Change Image
4.Barack’s Video
5.Springfield Video
6.Sam’s Video
Variations:Splash page experiment
Friday, January 31, 14
Media: “Get Involved”Friday, January 31, 14
Media: “Family”Friday, January 31, 14
Media: “Change”Friday, January 31, 14
Media: “Barack’s Video”Friday, January 31, 14
Media: “Springfield Video”Friday, January 31, 14
Media: “Sam’s Video”Friday, January 31, 14
Button:
1.Sign Up
2.Learn More
3. Join Us Now
4.Sign Up Now
Media:
1.Get Involved Image
2.Family Image
3.Change Image
4.Barack’s Video
5.Springfield Video
6.Sam’s Video
Variations:Splash page experiment
Friday, January 31, 14
Button:
1.Sign Up
2.Learn More
3. Join Us Now
4.Sign Up Now
Media:
1.Get Involved Image
2.Family Image
3.Change Image
4.Barack’s Video
5.Springfield Video
6.Sam’s Video
Variations:Splash page experiment
Friday, January 31, 14
Button:
1.Sign Up
2.Learn More
3. Join Us Now
4.Sign Up Now
Media:
1.Get Involved Image
2.Family Image
3.Change Image
4.Barack’s Video
5.Springfield Video
6.Sam’s Video
Variations:Splash page experiment
Friday, January 31, 14
Splash page experiment
Friday, January 31, 14
Splash page experiment
Friday, January 31, 14
Splash page experiment
Friday, January 31, 14
Email Subscriptions Volunteers Amount Raised
Original: 7,120,000 712,000 $143,000,000
+40.6% +2,880,000 +288,000 +$57,000,000
New: 10,000,000 1,000,000 $200,000,000
Splash page experiment
Friday, January 31, 14
Email Subscriptions Volunteers Amount Raised
Original: 7,120,000 712,000 $143,000,000
+40.6% +2,880,000 +288,000 +$57,000,000
New: 10,000,000 1,000,000 $200,000,000
Splash page experiment
Friday, January 31, 14
Email Subscriptions Volunteers Amount Raised
Original: 7,120,000 712,000 $143,000,000
+40.6% +2,880,000 +288,000 +$57,000,000
New: 10,000,000 1,000,000 $200,000,000
Splash page experiment
Friday, January 31, 14
Lesson #1:
Define quantifiable success metrics.
Friday, January 31, 14
Lesson #2:
Less is more. Reduce choices.
Friday, January 31, 14
+43.8% Purchases
Friday, January 31, 14
vs.
Friday, January 31, 14
vs.
+14.8% conversion rate
Friday, January 31, 14
Lesson #2:
Less is more. Reduce choices.
Friday, January 31, 14
Lesson #3:
Words matter. Focus on your call to action.
Friday, January 31, 14
vs.
Friday, January 31, 14
vs.
Friday, January 31, 14
vs.
+14.6% clicks
Friday, January 31, 14
Donation button experiment
Variations
Friday, January 31, 14
Donation button experiment
VariationsNever
Signed UpSigned Up,
Never DonatedPreviously Donated
0.0% 0.0% 0.0%+2.3% +27.8% +16.3%-27.8% N/A N/A+15.2% -24.6% +11.9%+8.5% +2.9% +18.4%
Friday, January 31, 14
Lesson #3:
Words matter. Focus on your call to action.
Friday, January 31, 14
Lesson #4:
Fail fast.
Friday, January 31, 14
vs.
Friday, January 31, 14
vs.
Friday, January 31, 14
vs.
-10% conversions
Friday, January 31, 14
Lesson #4:
Fail fast.
Friday, January 31, 14
Lesson #5:
Seek the Global Maximum.
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
+0.5% Retention +1.4% Engagement
Friday, January 31, 14
Lesson #5:
Seek the Global Maximum.
Friday, January 31, 14
1. Define quantifiable success metrics.2. Less is more. Reduce choices.3. Words matter. Focus on your call to action.4. Fail fast.5. Seek the Global Maximum.
Lessons Learned
Friday, January 31, 14
Thank You
Friday, January 31, 14
vs.
Friday, January 31, 14
vs.
+8% engagement
Friday, January 31, 14
2009! 2010! 2012!2011! 2013!
Friday, January 31, 14
Recommended