corporate.kaboose.com
KABOOSE Valentine’s Day 2009 A/B Testing Campaign
Case StudyApril 01, 2009
corporate.kaboose.com
Introduction:
Jose DavilaWeb Analytics - A/B Testing Strategist
Kaboose Inc.
corporate.kaboose.com
About Kaboose Inc.
corporate.kaboose.com
Business GoalsOnline Publisher:
• Increase the reach of our network of web properties
• Engage more our audience: • View more content, • Stay longer • Use our services• Interact.
• Business Model:• Advertisement...
corporate.kaboose.com
Questions & ChallengesBefore A/B Testing:
• Testing process was lengthy and cumbersome.
• Most of the decisions were made with limited testing.
• Potentially great ideas were not being tested.
• Organization ready for further optimization.
corporate.kaboose.com
Questions & ChallengesValentine’s Day Traffic:
• Valentine’s Day is an important seasonal source of traffic.
• Previous optimization for traffic and engagement (Marketing, Editorial, SEO, Web Analytics).
• Question: Can we further improve engagement on our Valentine’s page? (PVs/Visit, stickiness)
• List of “improvement” ideas to test: Valentine’s Day landing page.
corporate.kaboose.com
Valentine’s Day Page• Control Recipe: The original page.
corporate.kaboose.com
Valentine’s Day Page• Control Recipe: The original page.
corporate.kaboose.com
Test StrategyValentine’s Day Landing Page:
• Test variations of the center content.
• Incorporate multiple “improvement” ideas.
• Content area broken down in three modules/blocks.
• Variations for each module/block (based on previous tests)
• Success metrics: PVs/Visit, stickiness
corporate.kaboose.com
Test Strategy• Modules/blocks of content...
corporate.kaboose.com
Recipes: Module 1 -Top ModuleA: Feature
Rotator• Flash based rotator.
B: Static Image and Links
• Minimalist version.
C: Tab Based Hub
• Flash based. • Static.
corporate.kaboose.com
Test Strategy• Modules/blocks of content...
corporate.kaboose.com
Recipes: Module 2 - Middle ModuleA: Small Index• List of linked categories
and short description.
B: Extended Index• List of categories with
additional links.
corporate.kaboose.com
Test Strategy• Modules/blocks of content...
corporate.kaboose.com
Recipes: Module 3 - Bottom ModuleA: Empty space B: Bottom content
element• Group of three additional links
with images and a short descriptions
corporate.kaboose.com
The Results...!Can you guess?
• From the 12 possible combinations (+ control) the winner was….
… nobody on our team was able to guess it …
First some interesting findings….
corporate.kaboose.com
The Results...!Some interesting findings…• All the new versions performed better.
• One recipe showed the highest performance across the board(+ 54% Lift – PVs/Visit)
• More influential modules/elements identified.
• Behavioral patterns from key visitor segments extracted
corporate.kaboose.com
The Results...!Which recipe won?
• Module 1: Tabbed image features (C)• Module 2: Small Index (A)• Module 3: Blank (A)
• Recipe CAA had the highest performance: engagement + stickiness
• Recipe CAA pushed live
• Over 45% increase in engagement (PVs/Visit) during the whole campaign (including test).
corporate.kaboose.com
Long Term Outcomes.From our A/B testing experience.
• Start small + Value + Allocate resources
• Build awareness based on a sequence of success stories
• Learning process (“failures” can be extremely valuable)
• Don’t assume “best practices” are the best solution. Test them.
• Create testing business culture
• A/B Testing is now part of our process.
corporate.kaboose.com
Questions?
corporate.kaboose.com
Thank you!
Jose DavilaWeb Analytics - A/B Testing [email protected]
Kaboose Inc.