Embed Size (px)
Citation preview
#opticon2015
Scaling your Testing Program for Maximum ImpactAndy NelsonDirector of Growth Marketing, Moz
Martijn ScheijbelerLead Growth, SEO & Analytics, The Next Web
@martij[email protected]
Khattaab KhanStrategy Consultant, Optimizely
Increase Your Impact Without More Resources Andy NelsonDirector of Growth Marketing, Moz
#opticon2015
~2M visits/month
#1 Go Big
Evolution vs. Revolution
Quality > Quantity
#2 Prioritize Intelligently
Elements of good prioritization
Core• Potential impact to success metric• Estimated runtime required• Estimated level of effort
Elements of good prioritization
Modifiers• Alignment with strategy• Support from VOC, analytics, prior
testing• Urgency
Core• Potential impact to success metric• Estimated runtime required• Estimated level of effort
#3 Swim Lanes
Homepage
Blog
Conversion Funnel
In Product
Visitors 1-25
Visitors 26-50
Visitors 51-75
Visitors 76-100
Homepage
#4 Multiple Streams of Work
1 week
1 day1 day
Checkout Optimization
Product Page Optimization
Test Analyze
Design/Build
Test 1
Test 2
Test 3
Test 4
Test Analyze
Design
Test
Design/Build
#5 Prevent Busted Tests (or at least catch them early)
Chrome
Safari
IE
Firefox
Chrome
Safari
IE
FirefoxBUSTED
Have a process
Pre-launch QA
Final Sign-Off
Post-Launch QA
Weekly Check-Up
Test across browsers/devices
0
22500
45000
67500
90000
5/3 5/10 5/17 5/24 5/31 6/7
Watch for irregular events
Test big, bold changesImprove your prioritizationRun multiple testsEliminate dead timeImprove your QA
Takeaways
#opticon2015
How The Next Web’s Testing Program ScaledMartijn ScheijbelerLead Growth, SEO & Analytics - The Next Web
@martij[email protected]
Yearly 5.000 attendees, 75.000 products sold. Monthly 6 million unique users, 12 million pageviews.
The Next Web: 2006 > 2015
What’s happened in the last 12 months?
Scaling our testing program
That clearly wasn’t good enough
1 test a month :(
Currently 5-6 tests a week, how did we get there?
15-20 tests a month :)
What did we learn along the way?
• Swim lanes, KPIs, Goals.
• Segments: traffic sources, etc.
• Idea creation: backlog + prioritisation.
• Scaling > Analysing results.
• Do it live!
Unique Users, Page Views, Emails & Revenue.
Goals: UU, PV, Emails & $
Related stories, pagination, sharing, email signups.
KPI: CTR, Engagement, Funnels
New v.s. Returning - Europe v.s. USA - Facebook v.s. SEO - Mobile v.s. Desktop
Type, Regions, Source, Device
Idea creation v.s. Planning
• Different templates + segments.
• How much time per test? Run tests for a fixed predefined time.
• The maximum of your program.
• Take away: create a backlog!
The part that we can’t scale yet
Analysing > Scaling!
Well, this is both the most boring and most exciting part!
1-2 main KPIs, 15 secondary KPIs
Rails, Google Analytics API Built with love by the TNW Marketing team.
TNW Analytics
TNW Analytics was born!
Adding a new idea
Schedule
Test
Saving 20% of time per test!
Take away: Be able to push code!
What’s next?
Scaling our testing program to the next level?• Even more templates, across more projects.
• 25+ tests a month with current growth of traffic.
• Bigger team? We can’t keep up with coding ;-)
• More automation? Push test variant to GTM.
• Mix it up with personalisation/ behaviour.
• Push our experiments to Slack for sharing.
• Create a better style guide + Bootstrap guidelines.
Summary & Take aways
• Know your KPI framework, expectations and limitations.
• Know your segments, CR% so you know how to prioritise.
• Follow your planning, priorities.
• Scale = 40% - 50% automation: analyse fast!
• Be able to push code live!
#opticon2015
Have fun & Thank you!
Martijn ScheijbelerLead Growth , SEO & Analytics - TNW
@martij[email protected]
Strategies Essential to Achieve Testing Program Scale
#opticon2015
Khattaab KhanStrategy Consultant, Optimizely
Focus on Optimization Opportunities that Matter
44% 42%
8%
52%
87% 81%
41% 31%
44%
32%
11% 12%
15% 27%
48%
16% 7%
Homepage to Category
Category to Product Product to Shopping Cart
Shopping Cart to Checkout
Checkout to Payments
Payments to Conversion
Continue down funnel
Visit different page
Exit website
EXAMPLE
Homepage Category
Page Product
Page Shopping
Cart Checkout
Page Payment
Page
With a conversion funnel, we could analyze all steps to see how and where users drop off Evaluate User Path
44% 42%
8%
52%
87% 81%
41% 31%
44%
32%
11% 12%
15% 27%
48%
16% 7%
Homepage to Category
Category to Product Product to Shopping Cart
Shopping Cart to Checkout
Checkout to Payments
Payments to Conversion
Continue down funnel
Visit different page
Exit website
EXAMPLE
Homepage Category
Page Product
Page Shopping
Cart Checkout
Page Payment
Page
With a conversion funnel, we could analyze all steps to see how and where users drop off
Focus Testing on Problem Areas that are Drivers of Business Value
Intelligent Documentation for
Intelligent Prioritization
Roadmap
Roadmap + Resource Planning
Custom Prioritization Scoring
Pauline Marol Lead PM, Site Optimization
“I don’t want to be the one making decisions on what experiments I run and don’t run because I am biased.”
Results Analysis
Results Page Best Practices • Evaluate opportunity costs
• Analyze segment performance
• Shareable results link
Results Sharing
Communicating Results + Roadmap Iteration • Hold team accountable to strategy that inspired variation design
• Validate/invalidate hypothesis to evaluate goal effectiveness
• Define where results output will live (e.g. database, wiki, etc.)
Simultaneous Testing
Simultaneous Testing Use Cases
Experiments on same page
Experiments on same flow (e.g. checkout / registration flow)
Multivariate experiment
Beware of Interaction Effects
Elements on same page
Elements on subsequent pages in a funnel
When multiple points of user interaction influence the same goal, interaction effects can taint data
Evolution of Testing Org Structure
Core Testing Team
MEASURE RESULTS
Analytics
Dev + QADesign
Program Manager
CRO Analystoptional
Evolution intoOptimization Center of Excellence
MEASURE RESULTS
Analytics
Dev + QADesign
Program Manager
ProductManager
ProductManager
Dedicated optimization resources serving
multiple BUs
Leverage data to focus on goals that directly drive business valueHold your team accountable with effective planning documentsKnow when to run simultaneous tests
Scale your program structure to support multiple business units
Takeaways
#opticon2015
Thank you!
Khattaab KhanStrategy Consultant, Optimizely
#opticon2015
Andy NelsonDirector of Growth Marketing, Moz
Martijn ScheijbelerLead Growth, SEO & Analytics, The Next Web
@martij[email protected]
Khattaab KhanStrategy Consultant, Optimizely
Now for some Q&A
