Embed Size (px)
Citation preview
1
The ROI of Beta Testing
2
Meet the Speaker
Emily Hossellman
Director of Marketing @ Centercode
@ehossellman
Agenda
1. Where beta impacts your product
2. The tangible vs. intangible effects of beta
3. Measuring the impact of beta
a. The framework
b. The inputs
c. The results
4. Presenting ROI to stakeholders & executives
5. Q&A3
4
Who is Centercode?
Services Fully managed beta tests and programs
Software Complete beta program management platform
Testers Access to 150k+ deeply profiled global candidates
Content Free tools and best practices
We've been running beta tests since 2001
Beta test management solutions are our core competency
5
Beta Testing =Customer Validation
Targeted customers using your product in real environments
Increases quality, improves user experience, validates performance
Many names: CAT, UAT, FUT, Customer Evaluation, Field Trials, Pre-release
6
What Beta Brings to the Table
1. Introduces you to your target audience
2. Facilitates a real meaningful conversation
3. Gives you the opportunity to really listen
7
What Makes Beta ROI Challenging
1. Hard to define
2. Hard to measure
Beta Test ROI Calculator
8
9
The Impact of Beta
10
What does releasing a better product get you?
The Impact of Beta
11
The Impact of Beta
Development
Product Management
Development/QA
Usability
Product Management
Support
Marketing
Sales
Beta impacts every part of your product:
Development/QA
12
What you can achieve with beta:
● Identify bugs to improve quality (with context and frequency)● Evaluate real-world impact of known issues● Regression testing on solved issues● Analyze and improve real-world performance● Test real-world compatibility
Leading to:
● More effective development and quality assurance teams● Less time spent post-release on fixing issues
Usability
13
What you can achieve with beta:
● Study and improve the user experience● Assess the new user experience
Leading to:
● Higher customer retention● Lower support costs● Fewer returns● Better reviews
Product Management
14
What you can achieve with beta:
● Evaluate the total customer experience● Assess customer acceptance or product readiness● Solicit feature requests/suggestions for roadmap validation● Perform competitive analysis
Leading to:
● Better product release● Higher customer retention● Improved product planning● Decreased time to market
Support
15
What you can achieve with beta:
● Test documentation and support materials● Test support processes
Leading to:
● Lower support costs● Higher customer retention● Increased customer satisfaction
Marketing/Sales
16
What you can achieve with beta:
● Collect mock reviews and NPS scores● Collect testimonials, references, or case studies● Acquire early adopters/evangelists● Generate product/launch awareness
Leading to:
● Better product reviews● Increased buzz and goodwill● Increased sales● Fewer returns● Increased customer satisfaction
17
Visualizing the ROI of Beta
18
Tangible vs. Intangible
19
● Product Defects
○ Bugs
○ Interoperability
○ Usability
○ Alignment
Tangible Benefits of Beta Testing
20
● Testimonials
● User-Generated Content
● Press and Analyst Access
● Pre-Launch Customer Facetime
● Documentation Review
● Product and Brand Strength
Intangible Benefits of Beta Testing
● Infrastructure Testing
● Product Review Simulation
● Marketing Messaging Refinement
● Customer Goodwill
● Tester Community
● Feature Requests (Maybe)
21
Measuring ROI:The Framework
22
Two definitions of quality:
1. A product’s features and characteristics that influence customer satisfaction
2. Freedom from defects
Cost of Quality Framework
23
Cost of quality includes the costs invested in attaining a given level of quality, as well as the consequences of failing to meet expectations.
Cost of Quality Framework
ROI Calculation
24
Gains
Costs of Beta + Costs of ChangesX 100
ROI Calculation
25
Gains
Costs of Beta + Costs of ChangesX 100
Increased Sales
Decreased Returns
Decreased Post-Release Development/QA Cost
Decreased Support Costs
Increased Customer Retention
Beta Management Time
Shipping/Logistics
Beta Units
IncentivesTools
Increased Pre-Release Development/QA Cost
26
Measuring ROI:The Inputs
27
● Beta Management Time
● Beta Units
● Shipping/Logistics
● Incentives
● Tools
● Additional Dev/QA Time
Costs (Beta + Additional Dev/QA)
28
● Decreased Post-Release Dev/QA Cost
● Decreased Support Costs
● Decreased Returns
● Decreased Warranty Reserves
● Increased Customer Retention
● Increased Sales
Gains
29
Measuring ROI:The Results
30
Product: Wi-fi Router, $100, 500,000 units sold, $50 million in sales
Beta Test: 100 testers, 6 weeks
● Labor: $5,500● Shipping: $3,000● Units: $22,000● Incentives: $5,000● Tools: $500● Dev/QA Cost: $7,000● Total: $43,000
Example ROI Calculation
Gains
● Dev/QA Cost: decreased $8,500● Support Costs: decreased $35,000● Returns: save $77,000● Warranty Reserves: save $173,000● Sales Profit: increased $218,000● Total: $511,500
1.023%ROI of 1,090%
Optimizing Your Beta Test ROI
31
Optimizing Your Beta Test ROI
32
1. Beta Management Time2. Better Quality Feedback3. Feedback Workflow4. Beta Participation and Total Testers5. Length of Beta Phase6. Beta Management Tools
Optimizing Your Beta Test ROI
33
34
Presenting ROI
35
The ROI is high - be prepared to defend it
Tie it back to company initiatives (customer success, increasing sales, new market disruption)
Show predictive nature of tests
- Compare NPS/star ratings in beta to released product ratings
Don't forget about the intangibles
- Quotes from testers to illustrate points (bugs)- Future gains from product roadmap validation
Presenting ROI to Stakeholders
36
Q&A
Emily Hossellman
Director of Marketing @ Centercode
@ehossellman
Sign up for our newsletter and our future webinars
For more resources, visit our library
37
Thank You!
