A/B Testing – can you afford not to?#13NTCab

Hugh Dwyer,International Rescue Committee

Jacob Colie,Stage Coach Digital

Tom WilliamsonJackson River

Today’s Agenda

• Introductions

• What is A/B Testing?

• Testing Goals – what should we test?

• Testing versus Experimenting

• Crafting an A/B Test Plan

• Selling your plan

• Build, Measure, Learn

• Testing Tools

• Resources & Downloads

• QA

Slide 2A/B Testing

What is A/B Testing?A/B Testing:

A/B testing is an experimental approach to web design which aims to identify changes to web pages that increase or maximize an outcome of interest (e.g., donation conversion). As the name implies, two versions (A and B) are compared, which are identical except for one variation that might impact a user's behavior.

Slide 3A/B Testing

Standard Donation Page

Slide 4A/B Testing

A/B Test Donation Page

Slide 5A/B Testing

What is A/B Testing?Multivariate Testing

Multivariate testing is similar to A/B testing, but tests more than two different versions at the same time.

Slide 6A/B Testing

Multivariate

Slide 7A/B Testing

ChallengesWhat are some common challenges to doing A/B Testing?

• Resources – technology, time and people

• Data sampling size

• Is my list big enough?

• Are my results significant?

• “Do no harm”

• Lack of existing test data to build on

• Risk vs. Reward

To do A/B testing you need to have an Entrepreneurial Spirit!

Slide 8A/B Testing

A/B Testing ChannelsChannel Testing

• Email

• Advertising

• Web

• Offline/Direct Mail

• Integrated (On and Offline)

• Segmentation

Slide 9A/B Testing

Testing Goals – what should we test?

Channels and Audience Segments

Slide 10A/B Testing

Snail MailEmailSearch

Website/Landing Environment

Advertising

New Audiences

Leads

New Donors

Multi-Year Donors

Testing Goals – what should we test?

Metrics and KPIs

• Establish benchmark metrics: what are you trying to beat?

• Average time on site/page

• Open rates

• Click-through rates

• Form conversion (donations, email signups, etc.)

• Average Gift

• Cost to acquire

• Cost per lead

Slide 11A/B Testing

Testing Goals – what should we test?

Evaluating options

Slide 12A/B Testing

• Alignment with strategy, context

• Examine conversion funnels and donor lifecycle: where are your challenges?

• Consider potential returns, & calculate ROI

• Where can you move the needle the most?

• Quick wins and low hanging fruit

Testing Goals – what should we test?

Variables

Slide 13A/B Testing

Design elements

Persuasion assets

Offer elements

Benefits

Default ask

Products

Price points

Ask string

CTA

Multi-step forms

CTA Positioning

ImagesCopy

Media mentions

Premiums

Security seals

Testimonials

Achievements

Page layout

Expert ratings

Subject lines

Video

Keywords

Communications strategies

TimingFrequency

Cross-channel

Usability

Modal windows

Content

Testing Goals – what should we test?

Considerations

• Evolutionary vs. revolutionary changes

• Frequency and pace of testing

• Short-term vs. long-term

• Easy road vs. hard road

• Embrace (calculated) risks

Slide 14A/B Testing

Testing vs. EXPERIMENTS

Slide 15A/B Testing

Slide 16A/B Testing

When to test? When to experiment?

Slide 17A/B Testing

GOALSAccelerate your REVENUE

Provide VALUE for your users

Slide 18A/B Testing

The DIGITAL REVENUE CYCLE

BUILD. MEASURE. LEARN*

Slide 19A/B Testing

* Read THE LEAN STARTUP

BUILD an MVP

Slide 20A/B Testing

MEASURE revenue andengagement

Slide 21A/B Testing

LEARN what did and didn’t work

Slide 22A/B Testing

Baseline

IDEAL

Slide 23A/B Testing

Slide 24A/B Testing

vs.

short-term vs. LONG-TERM

Slide 25A/B Testing

5% vs. 35%

Slide 26A/B Testing

Examples

Slide 27A/B Testing

MERCY CORPS Retention

Slide 28A/B Testing

IRC Rescue Gifts

Slide 29A/B Testing

Getting started with A/B Testing• KISS – Keep it simple silly

• Define metrics to test

• Sustainer Acquisition

• Average Gift

• Gift String

• Conversion %, etc.

• Identify your top 3-5 items to test, and schedule them over X months!

• Decide how you will test each scenario

• Control group

• Minimum data required to validate

• Resources necessary

• Internal communications

• Implement each test

• Build, measure, learn – be patient

Slide 30A/B Testing

Individual Test Cycle

Slide 31A/B Testing

Identify KPI/Metric

How will you test?

Coordinate ResourcesImplement

Measure, Report, Analyze

Selling your plan

Slide 32A/B Testing

Objectives Inputs Plan

Developing a business case

Analysis Recommendation

Resources & Downloads

Presentation Materials

• PowerPoint - www.jacksonriver.com/nten/ab_testing

Third Party Tools:

• Optimizely - www.optimizely.com

• Visual Website Optimizer - http://visualwebsiteoptimizer.com

• Google Content Experiments

• 8 Seconds - http://www.8seconds.net

Slide 33A/B Testing

Questions and Answers

Contact info

• Hugh Dwyer

• Sr. Director, Information & Operations, International Rescue Committee

• hugh.dwyer@rescue.org

• Jacob Colie

• Senior Strategist & Principal, Stagecoach Digital

• jacob@stagecoachdigital.com

• @jacobcolie

• Tom Williamson

• CEO, Jackson River

• tom.williamson@jacksonriver.com

• @tomwilliamson, @jacksonriver

Slide 34A/B Testing

Evaluate This Session!Enter for a chance to win an NTEN engraved mini iPad! 

Or, search by session title at www.nten.org/ntc/eval