Building a Testing Playbook by Andrew Richardson

I CAN’T EAT LUNCH

YET

THIS GUY KEEPS

TALKING…

This is your

BRAIN

An attention span of

8 seconds

*https://advertise.bingads.microsoft.com/en-us/insights @analyticandrew #eMetrics

IS MY CONVERSION

RATE GOOD OR BAD?

WHY DO I NEED TO

TEST?

BUT NO ONE WANTS TO

DO TESTING AT MY

ORGANIZATION.

PLATFORM, PEOPLE,

PROCESS

WHAT DO I TEST AND

HOW DO I TRACK IT?

TIPS AND

SUGGESTIONS FOR

GETTING STARTED

TAKEAWAYS

@analyticandrew #eMetrics

IS MY CONVERSION

RATE GOOD OR BAD?

WHY DO I NEED TO

TEST?

BUT NO ONE WANTS TO

DO TESTING AT MY

ORGANIZATION.

PLATFORM, PEOPLE,

PROCESS

WHAT DO I TEST AND

HOW DO I TRACK IT?

TIPS AND

SUGGESTIONS FOR

GETTING STARTED

TAKEAWAYS

@analyticandrew #eMetrics

2.6%

2.8%

2.8%

3.2%

3.3%

3.4%

5.0%

5.0%

5.5%

6.0%

Higher Education

Real Estate

Health

Legal

Home Improvement

Business Services

Travel

Business Consulting

Credit & Lending

Vocational Studies & Job Training

*Unbounce Conversion Benchmark Report – March 2017

HOW DO YOU COMPARE?

Never stop

testing,

and your

advertising will

never stop

improving.

- David Ogilvy -

IS MY CONVERSION

RATE GOOD OR BAD?

WHY DO I NEED TO

TEST?

BUT NO ONE WANTS TO

DO TESTING AT MY

ORGANIZATION.

PLATFORM, PEOPLE,

PROCESS

WHAT DO I TEST AND

HOW DO I TRACK IT?

TIPS AND

SUGGESTIONS FOR

GETTING STARTED

TAKEAWAYS

@analyticandrew #eMetrics

DAYLIGHTYour ideas see

No more

HIPPO

5xHigher ROI than

PPC

Up to

IS MY CONVERSION

RATE GOOD OR BAD?

WHY DO I NEED TO

TEST?

BUT NO ONE WANTS TO

DO TESTING AT MY

ORGANIZATION.

PLATFORM, PEOPLE,

PROCESS

WHAT DO I TEST AND

HOW DO I TRACK IT?

TIPS AND

SUGGESTIONS FOR

GETTING STARTED

TAKEAWAYS

@analyticandrew #eMetrics

Not enough

TRAFFIC

We have

NO IDEAwhat to test

IS MY CONVERSION

RATE GOOD OR BAD?

WHY DO I NEED TO

TEST?

BUT NO ONE WANTS TO

DO TESTING AT MY

ORGANIZATION.

PLATFORM, PEOPLE,

PROCESS

WHAT DO I TEST AND

HOW DO I TRACK IT?

TIPS AND

SUGGESTIONS FOR

GETTING STARTED

TAKEAWAYS

@analyticandrew #eMetrics

Nothing should be

OFF LIMITS

T-shirt

TEST

size your

ING

Testing is often about

CHANGINGpolicy & perception

IS MY CONVERSION

RATE GOOD OR BAD?

WHY DO I NEED TO

TEST?

BUT NO ONE WANTS TO

DO TESTING AT MY

ORGANIZATION.

PLATFORM, PEOPLE,

PROCESS

WHAT DO I TEST AND

HOW DO I TRACK IT?

TIPS AND

SUGGESTIONS FOR

GETTING STARTED

TAKEAWAYS

@analyticandrew #eMetrics

Ok, I am

ALL INNow what?

Tool Selection:

BUILD or BUY

7 TOOLSMarTech 2011:

175 TOOLSMarTech 2017:

THE PEOPLEThe most important thing

THE PEOPLEThe most important thing

BUT!

THE PEOPLEThe most important thing

one: focus efforts where they matter most

two: avoid conflict-put opinions to the test

three: empower people to optimize & be data-driven

four: align teams around shared goals

five: collaborate around hypothesis generation,

test

ideas & execution

PROCESS

IS

CRUCIAL

FOR

SUCCESS

RESEARCH1.

PRIORITIZE2.

EXPERIMENT3.

ANALYZE, LEARN,

REPEAT

4.

RESEARCH1.

“If A/B tests are used in lieu of research, the variations

are essentially guesses. We can improve outcomes of

A/B tests by incorporating UX research to:

•improve cause identification

•develop more realistic hypotheses

•identify more opportunities for experimentation”

Nielsen Norman Group: www.nngroup.com

RESEARCH1.

RESEARCH1.

RESEARCH1.

RESEARCH1.

RESEARCH1.

A theory can be

proved by

experiment; but no

path leads from

experiment to the

birth of a theory.

RESEARCH1.

Issue

• What is the conversion problem?

Cause Theory

• What do you think is causing the problem?

Variation Hypotheses

• What do you propose changing?

Goals

• What outcomes are expected?

RESEARCH1.

Issue

• What is the conversion problem?

Cause Theory

• What do you think is causing the problem?

Variation Hypotheses

• What do you propose changing?

Goals

• What outcomes are expected?

RESEARCH1.

RESEARCH1.

User-Experience to Improve Optimization Testing

• Defining user intent and objections• Why did you visit? Were you successful? If not, why?

• Exposing interface flaws• Changing colors vs. changing form fields

• Measuring findability• Tree testing, site search, heatmaps

• Cleaning up design variations• Reduce friction, maximize testing

RESEARCH1.

Bug tracking Brainstorming Site Search

Web Analytics Site Surveys Chat

PRIORITIZE2.

• TEST• Where you put stuff for testing

• INSTRUMENT• Fixing, adding, improving tagging, events, etc.

• HYPOTHESIZE• Found something, but no clear solution

• JUST DO IT (JDI)• No brainers

• INVESTIGATE• Need to dig further, ask more questions

Craig Sullivan: @optimiseordie

PRIORITIZE2.

PRIORITIZE2.

Craig Sullivan Hypothesis Kit

one: Because we saw (data/feedback)

two: We expect that (change) will cause (impact)

three: We’ll measure this using (data metric)

https://medium.com/@optimiseordie/hypothesis-kit-2-eff0446e09fc

PRIORITIZE2.

Craig Sullivan Advanced Kit

one: Because we saw (qual and quant data)

two: We expect that (change) for (population)

will cause (impact(s))

three: We except to see (data metric(s)

change) over a period of (x business

cycles)

https://medium.com/@optimiseordie/hypothesis-kit-2-eff0446e09fc

EXPERIMENT3.

EXPERIMENT3.

EXPERIMENT3.

EXPERIMENT3.

ANALYZE, LEARN,

REPEAT

4.

Insert random cycle chart here

IS MY CONVERSION

RATE GOOD OR BAD?

WHY DO I NEED TO

TEST?

BUT NO ONE WANTS TO

DO TESTING AT MY

ORGANIZATION.

PLATFORM, PEOPLE,

PROCESS

WHAT DO I TEST AND

HOW DO I TRACK IT?

TIPS AND

SUGGESTIONS FOR

GETTING STARTED

TAKEAWAYS

@analyticandrew #eMetrics

• Headlines

• Sub headlines

• Paragraph Text

• Testimonials

• Call to Action text

• Call to Action Button

• Links

• Images

• Content near the

fold

• Social proof

• Media mentions

• Awards and badges

DESKTOPSegment

From

MOBILE

SWEATDon’t

the small stuff

IS MY CONVERSION

RATE GOOD OR BAD?

WHY DO I NEED TO

TEST?

BUT NO ONE WANTS TO

DO TESTING AT MY

ORGANIZATION.

PLATFORM, PEOPLE,

PROCESS

WHAT DO I TEST AND

HOW DO I TRACK IT?

TIPS AND

SUGGESTIONS FOR

GETTING STARTED

TAKEAWAYS

@analyticandrew #eMetrics

audience.ask(questions);

@analyticandrew #eMetrics

Never stop

testing,

and your

advertising will

never stop

improving.

- David Ogilvy -