Testing Beyond The Button Color - The Definitive Guide to A/B Testing

TESTING BEYOND THE BUTTON COLOR

The definitive guide to A/B testing

Ron Schneidermannhttps://angel.co/ron-schneidermann

https://www.linkedin.com/in/schneidermann

The Easy StuffPart 1

What is A/B testing?

A/B testing (also known as split testing or bucket testing) is a method of

comparing two versions of a webpage or app against each other to determine

which one performs better. 

optimizely.com/ab-testing/

Alright, that’s easy enough.

But tell me again why I care about this?

But the real secret…

Data > ego

Ego in a startup???

The truth is you don’t know shit.

And I don’t know shit.

But your users know what’s up

Three Pre Testing Questions

Part 2

c/o http://www.coelevate.com/essays/5-steps-to-choose-your-customer-acquisition-channel

Question #1:Are we at the right phase in our

lifecycle for testing?

Question #2:What are we optimizing for?

Different phases of your company lifecycle call for optimizing different things:

Different phases of your company lifecycle call for optimizing different things:

Learning

Different phases of your company lifecycle call for optimizing different things:

LearningProduct channel fit?

Different phases of your company lifecycle call for optimizing different things:

LearningProduct channel fit?Audience segmentation?

Different phases of your company lifecycle call for optimizing different things:

LearningProduct channel fit?Audience segmentation?Engagement?

Different phases of your company lifecycle call for optimizing different things:

LearningProduct channel fit?Audience segmentation?Engagement?

Monetization?

Different phases of your company lifecycle call for optimizing different things:

LearningProduct channel fit?Audience segmentation?Engagement?

Monetization?Conversion rate?

Different phases of your company lifecycle call for optimizing different things:

LearningProduct channel fit?Audience segmentation?Engagement?

Monetization?Conversion rate?Messaging?

Different phases of your company lifecycle call for optimizing different things:

Learning

Volume

Different phases of your company lifecycle call for optimizing different things:

Learning

VolumeHow far can I crank the dial on a given channel?

Different phases of your company lifecycle call for optimizing different things:

Learning

Volume

Profitability

Different phases of your company lifecycle call for optimizing different things:

Learning

Volume

ProfitabilityCan I drive down CPAs, reduce churn, increase AOV?

Question #3:What are our constraints?

Time

Money

Target Audience

Legal

Part 3

How to Find Good Test

Hypotheses

Analytics

User testingWhat you think your users do

What your usersactually do

Heat Map Tracking

Qualitative Feedback

Competitor Research

Ideas to TestPart 4

1. Landing page ideas to test:

Messaging

1. Landing page ideas to test:

Messaging

Form fields

1. Landing page ideas to test:

Messaging

Layout (long scroll vs. short, # of columns)

Form fields

1. Landing page ideas to test:

Messaging

Layout (long scroll vs. short, # of columns)Plan background vs. rich imagery

Form fields

1. Landing page ideas to test:

Messaging

Layout (long scroll vs. short, # of columns)Plan background vs. rich imagery# of CTAs

Form fields

1. Landing page ideas to test:

Messaging

Layout (long scroll vs. short, # of columns)Plan background vs. rich imagery# of CTAs

CTA copy

Form fields

1. Landing page ideas to test:

Messaging

Layout (long scroll vs. short, # of columns)Plan background vs. rich imagery# of CTAs

CTA copy

Form fields

Social proof (testimonials/endorsements/as-seen-in)

1. Landing page ideas to test:

Messaging

Layout (long scroll vs. short, # of columns)Plan background vs. rich imagery# of CTAs

CTA copy

Video vs. none

Form fields

Social proof (testimonials/endorsements/as-seen-in)

1. Landing page ideas to test:

Messaging

Layout (long scroll vs. short, # of columns)Plan background vs. rich imagery# of CTAs

CTA copy

Access to site wide navigation vs. locked down navigation

Video vs. none

Form fields

Social proof (testimonials/endorsements/as-seen-in)

1. Landing page ideas to test:

II. Home page ideas to test:

Messaging

II. Home page ideas to test:

Messaging

Imagery

II. Home page ideas to test:

Messaging

Layout (long scroll vs. short, # of columns)

Imagery

II. Home page ideas to test:

Messaging

Layout (long scroll vs. short, # of columns)Primary navigation (search vs. browse)

Imagery

II. Home page ideas to test:

Messaging

Layout (long scroll vs. short, # of columns)Primary navigation (search vs. browse)# of CTAs

Imagery

II. Home page ideas to test:

Messaging

Layout (long scroll vs. short, # of columns)Primary navigation (search vs. browse)# of CTAs

CTA copy

Imagery

II. Home page ideas to test:

Messaging

Layout (long scroll vs. short, # of columns)Primary navigation (search vs. browse)# of CTAs

CTA copy

Social proof (testimonials/endorsements/as-seen-in)

Imagery

II. Home page ideas to test:

Messaging

Layout (long scroll vs. short, # of columns)Primary navigation (search vs. browse)# of CTAs

CTA copy

Testimonials/endorsements/as-seen-inVideo vs. none

Imagery

II. Home page ideas to test:

III. Checkout flow ideas to test:

Form fields and required steps

III. Checkout flow ideas to test:

Form fields and required steps

Fewer long pages vs. more short pages

III. Checkout flow ideas to test:

Form fields and required steps

Access to site wide navigation vs. locked down navigation

Fewer long pages vs. more short pages

III. Checkout flow ideas to test:

Form fields and required steps

Access to site wide navigation vs. locked down navigationBreadcrumbs

Fewer long pages vs. more short pages

III. Checkout flow ideas to test:

Form fields and required steps

Access to site wide navigation vs. locked down navigationBreadcrumbs

Cross-selling of other products

Fewer long pages vs. more short pages

III. Checkout flow ideas to test:

Form fields and required steps

Access to site wide navigation vs. locked down navigationBreadcrumbs

Cross-selling of other products

Phone number/live chat for assisted bookings vs. self serve

Fewer long pages vs. more short pages

III. Checkout flow ideas to test:

Form fields and required steps

Access to site wide navigation vs. locked down navigationBreadcrumbs

Cross-selling of other products

Phone number/live chat for assisted bookings vs. self servePost-conversion marketing

Fewer long pages vs. more short pages

III. Checkout flow ideas to test:

IV. Product/pricing ideas to test:

Free trial

IV. Product/pricing ideas to test:

Free trial

Money back guarantee

IV. Product/pricing ideas to test:

Free trial

Money back guarantee

Annual subscription vs. monthly

IV. Product/pricing ideas to test:

V. Email ideas to test:

Subject line

V. Email ideas to test:

Subject line

Content (short vs. long, imagery vs. plain text, etc.)

V. Email ideas to test:

Subject line

# of CTAs

Content (short vs. long, imagery vs. plain text, etc.)

V. Email ideas to test:

Subject line

# of CTAs

From name

Content (short vs. long, imagery vs. plain text, etc.)

V. Email ideas to test:

Subject line

# of CTAs

From name

Send time

Content (short vs. long, imagery vs. plain text, etc.)

V. Email ideas to test:

Subject line

# of CTAs

From name

Send time

Cart abandonment

Content (short vs. long, imagery vs. plain text, etc.)

V. Email ideas to test:

Subject line

# of CTAs

From name

Send time

Cart abandonment

Content (short vs. long, imagery vs. plain text, etc.)

Post purchase offer

V. Email ideas to test:

How to Set Up Your Tests

Part 5

Pick a platform that works for you

https://conversionxl.com/conversion-optimization-tools/

http://mashable.com/2015/01/30/ab-testing-tools/

http://conversionsciences.com/blog/ab-testing-tools/

http://www.conversion-rate-experts.com/split-testing-software/

Generate Hypotheses

Prioritize

Create Variations

Run Experiment

Analyze Results

Watch out for the gotchas!

Watch out for the gotchas!Don’t test too many variations concurrently!

Watch out for the gotchas!Don’t test too many variations concurrently!Don’t end your test too early!

Watch out for the gotchas!Don’t test too many variations concurrently!

Don’t ignore external factors!

Don’t end your test too early!

Watch out for the gotchas!Don’t test too many variations concurrently!

Don’t ignore external factors!

Don’t give up after the first test!

Don’t end your test too early!

Additional Reading

Appendix

http://www.coelevate.com/essays/5-steps-to-choose-your-customer-acquisition-channelhttp://www.coelevate.com/essays/traction-vs-growthhttp://www.coelevate.com/essays/customer-acquisition

https://conversionxl.com/12-ab-split-testing-mistakes-i-see-businesses-make-all-the-time/https://conversionxl.com/ab-testing-guide/

http://www.conversion-rate-experts.com/understanding-your-visitors/http://www.conversion-rate-experts.com/website-usability/

https://moz.com/blog/most-entertaining-guide-to-landing-page-optimization

ANY QUESTION

S?

GOOD LUCK!

Ron Schneidermannhttps://angel.co/ron-schneidermann

https://www.linkedin.com/in/schneidermann