TESTING IS NOT CROBUT IT’S STILL DARN IMPORTANT

FIRST LET’S COVER SOME

A/B TESTING BASICS

Types Of Tests

Synchronous Asynchronous

Sequential Testing A/B/…/N TestingMultivariate

Testing

Test Types

Synchronous Testing

July 1st - July 15thJuly 16th - July 30th

A/b/…/n Split Testing

$$$$$$$$$$$$

$$$

A/b/…/n Testing

You’ll Get Page ConversionData (Not Element By Element)

Better For Low Traffic Sites

A/b/…/n Testing

Some Rules:Isolate Individual Elements

If You Don’t Isolate Elements, DO NOT attribute the lift or loss to any single element

Multivariate Testing

You’ll Get Element By Element Data

Don’t Run If Less Than 100k Pageviews Per Month

NOW LET’S GIVE A/B TESTING SOME LOVE

I BUILT MY CAREER AROUND SPLIT TESTING

IT’S THE ONLY WAY TO GET DEFINITIVE ANSWERS IN A

CLEAN ENVIRONMENT

TESTING HELPS GUIDE DECISIONS AND IMPROVE

THE USER EXPERIENCE WHILE BOOSTING PROFITS

TESTING GOT THE CONVERSATION

STARTED!

A/B TESTS ARE SEXY

89.97% LIFTIN

SALESSource: vwo.com

12.6% LIFTIN

SUBMISSIONS

Source: Unbounce.com

18.70% DROP

IN SUBMISSION

SSource: ContentVerve.com

19.47% LIFTIN

SUBMISSIONS

Source: ContentVerve.com

AWESOME STUFF RIGHT?

THERE’S ONE BIG PROBLEM

EVERYONE WANTS TO TEST

EVEN WHEN THEY CAN’T

SIMPLY PUT: NOT EVERYONE CAN OR

SHOULD BE SPLIT TESTING

NOT BEING ABLE TO TEST DOESN’T MEAN…

…NOT BEING ABLE TO OPTIMIZE

THERE ARE PLENTY OF WAYS TO BOOST CONVERSIONS

OUTSIDE OF SPLIT TESTING

TODAY IN A NUTSHELL1. When You Can Test2. Should You Actually Test3. What To Do When You

Can’t/Shouldn’t Test

Part PART 1Can I Actually Run A Test

Testing Eligibility

• ~100 Conversion Per Variation• Run for at least a full week • Not run longer than 8 weeks• Test can reach 95% confidence in this time frame

Testing Eligibility

• ~100 Conversion Per Variation• Run for at least a full week • Not run longer than 8 weeks• Test can reach 95% confidence in this time frame

Conversion Per Day Chart

Minimum Number Of Conversions Per Day

Test Period (Days)

2 Variants 3 Variants 4 Variants 5 Variants

7 29 43 58 72

14 15 22 29 36

21 10 15 20 24

28 8 11 15 18

35 6 9 12 15

42 5 8 10 12

49 5 7 9 11

56 4 6 8 9

Problem: Not Enough Conversions

Solution:Run Longer

Secondarily:If Outside Of 8 Week Testing Window - Don’t Test It

VARIABLE NUMBER ISN’T THE ONLY THING THAT DICTATES

TIME!

THE PERCENT LIFT IS A HUGE TIMING

FACTOR

In order for this test to get statistical results, it would require a 10% lift

AND SO IS THE CONFIDENCE RATE

But we’ll talk more about that later

Does Your Page Get Enough Daily Conversions?

YES No

Can’t Test - Sorry!Is There A Big Enough Lift Potential?*

YES No

This Is A Testable Page

Can’t Test - Sorry!

*With An Appropriate Confidence Rate/Index

Part PART 2Should I Actually Run That

Test

THE CONVERSION HIERARCHY

FunctionalDoes the page work and do what it’s supposed to?

Accessible Can people of all skill levels on any device use the site?

What are the barriers on the site?

UsableCan people understand your site, i.e., is it user friendly?

IntuitiveDoes the site have a logical flow?

Does the offer match the source?

PersuasiveIs your site convincing people to buy or take an action?

“For every test you run, there are an infinite number of other tests that you could have run” - Justindamus 120 AD

Part PART 3How To Optimize If I Can’t/Shouldn’t Test

You Have A Few Options

• Evaluate Your Offer• On-Site Retargeting• Changes Based On Qualitative Research

EVALUATE YOUR OFFER

It’s not a traffic problem, it’s an offer problem

It’s not a testing problem, it’s an offer problem

You might have to head back to the drawing board completely.

Does your product/service provide real value?

You also might be going after the wrong traffic source.

Identify if your ideal customer is ‘hanging out’ on the traffic medium

TRY ON SITE RETARGETING

What is On Site Retargeting?

Evaluating current interactions and providing a relevant offer based on these interactions.

Time On Site

Number Of Pageviews

Type Of Content Consumed

Exit Intent

Referring Source

Just Under Half Of All Offer Pageviews Sent By Optimonk

Typical Campaign Conversion:69%With Exit pop:80.7%

Using this method we got 2,689 more leads in just 14 days!

“Respondents targeting more than 20% of their visitors with personalized content achieved a 5% conversion rate, which was 2x the average conversion rate of 2.5% in their study” [Adobe 2014 Survey]

PAGE CHANGES BASED ON

OPTIMIZATION RESEARCH

Here are a few things to try

• Click/Scroll Tracking• Usability Testing• User Surveys• Session Recordings

CLICK/SCROLL TRACKING

People who actually scroll to the bottom clickNot enough people are scrolling that far down

~15% of all clicks on this page are on the nav bar

Try a shorter variation to get people to scroll to the CTA.

Add multiple CTA buttons at areas where scrolling is highest.

Hide the nav bar on this page

USABILITY TESTING

USER SURVEYS

• Was it was easy to find what you were looking for?• Were the words/vocab used to define categories/sub-categories clear to you?• Do you have any suggestions to improve our website navigation?• Does our website look credible to you?• Is our pricing clear?• Is there anything else you’d like to know before signing up with us?• Will you shop with us again? Why/why not?• Do you think the form has any confusing/unnecessary input fields?

QUESTION IDEAS

SESSION RECORDINGS

Shows individual recordings of a user interacting with your site.

Unlike heatmaps, this isn’t aggregated data

Really blow theirmind with a real life user recordingwith audio

TRICKS TO TEST WHEN YOU DON’T HAVE THE TRAFFIC

USE MICRO CONVERSIONS AS

INDICATORS

Test LegnthMinimum Conversions Per Variation

Per Day

7 Days 29

14 Days 15

21 Days 10

28 Days 8

35 Days 6

42 Days 5

49 Days 5

56 Days 4

LOWER THE CONFIDENCE THRESHOLD

Lift: 20%Confidence Rate: 95%_________________

Required Sample Size:7,682

Lift: 20%Confidence Rate: 90%_________________

Required Sample Size:6,052 21% Decrease

REMEMBER - SPLIT TESTING IS NOT OPTIMIZATION…

IT’S AN OPTIMIZATION TOOL

Split Testing

PersonalizationUser Surveys

Funnel Analysis

Web AnalyticsPersonas

Neurotesting Advanced Segments

Eye Tracking

Scroll Tracking

CRO

SPLIT TESTING CAN ACTUALLY HURT A

COMPANY

SO PRACTICE SAFE TESTING WITH A PAGE THAT

MATTERS

• Email: justin.rondeau@digitalmarketer.com• Twitter: @jtrondeau• Website: digitalmarketer.com

Justin RondeauOptimization Manager,

Digital Marketer

Let’s Talk More!