1
Direct Mail Testing:Innovations and Insightsfor Challenging Markets
• Richard Varey Vi P id t C M k ti
2
• Richard Varey, Vice President, Consumer MarketingFinancial Times
• Chris Law, Senior Director, Marketing and AnalysisTHD
• Gordon Bell, PresidentLucidView
Tuesday, October 16, 2007
Scientific DM Testing – Opportunity
• Perhaps one time in fifty a guess may be right. But fiftytimes in fifty an actual test tells you what to do and avoid.
(Claude Hopkins, My Life in Advertising, 1927)
• The more you test, the more profitable your direct mail
3
will become. Innovations, provided you test them, canwork wonders.
(David Ogilvy, Ogilvy on Advertising, 1983)
• “Test everything” has been a rallying cry in the marketingand advertising industry throughout the 20th century.
(Ledolter & Swersey, Testing 1-2-3, 2007)
Scientific DM Testing – Challenge• Improving upon success
The secret of all effective originality in advertising is not thecreation of new and tricky words and pictures, but one of puttingfamiliar words and pictures into new relationships. (Leo Burnett)
• Constant change in the marketplaceIt is not the strongest of the species that survives nor the most
4
It is not the strongest of the species that survives, nor the mostintelligent that survives. It is the one that is the most adaptable tochange. (Charles Darwin)
• Tight budgetsThe man who stops testing to save money is like the manwho stops the clock to save time.
• RiskNo one tests the depth of a river with both feet. (African proverb)
Scientific Testing = structured frameworkbeneath each campaign…
5
… But in the marketplace it can get messy
6
Strategic Testing – approach
1. Robust statistical design underlying each test2. Clear, consistent execution3 Fast tests � short term metrics
7
3. Fast tests � short-term metrics4. Strategic cycle of tests
• Screen � Refine � Adapt
Strategic Testing – test designs
• Split-run tests• Scientific multivariable tests
Key decisions
8
– Key decisions• Number of test elements• Levels for each element (optimal = 2)• Effects of interest: main effects and/or interactions• Number of recipes (test cells)• Statistical complexity
THD: Approach to DM testing
• Philosophy: Testing is paramount to a successfuland ever-growing file.
• Live by the adage: Test, test more, then test again• Become intimately involved in your client’s data
9
Become intimately involved in your client s data– Gain a crystal clear understanding of all that has occurred
in the past and use that learning to develop future testingstrategies.
• Everything is imminently testable: Creative, Offer,Message, Ask, Audience, etc..
• Apply same vigilance and methodical approach inall facets of the program
Case study #1: Non-profit fundraising
• Test was for the American Lung Association– Holiday name sticker renewal campaign– 3.1 million pieces mailed in the campaign
10
• One 15-element test within this campaign– Mailed in 16 different recipes– 500,000 mailed within this test
• Challenge: improve upon on-going creativetests
Test ElementsTest Elements (-) Control (+) New Idea
A Address label on envelope Window Mailing label on closed faceB Envelope graphic and teaser Control (label graphic and teaser) No art, new teaserC "Urgent" stamp on envelope No stamp "Urgent" stamp on OED Copy theme Control copy "Urgent" copy themeE Campaign name Generic, unbranded appeal Holiday Drive campaign nameF Add deadline No deadline "Please respond by" date
11
F Add deadline No deadline Please respond by dateG Gift ask Control ask array One gift askH Accentuate gift ask Control reply slip Highlight gift askJ Check boxes on reply slip Control (check boxes on front) Move check boxes to backK Backer color Control (black) Red and blackL Label design Control label design New label designM Monograms on labels Labels with monograms No monogramsN Additional labels Control (6 panel form) Add more labelsO Lift note None Add lift noteP Reply envelope CRE ("place stamp here") BRE (postage paid)
Creatives – control package (recipe 2)
12
Creatives – control package (recipe 2)
13
Recipe #8 outer envelope (A+, B+, C+, L-)
(� versus control OE)
A+
14
Recipe #8 outer envelope (A+, B+, C+, L-)
(� versus control OE)
15
B+
Recipe #8 outer envelope (A+, B+, C+, L-)
(� versus control OE)
C+
16
Recipe #6 outer envelope (A-, B+, C-, L+)
(� versus control OE)
17
A-L+
Recipe #3: reply slip and copy (D+, E-, F+, G+, H+, J+)
G+
J+
H+
18
D+
F+
Test Design
Addr
ess
labe
l on
enve
lope
Enve
lope
gra
phic
and
teas
er
"Urg
ent"
stam
p on
env
elop
e
Cop
y th
eme
Cam
paig
n na
me
Add
dead
line
Gift
ask
Acce
ntua
te g
ift a
sk
Che
ck b
oxes
on
repl
y sl
ip
Back
er c
olor
Labe
l des
ign
Mon
ogra
ms
on la
bels
Addi
tiona
l lab
els
Lift
note
Rep
ly e
nvel
ope
(disguised data)
Recipe A B C D E F G H J K L M N O P Volume Responserate
AvgGift
19
rate Gift1 + - + + - - + - + + - - + + - 31250 1.24 2.7%2 - - - - - - - - - - - - - - - 31250 1.00 0.0%3 - - - + - + + + + - - + - + + 31250 0.91 3.9%4 - + - + + - + - - + - + + - + 31250 0.99 9.6%5 - - + - + - + + + - + - + - + 31250 1.09 7.0%6 + - - + + - - + - + + - - + + 31250 0.96 3.2%7 - + - - + + - + + + - - + + - 31250 1.19 3.4%8 + + + + + + + + - - - - - - - 31250 1.09 1.8%9 + + - + - + - - + - + - + - + 31250 0.93 4.5%10 + - + - - + - + - + - + + - + 31250 1.12 5.6%11 + - - - + + + - + + + + - - - 31250 1.00 4.1%12 - + + - - + + - - + + - - + + 31250 0.80 8.9%13 - + + + - - - + + + + + - - - 31250 0.95 5.2%14 - - + + + + - - - - + + + + - 31250 1.09 6.3%15 + + + - + - - - + - - + - + + 31250 0.91 4.9%16 + + - - - - + + - - + + + + - 31250 1.09 6.3%
500,000 1.02 4.8%
ALA test results: 15 main effects (and interaction)ALA holiday test: Response Rate
optimal = 30.1% lift
--
-
++
+
A: OE address label (new)H: Accentuate gift ask (yes)
B: OE graphic/teaser (control)L: Label design (control)
P: Reply envelope (CRE)N: Additional labels (yes)
LM t i t ti ( t )
20
-
-
-
+
++
++
+
O: Lift noteD: Copy theme
J: Check boxes on reply slipG: Gift ask
F: Add deadlineK: Backer color
C: OE urgent stamp (yes)M: Label monograms (yes)
E: Campaign name (yes)A: OE address label (new)
Significant effects(beyond line)
LM two-way interaction (response rate)
L-: Control L+: New DesignLabel Design
M-: Include monograms M+: No monograms
ALA test results: Net RevenueALA holiday test: Net/M
--
+-
+-
M: Label monograms (yes)
L: Label design (control)
H: Accentuate gift ask (yes)
B: OE graphic/teaser (control)
E: Campaign name (yes)
P: Reply envelope (CRE)
21
-++
+-+-
++
J: Check boxes on reply slip
O: Lift note
N: Additional labels
K: Backer color
F: Add deadline
A: OE address label
D: Copy theme
C: OE urgent stamp (yes)
G: Gift ask (one)
Response Rate versus Net Revenue
ALA holiday test: Response Rateoptimal = 30.1% lift
+
++
--
-
A: OE address label (new)H: Accentuate gift ask (yes)
B: OE graphic/teaser (control)L: Label design (control)
P: Reply envelope (CRE)N: Additional labels (yes)
ALA holiday test: Net/M (in same order as response)
++
--
-+
22
+
++
++
+
-
-
-O: Lift note
D: Copy themeJ: Check boxes on reply slip
G: Gift askF: Add deadlineK: Backer color
C: OE urgent stamp (yes)M: Label monograms (yes)
E: Campaign name (yes)A: OE address label (new)
Significant effects(beyond line)
+-
-+
-+
+
-+
Test Design – pros and cons
Pros• Tested 16 out of 32,768 possible combinations
– Yet could predict outcome for all other combos• Quicker to market – confidence in results allows for larger
roll-outsEach main variable tested to over half of test volume or 250 000 donors
23
– Each main variable tested to over half of test volume or 250,000 donors• In contrast, split-run testing:
– Limits the number of variables tested or else requires sample size> 4.7 million or 15 drops for equal confidence
– Cannot analyze interactionsCons• Greater cost and effort than split-run testing• Complex statistical test design requires guidance
THD: Agency Insights
• Don’t be afraid of the water– It wasn’t as bad as it looks!
• Production staff needs to be relied upon
24
• Production staff needs to be relied uponmuch more than current process
• Need a capable vendor network– Data processing, printing, & lettershop
Financial Times:campaign test strategy
• The Financial Times (FT) is a daily newspaper offering businessnews and analysis to a global audience, with a third of itscirculation in the US. For subscription growth, direct mail is stillthe primary DTP source.
• Direct mail strategy relies on database marketing (modeling
25
• Direct mail strategy relies on database marketing (modeling,optimization) and continual testing, historically on a split-run basis.
• The prime control package has been a voucher or statement ofbenefits format for 5 years, with a “hard” offer of $49 for 26 weeks(cash with order or bill-me).
• A second “soft” control, a different package with a 4 week risk-freetrial offer, is used selectively on some lists.
• In this period we increased mail volume, and improved efficiency,but to quicken the process, decided to try testing with LucidView.
Case study #2: Subscriptions DM test
• The Financial Times• “Hard offer” direct mail package• 1 creative test + 1 price test grid
26
• 1 creative test + 1 price test grid• Challenge: allow adequate sample size with
relatively low response rates• Challenge: how to improve already-strong
control
Creative test – elements
Test Elements (-) Control (+) New Idea
A Live stamp Meter StampB Teaser on envelope (front and back) No teaser Add teasers (front and back)C Picture of newspaper No Yes, add graphic of newspaperD Benefits copy Control New benefits copy
27
E Reply slip layout Control New layoutF Reply slip creative Control (Prof Rate Offer) New (green Savings Certificate)G "Reply by" date Control Hype deadlineH Call-to-action Control (no URL) Add URLJ Request e-mail address No YesK Brochure Control (mini-newspaper) New brochureL Letter from Robin No Yes, add letter
Creative test – design & recipesLi
ve s
tam
pTe
aser
s on
env
elop
ePi
ctur
e of
new
spap
erBe
nefit
s co
pyR
eply
slip
layo
utR
eply
slip
cre
ativ
e"R
eply
by"
dat
eC
all-t
o-ac
tion
Req
uest
e-m
ail a
ddre
ssBr
ochu
reLe
tter f
rom
Rob
in
Recipe A B C D E F G H J K L1 – – – – – – – – – – –2 + + + – + + – + – – –3 + – + + – + – – – + +4 – – – + + + – + + – +5 + + – + + – + – – – +
Package A C D ERecipe 1(control) Recipe 2 Recipe 3 Recipe 4
A: Live stamp Meter Stamp Stamp Meter
B: Teasers onenvelope No teaser Add teasers
(front and back) No teaser No teaser
C: Picture ofnewspaper No Yes, add graphic
of newspaperYes, add graphic
of newspaper No
D: Benefits copy Control Control New benefits copy New benefits copy
28
6 + – – – + + + – + + –7 + – + – – – + + + – +8 – + + – + – – – + + +9 – – + + + – + + – + –10 – + – – – + + + – + +11 – + + + – + + – + – –12 + + – + – – – + + + –
E: Reply sliplayout Control New layout Control New layout
F: Reply slipcreative
Control(Prof Rate Offer)
New (greenSavings Certificate)
New (greenSavings Certificate)
New (greenSavings Certificate)
G: "Reply by"date Control Control Control Control
H: Call-to-action Control(no URL) Add URL Control
(no URL) Add URL
J: Requeste-mail address No No No Yes, request
e-mail address
K: Brochure Control(mini-newspaper)
Control(mini-newspaper) New brochure Control
(mini-newspaper)L: Letter fromRobin No No Yes, add letter Yes, add letter
Creatives – control package (recipe 1)
29
Recipe #4 (C-, D+, E+, F+, G-, H+, J+) and #7 (C+, D-, E-, F-, G+, H+, J+)
30
FT creative test: 11 main effectsFT hard offer Creative Test: Gross response rate
+
+
+
-
-
C: Picture of newspaper
E: Reply slip layout
D: Benefits copy
B: Teasers on envelope (none)
F: Reply slip creative (control) 11.3% drop in response versus the control
9.8% drop
8.4% lift
Positive, butnot significant� t it
31
-
-
+
-
-
+
+
J: Request e-mail address
L: Letter from Robin
K: Brochure
G: Reply by date
A: Live stamp
H: Call-to-action
C: Picture of newspaper
Significant effects(beyond line)
� opportunityfor “refining”
Price test – elements
Test Elements (-) Control (+) New Idea
A Price $49 for 26 weeks $99 for 12 monthsB Longer term (+25%) No Yes (32 weeks / 15 months)C Shi i & h dli f N Y $4 95
32
C Shipping & handling fee No Yes, $4.95D Streamline price box
(3 lines vs 5)No Yes, redesign price box and
reply slip
Price test – design & recipesPr
ice
Long
er te
rm (+
25%
)
S&H
(del
iver
y) fe
e
Stre
amlin
e pr
ice
box
Recipe A B C D1 – – – –2 + – – +
A P R SRecipe 1(control) Recipe 2 Recipe 3 Recipe 4
A: Price $49 $99 $49 $99
B: Longer term(+25%)
No($49 for 26 weeks)
No($99 for 12 months)
Yes($49 for 32 weeks)
Yes($99 for 15 months)
C: S&H (delivery)fee No No No No
D: Streamlineprice box No Yes, redesign price
box and reply slipYes, redesign pricebox and reply slip No
33
3 – + – +4 + + – –5 – – + +6 + – + –7 – + + –8 + + + +
price box box and reply slip box and reply slip
T U V W
Recipe 5 Recipe 6 Recipe 7 Recipe 8
A: Price $49 $99 $49 $99
B: Longer term(+25%)
No($49 for 26 weeks)
No($99 for 12 months)
Yes($49 for 32 weeks)
Yes($99 for 15 months)
C: S&H (delivery)fee Yes, $4.95 Yes, $4.95 Yes, $4.95 Yes, $4.95
D: Streamlineprice box
Yes, redesign pricebox and reply slip No No Yes, redesign price
box and reply slip
Creatives – control (recipe 1) vs. “all plus” recipe 8
34
FT price test: main effects & interactionsFT hard offer Price Test: Gross response rate
optimal = 32.7% lift
-
-
+
BC interaction
C: Shipping &handling fee (no)
B: Longer term(yes, +25%)
18.9% increase in response vs the control
17.2% drop
13.8% further increase in response if...BC Interaction
35
+
+
-
-
AB interaction
AC interaction
A: Price
D: Streamline pricebox
B-: Current(26 wk / 12 mo)
B+: Longer term (+25%)
Subscription Term
C-: No S&H fee C+: Add $4.95 S&H fee
Overall Test Results
• 5 significant main effects + 1 interaction• 41% overall increase in net response
• Creative test: 13% increase in net response (call-to-action: add URL)+ 7% increase in % payup
• Price test: 28% increase in net response
36
• Price test: 28% increase in net response
• If these 15 elements were tested with split-run techniques…• Not one effect would have been statistically significant• Interactions would be impossible to see• Only 3 or 4 elements could be tested with equal confidence
The Financial Times:Insights & advice
• Faster deeper learning: we raised the number of test panels nearly3-fold to 19, but tested 2,064 combinations. LucidView gave scopeto analyze main effects, comparative effects and multipleinteractions with confidence.
• Initial investment (consultancy & production) pays off on roll out
37
• Initial investment (consultancy & production) pays off on roll-out.• Get brainstorming participation from a wide group, not just the
direct marketers, and think bold. Use LucidView’s flexibility totest a wide range of designs. Scale the experiment to your budget.
• Expect a few variables to make a positive difference, a fewnegative and most none.
• Continue: next steps to confirm and refine tests (Fall 07).• If you are not a statistical genius, consult an expert, ideally with the
relevant industry experience.
Direct Mail Testing:Innovations and Insightsfor Challenging Markets
• Richard Varey Vi P id t C M k ti
38
• Richard Varey, Vice President, Consumer Marketing,Financial Times
• Chris Law, Senior Director, Marketing and AnalysisTHD
• Gordon Bell, PresidentLucidView
Tuesday, October 16, 2007