4
This is a tool used BEFORE any live, direct mail testing to pre-identify the best test ideas, those most likely to compete with and beat the control. Non-profits greatly reduce cost by NOT mailing test packages likely to perform poorly and increase net revenue by increasing volume on likely winners. The pre-identification of likely winners and losers is done in two parts 1) First, surveying donors who are representative of those who will receive the actual mailing, showing them visuals of the direct mail package and measuring preference using a very specific and battle tested methodology. 2) Using the survey data to build a statistical model to assign a score to every single element that was evaluated. This methodology is well established in the commercial sector and used by large, consumer companies (e.g. Coca Cola, General Mills, Proctor & Gamble) to guide product development for many of the sodas, cereals and detergents on grocery store shelves. Building a successful direct mail package is conceptually identical to a winning tube of toothpaste with size, shape, message and color considerations to name but a few of the moving parts you need to consider. A midsize, non-profit in the disease and medical research sector, had an acquisition control package with a family (whose child had the disease) who no longer wanted to featured. The organization faced a serious marketing challenge, identifying not just a new family to feature but one that could compete with the current control. They had several different families open to the possibility and a wildcard option of a researcher in the field. The cost to live test all the possibilities plus the control, even with their relatively small panel sizes (10k per), was going to be $46,000 and they only had budget to test 2 concepts against the control. DonorVoice used the pretest tool with a sample of house file donors (to evaluate renewal implications) and a universe of donors representing their acquisition target. We evaluated all the test options and, because this survey testing environment can accommodate far more moving partsthan could ever be afforded or logistically managed in live testing, we also included prospective taglines the organization was considering as additional test elements.

Pretest tool case studies

Embed Size (px)

Citation preview

Page 1: Pretest tool case studies

This is a tool used BEFORE any live, direct mail testing to pre-identify the best test ideas, those most

likely to compete with and beat the control. Non-profits greatly reduce cost by NOT mailing test

packages likely to perform poorly and increase net revenue by increasing volume on likely winners.

The pre-identification of likely winners and losers is done in two parts

1) First, surveying donors who are representative of those who will receive the actual mailing,

showing them visuals of the direct mail package and measuring preference using a very specific

and battle tested methodology.

2) Using the survey data to build a statistical model to assign a score to every single element that

was evaluated.

This methodology is well established in the commercial sector

and used by large, consumer companies (e.g. Coca Cola, General

Mills, Proctor & Gamble) to guide product development for

many of the sodas, cereals and detergents on grocery store

shelves.

Building a successful direct mail package is conceptually

identical to a winning tube of toothpaste with size, shape,

message and color considerations to name but a few of the

moving parts you need to consider.

A midsize, non-profit in the disease and medical research sector, had an acquisition

control package with a family (whose child had the disease) who no longer wanted

to featured.

The organization faced a serious marketing challenge, identifying not just a new

family to feature but one that could compete with the current control. They had

several different families open to the possibility and a wildcard option of a

researcher in the field.

The cost to live test all the possibilities plus the control, even with their relatively

small panel sizes (10k per), was going to be $46,000 and they only had budget to test

2 concepts against the control.

DonorVoice used the pretest tool with a sample of house file donors (to evaluate renewal implications)

and a universe of donors representing their acquisition target. We evaluated all the test options and,

because this survey testing environment can accommodate far more “moving parts” than could ever be

afforded or logistically managed in live testing, we also included prospective taglines the organization

was considering as additional test elements.

Page 2: Pretest tool case studies

The results identified one test family with real potential to beat the control and another with a

chance to be competitive. Conversely, it identified the rest as likely “losers”, including the one the

organization staff considered to be best.

The organization, with only enough budget for two test panels, elected to believe in the voice of the

donor as represented in the pretest findings even though their own personal preference and

internal, conventional wisdom would have led them to test different families.

And the results?

The Pretest tool identified Test A (one of the families) having a high likelihood to beat the

control and it did.

The Pretest Tool identified Test B as being competitive with, but still behind the control –

exactly what happened in the live test.

Savings from not producing & mailing all 6 test packages against the control $46,000

Anticipated annual revenue increase with rollout of better performing control $72,000

Peace of mind with having a control beating, new acquisition package

that would NOT have been identified without the pretest tool Priceless

Contact info: [email protected]

www.thedonorvoice.com

“We are extremely pleased with the information gathered

from the DonorVoice testing, which allowed us to solve a

tricky marketing dilemma with a reduced level of risk”

Page 3: Pretest tool case studies

This is a tool used BEFORE any live, direct mail testing to pre-identify the best test ideas, those most

likely to compete with and beat the control. Non-profits greatly reduce cost by NOT mailing test

packages likely to perform poorly and increase net revenue by increasing volume on likely winners.

The pre-identification of likely winners and losers is done in two parts

3) First, surveying donors who are representative of those who will receive the actual mailing,

showing them visuals of the direct mail package and measuring preference using a very specific

and battle tested methodology.

4) Using the survey data to build a statistical model to assign a score to every single element that

was evaluated.

This methodology is well established in the commercial sector

and used by large, consumer companies (e.g. Coca Cola, General

Mills, Proctor & Gamble) to guide product development for

many of the sodas, cereals and detergents on grocery store

shelves.

Building a successful direct mail package is conceptually

identical to a winning tube of toothpaste with size, shape,

message and color considerations to name but a few of the

moving parts you need to consider.

A large, well established non-profit helping children with disabilities

applied the pretest tool as a back-test to validate several test packages

for their year-end calendar appeal for renewal and acquisition.

The organization spends a lot of money testing packages that are

expensive to produce and fulfill given the use of front and back-end

premiums. The cost of production and fulfillment is magnified with

every test package that does not outperform the control, since money

was“left on the table.”

A mail package containing this many elements means an exponential

number of choices being made, whether they are all acknowledged or

not, about what to test. The inefficiency of A/B testing almost assures

a sub-optimal combination of all these choices.

Validating the accuracy of this tool against actual mail results means this organization can save hundreds

of thousands of dollars from mailing less AND mailing “better” by only choosing packages empirically

identified, from among thousands of possibilities, as having a chance to beat the control or deliver the

same results for less cost (production and fulfillment).

Page 4: Pretest tool case studies

DonorVoice used the pretest tool with a sample of house file donors (to evaluate renewal implications)

and a universe of donors representing the client’s acquisition target. We evaluated all the test package

elements including outer envelope color, letter and reply form changes, labels with different gender based

themes, calendar inserts and back end premium offers.

The pretest tool correctly identified winning and losing test packages with only one exception. The

tool correctly determined,

Outer envelope color matters to response but the test colors would not beat the control

Incremental calendar changes were no better than the control calendar

Back end premium offers beat the control with no back-end offer and that the two – duffle

and tote bag – were not equal performers.

Outer envelope personalization beats the control and is the 2nd most important element

after back-end premiums.

Applying these findings before doing the mailings means the “losing” tests could have been avoided

saving production time and money along with the opportunity cost of lost revenue for those mailed

poor performing tests instead of control. Furthermore, volume for the predicted, winning test ideas

could have been increased to take advantage of the better performance.

Savings from not producing & mailing predicted, losing test packages $45,000

Increase in net revenue by doubling panel volume on predicted winners $20,400

Increased year one annual revenue from getting to rollout volumes faster $257,000

Peace of mind knowing that, based on initial project insights,

client can use the tool to find premiums with lower cost and high response Priceless

Contact info: [email protected]

www.thedonorvoice.com

“The DonorVoice pretest tool is a dramatic improvement

to the testing process – we can test more ideas, faster and

cheaper than live testing and get to rollout faster by

putting more volume on the likely winners.