The Case for Moderation

The case for!moderation !!A view on pre-testing research!!!James Hurman!Head of Planning, Colenso BBDO, New Zealand!Author of The Case for Creativity

Disclaimer…

This isn’t an attempt to persuade you to abandon pre-testing research! It’s a discussion about how best to interpret and apply the conclusions and recommendations that come out of pre-testing research.

Why are great brands so skeptical of pre-testing?

“We don’t ask consumers what they want. They

don’t know. Instead we apply our brain power to what they need and will want

and make sure we’re there, ready.”

- Akio Morita, Sony

founder

“We do no market research. We figure out what we want. And I think we’re pretty good at

having the right discipline to think through whether a lot of other people are going to want it too. That’s what we

get paid to do.”

- Steve Jobs, former Apple CEO

“We never pretested anything we did at Nike,

none of the ads. Dan Wieden (the founder of Nike’s agency Wieden & Kennedy) and I had an agreement that as long as our hearts beat, we would never pretest a word of copy. It makes you dull. It makes you

predictable.”

- Scott Bedbury, Nike’s former worldwide

advertising director

“Geoff believes research is a blunt

instrument that bludgeons good

ideas to death. He was determined that

42 Below would never be subjected

to pre-testing.”

-  Justine Troy on Geoff Ross,

founder of 42 Below Vodka

We humans aren’t great at picking successes…

“It’s a great invention, but who would want to use it?” -  US President Rutherford Hayes

Six years later, as the telephone was transforming America… “The Americans have need of the telephone, but we do not. We have plenty of messenger boys.” - British Post Office’s Chief Engineer

“Everyone acquainted with the subject will recognize it (the light bulb) as a conspicuous failure.”

- The President of Stevens Institute of Technology, notable for producing several Nobel Prize winners, 1880


“We don’t like their sound, and guitar music is on the way out anyway.”

Decca Records’ leading A&R man explaining his rejection of the Beatles, 1962


“The market researchers concluded that no other product had ever performed so poorly in consumer testing: the look, taste and mouth-feel were regarded as ‘disgusting’ and the idea that it ‘stimulates mind and body’ didn’t persuade anyone that the taste was worth tolerating.” - Philip Graves on Red Bull in his book Consumer.ology

In the two decades that followed, Red Bull sold over three billion cans of its ‘disgusting’ drink, achieving sales of €2.6B.

Source: Consumer.ology by Philip Graves


“Americans aren’t interested in Swedish vodka, with many people unaware of where Sweden even is.”

- Conclusion of Manhattan’s Carillon Importers $80,000 Absolut Vodka pre-testing research project

Absolut went on to sell over 70 million litres of vodka to the US annually.

Source: Consumer.ology by Philip Graves


A history of pre-testing research

•  Conceived in the 1950’s by American psychologist Horace Schwerin

•  He created a research product he called ‘persuasion testing’ and sold it to advertisers as a way to measure the potential sales impact of an advertisement

•  The method was analysed by university researchers in 1965 and found to be barely more reliable than flipping a coin.

Source: Excellence in advertising: the IPA guide to best practice by Leslie Butterfield, p17


•  In the 1990s, Beecham (now GSK), carried out a long term global review of advertising testing methods.

•  They concluded “It ought to be emphasised that no reliable pre-testing technique exists for assessing the sales effectiveness of a specific advertisement.”

•  In 2004, researchers from the London Business School noted that there was still “no evidence in the public domain that pre-testing is predictive.”

Source: Why Pre-Testing is Obselete by Tim Broadbent, published in Admap Magazine, October 2004


•  In 2007, the UK’s Institute of Practitioners in Advertising produced the largest ever study of historical marketing effectiveness case studies (880 in total). They compared pre-tested campaigns with those that weren’t pre-tested.

•  “Beware of pre-testing”, the study concluded. “If pre-testing really did lead to more effective campaigns, then one would expect cases that reported favourable pre-testing outcomes to show bigger effects than those that did not. In fact the reverse is true. Cases that reported favourable pre-testing results actually did significantly worse than those that did not.”

•  This shows that the judgement of marketers is in fact much more reliable than positive pre-testing outcomes

Source: “Marketing in the Era of Accountability” by Les Binet & Peter Field


“I can never get a positive result. No matter how I cut the data, no matter how I stack the odds in favour of pretesting by doing fine cuts of the data, I can never get a result that says that pre-tested campaigns are more effective than non-pretested campaigns.” “If pre-testing really did work, we should at least get some positive correlations, but we only ever get negative ones. After a while you think, well, there’s an obvious conclusion to draw from all of that.”

-  Peter Field, author of ‘Marketing in the Era of Accountability’

Source: Interview with the author

Equally, there are all sorts of pre-testing successes

Among others, Cadbury Gorilla and Old Spice flew through pre-testing, and went on to become highly effective campaigns.

It isn’t that pre-testing always gets it wrong.

It’s just very difficult to predict whether the pre-testing conclusions are right or wrong.

So why is pre-testing so fickle?

We are biased toward the familiar

American social psychologist Robert Zajonc studied what he called ‘the exposure effect’ in the 1970s. His experiments showed that simply exposing subjects to a familiar stimulus led them to rate it more positively than other, similar stimuli that had not been previously presented.

In one experiment, people were shown a random sample of squiggle drawings. Some time later, they were shown the same sample, but this time the squiggles were placed randomly among a selection of other, similar squiggles. The subjects were asked whether they could remember which of the squiggles were the ones they were previously shown. As you’d expect, they had a hard time with the exercise and rarely chose correctly. Then they were asked to show the interviewer which squiggles they preferred. They found this test considerably easier, and unbeknownst to them, chose the squiggles that they’d seen the first time around.

Zajonc’s work concluded that people tend to develop a preference for things merely because they’re familiar with them.

Source: Affective Discrimination of Stimuli That Cannot Be Recognized”, Kunst-Wilson & Zajonc, published in Science, Vol. 207

We tend to be wrong about what we think we want

Google asked customers how many results they wanted the search engine to throw back on the first page.

“Since conventional wisdom says more is always better, people naturally said ‘more’. When Google tripled the number of results, however, it found that traffic actually declined. Not only did the results take a fraction of a second longer to load, but having more options led people to click on links that were less relevant. The respondents in Google’s research didn’t intentionally lead researchers down the wrong path; they just didn’t understand the real-world implications of their choices.”

- Steve McKee, BusinessWeek, 2010.

Often things we dislike ‘grow on us’

There’s an example of this effect that most of us are familiar with. Upon initially listening to a new album, we prefer certain songs to others. On subsequent listens this preference usually changes, and our long term favourite songs tend not to be the ones we liked at first.

"It is easy to slip into the comfortable belief that 'I like it' comments after the first exposure of a new execution are a must. In researching Levi's executions over the years, it has become abundantly clear that such findings should be treated with extreme caution.” -  Kirsty Fuller, Managing Director of RDS International Research, in 1995.

Source: Walking the creative tightrope: the research challenge”, Kirsty Fuller, published in Admap Magazine, March 1995

Often things we dislike ‘grow on us’

eg Levi’s ‘Swimmer’ "In pre-launch qualitative research, response from the consumer on first exposure to Swimmer was one of stunned silence. The hero's status was initially seen to be seriously undermined - he did little to earn his colours. Moreover the slow music did not have the immediate appeal of previous executions. Perhaps the most fitting description of response was disappointment. Swimmer broke the mould of the campaign to date, and consumers claimed not to like it.”

“At this stage the weight of negative reactions was strong. Then two months after airing, research uncovered a marked shift in response. Swimmer had become a talking point: new, different, challenging. A further four months later and Swimmer was being widely described as one of the best ever Levi's ads, destined to live among the greats, such as the universally acclaimed Launderette.”

“Research must therefore seek to evaluate the potential of an execution, not its immediate impact on one viewing. Challenging advertising is not necessarily either immediately liked or fully understood. It may however, be rich and long-lasting.”

So…

•  A long history of pre-testing research being studied and proven unreliable •  Pre-testing often gets it right, but it’s very difficult to predict when that will be the case

•  Pre-testing is hampered by a few inconvenient realities…

•  We’re biased toward the familiar, not the effective

•  We often think (and will report) we want things that we actually don’t

•  Pre-testing only offers a ‘first impression’ whereas new ideas tend to ‘grow on us’

•  The numbers show that marketers’ judgment is significantly more reliable than positive pre-testing outcomes

The case for moderation

Alcohol has positive and negative effects. When we use it moderately, it’s great.

When we use it immoderately, we get into trouble. Pre-testing is the same.

It’s useful when used as part of a wider decision and development process

But when used ‘immoderately’ – as a decision maker – it’s dangerous.

Let’s be moderate in how we use the outcomes from our pre-testing research.