A part of Creston Unlimited

Confidential: For research purposes only British Polling Council & Market Research Society Inquiry

Evaluating the ICM pre-election pollsJune 2015

Martin Boon

Confidential: For research purposes only 2

Campaign poll 1 Campaign poll 2 Campaign poll 3 Prediction poll Result0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

39% 34% 35% 34% 38%

33%32% 32% 35% 31%

8%10% 9% 9% 8%

7% 11% 13% 11% 13%

13% 13% 11% 11% 10%

Not good; where’s the rogue?

2015 ICM/Guardian campaign shares

ICM campaign polls – April-May 2015Base: All participants expressing an intention to vote: turnout weighted, past vote weighted & adjusted.

6% 2% 3% 1% 7%

Confidential: For research purposes only 3

What happened – a failure of tried & trusted? Nope. The ICM adjustment techniques all worked pretty much as expected, and in the right direction.

Raw data Demographic weighting

Past vote weighting

Turnout weighting

Adjustment

Conservative 35% 32% 32% 33% 34%

Labour 35% 38% 37% 36% 35%

Lib Dem 8% 7% 8% 8% 9%

UKIP 11% 12% 12% 12% 11%

Green 3% 3% 3% 4% 4%

SNP/PC 6% 6% 7% 6% 6%

Other 1% 1% 1% 1% 1%

Total 99% 99% 100% 100% 100%

Confidential: For research purposes only 4

What was different? Voting patterns among up-weighted C1/C2 social grades directly contributed to final poll waywardness.

Campaign Poll 1 Campaign Poll 2 Campaign Poll 3 Prediction Poll0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%C1 break by poll

Conservatives Labour

Campaign Poll 1 Campaign Poll 2 Campaign Poll 3 Prediction Poll0%

5%

10%

15%

20%

25%

30%

35%

40%

45%C2 break by poll

Conservatives Labour

Confidential: For research purposes only 5

Recall poll evidence

- Reported 2015 vote does not match the election result, with under-estimate of the Conservative lead still very much present (Con: 34% vs Lab: 33%). Thus, late swing cannot account for our error. - Implies that sampling and weighting no longer adequate in predicting Con/Lab

shares.

- Vote switching: 9 in 10 did what they said they’d do – although minor and largely off-setting switching very slightly in favour of the Conservatives.

- ‘Shy Tory’ effect worth 1% swing and once again vindicated our ‘adjustment’. May need to be up-scaled and relative adjustments by party applied.

- Recalled turnout of 86%. Despite over-statement, 10-point probability scale worked quite well, but halving the probability of 2010 non-voters did not improve voting probabilities.

- Differential turnout model worth 1% swing to Conservatives.

2,914 recall interviews by telephone on 8-14th May with people we spoke to during the campaign: polls 1-4

Confidential: For research purposes only 6

Recall poll evidenceTwo rights do not explain the wrong. ‘Shy Tory’ & ‘Lazy Labour’ are modest & incomplete explanations of ICM’s error.

Confidential: For research purposes only 7

What’s to blame?

- Initial fear that sampling error was to blame remains leading suspect.

-c.30,000 randomly generated telephone numbers were dialled at lest once to generate 2,000 interviews over a weekend.

- The ability to reach a representative sample by telephone is now open to the same accusation as online polls: are the interested and the willing ‘different’ to the disinterested and the unwilling?

- ICM includes 33% of random mobile numbers in samples. But higher levels of phone poll volatility may indicate that the solution lies elsewhere.

- Weekday interviewing may have caused us to access the ‘wrong type of respondent’. Did we reach enough working people?

Are orthodox data collection techniques still capable for reaching a representative sample?