The Performance of The Polls

John Curticewhatscotlandthinks.org

@whatscotsthink

Why Polls Matter

• Can affect the extent and nature of media coverage

• Can affect the ability of campaigns to raise money and motivate activists

• Can influence the campaign’s strategy and tactics

• Can influence whether people vote

The Challenge

• Regular monthly political polling in Scotland ended in 2003. Partly thanks to perceived poor performance, and partly the result of worsening newspaper finances.

• Thereafter mostly episodic and concentrated at election times.

• So companies having to estimate the attitudes of a population most have not been regularly monitoring.

• And for a ‘vote intention’ that they have not previously attempted to estimate.

• While the industry is still coming to terms with the internet

The Performance - 1

The Performance - 2

A Time Trend?

Feb-May July-Sept Oct-Dec Jan-mid Feb mid Feb-Mar0

10

20

30

40

50

60

70

38 39 39 41 43

62 61 61 59 57

YesNo

Based on 9 polls conducted Feb-May, 10 polls July-Sept, 8 polls Oct-Dec, and 7 polls in Jan-mid Feb and 10 polls mid-Feb-Mar. Don’t Knows excluded.

House Differences

Panelbase ICM/Survation TNS/YouGov Ipsos MORI0

5

10

15

20

25

30

35

40

45

5045

4339

36

% Y

es

Based on all polls since Sept. Don’t Knows excluded

Panelbase’s Record

Mar. 13 May. 13 Jul. 13 Aug. 13 Sept. 13 Nov. 13 Jan. 14 Feb. 14 Mar. 140

10

20

30

40

50

60

44 45 4551

44 45 43 4447

% Yes

Ipsos MORI’s Record

Feb. 13 May. 13 Sept. 13 Dec. 13 Feb. 130

5

10

15

20

25

30

35

4038

34 3437 36

% Yes

The Don’t Knows

TNS Others YouGov - AV ICM - AV0

5

10

15

20

25

30

35

30

15

33

27

Source: 1st 2 cols: Average of all polls since September

Methodological DifferencesCompany Method Political

Weighting/Selection

Question

Panelbase Internet 2011 Holyrood Vote; 8/10 likely to vote

Was current vote; now intend to vote Sept

ICM Internet 2011 Holyrood Vote Think will vote Sept

Survation Internet 2011 (1st poll: 2010) & likelihood of voting

Vote now

TNS BMRB Face to face quota 2011, incl abstention rate

Intend to vote Sept

YouGov Internet (18+ only) 2011, + extra Lab 10, SNP 11 group

Vote now

Ipsos MORI Telephone (RDD) None. Figures based on certain to vote

Vote now

Record

• Panelbase – no previous polling close to election day• ICM – previously all election polling done by phone• Survation – new company since 2010• TNS BMRB – still using System Three’s (not always

successful) approach• YouGov – over 10 years experience. Panellists’

previous vote collected on joining/after election• Ipsos MORI – using same approach as for GB polls

since 2008

How Their Last Poll Performed In 2011

% SNP Const Error % SNP List Error

TNS BMRB 45 0 38 -6

YouGov 42 -3 35 -9

Ipsos MORI* 45 0 42 -2

* Last Poll Conducted 14-17.4.11

The Extent and Impact of Weighting

Company Men 16-34 C2DE Impact on % Yes

Panelbase 1.16 1.17 1.40 +2

ICM 1.03 1.34 1.33 +3

Survation 1.17 1.42 1.01 +3

TNS BMRB 1.03 1.15 0.96 +3

YouGov 0.90 1.21* 1.26 +1

Ipsos MORI 0.98 1.08 - +2

* Based on 16-39

Reported Turnout

Feb May Sept Dec Feb0

10

20

30

40

50

60

70

80

90

73 74 7379 78

65 64 6369 68

54 5752

56 54

Referendum Holyrood Westminster

% c

erta

in to

vot

e

Source: Ipsos MORI (Scottish & British Polls)

Polls of The Interested?

Panelbase ICM Survation TNS BMRB YouGov0

5

10

15

20

25

30

35

40

2427

29

37

20

% DNV etc 2011

Survation & YouGov figures may include a few Other party voters

Tentative Conclusions

• Particularly large house effects on % Yes vote• Do not simply correspond to internet vs non-

internet• But do correspond to experienced vs. less so• Internet samples more heavily weighted• But weighting etc has pushed all recent polls in

a pro-Yes direction• Some polls (at least) look set to be wrong – but

which ones?