Internet Surveys and Political Attitudes:

Evidence from the 2005 British Election Study

David Sanders, Harold Clarke, Paul Whiteley and Marianne Stewart

University of EssexUniversity of Texas at Dallas

Co-funding from ESRC, Electoral Commission and Newsnight

Preliminary Findings – not for citation or quotation

Four sections1. Study design – comparing a probability face-to-

face sample (response rate 62%) with an internet sample

2. Comparison of marginal distributions on key variables

3. False causal inferences? Comparison of models of turnout and vote using face and internet modes

4. Internet measurement experiments – feedback to respondents

Wave 1 Pre-electionProbability Sample,Face-to-Face N=3500128 PSUs

Wave 2 Post-electionProbability Sample,Face-to-Face N=3220Including top-up, mail-back; 128 PSUs

Wave 3 One Year OutInternet users from Wave2 Probability Sample, InternetSurvey method N=c2000

BES 2005 CORE FACE-TO-FACE PANEL SURVEY:

BES 2005 INTERNET CAMPAIGN PANEL SURVEY:

Wave 1 Pre-campaignBaseline SurveyN=c7000

Wave 2Campaign survey200 interviews perDay for 30 daysN=c7000

Wave 3Post-electionInterviewN=6117

Wave 4One Year OutInterviewN=c4000

Probability Internet sample versus traditional Internet sampleSampling Experiment:

Face-to-face vsInternet sampling experiment (2)

Face-to-face vsInternet sampling experiment (1)

Pre Face-to-Face Survey released June 1st 2005; Post released August 10th 2005Internet Panel (Pre/Campaign/Post) released June 1st 2005

2. Comparisons of NatCen and YouGov marginals on key variables

• Party vote shares, partisanship and reported turnout• Assessments of Government competence• Liking of party leaders• Position of respondent and parties on tax/spend scales• Position of respondent and parties on authoritarian/liberal scales• Interest in politics and personal political influence• Attitudes to voting and sense of civic duty

Vote Shares and Percentage of Party Identifiers: NatCen-YouGov Comparison

Reported Vote (%) Party Identifiers (%)

NatCen YouGov NatCen YouGov

Labour 40 36 38 36 Conservative 32 30 26 24 LibDem 22 25 13 12 Other 7 9 6 7 None 17 21 Chisq [sig] 30.6 [.000] 30.4 [.000]

Actual GB shares: Lab 36%; Con 33%; LibDem 23%. Significant differences but YouGov more accurate on Vote shares

Comparison of Voting and Party Identification marginals….

Percentage Voting versus Non-voting: NatCen-YouGov Comparison

NatCen YouGov

Did Not Vote 28.3 16.2 Voted 71.7 83.8

Chisq [sig] 193 [.000]

Actual: Voted 61.3%; Non-Voting: 38.7% Significant differences but NatCen more accurate

Comparison of Turnout marginals….

Mean scores on Government Competence (1-5) Scales

Survey Mode N Mean Sig

Government handle crime natcen 4138 2.73 .000 yougov 4919 2.59 Government handle asylum natcen 4105 2.12 .621 yougov 4919 2.11

Government handle nhs natcen 4136 2.97 .000 yougov 4919 2.80 Government handle terror natcen 4093 3.25 .000 yougov 4919 3.05 Government handle economy natcen 4072 3.60 .000 yougov 4919 3.49 Government handle tax natcen 4037 2.91 .000 yougov 4919 2.81

Significant differences apart from on Asylum: YouGov respondents less impressed by Labour

Assessments of Government Policy Competence…

Mean scores on Leader Liking 0-10 Scales

Survey Mode N Mean Sig Feelings About Blair natcen 4129 4.92 .000 yougov 4784 4.08 Feelings About Howard natcen 4002 4.43 .000 yougov 4693 3.61 Feelings About Kennedy natcen 3969 5.52 .000 yougov 4649 5.39

Significant differences: YouGov respondents more cynical about all three leaders

Likeability of Party Leaders…

Mean scores on Raise Tax/Improve Services 0-10 Scales

Survey Mode N Mean Sig Cut Tax/Spend More-Self natcen 4063 6.04 .000 yougov 5737 5.53 Cut Tax/Spend More-Labour natcen 3920 6.07 .000 yougov 5187 6.57 Cut Tax/Spend More-Cons natcen 3822 5.00 .000 yougov 5032 4.06 Cut Tax/Spend More-LibDem natcen 3685 6.05 .000 yougov 4665 6.68

Significant differences: YouGov respondents slightly more in favour of tax cuts themselves and place parties nearer extremes of the spectrum.

Comparison of positions on (left-right) raise taxes and improve services scale…

Mean scores on Punish Criminals/Protect Rights 0-10 Scales

Survey Mode N Mean Sig

Crime/Rights-Self natcen 4085 3.65 .000 yougov 5843 2.32 Crime/Rights-Labour natcen 3858 5.13 .000 yougov 5133 5.42

Crime/Rights-Conservatives natcen 3762 4.09 .000 yougov 4923 3.19 Crime/Rights-LibDems natcen 3573 4.99 .000 yougov 4426 5.56

Significant differences: YouGov respondents more in favour of punishing criminals and place parties nearer extremes of the spectrum.

Comparison of positions on authoritarian/liberal scale…

Mean scores on Interest and Influence Scales

Survey Mode N Mean Sig Political interest (1-5) natcen 4159 3.07 .000 yougov 4909 3.35 Election Interest (1-5) natcen 4158 2.95 .000 yougov 4919 3.23 Influence On Politics (0-10) natcen 4136 2.71 .000 yougov 4806 2.36

Significant differences: YouGov respondents more interested and less efficacious.

Comparison of interest in politics and political efficacy…

Attitudes to Voting and Civic Duty

Survey Mode N Mean Sig

Sense of satisfaction voting natcen 4121 3.63 .817 yougov 5811 3.62 I feel guilty if I don’t vote natcen 4150 3.47 .921 yougov 5962 3.47

Serious neglect duty if not vote natcen 4148 3.70 .503 yougov 5987 3.69

No significant differences

Comparison of attitudes to voting and civic duty…

Conclusions on Comparisons

• These are two different sets of measures in terms of the overall pattern of marginal distributions.•YouGov sample is more politically interested, which may help to explain why YouGov vote share is more accurate and its turnout share less accurate.• YouGov sample is less ‘Labour’ and less left generally in its attitudes than the NatCen sample.• But the electorate seems less Labour and less left than the NatCen sample implies. • Because of survey non-response in the probability sample, there is no telling which is ‘correct’.

3. Do the two samples produce different causal inferences?

1. Turnout model: based on discounted benefits minus costs; civic duty; exposure to personal campaigning and demographics

2. Model of (Labour) vote: based on party identification; policy competence/valence; Downsian ideological distance on tax/spend and authoritarian/liberal scales; leader ratings; economic perceptions; and demographics

Logistic Regression Model of Turnout/Not, NatCen and YouGov Datasets Combined

B S.E. Sig. Exp(B)

Benefits*Efficacy .053 .007 .000 1.055 Costs of Participation -.204 .031 .000 .815 Sense of Civic Duty .491 .025 .000 1.635 Exposure to Personal Campaigning .324 .094 .001 1.383 Age .025 .003 .000 1.026 Owner-occupier .863 .094 .000 2.371 Education Qualifications .090 .026 .001 1.095 White .529 .161 .001 1.696 YG Interaction Benefits*Efficacy .011 .009 .216 1.011 YG Interaction Costs of Participation .046 .045 .299 1.047 YG Interaction Civic Duty -.132 .032 .000 .876 YG Interaction Campaign Exposure -.004 .141 .978 .996 YG Interaction Age -.006 .004 .173 .994 YG Interaction Owner-occupier -.267 .132 .043 .766 YG Interaction Education -.016 .037 .674 .984 YG Interaction White -.037 .264 .890 .964 YG Respondent dummy 1.984 .511 .000 7.269 Constant -4.482 .351 .000 .011

Nagelkerke R2=.40 N=9032 Percent correct 83.9

Turnout ModelEffects virtually identical, though civic duty and owner have slightly smaller effects in YouGov sample

Logistic Regression Model of Labour/Not Vote, NatCen and YouGov Datasets Combined

B S.E. Sig. Exp(B) Labour identifier/not 2.080 .179 .000 8.004 Conservative identifier/not -1.871 .304 .000 .154 LibDem identifier/not -1.446 .266 .000 .235 Lab/Con difference in competence .107 .019 .000 1.113 Distance R and Lab on tax/spend -.111 .057 .051 .895 Distance R and Lab on authorit/lib -.106 .040 .008 .900 Blair liking ratings .376 .039 .000 1.457 Kennedy liking ratings -.208 .044 .000 .812 Economic perceptions .061 .036 .091 1.063 Educational qualifications -.107 .043 .013 .899 YG interaction Lab identifier -.392 .241 .104 .676 YG interaction Con identifier .622 .424 .143 1.863 YG interaction LibDem identifier .740 .363 .042 2.096 YG interaction Competence -.037 .022 .092 .964 YG interaction Tax/spend distance .033 .069 .630 1.034 YG interaction Authorit/lib distance .073 .048 .129 1.075 YG interaction Blair ratings -.045 .048 .355 .956 YG interaction Kennedy ratings .058 .055 .291 1.060 YG interaction Econ Perceptions .064 .045 .161 1.066 YG interaction Education .006 .056 .918 1.006 YG respondent dummy -1.167 .666 .080 .311 Constant -1.714 .515 .001 .180

Nagelkerke R2=.75 N=5311 Percent correct 89.4

Vote ModelNo significant differences in coefficients across the two samples. Only exception is LibDem identification – slightly smaller effect in YouGov sample

Conclusions on Causal Inferences• Even though the marginals on key variables are significantly different across the two samples, the inter-correlations among variables are similar.

• As a result, the estimates of the two ‘typical’ models are very similar across the two surveys.

• Even in the turnout model, where the marginal distributions are very different, almost exactly the same patterns of effect are observed across the two modes.

• Crucially, if the probability sample gets ‘close to reality’, we would not draw false causal inferences by using the internet sample.

4. Internet Survey Experiments• Ask respondent to self-locate on two 0-10 scales: tax/spend and liberal/authoritarian

• Logic: improve measurement of multidimensional scale responses by inviting respondent to adjust her/his assigned position in a 2-d space [WRONG]

• Later in survey show respondent where s/he is located in 2-d space defined by earlier responses

• Ask if respondent wishes to re-locate self

• Split sample on cues provided (eight experimental groups plus control)

Earlier in the survey we asked you for your views about taxes and public spending. We also asked you for your views about the importance of reducing crime versus the importance of protecting the rights of the accused.

The graph below indicates the point that we think best summarises your position. The graph also shows the positions of the Labour Party, the Conservative Party and the Liberal Democrat party:

Reducingcrimemoreimportant

Rights of accusedmoreimportant

Raise taxes, morespent on services

Cut taxes, less spent on services

Did we locate you in the right place?• Yes: please click to submit• No: Please point and click your mouse to indicate the point on the graph that you think best summarises your position• Don’t know

•Conservative Party

•Labour Party

•Liberal Democrat Party

•YOU

Experimental treatments

Treatment 1:CONTROL = Respondent’s position only

Treatment 2:Respondent plus ‘average voter’

Treatment 3:Respondent plus party supporters

Treatment 4:Respondent plus party leaders

Treatment 5:Respondent plus leaders with party labels

Treatment 6:Respondent plus parties 1983 scenario

Treatment 7:Respondent plus parties 1964 scenario

Treatment 8:Respondent plus parties 2005 (LD left)

Treatment 9:Respondent plus party supporters/leaders

Average Scores on Taxes versus Services Scale and Punish Criminals versus Protect Rights Scale, before and after Experimental Manipulations

Scale All Exp1 Exp2 Exp3 Exp4 Exp5 Exp6 Exp7 Exp8 Exp9 Pre tax 5.53 5.52 5.63 5.56 5.48 5.61 5.48 5.55 5.46 5.51 Post tax 5.57 5.59 5.63 5.68 5.54 5.61 5.56 5.63 5.40 5.50 Pre rights 2.23 2.33 2.37 2.31 2.36 2.47 2.36 2.22 2.24 2.41 Post rights 2.53 2.29 2.49 2.69 2.47 2.64 2.62 2.61 2.34 2.59 N 619 650 638 600 637 636 626 612 614 All scales are 0-10. Total effective N is 5613. A higher score on the Pre and Post Tax scales denotes a stronger preference for higher taxes and more spent on services; a lower score denotes a stronger preference for lower taxes and less spent on services. A higher score on the Pre and Post Rights scales denotes a stronger preference for the rights of the accused; a lower score denotes stronger preference for punishing criminals.

Raw Changes in taxes/services and authoritarian/liberal scores, before and after experimental stimuli

Are the changes significant? See next slide…

Average Change in Score on Taxes versus Services Scale, Pre-Post Split Sample Experiments

All Exp1 Exp2 Exp3 Exp4 Exp5 Exp6 Exp7 Exp8 Exp9 Change +.04 +.06 -.00 +.12 +.06 -.00 +.08 +.08 -.06 -.01 df 5630 618 649 637 599 636 635 625 611 613 p .00 .01 .93 .00 .05 .97 .01 .04 .08 .73

Average Change in Score on Punish Criminals versus Protect Rights Scale, Pre-Post Split Sample Experiments

All Exp1 Exp2 Exp3 Exp4 Exp5 Exp6 Exp7 Exp8 Exp9 Change +.19 -.04 +.11 +.39 +.11 +.17 +.26 +.39 +.10 +.18 df 5630 618 649 637 599 636 635 625 611 613 p .00 .31 .00 .00 .00 .00 .00 .00 .02 .00 Scale is 0-10. A higher score on scale denotes a stronger preference for the rights of the accused; a lower score denotes stronger preference for punishing criminals. Statistically significant changes (at .05 or better) in bold.

Effects of experimental manipulations on respondents’ self-placements on tax/services and auth/liberal scales

Some become more pro-taxes; all except control become more liberal

Summary of Pre-Post Experimental changes on Tax/Spend and Punish Criminals/Protect Rights Scales, by Voter type

Labour (N=1308)

Cons (N=1265)

LibDem (575)

Other (296)

Change in Tax/Services scale

.05 .00 .06 .02

Change in Punish Criminals/Protect Rights scale

.30 .11 .25 .14

Significant experimental changes in bold. Increase in Tax/Services scale denotes stronger preference for higher taxes and more spent on services when confronted with the experimental stimulus. Increase in Punish Criminals/Protect Rights denotes stronger preference for rights of the accused when confronted with the experimental stimulus.

Regardless of the specific stimulus, Conservative and other party supporters respond differently to feedback from Labour and LibDem supporters….

Reduced Logistic Regression Model of Post-Election Labour Vote/Not; Pre-experimental exposure measures of tax/spend and

authoritarian/liberal distances only

B S.E. Sig. Exp(B)

Distance R and Lab on tax/spend -.376 .026 .000 .686

Distance R and Con on tax/spend .235 .018 .000 1.265

Distance R and Lab on authorit/lib -.219 .017 .000 .804

Distance R and Con on authorit/lib .140 .020 .000 1.151

Constant -.144 .093 .122 .866

Nagelkerke R2=.37 N=3294 Percent correct=74.7

Do the pre-stimulus or post-stimulus measures best explain party choice?

Similar effects observed when further controls applied for party identification; policy competence; leader ratings; econ perceptions.

The model:

R2 Values from Logistic Regression models of Post-Election Labour/not vote and Conservative/not vote; Pre-experiment and Post-Experiment measures of tax/spend and authoritarian/liberal distances only; by experimental treatment

group

Lab Con N Experimental treatment PreE PostE PreE PostE

364 1. Control: R (Respondent) only .34 .27 .53 .46 397 2. R + ‘average voter’ .39 .35 .68 .59 350 3. R + ‘party supporters’ .43 .42 .59 .52 351 4. R + named party leaders .46 .40 .53 .50 364 5. R + leaders + party labels .42 .35 .54 .47 374 6. R + ‘parties in 1983’ .38 .32 .49 .46 368 7. R + ‘parties in 1964’ .29 .25 .60 .55 357 8. R + ‘parties in 2005’ .37 .32 .50 .41 369 9. R + ‘supporters+leaders’ .38 .33 .54 .52

3294 All cases combined .37 .33 .55 .49

The pre-stimulus measures clearly explain party choice better than the post-stimulus measures….

Note how post-experiment R2 values are all lower than their pre-experiment counterparts. Same pattern with coeffs, sigs and dps.

Summary of Experimental Results• Regardless of treatment, Labour and Lib-Dem voters wish to re-position themselves as being more in favour of higher taxes/better services (i.e. they move themselves left).

• Control group respondents do not wish to change their position on the liberal-authoritarian axis.

• But all treatment groups wish to re-position themselves as being more liberal on the liberal-authoritarian axis than they first indicated.

• The effects are most pronounced for Labour and Liberal Democrat voters: when they see where they have located themselves in relation to the parties, they wish to re-describe themselves as more liberal.

• Pre-treatment measures are better predictors of vote preference than post-treatment measures.

• Conclusion: additional party-cue information encourages a more ‘politically correct’ (liberal) response. It doesn’t improve measures.

Conclusions

1. We conducted the comparison between internet and face probability modes because the low/declining response rate problem means there is a real need systematically to explore the potential and limitations of internet polling.

2. The marginal distributions of our probability and internet polls were typically significantly different from one another.

3. The key difference is that internet respondents are more politically interested than ‘probability’ sample respondents.

4. The relationship between key dependent variables and ‘big beast’ predictor variables does not vary significantly across the two modes.

5. There is clearly a big role for internet polling in future academic research. Further studies are needed to assess inter-mode similarities and differences.