Is it a good idea to optimise question format for mode of data collection?

Gerry Nicolaas National Centre for Social ResearchPamela Campanelli Independent Survey Methods ConsultantSteven Hope University College LondonAnnette Jäckle University of EssexPeter Lynn University of Essex

Overview

• Background

• Research question• ‘Mark all that apply’ vs ‘yes/no for each item’

• Literature, Hypotheses, Methods, Results

• Branched questions vs non-branching• Literature, Hypotheses, Methods, Results

• Cognitive interview results• Methods, Results

• Conclusion

Background

• Face-to-face interviewing is dominant mode in UK• Interviewer presence• Aural stimuli & Visual stimuli (e.g. show cards)

• Many questions designed to be ‘optimal’ for face-to-face interviews• abundant use of show cards, long response lists, ‘mark all

that apply’, fully labelled scales, etc

• Push towards cheaper data collection modes and mixed modes• Some questions used in face-to-face interviews need to be

adapted for other modes

Research question

• Is it a good idea to ‘optimise’ question format for mode of data collection? • Increasing risk of differences in measurement across modes?

• 2 question formats that tend to be changed for telephone interviews• Mark all that apply • Scales

‘Mark all that apply’versus

‘yes/no for each item’

Example of ‘mark all that apply’

F2F & Self-CompletionThis card shows a number of different ways for reducing poverty. In your opinion, which of the following would be effective in reducing poverty?

SHOW CARD / MARK ALL o Increasing pensionso Investing in education for childreno Improving access to childcare o Redistribution of wealtho Increasing trade union rightso Reducing discriminationo Increasing income supporto Investing in job creation o None of these

Telephone interview

o Yes o No

Would investing in education for children reduce poverty?

Would increasing pensions reduce poverty?

o Yes o No

etc……….

I am now going to ask you a number of questions about different methods for reducing poverty. In your opinion, which of the following would be effective?

‘mark all’ vs ‘yes/no for each’

• Difficult to interpret absence of being marked with ‘mark all’• Does not apply or item non-response?• Sudman & Bradburn (1982)

• Deeper processing of each item with ‘yes/no’• More items selected with ‘yes/no’• Respondents take more time to complete ‘yes/no’• Primacy effects with ‘mark all’• Rasinski et al (1994), Smyth et al (2006), Smyth et al

(2008)

• ‘yes/no’ performs similarly in tel & web modes• Smyth et al (2008)

Our research

• Replicate & extend Smyth et al (2006 & 2008)• Face-to-face mode as well as telephone and web• UK general population with internet access• Cognitive interviews as well as experimental data

Hypotheses

• Higher % of items chosen in tel mode with ‘y/n’than in F2F & web with ‘mark all’

• If using ‘y/n’ in all 3 modes, no differences • If using ‘mark all’ in F2F & web, no differences• Deeper processing with ‘y/n’

• ‘y/n’ takes longer to complete than ‘mark all’• more primacy with ‘mark all’ respondents with below

average completion times• ‘mark all’ respondents with at least average completion

times select as many items as ‘y/n’ respondents

• Differences greater for difficult questions

Methods

• 2 split ballot experiments• Easy attitude question about neighbourhood with 8 items• Difficult attitude question about poverty with 8 items

• Mean number of endorsements for each format in each mode

• Response latencies for each format in each mode

• OLS regression

• Control variables for differential non-response

2 split ballot experiments

(1) Neighbourhood

What are the things that you like about your neighbourhood?

SHOW CARDo Its community spirito It feels safeo The neighbourso The character of its buildingso Its cleanlinesso Its locationo Its quietnesso Its transport facilities

(2) Reducing povertyThis card shows a number of different ways for reducing poverty. In your opinion, which of the following would be effective in reducing poverty?

SHOWCARD o Increasing pensionso Investing in education for childreno Improving access to childcare o Redistribution of wealtho Increasing trade union rightso Reducing discriminationo Increasing income supporto Investing in job creation o None of these

Results

• Hypotheses supported:• higher % of items selected in tel with ‘y/n’ than F2F &

web with ‘mark all’• ‘y/n’ takes longer to complete than ‘mark all’• no differences when using ‘mark all’ in F2F & web

• But:• when using ‘y/n’ in all modes, differences observed

between tel (& F2F) and web• differences were greater for the difficult question• no primacy among ‘mark all’ respondents with less than

average completion times• ‘mark all’ respondents with at least average completion

times still selected fewer items

Differences with Smyth et al (2006 & 2008)

• Differences in study population• Smyth et al: students at USA university• This study: general population who used internet

• Differences in question design• Smyth et al: 9-15 items • This study: 8 items for both questions

Full response list in 1 step versus

Branching into 2 steps

Example of scalar question

Fully labelled scale

Please tell me whether you consider your local shopping facilities to be …….o extremely poor,o very poor,o poor,o good,o very good, oro extremely good?

Branched question

Please tell me whether you consider your local shopping facilities to be …….o poor oro good?

Would this be …….o (poor/good), o very (poor/good), oro extremely (poor/good)?

Full scale vs branching

• Branching makes the task easier for respondents• Decomposition principle• Armstrong et al (1975), Groves & Kahn (1979), Krosnick &

Berent (1993), Yu et al (2003), Malhotra et al (2008)

• Branching produces more extreme responses• Reduces reluctance to select extreme responses• Nicolaas et al (2000), Yu et al (2003)

Hypotheses

• More extreme responses in tel mode with branching than in F2F & web with no branching• This effect is more prevalent for attitudinal than factual

questions

• Within each mode, more extreme responses with branching• This effect is more prevalent for attitudinal than factual

questions

• If using branching in all 3 modes, no differences

Methods

• 4 split ballot experiments• Easy attitude question about shopping facilities• Difficult attitude question about standard of living• Easy factual question about rent/mortgage• Difficult factual question about grocery shopping

• Proportion of extreme & non-extreme responses• Highest versus the lowest category• Two highest versus the two lowest categories

• Logistic regression

• Control variables for differential non-response

4 split ballot experiments

(1) Please tell me whether you consider your local shopping facilities to be ...

READ OUT oextremely poor,o very poor,o poor,o good,o very good or o extremely good?

(2) Thinking back to the last general election, would you say that the standard of living has ...

SHOW CARD oincreased by a large amount,o increased by a medium amount,o increased by a small amount,o stayed the same,o decreased by a small amount,o decreased by a medium amount, oro decreased by a large amount?

4 split ballot experiments (continued)

(3) How much did your household spend last month in rent or mortgage for the accommodation you live in?

READ OUT oless than £100o £100 - £199o £200 - £299o £300 - £399o £400 - £499o or £500 or more?

(4) How much did your household spend last month on grocery shopping?

READ OUTo less than £100o £100 - £199o £200 - £299o £300 - £399o £400 - £499o or £500 or more?

Results

• Hypotheses supported or partly supported:• more extreme responses in tel with branching than F2F

& web with no branching• difference was greater for attitudinal than factual

questions• within each mode, branching produced more extreme

responses but only for attitudinal questions

• Hypotheses not supported:• within each mode no branching seemed to produce

more extreme responses than branching for factual questions

• many differences observed across modes when using branched format in all 3 modes

Cognitive interviews

Mode effects with Y/N (CAPI vs CAWI) (1)

Subtleties going on that could affect aggregatemode comparisons. 1. Instances of clear and possible satisficing

• Most of these were in the ‘yes’ category• Most of these were in CAWI

2. Instances of Rs in the middle ground between ‘yes’ and ‘no’ (e.g., qualified their answer or said it depends)• Almost all chose ‘yes’ rather than ‘no’• More in CAPI than CAWI

Mode effects with Y/N (CAPI vs CAWI) (2)

3. Of 6 questions, 2 were more sensitive (increasing income support benefit and redistribution of wealth), • slightly more ‘yes’ answers in CAPI than in CAWI

Finding (1) –• Opposite to quant findings, but not surprising - more

satisificing in CAWI on other hypotheses

Findings (2) and (3) –• In the direction of the quantitative findings

Resultant quantitative effects would depend on prevalence of respondents in these different categories

Branching versus No Branching (1)

• Needed direct evidence rather than aggregate evidence

• 12 Rs asked the 2 attitude questions in branched format as part of the survey questions

• Much later in cognitive interviews, R given a showcard with ‘non-branched’ format of same question

• Interviewer was to probe for any inconsistencies

Branching versus No Branching (2)

• Some Rs were inconsistent• 2 themes:

• Vagueness of the answer categories• EXAMPLE: “To me, extremely good and

very good are the same thing”• Confusion in scope of question

• EXAMPLE: Were ‘local shopping facilities’for food shopping or clothes shopping?

• Although not explaining the branching effect, per se, findings suggest how variable respondents’answers can be.

Conclusions

Conclusions

• ‘Y/N for each’ and branching formats are not necessarily functionally equivalent across all three modes

• Optimising question format for each mode should not be done without empirical evidence that this will produce comparable data across modes

Thank you

www.natcen.ac.uk