RESULTS FROM AN EXPERIMENT TO PREVENT REFUSALS IN A CONTINUING LONGITUDINAL STUDY

Presentation to PHSRN Workshop on Attrition in Cohort Studies, Royal Statistical Society, October 2009

The Research Team

Principal Investigator:

IAN PLEWIS, CCSR/SOCIAL STATISTICS, UNIVERSITY OF MANCHESTER.

Co-Investigators:

LISA CALDERWOOD, CLS, INSTITUTE OF EDUCATION, LONDON

REBECCA TAYLOR, NATCEN, LONDON

Research Officer:

SOS KETENDE, CLS

Preventing Refusals

• Seeking to prevent/convert refusals in an ongoing cohort study – the Millennium Cohort Study – with two interventions applied in an experimental framework.

• Testing the hypothesis that devoting extra field resources to the problem of increasing cooperation will bring benefits in the forms of increased precision and less attrition bias.

• Guided by the view that failure to cooperate in later waves of a longitudinal study, conditional on initial cooperation, will be largely circumstantial.

• Intervention 1: providing extra information in the form of a letter/leaflet that addresses previously reported concerns and reasons for not continuing to participate.

• Control condition: no leaflet

Piloted in wave 4 dress rehearsal

Reasons for refusal, wave 3

1= “Too busy” (36%)1= “Nothing has changed” (36%)1= “Don’t see public benefit” (36%) 4 “Don’t want to bother” (23%)5 “Stressful family situation” (15%)6 “Survey too long” (10%)7 “Looking after children” (8.9%)8 “Don’t see personal benefit” (5.5%)9 “Questions too personal” (4.8%)10 “Survey not important” (4.3%)

• Intervention 2: reissue all refusals to the interviewer (except ‘hard’ refusals), usually to a different interviewer.

• Control condition: standard NatCen reissue strategy (MCS, Wave 3 – just 4.4%).

Crossed designI1, I2. All initial and potentially convertible refusals reallocated to a different interviewer; leaflet sent in advance of the visit from the new interviewer.

C, I2. All initial and potentially convertible refusals reallocated to a different interviewer; no prior leaflet.

I1, C. Reissues of initial refusals follows NatCen standard practice of reissuing a proportion of all initial refusals; leaflet sent to all initial refusals prior to visit from new interviewer and to all non-issued and potentially convertible refusals with freephone number for those choosing to participate after all.

C, C. Reissues of initial refusals follows NatCen standard practice of reissuing a proportion of all initial refusals; no leaflet. This group was likely to contain very few reissued cases.

Issued sample at wave 4 of MCS: n = 15350 (GB only) – this was allocated to each of the four experimental groups within the seven GB strata.

‘Intention to treat’ sample – all refusals at first issue: n = 1660 (11%).

Experimental Group by Stratum

England Wales Scotland

GROUP Ad. Disad. Ethnic Ad. Disad. Ad. Disad. Total

Control (n) 91 133 70 14 43 25 44 420

Leaflet (n) 93 129 77 17 57 24 40 437

Reissue only (n)

79 120 63 24 48 22 33 389

Reissue + leaflet (n)

88 119 70 22 61 24 30 414

Total (%) 21 30 17 5 13 6 9 100

Issued sample (%)

28 28 15 5 11 6 7 100

Outcome numbers by Intention to Treat Group

Control Leaflet Reissue only

Reissue + leaflet

Fully productive 2 3 44 54

Partially productive 0 0 24 17

Unproductive 418 434 321 343

No evidence to support any effect of the leaflet. 

Reissuing appears to be effective.

Exclusions by Experimental Group

Note * - p < 0.01

Control Leaflet Reissue only

Reissue + leaflet

‘Hard’ refusals * 67 (16%)

61 (14%)

80 (21%)

100 (24%)

Other exclusions 0 9 5 8

Productive % (eligible) 0.56 0.82 22 23

Conversion attempt rate: 76% (for the two reissue groups). Burton et al. (2006), BHPS, waves 4 to 6: 40%. Productive rate (MCS) : 23% Productive rate (BHPS): 34% but only 13% for F2F interview.

Intensive reissuing reduced refusal rate from 11% to 10%. However, ratio of full to partially productive interviews = 2.4 compared with 8.0 for cases not refusing initially.

Conclusions

Intensive reissuing in an ongoing birth cohort study is effective, possibly more so for main respondents than for partners.

 Leaflet addressing respondents’ reasons for

refusing is not effective.

Further analyses

1. What was the previous response pattern for the converted refusers?

 2. What are the characteristics of the converted

cases? 3. Is refusal conversion cost effective?