Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
Reasons for Late Response and Nonresponse in Surveys of
Government Agencies Tim Smith (RTI International)
Christian Genesky (RTI International) Danielle Kaeble (Bureau of Justice Statistics) Anthony Whyde (Bureau of Justice Statistics)
www.rti.org RTI International is a registered trademark and a trade name of Research Triangle Institute. 1
Establishment Response Rates
▪ Annual Survey of Public Employment and Payroll (n=50 state governments and about 90,000 local governments)
100 95 90 85 80 75 70 65 60
Res
pons
e R
ate
(%)
2010 2011 2012 2013 2014 2015 2016 Year
▪ Medical Expenditure Panel Survey – Private Sector Establishments
100 95 90 85 80 75 70 65 60
Res
pons
e R
ate
(%)
2010 2011 2012 2013 2014 2015 2016 Year
2
Establishment Response Rates
▪ 2015 Occupational Information Network (O*NET)
-100
80
60
40
20
0 Res
pons
e R
ate
(%)
Number of Employees
▪ Schools and Staffing Survey (SSAS) 2011-12
■ El
Res
pons
e R
ate
(%) 100
80 60 40 20
0 Teacher Listing School Survey Principal Survey Teacher Survey
Form Data Collection
Public Private
3
Risks Due to Late Response/Nonresponse
▪ Higher data collection costs
▪ Longer data collection period
▪ Lower data quality
▪ Staff turn-over
▪ Biased estimates
4
Research Goals
▪ Identify reasons for late response/nonresponse among probation and law enforcement agencies
▪ Assess differences by agency size and other characteristics
▪ Define tailored strategies to encourage survey participation
5
- 95 ~ 0 -~ co 90 0:: Q) 1/) C: 85 0 a. 1/) Q) 80 0::
75 First Second Third Fourth
Agency Size (Probation Population)
---2014 ..... 2015 -+-2016
Methods – Focal Studies
▪ Annual Probation Survey (APS) – Census of 456 agencies nationwide
• 42 provide state-level data • 13 provide data for multiple agencies • 401 provide data for a single agency
– Burden between 30-90 minutes
– Annual confirmation of agency head contact information for survey pre-notification
– Web survey, with telephone follow-up
– 5-month data collection period
– 92% response rate
6
Methods – Focal Studies
▪ Law Enforcement Management and Administrative Statistics (LEMAS) Core Survey – Sample of 3,500 state and local law enforcement agencies
▪ 50 state agencies ▪ 2,640 local police departments ▪ 810 sheriff’s offices
– Burden of 3 hours – Online search for
agency head contact information for survey pre-notification
– Web survey, with telephone follow-up
– 7-month data collection period
– 80% response rate (2016)
--- .....
100
Res
pons
e R
ate
(%) 80
60
40
20
0
Local Police Sheriffs' Office
1
2–4
5–9
10–2
4
25–4
9
50–9
9
100+
1
2–4
5–9
10–2
4
25–4
9
50–9
9
100+
Agency Type and Size (FTE)
2013 2016
7
Methods – Reasons for Late Response/Nonresponse
▪ Codes based on experiential/anecdotal evidence
– Perception that limited resources (e.g., budget and staff), benefits, and role of federal government in state/local issues might impact cooperation
– Tested on 2016 LEMAS Body-worn Camera Supplement Survey
▪ Verbatim reasons
▪ Double-coded into 21 categories, with adjudication
– Refined for APS and LEMAS-Core
8
Methods – Reasons for Late Response/Nonresponse
▪ Final code list – COMPLETE — Claims that survey has been submitted – DUE DATE — Cannot respond by due date – LIMITED TIME/RESOURCES — Not related to due date – APPLICABILITY — Thinks survey does not apply to agency – NO INTEREST – Uninterested in survey topic/goals – NO BENEFIT – No benefit to agency from participation/survey – VOLUNTARY – Participation is not mandated by law – SURVEY FATIGUE — Agency receives too many survey requests – LACK OF DATA (IN PERIOD) — Data are not available during survey period – LACK OF DATA (EVER) — Data do not exist or are not maintained – INACCESSIBLE DATA – Data exist, but are not easily accessible – POOR QUALITY DATA – Data exist, but are of questionable/poor quality – CONFIDENTIALITY – Data cannot be shared outside of agency/authority – FEDERAL ROLE – Federal government should not be involved in local issues/operations – JURISDICTION RULE – Jurisdiction does not participate in research – OTHER – REFUSED TO GIVE REASON FOR DELAY/REFUSAL
9
Findings – Overview
▪ Late respondents – APS: 165 cases; 129 contacts gave reasons – LEMAS: 679 cases; 339 contacts gave reasons (240 “did not receive”)
▪ Nonrespondents – APS: 38 cases; 29 contacts gave reasons – LEMAS: 572 cases; 310 contacts gave reasons (223 “did not receive”)
▪ Five reasons were not cited by late respondent or nonrespondent contacts on either study – APPLICABILITY — Thinks survey does not apply to agency – NO BENEFIT – No benefit to agency from participation/survey – CONFIDENTIALITY – Data cannot be shared outside of agency/authority – FEDERAL ROLE – Federal government should not be involved in local issues/operations – JURISDICTION RULE – Jurisdiction does not participate in research
10
SURVEY FATIGUE
REFUSED
POOR QUALITY DAT A
OTHER
NO INTEREST --1-----------------LIMITED TIME/RESOURCES ------------,
LACK OF DATA (IN PERIOD)
LACK OF DATA (EVER)
INACCESSIBLE DATA --~ ----------------
DUE DATE
COMPLETE I I I I I
0% 25% 50% 75% 100% % Contacts Mentioning Reason
Survey 9 APS
■ LEMAS
Findings – Late Respondents
11
SURVEY FATIGUE
OTHER
NO INTEREST
---.....--------------- -----------------
LIMITED TIME/RESOURCES ------------4
LACK OF DATA (IN PERIOD) - -----------------
LACK OF DAT A (EVER)
INACCESSIBLE DATA ~ - ---------------
DUE DATE
CONFIDENTIALITY
COMPLETE ------------ ■---------
• I I I I
0% 25% 50% 75% 100% % Contacts Mentioning Reason
Survey 9 APS
■ LEMAS
Findings – Nonrespondents
12
POOR QUALITY DATA
OTHER
LACK OF DATA (IN PERIOD)
LACK OF DATA (EVER)
INACCESSIBLE DATA
DUE DATE
COMPLETE I I I I I
0% 25% 50% 75% 100% % Contacts Mentioning Reason
Agency Size - All ■ 01
02 Q3
04
Findings – APS Late Respondents, by Agency Size
13
SURVEY FATIGUE
NO INTEREST
LACK OF DATA (EVER)
INACCESSIBLE DAT A
COMPLETE I I I I I
0% 25% 50% 75% 100% % Contacts Mentioning Reason
Agency Size - All
■ 01 Q2
Q3
Q4
Findings – LEMAS Late Respondents, by Agency Size
14
LUNTARY
OTHER
NO INTEREST
LIMITED TIME/ RESOURCES
LACK OF DAT A (EVER)
INACCESSIBLE DATA
DUE DATE I
0% I I I
25% 50% 75% % Contacts Mentioning Reason
I
100%
Agency Size - All
■ 01 Q2
Q3
Q4
Findings – APS Nonrespondents, by Agency Size
15
SURVEY FA TIGUE
NO INTEREST
LIMITED TIME/RESOURCES ----------
LACK OF DA TA (EVER)
INACCESSIBLE DAT A
CONFIDENTlALITY
COMPLETE I
0% I I I
25% 50% 75% % Contacts Mentioning Reason
I
100%
Agency Size
- All
■ 01 02 03
04
Findings – LEMAS Nonrespondents, by Agency Size
16
.1%
QUARTILE 1
73.5%
QUARTILE 2
8.0% 5.3%
10:69/,
76.1%
QUAR ILE 3
■ Respondent, Did Not Receive ■ Non-Respondent, Did Not Receive ■ Non-Respondent, Received □ Respondent, Received
3.5% 4.6'/o
11.0'k
81.0%
QUARTILE 4
Findings – LEMAS - Did Not Receive Survey Request, by Agency Size
17
Summary
▪ APS and LEMAS demonstrated differential response rates by agency size
▪ Many experiential/anecdotal reasons for nonresponse are not borne out by evidence (e.g., no benefit, confidentiality, federal role)
▪ Reasons for late response and nonresponse vary by agency size
– Large agency late respondents often report data availability
– Small agency late respondents often report staff resource limitations
– Frame address quality did not influence response, even though LEMAS agencies (especially small agencies) often reported not receiving survey materials
18
Lessons
▪ Test data availability when defining survey content
▪ Communication should stress flexibility/support for due dates and mode of data submission
▪ Confirm receipt of survey materials early
19
Limitations
▪ Data are from only two specific kinds of government agencies
▪ Data on reasons restricted to those contacted
▪ Analysis to-date only related to agency size
▪ Solutions are undefined/untested
– Restricted survey content
– Tailored survey correspondence/communication
– Interviewer procedures/materials
20
Next Steps
▪ Identify/analyze additional characteristics
▪ Develop tailored communications strategies
▪ Conduct randomized tests
▪ Investigate with other establishment/agency populations
21
More Information
Tim Smith, Director Security and Resilience Program RTI International, Survey Research Division [email protected] 919-316-3988
Information on RTI data collection on criminal justice projects: https://www.rti.org/sites/default/files/brochures/rti_data_cj.pdf
22