Nadja Lamei The hard-to-survey in EU -SILC · 2018-11-07 · We provide information The hard-to-survey in EU -SILC Challenges and potential solutions for the Austrian case. Summer

www.statistik.at We provide information

The hard-to-survey in EU-SILCChallenges and potential solutions for the Austrian case

Summer school 'Reaching out to hard-to-survey groups among the poor' 30 May – 3 June, 2016

Nadja LameiStatistics Austria,

Directorate Social Statistics

www.statistik.at slide 2 | 3 June 2016

Outline

• _ EU-SILC: the project, its political context and someimportant facts of fieldwork

• _ The „hard-to-survey“: who are they and what is so difficult about them?

• _ Any solutions?: strategies for different groups

Folie 3 | 30.03.2016

EU-SILCEuropean Community Statistics on Income and Living Conditionshttp://ec.europa.eu/eurostat/web/income-and-living-conditions/overview

http://ec.europa.eu/eurostat/web/income-and-living-conditions/overview


Objectives

Comparable statistical information on income and living conditions

Some typical questions at hand: How are household incomes composed and distributed? What about the

income situation of families, singel-parents households, older persons…? For which groups of the population are certain goods and services not

affordable, e.g. going to the doctor‘s, owning a car, having a holiday…? How often are children socially excluded because of their parents‘ economic

situation? How often are social risks passed on between the generations? How much are tenants burdened by their rents? How satisfied are persons with their life, what roll does the financial

situation play in this? How much risk of poverty and social exclusion is there really in a wealthy

country like Austria?

EU-SILC as data source for (social) politics in MS and the EU!


Regulation (EC) No 1177/2003 of the European Parliament and of the Council of 16 June 2003 concerning Community statistics on income and living conditions (EU-SILC)

plus implementing regulations: Definitions Fieldwork and imputation proceduresSampling and tracing rulesList of permanent variablesQuality reportsNew material deprivation items from 2016 onwardsYearly Modules

Ongoing revision of the European legal documents > Integrated European SocialStatistics (from 2019?)

National Regulation of the Federal Minsistry of Labour, Social Affairs andConsumer Protection (ELStV, BGBL. 277/II/2010)

Legal Background


1999 Treaty of Amsterdam: Social Politics on the EU‘s Agenda

2000 European Councils of Lisbon and Nice: Poverty must me reduced until2010

2001 European Council of Laeken: Decision on common indicators in the fieldof social protection and social inclusion

2010 Europe 2020-Strategy for smart, sustainable and inclusive growth:Emphasis on social situation than just economic indicatorsTargets and indicators: Europe-2020-Targets

Fighting poverty and social exclusion in the EU:At least 20 million fewer people in or at risk of poverty and social exclusion

Translated into National Action Plans

Political Background

http://ec.europa.eu/europe2020/europe-2020-in-a-nutshell/targets/index_en.htm


Measuring Poverty in Austria and the EU

„Official“ reporting on poverty since 1990ies

1995: AT‘s EU-accession, European Community Household Panel (ECHP)2003: new instrument EU-SILC, continous reporting2004: start of the integrated (cross-sectional and longitudinal) rotational design, 18 participating countries.Now: 28 EU-MS + Norway, Island, Turkey, Switzerland, Macedonia, Serbia andMontenegro.

Further development of concepts:• OECD vs. EU-scale for equivalised income• Risk of poverty > risk of poverty and social exclusion

Further development of data and methods:• Survey and administrative data combined• CAPI, CATI and in the future also CAWI• Optimizing field work, sampling, weighting > better representativeness and

validity

Ever faster, many stakeholders!


Latest results

Short English content: http://www.statistik.at/web_en/statistics/PeopleSociety/social_statistics/poverty_and_social_inclusion/index.htmlMore detailed German Version:http://www.statistik.at/web_de/statistiken/menschen_und_gesellschaft/soziales/armut_und_soziale_eingliederung/index.html

http://www.statistik.at/web_en/statistics/PeopleSociety/social_statistics/poverty_and_social_inclusion/index.html

http://www.statistik.at/web_de/statistiken/menschen_und_gesellschaft/soziales/armut_und_soziale_eingliederung/index.html


Latest results (2)

Risk of poverty

Low work intensity

Severely materiallydeprived

Risk of poverty or social exclusion


Latest results (3)

Q: STATISTIK AUSTRIA/EUROSTAT, EU-SILC 2008-2015.

Risk of poverty or social exclusion


Fieldwork for EU-SILC in Austria – overview2003 Cross sectional survey

2004 Start of integrated rotational design

2005

2006

2007 Beginning of in-house field work

2008

2009

2010 National regulation

2011 Register use (for pension variables only)

2012 Extensive register use

2013

2014

2015

2016

2017 New survey infrastructure

2018 Modular questionnaire design and CAWI

2019 New European Legal Act?

Mixed Mode Design, field work 100% in-housenational financing (before: 2/3 Eurostat)

Some numbers…Min. effective sample size: 4,500 (cross-sectional)

3,250 (longitudinal) HHs.Actual sample size reached: ~6,000 HHs net/ year4 wave panel: ~1,200 HHsVoluntary participation, all persons aged 16+ are surveyedProxy rate ~ 10%Since 2012: 85% of total sum of household income from registers

Rotational design


First wave

Dwellings /households – see in further detail next slide

Personal computer assisted interviewing (CAPI)

Gross sample 2016: 3,528 adresses

Expected response rate: 65%


Type of sampling: one-stage stratified probability sample

Sampling units: dwellings registered in the central residence register (ZMR)

Stratification criteria: _Interviewer units (geographical units below NUTS2 level)

_Since 2016: also information on Household income from registers (firstquartile or above)_Disproportional allocation per NUTS2 level according to expected response rates (based on average response of two preceding year)

Sampling: Selection of first wave sample


Follow up waves (2-4)

Persons sample_ Tracing of sample persons (movers, splits)_ All households with at least one sample person take part in the survey (i.e. sample persons and “co-residents” are interviewed)_ Non eligible if moved to institutional household or outside Austria

Computer assisted telephone interviewing (CATI) except for:_ Households with no valid telephone number

_ Households explicitly asking for a CAPI (f2f)-interview

_ Method changes are possible throughout the fieldwork period

Gross sample 2016: 4,787 households, 2-3% split off-households per year• therof CAPI: 965

• therof CATI: 3,822 (80%)

Expected response rate: 85%


Fieldwork period: February – June/July

Information for households_ Seprate letters for each wavealso in Bosnian/Croatian/Serbian or Turkish_ Leaflet on EU-SILC_ First wave: Booklet „Austria in Figures“_ Follow up waves: EU-SILC newsletter Website and information video: www.statistik.at/silcinfo Hotline and e-mail

Information for local administrative units

15 Euro incentive (voucher) for each household upon successfull interview

Fieldwork

http://www.statistik.at/silcinfo


Mode (only follow up waves)

planned assignment

to modemode changes response rate by

mode changes

response rate by planned

mode

actual assignment

to mode

response rate by actual mode

2% changes to CATI response rate: 62%

n=4.699 (follow-up) n=3.941(n=7.936 total cross section) (n=5.909 total cross section)

100%

response rate: 86%

response rate: 81%

S: Statistics Austria, EU-SILC 2014.*Follow -up w aves excl. 8 households w hich w eren't processed (reported refusals betw een the w aves etc.).

60% CATI

40% CAPI

100%

response rate: 83%

response rate: 84%

total response rate follow-up waves:

84%

18% changes to CAPI

82% remain CATI

98% remain CAPI

100%

100%

response rate: 84%

response rate: 86%

response rate: 75%

follow-up gross

sample 100%

27% CAPI

73% CATI

Folie 18 | 30.03.2016

Who are those hard-to-survey?… and what effect might they have on the statistic‘s outcome?


Nonresponse and Total Survey Error

Groves et al. 2004:48

_ Item-Nonresponse:Missings in Variables Counter-measure -> Imputation

Item Non-Response 2014:2% for Employment Income(from admin data > lack of identifier for linking but income receipt seems likely),<1% for Unemployment benefits (same reason as forempl. income),10% for Self-employed income (surveyed)

_ Unit-Nonresponse:Missing Persons or complete HouseholdsCounter-measure -> Weighting

Is Nonresponse selective?


Nonresponse Error and Bias

Two kinds of Nonresponse Error: Variance due to Nonresponse

Random deviation from one net sample compared to all potential net samples due to nonresponse (cf. Groves 2006)

Nonresponse Bias • Systematic deviation of the expected value of the estimate in the

net sample from the expected value in the gross sample due to nonresponse. (cf. Groves 2006, Eurostat 2009, Särndal & Lundström 2005)

Size of Nonresponse Error gets bigger with

_ Lower response rates AND

_ Bigger variance between respondents and nonrespondentsImportant to know if nonresponse error is systematic, i.e. variable of interest iscorrelated with nonresponse


Missingness at random?

MCAR MAR NMAR

non-ignorable NR


Who is surveyed – who not: Response rates

Rotational group 1. w ave 2. w ave 3. w ave 4. w aveFirst w ave 2014 2013 2012 2011Household non-responseTotal sample 3.229 1.887 1.473 1.347 7.936Address not existent (DB120 = 23) 132 0 0 0 132NRh - Household non-response rate in % 36,5 23,6 12,2 10,5 24,3Rh - Household response rate in % 63,6 76,4 87,8 89,5 75,7

Individual non-response Eligible persons (RB245 = 1+2+3) 3.568 2.615 2.344 2.218 10.745Personal interview s (RB250 = 11+12+13) 3.557 2.613 2.343 2.216 10.729Rp - Complete personal interview s in % 99,7 99,9 100,0 99,9 99,9

Source: Statistics Austria, EU-SILC 2014

Total


Resasons for drop out

Rotational group 1. w ave 2. w ave 3. w ave 4. w aveFirst w ave 2014 2013 2012 2011Household non-responseTotal sample 3.229 1.887 1.473 1.347 7.936Address not existent (DB120 = 23) 132 0 0 0 132NRh - Household non-response rate in % 36,5 23,6 12,2 10,5 24,3Rh - Household response rate in % 63,6 76,4 87,8 89,5 75,7

Individual non-response Eligible persons (RB245 = 1+2+3) 3.568 2.615 2.344 2.218 10.745Personal interview s (RB250 = 11+12+13) 3.557 2.613 2.343 2.216 10.729Rp - Complete personal interview s in % 99,7 99,9 100,0 99,9 99,9

Source: Statistics Austria, EU-SILC 2014

Total

Rotational group 1. w ave 2. w ave 3. w ave 4. w aveFirst w ave 2014 2013 2012 2011

-2 Adress not used 41 25 13 10 89

11 Household sucessfully contacted

3.051 1.857 1.452 1.331 7.691

21 Adress cannot be found 5 5 8 6 24

23 Building does not exist 4 0 0 0 4

24 Not used for living purposes

19 0 0 0 19

25 Empty 75 0 0 0 75

26 No person w ith main residence

34 0 0 0 34

3.229 1.887 1.473 1.347 7.936

D002000 Adress contact status

Total

Total

Rotational group 1. w ave 2. w ave 3. w ave 4. w aveFirst w ave 2014 2013 2012 2011

-2 non eligible adress (D002000 <> 11)

178 30 21 16 245

11 sucessfull 1.968 1.442 1.293 1.206 5.90921 noone at home 158 64 24 21 26722 refusal 804 281 101 79 1.26523 break-off during interview 12 18 3 0 3324 language problems 18 2 0 0 2025 no person at home qualif ied for an interview

2 0 0 0 2

26 entire household temporarily aw ay

15 8 6 7 36

27 household unable to respond (illness, disability…)

73 24 11 9 117

28 other reason for drop-out 1 18 14 9 42

3.229 1.887 1.473 1.347 7.936Total

D003000 Household contact status

Total


Nonresponse-AnalysisComprison of gross and net sample: (Rich) Sampling frame, Screenings, Interviewer debriefings, Follow up surveys for nonrespondents…)

Estimate response rates for relevant groups

Results from previous research shows (cf. Glaser/Kafka 2015):

First wave

S: Statistics Austria, EU-SILC 2010

60% 61%58% 57%

61%

55%

62%66% 67% 66%

61%

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

1 2 3 4 5decile

6 7 8 9 10 Total

Mean response rates by income decile


Nonresponse-Analysis (2)Results from previous research shows (cf. Glaser/Kafka 2015):

Panel


Nonresponse-Analysis (3)Results from previous research shows (cf. Glaser/Kafka 2015):

Summary of results – Effects on panel participationNegative Being deprived Larger number of grown-ups in HH Main activity full time work or being

in education Household with foreigners Vienna Item-Nonresponse Proxyinterview

Positive Larger number of children in HH Partnership in the HH University degree Landline telephone No moving of the household Two-families or semi-detached

house Being contacted later during field

workNo impact Equivalised HH income Tenure status Response burden

Folie 27 | 30.03.2016

Solutions for (better) treatment of thehard-to-survey?What we already do and what could be done more


Different measures at different times

During field work:_ close monitoring of field work and response rates_ measures for groups that are hard to reach/interview (several contact attempts, allowing for mode change, incentive, translations,…)

After field work:_ weighting and calibration_ keeping-in-touch between waves


Treatment of nonresponse bias - fieldwork

High response means better net sample size

BUT: Higher response can lead to higher bias if response rate is very different for different (relevant) groups

So effect of fieldwork measures (letter design, incentives, number and mode of contact) has to be evaluated, on whichgroups does it have which effect?

R-Indicator: “Indicator of Representativity”“Definition (weak): A response subset is representative of a categorical variable X with H categories if the average response propensity over the categories is constant” (cf. Schouten et al., 2009)


Weighting and calibration (1)

Sampledesign

Nonresponse Adjustment Base weight Household X-weight

Nonresponse Base weightAdjustment Household X-weight

(t=1)

(t>1)

Nonresponse …

Individual L-weight

panel attrition


Weighting and calibration (2)

Marginal distributions used for calibrationHousehold level_ Household size (1, 2, 3, 4+ HH members) _ Tenure Status _ NUTS2Personal level:_ Age_ Sex_ No. of Persons with foreign citizenship (aged 16+) _ No. of Persons with receipt of unemployment benefits / employment inc. / pension inc._ PLUS in LONGITUDINAL WEIGHTS: income below the median equivalized income, income below 60% of median equivalizedincome (individuals at-risk-of-poverty), Individuals belonging to the population not covered in the panel (migrants and newborns)

household weight = design weight * non-response weight * adjustment weightdesign weight: inverse selection probability

non-response weight: inverse estimated response probability

adjustment weight: calibration to external sources


What more?

Some ideas for further research:_ (Non)response and Mode effects_ Tailored field work strategies for different groups_ Nonresponse from wave to wave and its effect on poverty dynamics_ Better use of register data in Sampling and weighting (Optimal allocation, calibration, quantification of nonresponse bias and nonrespnse adjustment)_ Analyse nonresponse bias and measurement error together (Total Survey Error)


Literature• European Commission (2009): ESS Handbook for quality reports. Luxembourg: Office

for Official Publications of the European Communities. (Eurostat methodologies and working papers).

• Glaser, Thomas; Kafka, Elisabeth (2015): Analyse und Behebung von selektivem Bias –EU-SILC in Österreich. In: Nonresponse Bias. Qualitätssicherung sozialwissenschaftlicher Umfragen. Schriftenreihe der ASI - Arbeitsgemeinschaft Sozialwissenschaftlicher Institute. 395-434.

• Groves, Robert M.; Fowler Jr., Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor & Tourangeau, Roger (2004): Survey Methodology. Hoboken: Wiley. (Wiley series in survey methodology).

• Groves, Robert M. (2006): Nonresponse Rates and Nonresponse Bias in Household Surveys. Public Opinion Quarterly 70 (5, Special Issue), 646–675. DOI: 10.1093/poq/nfl033.

• Särndal, Carl-Erik; Lundström, Sixten (2005): Estimation in Surveys with Nonresponse. West-Sussex: Wiley. (Wiley series in survey methodology).

• Schouten, Barry; Cobben, Fannie; Bethlehem, Jelke (2009). Indicators for the representativeness of survey response. Survey Methodology 35, 101-113.


Please address queries to:Nadja Lamei

Contact information:Guglgasse 13, 1110 Viennaphone: +43 (1) [email protected]

The hard-to-survey in EU-SILCChallenges and potential solutions for the Austrian case


Imputation of income components: Overview

calendar or imputation

How long?number of months/times

How much gross?

How much net? net amount

category?

category imputation

statistical imputation

n.a. = no answerN/G = net/grossG/N = gross/net

calendar etc.yes no

n.a.

n.a.

n.a.

n.a.n.a.

N/G-conversion orG/N-conversion

gross amount

Income component received?


Imputation of income components: Methods

Imputation (only) for income variables

Longitudinal and cross-sectional imputation

Longitudinal: using the information of previous years:

Little & Su (1989) Development of distribution of variables and income development of single case

Cross-sectional imputations

Multiple linear regression models Median imputation Always: adding an artificial error term (reduction of variance)

Documents

Nadja Lamei The hard-to-survey in EU -SILC · 2018-11-07 · We provide information The hard-to-survey in EU -SILC Challenges and potential solutions for the Austrian case. Summer