Upload
abdiel-drudge
View
219
Download
2
Tags:
Embed Size (px)
Citation preview
PIAAC: Delivering the Survey in Ireland
Kevin McCormackSenior Statistician - Social Analysis
Overview
• Background• Development• Survey Process• Content• Results• Dissemination
HandoutsPIAAC BrochureBackground QuestionnairePIAAC FactsheetOutline of international reportProposed thematic reports
• SLA 2010 - 2013– DES (and DETE originally) – CSO
• Statistics Act 1993– Data collection– Dissemination
• Timeline– 2009 – 2013 – Publication Oct 2nd 2013
Background
June 2010 – July 2011Main Study preparation:
TestingDeveloping IT systems Commissioning laptops
Training interviewers
PIAAC Timeline
2009 2010 2011 2012 2013
September 2009 CSO commenced formal engagement
Translation and Localisation tasks began
March 2010SLA signedCSO, DES, DETE
May 2010Field TrialInterviewer trainingSampling, maps, addressesQuestionnaire testing
October 2013PublicationInternationalNational&Databases
2009 2010 2013
May 2012 – May2013Data processing,
verification, cleaningTable generationNational report
August 2011 – March 2012Main Study
5,983 completed interviews72% Overall response rate70% Computer 30% Paper
• Instrument development• Translation and Localisation• Scoring testing and re-testing
• IT development• Virtual Machine • Screener and Case Management System development
• Field Test
Survey Development
Survey Development
• OECD oversight– “The Consortium”
• Sampling plans• Interviewer training• Quality Control• Weighting - Westat• Data Adjudication
• Main Study Aug 2011 – March 2012
• Sampling• Interviewer training• Data collection• Data processing
Survey Process
700 Blocks 15 Households 1 Person
Randomly
selecte
d !
!!
• Main Study• Sampling• Interviewer training• Data collection• Data processing
Survey Process
• Mix of new and experienced interviewers• 8 day training programme
• OECD specification
Focus on :•Gaining co-operation•Use of survey instrument
•Other important elements:•Nature of survey• Quality control•Confidentiality
• Main Study• Sampling• Interviewer training• Data collection• Data processing
Survey Process
•Encrypted data - file transfer & laptops
•Screener - random adult (16-65)•Targets and bonus payments•Incentive payment
•Management information system•Unlimited call –backs•Strict weekly monitoring
Quality control•Taped interviews•10% of all interviews verified•Telephone calls, field visits,
• Main Study• Sampling• Interviewer training• Data collection• Data processing
Survey Process
Industry and occupation codingManual AutomaticVerification against LFS
Paper booklet scoring Inter-coder reliability designAnchor booklets (international)
Data cleaningScreener vs PIAAC questionnaire
Survey Content
• Background Questionnaire
• ‘Core’– IT competency– Basic Skills assessment
• Direct Assessment– Computer Based Assessment– Paper Based Assessment
Background QuestionnaireA. General Information – Age & Gender
B. Education and Training – Formal Education & Training
C. Current status and work history
D. Current work – Occupation & Earnings
E. Last job - Occupation & Earnings
F. Skills used at work – Frequency of use of workplace skills
G. Literacy, numeracy and ICT skills used at work
H. Literacy, numeracy and ICT skills used in everyday life
I. About yourself – Respondent attitudes and activities
J. Background information – including parents’ occupations
FAIL <4 of 8
FAIL <3 of 6
PIAAC Assessment Design
BQ
Basic Skills (Computer)
Basic IT Skills
Assessment (Computer)
Reading Components
Basic Skills (Paper)
Assessment (Paper)
Everyone takes the BQ
Aged 16-65
6 tasks: 4 of 6 to PASSScrolling, clicking, highlighting,
4 literacy tasks4 numeracy tasks(same as computer tasks) 4 of 8 to PASS
FAIL
<4
3 literacy tasks3 numeracy tasks
3 of 6 to PASS
•Literacy and Numeracy•Numeracy and Literacy •Numeracy and Problem-solving
Literacy OR Numeracy booklet
Selected at random
•All those who fail the simple literacy or numeracy task on computer or paper are routed automatically to the Reading Components booklet
PIAAC Assessment Design
BQ
Basic Skills (Computer)
Basic IT Skills
Assessment (Computer)
Reading Components
Basic Skills (Paper)
Assessment (Paper)
Respondents had the option of skipping the Core IT assessment and proceeding directly to the Paper route
Results
Background Questionnaire• Economic• Education• Social
• Age• Gender
• Skills use (work & home)
Assessments• Literacy• Numeracy• Problem-solving in
Technology Rich Environments (PS-TRE)
• Proficiency – Item difficulty– Item Response Theory & scale scores– Cut-points (levels 1 to 5)
• IRELAND: Trends from IALS to PIAAC (linked items)
PIAAC Proficiency Scales
Trends (IALS vs PIAAC)
IALS ALL PIAAC
Prose Literacy Prose Literacy
Document Literacy Document Literacy
Literacy (rescaled for trend) Literacy (rescaled for trend) Literacy
Quantitative Literacy
Numeracy Numeracy
Problem solving
Problem solving in TRE
PIAAC Proficiency Scales
Level 1Level 2Level 3Level 4Level 5
0-225
226-275
276-325
326-375
376-500
Key findings of IALS
• 25% of Irish Adults at lowest levels of ‘literacy’
• Gender: Males more likely to be at higher levels in Quantitative Literacy – otherwise little difference between males and females
• Age: Older people more likely to be at lower levels of ability – related in IALS to educational attainment
• Education: Higher performance related to higher levels of educational achievement.
Prose Literacy (IALS 1997)Document Literacy (IALS 1997)
Level 1 Level 2 Level 3 Levels 4/50%
5%
10%
15%
20%
25%
30%
35%
40%
23%
30%
34%
14%
25%
32% 32%
12%
25%28%
31%
16%
Scales
Quantitative Literacy (IALS 1997)
IALS 1997
Canad
a
German
y
Irelan
d
Netherl
ands
Poland
Swed
en
Switz
erlan
d (Fren
ch sp
eakin
g)
Switz
erlan
d (Germ
an sp
eakin
g)
United St
ates
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
18.29
25.310.1
45.4
6.216.2 18.1 23.7
25.1 18.911.5
205.8
35.5
16 16.1 19
IALS 1997
International comparison Documents scale
International comparison Quantitative scale IALS 1997
Some data!
1996 2011
77.3
87.6
72.0
85.382.8
89.8
Percentage of those 20-24 with Leaving Certificate or equivalent
Both sexes Male Female
Source: CSO (Census 1996 & QNHS Q2 2011)
Source: DES (Education Statistics 2010/2011)
Highest level of Education Attained (Aged 25-64)
Primary (incl. no formal education)
Lower secondary Upper secondary Third level: Non-degree Third level: Degree or higher
25%
22%
31%
10%12%
10%
15%
24%
28%
23%
1996 2011
Source: CSO (Census 1996 & QNHS Q2 2011)
Population change 1996 to 2011
0 - 4 years
5 - 9 years
10 - 14 years
15 - 19 years
20 - 24 years
25 - 29 years
30 - 34 years
35 - 39 years
40 - 44 years
45 - 49 years
50 - 54 years
55 - 59 years
60 - 64 years
65 - 69 years
70 - 74 years
75 - 79 years
80 - 84 years
85 years and over
0
50000
100000
150000
200000
250000
300000
350000
400000
450000
Age profile of Population 1996 vs 2011
2011 1996
+1m (26%)
Country of Birth
Source: CSO (Census 1996 & 2011)
Ireland; 93%
Outside Ireland; 7%
1996
Ireland; 80%
Outside Ireland; 20%
2011
PIAAC results
International report • 2nd October 2013• Two volumes
– Results (I) and Technical (II)
• Distribution of skills (mean scores)
• International comparisons• Main focus on LITERACY• Other domains on the web• Problem-Solving in TRE for
later thematic releases
National report• Focus on Irish
performance
• Trends from IALS
• Relevant international comparisons
Profile 9 – Census 2011
A Study of Education & Skills in Ireland• November 22nd 2012
Thank you
Questions?