Upload
melissa-creek
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
PISA for Development
Technical Strand 1:
Contextual questionnaires
Overview of PISA instruments(constructs, indices and variables)
EDU/DCD
PISA for DevelopmentContextual questionnaires
2
Purpose of PISA contextual questionnaires
How are they used?
How they are developed
What will PISA for Development seek to do?
Questions for discussion
Student and school questionnaires are part of the core assessment design of PISA
Countries may also opt for additional questionnaires:
- Parents, Teachers, Health, Time Use, ICT familiarity, Career…
Student questionnaires: ~ 30 minute* stand-alone document (students respond after test questions)
School questionnaires: ~ 30 minute stand-alone document (school authority responds before or on day of testing)
* Rotated questionnaires in 2012 to cover additional material
PISA for DevelopmentPurpose of PISA contextual questionnaires
Contextualise the performance results in reading, mathematics and science*
Allow analyses of equity issues across sub-groups and populations
Provide indications of possible policy interventions
Some information collected is used to scale cognitive assessment data
Country-specific questions can be included for additional analysis
* Additional domains such as financial literacy and problem-solving
PISA for DevelopmentPurpose of PISA contextual questionnaires
Have been used in 74 countries/economies
In more than 40 languages
Publicly availablePISA 2009 Student Questionnaire
PISA 2012 Student Questionnaire
PISA 2012 School Questionnaire
PISA 2009 School Questionnaire
PISA for DevelopmentPurpose of PISA contextual questionnaires
National educ., social and economic context
Structures, resource allocation
and policies
Social & economic outcomes of education
Community and school
characteristics
Student learning, teacher working
conditions
Socio-economic background of
learners
Antecedentscontextualise or
constrained policy
The learning environment at
school
Teaching, learning practices and
classroom climate
Individual attitudes,
engagement and behaviour
Output and performance of
institutions
Quality of instructional
delivery
Quality and distribution of
knowledge & skills
Policy Leversshape educational
outcomes
Outputs and Outcomes
impact of learning
Individual learner
Instructional settings
Schools (other institutions)
Country or system
This is the PISA Analytic Framework that guides data collection and analysis
Leve
ls o
f an
ed
ucati
on
sys
tem
PISA Contextual QuestionnairesHow they are used
Outcomes Processes Antecedents
Students
• General recurring variables (all cycles)
• Domain recurring variables (main domain every 3 years)
• Extension variables (specific cycles)
• System-level data
Classrooms
Schools
Country
Main data collection
Student Questionnaire, School Questionnaires and System Level*
This is how the PISA Analytic Framework is operationalised for data collection and analysis
Variables included in instruments:
* Options include questionnaires geared for Teachers, Parents, Health, Time Use, ICT familiarity, Career
PISA Contextual QuestionnairesHow they are used
8
PISA Contextual QuestionnairesSystem-level data collection
• Ratio: students / teaching staff• Spending /student• Instruction time (by age)• Teachers’ salaries
(by education, experience)• Evaluation & accountability• Educational stratification• Decision-making
at different levels• School choice• Changes in education
policy• Others (…)
Standards and procedures for data collection in PISA Questionnaires for non-OECD economies Panel of experts to complete some data
collection
• Within-country disparities• Quality, supply and
distribution of educational materials and resources
• (Extra-)tuition• Parental support• Contractual arrangements
of school teaching staff
Sample of relevant areas to address in developing
contexts
These are examples of the types of information collected at the system level:
9
• Grades served• Public-Private• Funding sources• Community type• Enrollment (boys, girls)• Average class size• Non-native speakers• Grouping/tracking• Teachers, by qualifications• Teacher shortages • Physical building limitations• Instructional materials• Teacher attitudes, expectations,
relations• Computer/network/web access
• Absenteeism, disruptions• Teacher morale• Student morale• Teaching environment• Time for special programmes• Non-native speaker
offerings• Admissions requirements• Assessment types• Uses of assessment• Assessment of teachers• Decision making
authority• External decision makers
School resources, policies and practicesThis is the type of information collected through the school questionnaires, that can then be cross-referenced with other sources
These are the constructs that will be reviewed as part of the technical work
of PISA for Development
PISA Contextual QuestionnairesSchool questionnaires
10
PISA Contextual QuestionnairesStudent questionnaires
General and domain-related processes
Sense of belonging at schoolStudent-teacher relations at schoolDisciplinary climate in classesTeacher support in subject-specific classesOut-of-school activities (homework, etc...)
Self and domain-related cognitionsAnxiety towards “study subject”Instrumental motivation to learn “study subject”Interest in and enjoyment of “study subject”Self-efficacy in “study subject”Self-concept in “study subject”
Learning strategies
Preference for competitive learning situationsPreference for co-operative learning situationsControl strategiesPerseverance
Others
Health status, Time use
(…)
These are examples of the types ofnon-cognitive outcomeson which PISA collects information
These are the
constructs that will be reviewed as part of the technical work of PISA for
Development
Student attitudes, behaviours and approaches to learning
11
… How are questionnaire data used for relevant analysis and comparative policy insights?
(a few examples from PISA results)
PISA Contextual QuestionnairesExamples of analysis
12
Sh
an
gh
ai-C
hin
a
Sin
ga
po
re
Ma
cao
-Ch
ina
Du
ba
i (U
AE
)
Un
ited
Ara
b E
mira
tes
Qa
tar
Ch
ine
se T
aip
ei
Lith
ua
nia
Co
sta
Ric
a
Bu
lga
ria
Ro
ma
nia
Th
aila
nd
Co
lom
bia
Mo
nte
ne
gro
Jord
an
Arg
en
tina
Pa
na
ma
Aze
rba
ijan
Ind
on
esi
a
Ge
org
ia
Him
ach
al P
rad
esh
-In
dia
Kyr
gyz
sta
n
300
350
400
450
500
550
600
2009 2006 2003 2000Reading per-formance
Low-income economies ($1,025 or less)Lower-middle-income economies ($1,026 to $4,035)Upper-middle-income economies ($4,036 to $12,475)High-income economies ($12,476 or more)
Examples of analysisRange of performance among non-OECD countries
PISA Results for different years
-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2200
250
300
350
400
450
500
550
600Score
OECD average
OE
CD
avera
geS
tude
nt
perf
orm
ance
AdvantagePISA Index of socio-economic backgroundDisadvantage
School performance and students’ socio-economic background within schools
Student performance and schools’ socio-economic background
Private school Public school in rural area Public school in urban area
Mexico
Schools with a similar socio-economic profile
PISA 2009 ResultsExamples of analysisSchool performance and socio-economic background
-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2200
250
300
350
400
450
500
550
600Score
OE
CD
avera
ge
Stu
de
nt
pe
rfo
rma
nce
AdvantagePISA Index of socio-economic backgroundDisadvantage
Private school Public school in rural area Public school in urban area
OECD average
Examples of analysisSchool performance and socio-economic background
School performance and students’ socio-economic background within schools
Student performance and schools’ socio-economic background
Kyrgyzstan
PISA 2009 Results
Examples of analysisPolicy findings from contextual information
Policies and practices
Learning climate
Discipline
Teacher behaviour
Parental pressure
Teacher-student relationships
Dealing with heterogeneity
Grade repetition
Prevalence of tracking
Expulsions
Ability grouping (all subjects)
Standards /accountability
Nat. examination
Standardised tests
PolicySystem
RSchool
REquity
E
Policy briefs on these issues can be found in the PISA in Focus series
Examples of analysisPolicy insights using contextual information
Commitment to universal achievement
Goals, gateways,
instructional systems
Capacity at point of delivery
Incentives and
accountability
Resources where they yield most
A learning system
Coherence
Lessons from PISA from successful
education systems (equitable and improvers)
Based on the analysis of cognitive outcomes
and contextual information
17
PISA Contextual QuestionnairesHow they are developed
‘Crowd sourcing’ and collaboration– PISA draws together leading expertise and institutions from
participating countries to develop instruments and methodologies…
… guided by governments on the basis of shared policy interests
Cross-national relevance and transferability of policy experiences
– Emphasis on validity across cultures, languages and systems– Frameworks built on well-structured conceptual
understandingof assessment areas and contextual factors
Continuous review and updating of assessment frameworks (cycles)
– Guided by participating countries and various expert groups formed by leading international experts in different areas related PISA
– Technical Advisory Group, Reading Expert Group, Mathematics EG, Science EG, Questionnaire EG, and international contractors
• Student questionnaire: Rotated design (total of 65 questions – 25 core and 3 blocks of 27 questions)
• School questionnaire: 33 questions (5 additional questions on financial education at school)
• Parental questionnaire: 25 questions
• ICT questionnaire: 12 questions
• Education careers: 14 questions
PISA Contextual QuestionnairesHow they are developed
PISA 2012 Questionnaires
Linkages between cognitive and non-cognitive outcomes
• Selection of cognitive test items to be included in the PISA Main Study instruments (e.g. countryXcountry DIF, gender DIF)
• Validation of psychometric equivalence of translated instruments (and adaptations from source versions) issues addressed
• Identify constructs from contextual questionnaires (indices and variables) that are associated with performance for inclusion in the Main Study
• Determine cross-national validity of the questions in the questionnaire
• Technical standards ensure that data from participating countries are internationally comparable
PISA Contextual QuestionnairesHow they are developed
Purpose of the PISA Field Trial (for Main Study)
Adapt and enhance contextual instruments to “better fit” diverse contexts found in developing countries while maintaining the comparability with the main PISA scales and international results
Adapt existing constructs (and indices and variables)
Identify and introduce new constructs (indices and variables)
Confirm and validate enhancements through field trials and main data collection in participating countries
PISA Contextual QuestionnairesWhat will PISA for Development seek to do?
Student Questionnaire
Section 1: ABOUT YOU
Section 2: YOUR FAMILY AND YOUR HOME
Section 3: YOUR READING ACTIVITIES
Section 4: LEARNING TIME
Section 5: YOUR SCHOOL
Section 6: YOUR SCHOOL LESSONS AND SUBJECTS
Section 7: YOUR STRATEGIES IN READING AND UNDERSTANDING TEXTS
Section 8: YOUR VIEWS ON <BROAD SCIENCE>
Section 9: CAREERS AND <BROAD SCIENCE>
Section 10: YOUR MATHEMATICS EXPERIENCES
Examples of variables and indices
PISA Contextual QuestionnairesWhat will PISA for Development seek to do?
Student Questionnaire
SECTION 2: YOUR FAMILY AND YOUR HOME
2009-ST08: Family structure2009-ST09: Mother’s main job (ESCS, HISEI, BMMJ)2009-ST10: Mother’s education (ESCS, HISCED, PARED, MISCED)2009-ST11: Mother’s qualifications (ESCS, HISCED, PARED, MISCED)2009-ST12: Mother’s employment status (ESCS, HISEI, BMMJ)2009-ST13: Father’s main job (ESCS, HISEI, BMFJ)2009-ST14: Father’s education (ESCS, HISCED, PARED, FISCED)2009-ST15: Father’s qualifications (ESCS, HISCED, PARED, FISCED)2009-ST16: Father’s employment status (ESCS, HISEI, BFMJ)2009-ST17: Country of birth for student and parents (IMMIG)2009-ST18: Age at arrival in country of test2009-ST19: Language spoken at home2009-ST20: Home resources (ESCS, HOMEPOS, WEALTH, HEDRES, CULTPOS)2009-ST21: Family wealth (ESCS, HOMEPOS, WEALTH)2009-ST22: Books in home (ESCS, HOMEPOS)
PISA Contextual QuestionnairesStudent questionnaires
Examples of variables and indices
Comparability across countries (not a country-specific income indicator)
PISA index of Economic, Social and Cultural Status(ESCS)
HISEIHighest occupational status of parents
PAREDHighest educational
level of parents
HOMEPOSHome possessions
WEALTH
CULTPOSS
HEDRES
# of books in home
PISA Contextual QuestionnairesStudent questionnaires – examples of indices
WEALTH
CULTPOSS
HEDRES
# of books in home
Index of family wealtha room of their own, a link to the Internet, a dishwasher (treated as a country-specific item), a DVD player, and three other country-specific items (some items in ST20), and their responses on the number of cellular phones, televisions, computers, cars and the rooms witha bath or shower (ST21).
Index of cultural possessionsclassic literature, books of poetry and works of art (ST20)Index of home educational resourcesFor example, availability of a desk, a quiet place to study, a computer that students can use for schoolwork, educational software, books to help students, technical reference books and dictionary (some items in ST20)
PISA index of Economic, Social and Cultural Status(ESCS)
PISA Contextual QuestionnairesStudent questionnaires – examples of indices
2006-ST17: Self-efficacy in science (SCIEEFF)
2003-ST31: Self-efficacy in mathematics (MATHEFF)
Self-efficacy in science and mathematics
PISA Contextual QuestionnairesStudent questionnaires – examples of indices
Relationship between constructs, indices and variables (questions)
Stays the same in PISA 2012
06-ST17Q42 How easy do you think it would be for you to
perform the following tasks on your own?(Please darken only one circle in each row.)
I could do this easily
I could do this
with a bit of effort
I would struggle to do this
on my own
I couldn’t do this
a) Recognise the science question that underlies a newspaper report on a health issue
●1 ●2 ●3 ●4
b) Explain why earthquakes occur more frequently in some areas than in others
●1 ●2 ●3 ●4
c) Describe the role of antibiotics in the treatment of disease
●1 ●2 ●3 ●4
d) Identify the science question associated with the disposal of garbage
●1 ●2 ●3 ●4
e) Predict how changes to an environment will affect the survival of certain species
●1 ●2 ●3 ●4
f) Interpret the scientific information provided on the labels of food items
●1 ●2 ●3 ●4
g) Discuss how new evidence can lead you to change your understanding about the possibility of life on Mars
●1 ●2 ●3 ●4
h) Identify the better of two explanations for the formation of acid rain
●1 ●2 ●3 ●4
Self-efficacy in science
PISA Contextual QuestionnairesStudent questionnaires – examples of indices
2006-ST17: Self-efficacy in science (SCIEEFF)
2006-ST35: Instrumental motivation to learn science (INSTSCIE)
06-ST35
Q46 How much do you agree with the statements below?(Please darken only one circle in each row.)
Strongly agree Agree Disagree
Strongly disagree
a)Making an effort in my science class(es) is worth it because this will help me in the work I want to do later on
●1 ●2 ●3 ●4
b)What I learn in my science class(es) is important for me because I need this for what I want to study later on
●1 ●2 ●3 ●4
c) I study science because I know it is useful for me ●1 ●2 ●3 ●4
d)Studying science is worthwhile for me because what I learn will improve my career prospects
●1 ●2 ●3 ●4
e) I will learn many things in my science class(es) that will help me get a job ●1 ●2 ●3 ●4
Instrumental motivation to learn science
PISA Contextual QuestionnairesStudent questionnaires – examples of indices
Identify constructs in need of adaptation and important constructs that are missing
Obtain input and advice from experts in the field (grounded in developing-country contexts)
Obtain input and guidance from participating countries
Draw on existing work and empirical evidence
Test the constructs (indices and variables) to ensure comparability with PISA scales
PISA for DevelopmentHow will we do this?
PISA for Development
Technical Strand 1:
Contextual questionnaires
Are the intentions and rationale of this strand of work clear and understood?
Which components of the PISA questionnaires appear to demonstrate the greatest need of adaptation?
Where do you see the largest hurdle to overcome?
What existing studies and evidence should contribute to this strand of work?
PISA for DevelopmentQuestions for discussion