Upload
others
View
7
Download
0
Embed Size (px)
Citation preview
Statewide High School Exit Exams and Data-‐Based School Improvement:
A European Story Esther Dominique Klein
BEAR Seminar – UC Berkeley – February 23, 2016
Program
What is a “statewide exit exam”?
Why are statewide exit exams so popular?
One size doesn’t fit all -‐ Different statewide exit exam formats
Statewide exit exams and school improvement
State intenTons…
…and school responses: Does the message “hit home”?
What is a “statewide exit exam”?
Statewide exit exams...
(Defini'on adapted from Kellaghan and Madaus‘, 2003, defini'on of public exams)
are exams taken at the end of a course of studies that are…
mandatory for graduaTon
based on prescribed syllabi in curriculum
taken in a common test situaTon
controlled by an agency external to the schools
based on tasks or assignments that are set by an agency external to the schools
usually administered by a naTonal or state government
Exit Exams in Europe (as of 2010)
state-‐wide exams only
state-‐wide and school-‐based exams
school-‐based exams only
no exit exams or no common procedure
Germany before 2005 and aDer 2008
Länder with statewide Abitur exams (Zentralabitur) Länder with school-‐based Abitur exams Länder with a mix of both
before 2005 a^er 2008
Curriculum-‐based statewide exit exams 2002
New York Texas*
2008
Mississipi New York Tennessee Virginia
2012
Alabama* Arkansas Indiana Louisiana Maryland Massachusegs* Mississippi New York Oklahoma Virginia
(Zabala et al., 2008) (McIntosh, 2012)
Why are statewide exit exams so popular?
In recent years educa'onal policy makers have recognized what was known by the Imperial Chinese court over a thousand years ago (…): that a
na'onal external examina'on system can be a powerful instrument for moving schooling in a desired direc'on.
Noah & Eckstein, 1992
10
SWEE as potenJal “allrounders”
Achievement
Accounta-‐bility
InnovaTon
Professional learning
• Formal: e.g., cerTficates, comparability
• Contents: e.g., standards assurance, control of the delivered curriculum
For both students and teachers/schools
ImplementaTon of new curricula or new methods of delivery and assessment • Teachers: improving
diagnosTc skills, reduce uncertainTes, iniTate reflecTve thinking
• Schools: External validaTon and evaluaTon, improving collaboraTon and other organizaTonal rouTnes
Statewide exit exams are perceived as homogeneous
SWEE as a tool for school improvement
Heterogenous exam formats
Research focuses on the effects of student achievements and instrucJon
Research Project „Forms, FuncJons, and Effects of Statewide Exit Exams“, funded by the German Research FoundaJon
• Interview Study • Document Analysis
State IntenTons
• QuesTonnaire Survey
School ImplementaTon
Review of Exam Procedures in OECD countries ● Document Analysis ● Email Interviews
Exam Features
One size doesn‘t fit all:
Different Statewide Exit Exam Formats
Analyzing different SWEE formats
• e.g., structure of secondary educaTon, historical context, development of exam tasks, student choice, exam supervision, marking procedures, use of exam results, ...
Exam procedures
• e.g., specificity, standardizaTon, focal / core themes, ...
Curriculum
• e.g., task formats, complexity / difficulty, curricular validity, ... Tasks
• e.g., specificity, standardizaTon, and bindingness of marking schemes, actual compliance with marking schemes, ...
Marking
Document analysis
Interviews
Curriculum / Task analysis
ObservaTon of marking behavior
...
DescripTon of statewide exit exam procedures at the end of ISCED 3A
17 OECD countries in Europe**
→ 31 naTonal or state authoriTes
A. content analysis (Eurybase, countries‘ relevant websites, legal documents)
B. expert interviews in some countries
(email, unstructured)
** no federal countries except Germany
Cycle 1 Study Procedures
(Klein et al., 2009; Klein & van Ackeren, 2011)
What sub
jects • statewide vs.
school-‐based • core vs. specializaTon
• students‘ choice? W
hat type • wrigen vs. oral
• students‘ choice?
How diffi
cult • different
difficulty levels?
• role of difficulty level for final grade
• students‘ choice?
Differences in the organizaJon
Who
sets th
e pape
rs
• private vs. state-‐run • bogom-‐up vs. top-‐down
• unit sexng the papers
How are papers set
• guidelines and specificaTons • quality assurance • standards assurance
• procedures
Development of exam tasks
Who
doe
s it • marking stages
• role of class teacher • role of second marking • external marking • marker choice • marker training
How is it don
e • control • marking schemes • Standards assurance
Marking
Handling of results Final grade
• weighTng • role of school-‐based exams
• role of conTnuous assessment
Feed
back and
use
• System monitoring • Control and Accountability
• Benchmarking and ProfessionalizaTon
• Student selecTon
High or low
stakes?
Statewide exit exams as a tool to support improvement?
Purposes of statewide exit exams
Achievement
Accounta-‐bility
InnovaTon
Professional learning
• Formal: e.g., cerTficates, comparability
• Contents: e.g., standards assurance, control of the delivered curriculum
For both students and teachers/schools
ImplementaTon of new curriculua or new methods of delivery and assessment • Teachers: improving
diagnosTc skills, reduce uncertainTes, iniTate reflecTve thinking
• Schools: External validaTon and evaluaTon, improving collaboraTon and other organizaTonal rouTnes
“Educa@onal Governance” – Managing the mulJ-‐level school system
Macro level
Meso level
Micro level
§ Not one single “governing unit”, but instead mulTple actors with their own logics
§ Impulses from other actors are “recontextualized” (Fend, 2008)
§ Interdependencies between different actors
§ “Actor constellaTons” à acTons must be coordinated between different actors
(e.g., Altrichter & Maag Merki, 2010; Kussau & Brüsemeister, 2007)
External actors
Inter-‐mediary actors
How can quality and effec@veness be assured and improved in a system like this?
Actual Outcome
Target Outcome
Comparison of target and actual
outcome
NaTonal tests Statewide exit exams
School processes
ReflecTon Strategic decisions
for school development
Standards
Reinforcement
If congruent
If not congruent
InterpretaTon RecontextualizaTon
ModificaTon
Expected cause and effect model (Maag Merki, 2010; shortened version)
24
Accountability, support, incenTves
Individual, school, society, and policyfactors
“Technology” of the exams
Use of the exams (School, department, teacher level)
OrganizaTonal rouTnes (School, department, individual level)
Intended and unintended effects
of the exams
FuncTons and strategies
ImplementaTon process
Governance system, System of quality assurance
Local administraTon, district
Exam Use Model (Klein, 2013, adapted from Visscher, 2002)
Research quesJons IntenJons of educaJon policy • How do policy makers in different cultural and poliTcal sexngs want state-‐wide exit exams to affect schooling?
Responses of school actors • How do school actors respond to the exams, and what factors on system, school and subject level influence their responses?
26
Ø InternaTonal comparaTve case study of statewide exit exams at the end of upper secondary educaTon (ISCED 3A)
Ø MulT-‐method/mulT-‐perspecTve approach (Yin, 2009) Ø Exploratory approach
Context-‐free research quesTon à DescripTon à JuxtaposiTon and Comparison à Hypotheses (Philips & Schweisfurth, 2007)
The three cases Finland Ireland Netherlands
Exam Ylioppilastutkinto Leaving Cer'ficate Examina'ons Eindexamen vwo
Exam type Statewide exit exams
Statewide exit exams
Statewide (50%) und school-‐based (50%) exit exams
StandardizaTon of the exam procedures
moderate high SWEE: high SBEE: low
Governance* LegiTmacy type with Local Empowerment
LegiTmacy / Bureaucracy type with School Empowerment
LegiTmacy / Efficiency type with School Empowerment
*Schmid et al., 2007; Glager et al., 2003
State intenJons
Analyzing state intenJons
QuanJtaJve: • e.g., correlaTon between exam type and demographic data or achievement data
QualitaJve research: • Document analyses (websites, press releases, educaTon codes)
States with statewide high school exit exams have a higher unemployment rate and more diversity among students à SWEE are a means to counteract
disadvantages and inequiTes in the acquisiTon of cerTficates and access to higher educaTon
(Warren & Kulick, 2007)
SWEE are used for „a higher transpa-‐ rency of the requirements, a higher comparability of the achievements, and more equity in the allocaTon of cerTficates. [...] Peformance of the educaTon system as a whole: [...] Based on the data from SWEE, improvement measures can be iniTated.“ (h]p://www.standardsicherung. schulministerium.nrw.de/zp10/ziele/ ; transla'on by me)
DescripTon of state intenTons connected to statewide exit exams
Content analysis
(countries‘ relevant websites, legal documents, brochures, literature)
Expert interviews with representaTves of the exam
bodies (N = 9) semi-‐standardized interview
protocol
Cycle 2a Study Procedures
(Klein 2013)
FuncJons of the exams
Finland • Formal qualificaTon of students
• No agempt to influence school processes
• No school inspecTon • Data feedback as offer • No development expectaTon
Ireland • Formal qualificaTon of students
• Completely external • No agempt to control schools
• School inspecTon with low stakes for schools
• No development expectaTon
Netherlands • 50% statewide, 50% school-‐based exams
• Exam results as benchmark for student achievements
• MulT-‐dimensional internal and external evaluaTon system
• Clear development expectaTon
School responses:
Does the message “hit home“?
• Monitoring
• Policies • Strategic use • Professional Development
• Support conversaTons • ReflecTon • Curricular and instrucTonal decisions
(Schildkamp et al., 2012)
Different users with different logics
• AdministraTon / data-‐wise Leadership
• Teachers as the “deliverers” of instrucTon
• ...
Different types of data use by schools
Use of SWEE data for school improvement
Perceived usefulness and use of data
• England: Perceived benefit of retroacTve feedback is low (Saunders & Rudd, 1999)
• Germany: Perceived usefulness of data is higher for SWEE than for other tests (Wurster et al., 2013), but the actual use of the data is not higher and is restricted to decisions on the curriculum (Wurster et al., 2013; Wurster, 2015)
OrganizaJonal factors
• School organizaTonal factors lead to differences in uTlizaTon (DeBray, 2005; DeBray et al., 2000; Saunders, 2000; Saunders & Rudd, 1999; Schildkamp & Kuiper 2010; Schildkamp et al. 2012)
• Schools need support for data-‐based school development (DeBray et al., 2000)
• Even with training the use of data from exit exams is low (Tymms, 1995)
Is the school actors’ percepTon of the exam funcTons coherent with
the intenTons of the state?
How do school actors uTlize the exams for school and classroom
development?
Cycle 2b Study Procedures
(Klein 2013)
Standardized quesTonnaire survey
35 upper secondary university-‐bound schools • 15 schools in Finland • 13 schools in Ireland • 7 schools in the Netherlands • N = 35 principals • N = 385 teachers
Data use at school level
0%
25%
50%
75%
100% Benchmarking
Monitoring
Vergleich
Rechensch_L
EvaluaTon_L
EvaluaTon_SL
PublikaTon
U-‐entwickl.
Kursdiff.
U-‐ressourcen
Finland
0%
25%
50%
75%
100% Benchmarking
Monitoring
Vergleich
Rechensch_L
EvaluaTon_L
EvaluaTon_SL
PublikaTon
U-‐entwickl.
Kursdiff.
U-‐ressourcen
Ireland
0%
25%
50%
75%
100% Benchmarking
Monitoring
Vergleich
Rechensch_L
EvaluaTon_L
EvaluaTon_SL
PublikaTon
U-‐entwickl.
Kursdiff.
U-‐ressourcen
Netherlands
N = 15 N = 12
N = 6
Comparison Comparison
Comparison
Teacher Acc. Teacher Acc.
Teacher Acc.
Teacher Eval. Teacher Eval.
Teacher Eval.
Principal Eval. Principal Eval.
Principal Eval.
PublicaTon PublicaTon
PublicaTon
Instr. Develop. Instr. Develop.
Instr. Develop.
Streaming Streaming
Streaming
Instr. Resources Instr. Resources
Instr. Resources
(Klein, 2013)
Teacher collaboraJon in the context of the exams
2,0
2,5
3,0
3,5
4,0
A1 A2 A3 B1 B2 C1 C2 A1 A2 A3 B1 B2 C1 C2 A1 A2 A3 B1 B2 C1 C2
Finnland Irland Niederlande
Means and
95%
con
fiden
ce inervals
5 point Likert scale; 1 = not at all true; 5 = totally true
A: IntegraJng exams in lessons (exchange of experience, synchronize teaching of exam content) B: InstrucJonal improvement (using exams and data to improve teaching)
C: StandardizaJon (use of older exam tasks, adjustment state standards)
(Klein, 2013)
Exam use at individual level
3,0
3,5
4,0
4,5
5,0
Improvement Assessment Improvement Assessment Improvement Assessment
Finland Ireland Netherlands Finland
Means and
95%
con
fiden
ce intervals
Use of exam tasks and results for...
5-‐point Likert scale; 1 = strongly disagree; 5 = strongly agree
*** p < .001; ** p < .01; * p < .05
MA-‐B = 0,29 t (128) = 3.11**; d = .31
MA-‐B = 0,89 t (148) = 8.77***; d = .92
MA-‐B = 0,20 t (53) = 1.78, n.s.; d = .26
(Klein, 2013)
JuxtaposiJon and comparison
Finland Ireland Netherlands
FuncTon of the exams
Formal qualificaTon and development
Formal qualificaTon
Formal qualificaTon, moTvaTon, and development
Relevance in schools lower higher higher
School data use
Individualized rouTne
One-‐dimensional purposive
MulT-‐dimensional systemaTc
(Klein, 2013)
Discussion
• Heterogeneous Exams à Heterogeneous Inten@ons à Heterogeneous Responses
Ø The “governing effect” of the exam system becomes visible through the internaTonal comparison
• SWEE are only used for data-‐based school improvement under certain condi@ons
Ø Different funcTons and purposes Ø Data are used at the organizaTonal level, but to a lesser extent at the individual level
Ø Aspects of the macro level might help change collecTve rouTnes
• Variance within countries indicates school-‐specific use Ø Needs more in-‐depth analyses
Thank you!