53
OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes Synergies for Better Learning: An International Perspective on Evaluation and Assessment International Conference, Oslo, 11-12 April Synthesis Report: What have we learned? Deborah Nusche, Thomas Radinger, Paulo Santiago and Claire Shewbridge

OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Embed Size (px)

DESCRIPTION

Purpose: To explore how systems of E&A can be used to improve the quality, equity and efficiency of school education. Focus: A Review of national approaches to E&A in school education (primary and secondary schools) Comprehensive approach: The Review looks at the various components of E&A such as: Student assessment; Teacher appraisal; School evaluation; The appraisal of school leaders; Education system evaluation.

Citation preview

Page 1: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes

Synergies for Better Learning: An International Perspective on Evaluation and AssessmentInternational Conference, Oslo, 11-12 April 2013

Synthesis Report: What have we learned? Deborah Nusche, Thomas Radinger, Paulo Santiago and Claire Shewbridge

Page 2: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

1. Key Features of the Review2. Trends in Evaluation and Assessment (E&A)3. The evaluation and assessment framework4. Student Assessment5. Teacher Appraisal6. The Appraisal of School Leaders7. School Evaluation8. Education System Evaluation

Outline of Presentation

Page 3: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

1. Key Features of the Review

Page 4: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes

• Purpose: To explore how systems of E&A can be used to improve the quality, equity and efficiency of school education.

• Focus: A Review of national approaches to E&A in school education (primary and secondary schools)

• Comprehensive approach: The Review looks at the various components of E&A such as: – Student assessment; – Teacher appraisal; – School evaluation; – The appraisal of school leaders;– Education system evaluation.

• Large country participation: 26 education systems / 25 countries producing a CBR; 14 country reviews.

Page 5: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

1. Governance: Striking the right balance between central efforts and local initiative and between accountability and development

2. Procedures: Designing the right instruments to ensure E&A contribute to improvement of teaching and learning

3. Capacity: Developing competencies for E&A and for using feedback at all levels of the education system

4. Use of results: Organising evaluative information in such a way that it facilitates effective use by stakeholders; avoiding ‘misuse’ of E&A results

OECD Review on Evaluation and Assessment Frameworks: Key issues

Page 6: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

A wide range of groups involved• National co-ordinators and informal groups within countries to produce CBRs

• 28 external reviewers involved in OECD-led Review teams

• About 90 schools visited and over 2 800 persons interviewed

• Links with other international organisations and key stakeholder groups (BIAC, TUAC, EC, Eurydice, the World Bank, SICI, UNESCO)

• Collaboration with other OECD units (PISA, TALIS, CERI’s projects, NESLI)

A range of outputs• Synthesis report

• Background papers (11)

• Country Background Reports (26)

• Country Review Reports (14)

• Qualitative data collected (29 systems / 28 countries provided data)

• Meetings of the GNE on E&A (4)

• International Conference to launch synthesis report in Oslo 11-12 April

• Dissemination Conferences in countries

OECD Review on Evaluation and Assessment Frameworks: Stakeholders and outputs

Outputs available at www.oecd.org/edu/evaluationpolicy

Page 7: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

2. Trends in Evaluation and Assessment

Page 8: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Trends in Evaluation and Assessment

• Educational evaluation in school systems is expanding

– E&A is increasingly prominent in education policy– Dedicated agencies are being created as part of new approaches to govern E&A

• There is a greater variety of E&A activities

• Educational measurement and indicators development are rising in importance

– Student outcomes as increasingly the focal point for analysis– There is a growing emphasis on measuring student outcomes– There is a proliferation of education indicators

• Larger and more varied uses are given to E&A results

• Accountability as a purpose of E&A is gaining in importance

• There is greater reliance on educational standards

• Assessment is becoming more international

• Assessment involves greater technological sophistication

Page 9: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

3. The evaluation and assessment framework: Embracing a holistic

approach

Page 10: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

The evaluation and assessment framework: Components

1. Educational ContextThe educational context shapes the E&A framework

2. GovernanceObjectives

– Improving student outcomes! Often challenging to communicate.– Range of functions: Accountability, certification, development, diagnostic

– Challenge: balance between the development and the accountability functions

Responsibilities

– Shared among a wide range of agents– Dedicated intermediate agencies gain a prominent role– The devolution of responsibilities involves a variety of trade-offsIntegration of the non-public sector

3. Overarching reference: Goals for student learning

– General goals for education system (including equity); specific student learning objectives– Challenge: Alignment between goals for student learning and E&A

Page 11: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

The evaluation and assessment framework: Components

4. DesignPrinciples

Placing students at the centre; focussing on student outcomes; committing to transparency; culture of sharing classroom practice; relying on teacher professionalism; diverse student needs

Main components, Main elements within components

Articulations: Within components; between components; moderation processes; links to classroom

5. Capacity for Evaluation and AssessmentCompetencies for E&A

Skills for school leadership

Tools and guidelines for E&A

6. Use of resultsKnowledge management– Information systems; Identification of best practices; Innovation in education;

Evidence-based policy

7. Implementation of evaluation and assessment policiesDivergence of views and interests; consultations; involvement of professionals; clarity of purposes;

evidence to inform consensus-building; policy experimentation; evaluation of implementation; capacity building; resources; timing.

Page 12: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

The evaluation and assessment framework: Practices

Page 13: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

The evaluation and assessment framework: Practices

Page 14: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

The evaluation and assessment framework: Policy Options

1. Governance– Integrate the E&A framework, engaging in a strategic reflection– Align the E&A framework with educational goals and student learning objectives

– E&A to align with the principles embedded in educational goals, be based on fit-for-purpose procedures, and rely on a clear understanding of educational goals by school agents

– Secure links to the classroom and draw on teacher professionalism– Articulation of ways for the E&A framework to generate improvements in classroom practice

through the E&A procedures which are closer to the place of learning

– Promote national consistency while giving room for local diversity

2. Design and procedures

– Ensure core components are sufficiently developed within the E&A framework– Establish articulations between components of the E&A framework– Place the students at the centre of the E&A framework– Build on some key principles to effectively implement E&A

– Centrality of teaching and learning; importance of school leadership; equity as a key dimension; commitment to transparency

Page 15: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

The evaluation and assessment framework: Policy Options

3. Capacity

– Sustain efforts to improve capacity for E&A– Improve the articulation between levels of authority and assure support from the

centre

4. Use of results

– Maintain sound knowledge management within the overall E&A framework– Commit to the use of evidence for policy development

5. Implementation

– Anticipate potential implementation difficulties– Engage stakeholders and practitioners in the design and implementation of E&A

policies– Communicate the rationale for reform– Use pilots before full implementation and review implementation– Ensure adequate capacity and sufficient resources

Page 16: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

4. Student assessment: Putting the learner at the centre

Page 17: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Student Assessment

Evaluation and assessment framework

System

School

Classroom

Student assessment

Teacher appraisal

School evaluation

System evaluation

School leader appraisal

Internal and external assessment

Formative and summative assessment

1) Trends and current practices

2) The potential of student assessment

3) Common challenges in student assessment

4) Some policy options

Page 18: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Student Assessment: Practices

Trends

1) As national curricula increasingly emphasise ‘key competencies’ or ‘21st century competencies’ for lifelong learning, countries are seeking to adapt their assessment systems in order to capture such broader types of learning.

2) Research evidence on the benefits of using assessment results to inform teaching and learning has increased policy attention to formative assessment. 18/29 systems have central policy frameworks for formative assessment in place.

3) Summative assessment and reporting remain important at key stages of schooling in all countries. 21/29 systems had policy frameworks for internal summative assessment in place to ensure transparency in marking and reporting.

4) Concerns about the quality of learning and assessment across schools have led to a renewed focus on central standards and large-scale assessments to ensure high standards for all students. 26/37 systems have central standardised examinations in place.

Page 19: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Student Assessment: Analysis

The potential of student assessment

Formative assessment:• Diagnose learning needs and differentiate teaching; • Provide timely feedback to students• Actively engage students in their own learning; strengthen their self-monitoring • Help teachers and students adjust teaching and learning strategies

Summative assessment:

• Signal high standards and expected performance

• Motivate students to increase effort and achievement

• Provide information about performance to students, parents and others

• Certify learning and award qualifications

Page 20: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Student Assessment: Analysis

The challenges of student assessment

• Governance: Lack of alignment between central curricula, standards and assessment approaches; lack of clarity of purposes; tensions between summative and formative functions

• Procedures: Assessment formats tend to remain more traditional than curriculum goals; national assessments focus mostly on literacy and numeracy and predominantly use multiple choice and written tasks; limited use of ICT

• Capacity: Limited focus on students’ own assessment competencies; inadequacies in teachers’ and school leaders’ preparation and training

• Use of results: Parental concerns about inadequate reporting information; lack of transparency when using assessment results for high stakes decisions

Page 21: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Student Assessment: Policy Options

Governance: Establish a coherent framework for student assessment• Balance between formative and summative; internal and external assessment • Based on well-aligned reference documents (curriculum, standards, learning progressions, criteria to judge performance and exemplars illustrating different levels of achievement )• Clarity of purpose

Procedures: Ensure a comprehensive approach to student assessment• Draw on a variety of assessment types to get a rounded picture of student learning• Promote assessment formats that capture valued competencies• Build on innovative approaches to assessment, and tap into the potential of ICT• Provide tools and guidelines to support effective teacher-based assessment

Capacity: Make the assessment framework participatory and build assessment competencies across the system• Build student capacity to engage in their own assessment• Promote teacher professionalism in assessment;• Develop central expertise related to student assessment

Use of results: Ensure assessment is useful and informative• Provide clear reporting guidelines to ensure transparency and fairness in reporting results• Engage parents in education through adequate reporting and communication• Promote regular use of assessment results to foster further learning

Page 22: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

5. Teacher appraisal: Enhancing teacher professionalism

Page 23: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Teacher appraisal

Evaluation and assessment framework

System

School

Classroom

Student assessment

Teacher appraisal

School evaluation

System evaluation

School leader appraisal

1) Trends and practices

2) The potential of teacher appraisal

3) Common challenges in teacher appraisal

4) Some policy options

For performance management

For completion of probation

For rewards

Page 24: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Teacher Appraisal: Practices

Frameworks for teacher appraisal: trends• Teacher appraisal is the component of E&A frameworks where there is the most variation across countries

• Practices range from highly prescriptive national systems to informal approaches mostly left to the school level

• In many countries, there has been renewed focus on teacher appraisal in recent years, reflecting recognition that effective appraisal can contribute to improved teaching quality

• Many systems (21/29) have developed central standards for the teaching profession that can guide teacher appraisal processes

• Most systems (23/29) have policy frameworks for teacher appraisal in place.

Page 25: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Teacher Appraisal: Practices

Existence of policy frameworks for teacher appraisal, 2011-12

Reward scheme

Performance management

Probation

Country

Aust

ralia

Aust

ria

Belg

ium

(Fl.)

Belg

ium

(Fr.)

Cana

da

Chile

Czec

h R

epub

lic

Denm

ark

Esto

nia

Finl

and

Fran

ce

Hung

ary

Icel

and

Irela

nd

Isra

el

Italy

Kore

a

Luxe

mbo

urg

Mex

ico

Neth

erla

nds

New

Zeal

and

Norw

ay

Pola

nd

Portu

gal

Slov

ak R

epub

lic

Slov

enia

Spai

n

Swed

en

UK (N

orth

ern

Irela

nd)

Source: Information collected from countries participating in the Review

Page 26: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Teacher Appraisal: Analysis

The potential of teacher appraisal• Provide feedback to teachers, identify suitable professional development

• Allow teachers to progress in their career and take on new responsibilities based on solid evaluations of their performance

• Allow school leaders to be accountable for the quality of education in every classroom; move on persistently underperforming teachers

Page 27: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Teacher Appraisal: Analysis

The challenges of teacher appraisal• Governance: High stakes teacher appraisal may lead to a climate of stress and anxiety; tensions between professional learning and

accountability function of teacher appraisal

• Procedures: Lack of a professional profile or standards to guide appraisal in some countries; overreliance on one or two sources of information; insufficient guidance for classroom observation; appraisal may increase workload for teachers; inadequate use of student results to evaluate teachers

• Capacity: Limited preparation of teachers to benefit from their appraisal; school leaders insufficiently trained for HR management and appraisal; lack of legitimacy of evaluators

• Use of results: Difficulties in ensuring systematic follow-up with professional development; absence of career opportunities for teachers; controversies around the use of appraisal results for rewards.

Page 28: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Teacher Appraisal: Policy Options

Governance: Establish a coherent framework for teacher appraisal• Consolidate regular developmental appraisal at the school level• Establish periodic career-progression appraisal involving external evaluators • Resolve tensions between the developmental and accountability functions

Procedures: Ensure a comprehensive approach to teacher appraisal • Establish teaching standards to guide teacher appraisal• Use multiple instruments and sources of evidence• Provide support for effective classroom observations • Avoid simplistic use of student assessment results for teacher appraisal

Capacity: Build capacity of both evaluators and evaluatees• Prepare teachers for their role in the appraisal process • Strengthen the capacity of school leaders for teacher appraisal • Ensure that designated evaluators are qualified for their role

Use of results: Feed results into professional development, career advancement and accountability• Ensure that teacher appraisal feeds into professional development and school development • Establish links between teacher appraisal and career advancement decisions• Ensure that underperformance is identified and adequately addressed

Page 29: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

6. The appraisal of school leaders: Fostering pedagogical leadership in

schools

Page 30: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

School leader appraisal

Evaluation and assessment framework

System

School

Classroom

Student assessment

Teacher appraisal

School evaluation

System evaluation

School leader appraisal

1) Analytical approach

2) Trends and practices

3) The potential of school leader appraisal

4) Common challenges

5) Some policy options

Page 31: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

School Leader Appraisal: Analytical approach

Conceptualising school leadership– Impact of school leadership– The nature of effective leadership in schools– The development of new and diverse models of school leadership

Scope– Appraisal of individual school leaders– Performance-management, employment-related decisions and rewards purposes

Page 32: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

School leader appraisal: Practices

Approaches to the appraisal of principals in public schools 1. No appraisal (6/29 countries)

2. Local appraisal procedures and implementation (10/29 countries)

3. Central/state requirements for appraisal (17/29 countries)– Employment-related appraisal (3/17 countries)– Mandatory periodic appraisal (14/17 countries)

Existence of central/state policy frameworks (15/17 countries)1. Implementation through central/state, regional and/or intermediate authorities

(9/15 countries)

2. Implementation through local authorities and/or school organising bodies (9/15 countries)

Objectives of appraisal1. Summative purpose (8/17 countries)

2. Formative purpose (3/17 countries)

3. Combination of summative and formative purposes (7/17 countries)

Page 33: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

School leader appraisal: Analysis (1)

The potential of school leader appraisal• Tool to communicate a vision of effective school leadership

• Tool to influence school leaders’ practices and behaviours

• Opportunity for reflection, feedback and support

• Risk of increasing school leaders’ work load and stress levels

• Concerns about existing approaches to appraisal in various countries

Page 34: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

School leader appraisal: Analysis (2)

Challenges for school leader appraisal• Governance: Balancing consistency through central/state frameworks with local diversity; Combining developmental and

accountability functions

• Procedures: Ensuring a focus on pedagogical leadership and scope for local contextualisation; Gathering a feasible amount of information to yield an accurate, fair, valid and reliable picture of performance; Accounting for student outcomes; Reflecting new and diverse models of school leadership

• Capacity: Focussing attention towards the development of capacity among evaluators and school leaders

• Use of results: Establishing feedback as a core element of appraisal; Ensuring links between appraisal and meaningful professional development opportunities; Establishing opportunities for career advancement

Page 35: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

School leader appraisal: Policy Options

Governance: Promote school leader appraisal within the E&A framework • Develop a central/state policy framework to strengthen the systematic appraisal of school leaders• Ensure scope to adjust procedures to local, school and individual circumstances• Ensure links between school evaluation and school leader appraisal

Procedures: Promote the appraisal of pedagogical leadership with scope for local adaptation • Develop professional standards for school leaders• Ensure scope for the local selection of appraisal aspects and criteria in line with central/state frameworks• Provide school leaders with opportunities to further develop pedagogical leadership skills• Promote the appraisal of school leaders’ competencies for monitoring, evaluation and assessment

Capacity: Build capacity of both appraisers and school leaders• Prepare school leaders and evaluators for their role in the appraisal process • Promote school leader appraisal as an opportunity for peer learning

Use of results: Ensure links between appraisal and professional development, and develop career advancement opportunities• Ensure that appraisal informs meaningful professional development that pays attention to individual and school needs• Develop career advancement opportunities that are informed by appraisal procedures

Page 36: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

7. School evaluation: From compliancy to quality

Page 37: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

School evaluation

Evaluation and assessment framework

System

School

Classroom

Student assessment

Teacher appraisal

School evaluation

System evaluation

School leader appraisal

School self-evaluation / review

External evaluation

(inspection / review)

School performance measures

Page 38: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

School Evaluation: Practices

• External school evaluation is established in the vast majority of OECD countries (Except: FIN, GRC, HUN, ITA, JPN, LUX & MEX)

– Typically devised by central or state authorities and conducted by Education authorities and/or specific bodies (e.g. School Inspectorate, School Review Body)

– Publication of comparative school performance measures• Almost universal focus in national policy to stimulate school

self-evaluation (No requirements in GRC, ITA, MEX and ESP)

– Requirements vary significantly in nature, e.g. conduct self evaluation; produce specific report on school development; account for school quality

– Providing comparative information to schools on performance and other measures

Page 39: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

School Evaluation: Analysis - governance

• Fitting school evaluation policies to the wider governance context– Public demands for school performance information– Optimising use of resources for external school evaluation (cycles,

proportionality)

• Articulating external school evaluation and school self-evaluation

• Risks that compliancy dominates school evaluation– Multiple forms of school accountability– Demands for evidence in external evaluation / administrative burden in

schools– Documentation v. classroom observation– Requirements for schools to develop specific reports

Page 40: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

School Evaluation: Analysis - procedures

Refence standards Instruments and information sources

COMPLIANCY Legal standardsAdministrative school reporting; Verification during external school evaluation visits

QUALITY:

Educational practices

Criteria to evaluate educational processes, notably teaching and learning

Classroom observation - and evaluation indicators; Mechanisms for feedback from students, teachers and parents

Outcomes Student learning objectivesSchool-developed tests to monitor student progress;Standardised assessments /examinations

Student, parent and staff satisfaction

School-developed surveys, student and school councils/ centrally-developed surveys or questionnaires as part of standardised assessments

School capacity for self-evaluation

School leadership; Effectiveness of school self-evaluation; School capacity to improve

Verification of school self-evaluation processes and results during external school evaluation visits

Page 41: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

School Evaluation: Analysis - capacity

Related challenges for capacity

COMPLIANCYEnsuring legal knowledge/competencies within the external school evaluation body and school leadership;Reducing the burden of compliancy reporting on schools

QUALITY:

Educational practices

Recruitment and training of external evaluators/reviewers; Training school leadership in undertaking classroom observation;Stimulating and supporting peer reviews among schools;Central support tools for stakeholder surveys;

Outcomes

Engaging teachers in analysis of student results and giving them autonomy to make changes for improvement; Time and resources at the school level to analyse dataCentral capacity to develop suitable standardised assessments and capacity to report back results

School capacity for self-evaluation

Ensuring leadership of school self-evaluation activities:Building trust in self-evaluation processes;Engaging school leaders in external school evaluation activities;Creating roles and responsibilities within the school; Engaging the full community in school self-evaluation activities

Page 42: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

School Evaluation: Analysis – use of results

Related challenges for the reporting and use of results

COMPLIANCY Separating school audit and school self-evaluation reporting;Sanctions for schools not complying with legal standards

QUALITY:

Educational practices

Reporting of results of external school evaluation;Immediate feedback to teachers and school leaders;Feeding into professional development and school developmentMechanisms to follow up how schools act on external school evaluation results when educational practices are identified for improvement

Outcomes

School developed tests: Integrating these results into analysis of school development priorities, including professional development needsStandardised assessments/ examinations: Timeliness of feedback of data; validity and relevance of data; tools for analysis; accuracy of reporting and contextualisation of resultsIntegration of student, staff and parent satisfaction results into school reporting and development plans

School capacity for self-evaluation

School reporting to its community; Integrating the use of self-evaluation results in school development/strategic improvement cycles; School capacity and the cycle/intensity of external school evaluations

Page 43: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

School Evaluation: Policy Options

Governance: Underlies all options on procedures, capacity and use of results• Clarify the role and purpose of school evaluation• Focus on the improvement of teaching, learning and student outcomes• Evaluate and adapt external school evaluation to reflect the maturity of the school evaluation culture• Raise the profile of school self-evaluation

Procedures: Ways to minimise the burden and maximise the fairness• Develop nationally agreed criteria for school quality to guide school evaluation• Develop appropriate resources for school self-evaluation• Ensure transparency in external school evaluation procedures and a strong evidence base

Capacity: A priority for school improvement• Ensure credibility of external school evaluators and sufficient capacity and retraining as necessary• Strengthen school principals’ capacity to stimulate an effective school self-evaluation culture• Promote the engagement of all staff and students in school self-evaluation• Promote peer learning among schools

Use of results: Learning what to improve and where to intervene• Promote wider use of external school evaluation results and ensure systematic follow up• Optimise feedback of centrally collected data to schools for self-evaluation and development planning• Report a broad set of school performance measures with adequate contextual information

Page 44: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

School Evaluation: Policy Options

Governance: Underlies all options on procedures, capacity and use of results• Clarify the role and purpose of school evaluation• Focus on the improvement of teaching, learning and student outcomes• Evaluate and adapt external school evaluation to reflect the maturity of the school evaluation culture• Raise the profile of school self-evaluation

Procedures: Ways to minimise the burden and maximise the fairness• Develop nationally agreed criteria for school quality to guide school evaluation• Develop appropriate resources for school self-evaluation• Ensure transparency in external school evaluation procedures and a strong evidence base

Capacity: A priority for school improvement• Ensure credibility of external school evaluators and sufficient capacity and retraining as necessary• Strengthen school principals’ capacity to stimulate an effective school self-evaluation culture• Promote the engagement of all staff and students in school self-evaluation• Promote peer learning among schools

Use of results: Learning what to improve and where to intervene• Promote wider use of external school evaluation results and ensure systematic follow up• Optimise feedback of centrally collected data to schools for self-evaluation and development planning• Report a broad set of school performance measures with adequate contextual information

Page 45: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

8. Education system evaluation: Informing policies for system

improvement

Page 46: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Education system evaluation

Evaluation and assessment framework

System

School

Classroom

Student assessment

Teacher appraisal

School evaluation

System evaluation

School leader appraisal

Sub-national education system

e.g. local authority, schools in religious/pedagogical network

National education system

Page 47: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Education System Evaluation: Practices

• Major increase in use of monitoring systems across OECD– Large-scale student assessments

• By the late 1990s all OECD countries had participated in an international study• Development of national assessments

– Stakeholder surveys and longitudinal information

• Thematic evaluations in samples of schools

AUS

AUT

BFL

BFR

CAN

CHL

CZE

DNK

EST

FIN

FRA

HUN

ISL

IRL

ISR

ITA

KOR

LUX

MEX

NLD

NZL

NOR

POL

PRT

SVN

SVK

ESP

SWE

UK-NI

Nat Asst: Full cohort Nat Asst: Sample Survey: Students Survey: Teachers

Survey: Parents Longitudinal information

Page 48: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Education System Evaluation: Analysis - governance

• Calls to monitor performance in the public sector• Recognising the economic importance of education• Many systems do not have an overall framework for

education system evaluationIndicators of a strategic approach information collection

Countries

Mapping against system priorities and plan to prioritise new collection

Australia; Czech Republic; Hungary; Israel; Netherlands; Slovak Republic

Mapping against system priorities France; Iceland; Ireland; Northern Ireland (UK)

Plan to prioritise collection of new information

Belgium (French & Flemish Comm.); Chile; Finland; Slovenia; Spain

Neither Austria; Denmark; Italy; Korea; Luxembourg; Mexico; New Zealand; Norway; Poland; Sweden

Page 49: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Education System Evaluation: Analysis - procedures

• Reference standards may be many– Broad goals for education system; Student learning objectives / standards;

Objectives set for specific education policies– The use of targets for system performance

• E.g. European Union benchmarks• Improvement or decline over time in specific assessments

• Monitoring trends in equity and quality – Availability and reliability of contextual information– Prioritising reliability of national assessments, but validity?

• Multiple choice (27); Closed-format short answer (21); open-ended writing tasks/calculations (17); Performing a task (7); oral elements (6)

• Broader curriculum coverage with sample surveys v full cohort

– Capitalising on technology– Tensions: monitoring trends and holding schools accountable– Specific reviews on aspects of the school system

Page 50: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Education System Evaluation: Analysis - capacity

• Capacity to produce evidence on system performance and use results– The creation of specific bodies– Credibility and objectivity of evidence and reporting underlying policy

development– Capacity to synthesise and disseminate results

• Evaluation capacity at sub-national level may vary significantly– Local monitoring and school use of data

• Aligning goals• Building trust

Page 51: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Education System Evaluation: Analysis – reporting and use of results

• Communicating education system evaluation results clearly and comprehensively– Reporting principles: relevance; credibility; timeliness; accessibility and

interpretability– Avoiding media misinterpretation– Engaging key stakeholders in discussion of results

• Making better use of results in planning and policy development– Too much information! Findings ways to better channel results– Overall summative report on the education system– Political urgency versus availability of broad research base– Research and empirical analysis– Information systems to promote use of results at local levels

Page 52: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Education System Evaluation: Policy Options

Governance: Being systematic and strategic for better informed policy making• Ensure a broad concept of education system evaluation within the E&A framework• Ensure policy making is informed by high-quality measures, but not driven by their availability• Situate education system evaluation in the broader context of public sector performance requirements

Procedures: Developing an approach to learn from a broad evidence base• Develop a national education indicator framework • Design a national strategy to monitor student learning standards• Ensure the collection of: qualitative information; and contextual information to monitor equity• Assure the monitoring of changes over time and progress of particular student cohorts

Capacity: Ensuring continuity and credibility• Establish and secure capacity for education system evaluation• Promote the development of evaluation capacity at the local authority level• Ensure objectivity and credibility in education system evaluation activities

Use of results: Strengthening analysis for system improvement• Strengthen analysis of education system evaluation results for planning and policy development• Communicate key results of education system evaluation to stakeholders• Support feedback for local monitoring

Page 53: OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Synthesis Report: What have we learned?

Thank you for your attention!

www.oecd.org/edu/evaluationpolicy