Making self review work for you Julie Foley & Dave Shepherd Education Review Office October 2010 [email protected]

Making self review work Making self review work for youfor you

Julie Foley & Dave ShepherdJulie Foley & Dave ShepherdEducation Review OfficeEducation Review Office

October 2010October 2010

[email protected]@ero.govt.nz

ERO self review presentationERO self review presentation 22

Talk outlineTalk outline

• Recapping concepts and activities from the 2009 presentation to the CPPA

• Schools’ self review and ERO’s 2010/11 reviews - how they can complement each other.

• Continuing to build evaluation capacity in the sector – ERO’s strategic theme

Evaluation Capacity Evaluation Capacity BuildingBuilding

ERO’s current thinking about ERO’s current thinking about internal and external evaluationinternal and external evaluation

Ko te Tamaiti te Putake o te KaupapaKo te Tamaiti te Putake o te Kaupapa

ERO’s conceptual frameworkERO’s conceptual framework

• Complementary evaluation takes key aspects of internal and external evaluation, and accountability and improvement, and adjusts these to the relevant context. It is informed by other evaluation theories and aims to resolve the conundrum raised by Nevo (2000, p.28) “Everybody seems to hate external evaluation

while nobody trusts internal evaluation”.

44

ERO’s Reviews 2010ERO’s Reviews 2010An evaluation approach that balances external ERO review with school self review according to each school’s circumstances.

55

Schools experiencing difficulties with very limited self review

Schools sustaining high quality

performance and continuous

improvement through effective self review

Schools operating well with established processes of self review


Self review is…Self review is…• Self review involves investigating evidence

about student outcomes and current ways of doing things to find out where improvement is needed. Planning for school improvement requires schools to set goals and targets for better student outcomes and make changes that are necessary to bring about those improvements.

(MoE Planning and Reporting, 2003)


Or…Or…• …the deliberate and on-going process of

finding out how well our practice enhances children’s learning and development. Review allows us to see which aspects of our practice are working well and what we could do better. As a result we can make decisions about what to do to improve. Through review our practice is transformed, and ultimately, our children’s learning benefits. (MoE, ECE, 2006)


Or…Or…

• Kura are most successful when there is an robust process of on-going internal review that leads to further improvement. Effective internal review focuses on the key goals for students and the evidence of the extent to which they are actually being achieved…[It is] rigorous, informed by excellence in evaluation practice, with findings based on evidence…

(ERO, Te Aho Matua, 2008)


Or…Or…• Self evaluation will collect and analyse a combination of

qualitative and quantitative data that helps leaders and teachers develop themselves, their practices, promote high performance, help support student progress and develop intervention programmes from a proactive perspective. (Paphitis, n.d.)

• Self evaluation is a dialogic process, a meeting of hearts and minds, a forging of new ways of doing, vital and continuing because it is the core of a school’s educational life. It is the essence of the learning community, the intelligent school, schools that learn.

(MacBeath, 1994)


Summary. Self review is:Summary. Self review is:

• Centred around giving all students the most successful educational experiences possible

• Set in a collaborative culture of continuous self reflection and constructive critique

• Self initiated by a desire to seek constant improvement – mostly planned but sometimes spontaneous

• Celebrating and sustaining what is working well while striving to improve what is not

• A willingness to embrace challenge and change• Setting long term goals and mid-term targets and

gathering rigorous and wide-ranging evidence to examine the extent to which these have been achieved

•


Different types of reviewsDifferent types of reviewsbut all linked to the big picturebut all linked to the big picture


Balancing accountability and Balancing accountability and improvementimprovement

Complementary evaluation

Eg Assurance statements

Eg External audit

Eg Self review

Eg ERO education review

INTERNAL

EXTERNAL

ACOUNTABILITY IMPROVEMENT


Turning data into evidence…Turning data into evidence…

• Schools gather lots of data – ‘hard’ (quantitative/ numerical) and ‘soft’; (qualitative/narrative). The raw or aggregated data is only the first level of analysis. It describes ‘what is’ or ‘what is happening’.

• The second level of analysis takes data and turns it into information. It explains ‘what this means’. It allows schools to make key statements or comparisons.

• The third level uses this information as evidence to support judgments (how well/to what extent), to make decisions (if this so then we need to…) and to prioritise (the most compelling need is…).


On-going review cycles…On-going review cycles…

Considering

Informing

MonitoringImplementing

Organising


What are we learning from our What are we learning from our workshops?workshops?

Schools do it differently:


What are we learning from ERO’s good What are we learning from ERO’s good practice in self review case studies?practice in self review case studies?

• Importance of leadership to drive the vision• Importance of ‘buy-in” from all involved• Belief that students can achieve• Belief that teachers can make a difference• Importance of the big picture• Commitment to the long term• Importance of prioritising• Remembering to celebrate and build on successes• Importance of gathering useful data and analysing it

effectively• Using strengths from in and outside the school• There is no one way- each school and community is

unique but we can learn from each other

DiscussionDiscussion

Developing your own definition – Developing your own definition – purpose statement purpose statement •Mapping your review activities – Mapping your review activities – quandrant diagramquandrant diagram•Finding an overall structure that Finding an overall structure that integrates your review activitiesintegrates your review activities•What process works for your school?What process works for your school?

Recapping Recapping

• Thinking about self review

• Tools and strategies suggested

• Feedback – • What did you use?• How well did it work for your school?• What changes have you made as a

result? • Share ideas


What ERO is finding in schoolsWhat ERO is finding in schools

• Pivotal Questions for internal review

• What is so?

• Why is it so?

• So what?

• Now what?


Planning reviewsPlanning reviews

• What do we need to review?

• Why?

• How?


Focus of reviewsFocus of reviews

• Balance needed

• What is on paper / documented

• School practices

• Outcomes for students


ERO’s reviewsERO’s reviews

What is the place of What is the place of

school’s self reviewschool’s self review

in 2010 and ahead?in 2010 and ahead?

Principles underpinning ERO’s Principles underpinning ERO’s reviewsreviews

2323

Enquiry focused

Responsive

Transparent

Collaborative & constructive

Building evaluation capacity


What do we know about effective What do we know about effective schools?schools?

• National and international research, eg, BES• ERO’s Evaluation at a Glance: Effective

Schools:– A focus on the learner– Strong leadership within an inclusive culture– Effective teaching– Quality engagement with parents, whānau and

communities– Coherent and consistent policies and practices within

a cycle of continuous self review


ERO’s differentiated review cycleERO’s differentiated review cycle• Rather than being based on a “one-size-fits-all”

three-yearly cycle, the differentiated review cycle is based on ERO’s confidence that the school is providing a quality education for all students and maintaining an on-going process of continuous self improvement

• Self review is the tool through which schools monitor and sustain this continuous improvement and provide evidence of what they are doing and what is working for their students

Deciding on Future ActionDeciding on Future Action• ERO intends to carry out another review over a

period of one-to-two years:

Building capacity• ERO is likely to carry out the next review in three

years: Developing capacity • ERO is likely to carry out the next review in four-

to-five years: Sustaining capacity – Criteria– Process Trends– Moderation


ERO’s Reviews 2010ERO’s Reviews 2010An evaluation approach that balances external ERO review with school self review according to each school’s circumstances.

2727

Schools experiencing difficulties with very limited self review

Schools sustaining high quality

performance and continuous

improvement through effective self review

Schools operating well with established processes of self review

Meta evaluation and designMeta evaluation and design

2828

Schools operating well with Schools operating well with established self review processesestablished self review processes

2929

Scoping & review design phase•using self review to set the scene•plan the evaluation, balancing the need for first hand evidence with evidence from self review

On site external review phase•Gathering evidence•triangulate data•ongoing & collaborative synthesis

Reporting phase•internal reporting•external reporting

use self review as the basis for external review

use external review to test, affirm, strengthen & broaden self review

Building & using school capacity to:

account for progress & achievement of goals, & generate further priorities for development & reviewSchool self reviewing & reporting to school

community

School self reviewing & reporting to school community

Schools performing very well with Schools performing very well with high quality self review processeshigh quality self review processes

3030

School builds & presents “a case” comprised of valid & significant examples of using self review to:•maintain high levels of student progress & achievement•make sound decisions & build continuous school improvement

External review phase•scope & design review using the school’s “case”•meta-evaluation of school’s self review•external reporting

challenge their own assumptions about the quality & efficacy of self review processes & findings

Using & building school capacity to:

engage with & use external meta- evaluation to enhance self review

School self reviewing & continuously improving & reporting to school community

School self reviewing & continuously improving & reporting to school community

Schools experiencing difficultiesSchools experiencing difficulties

3131

Development & review planning phase•planning the action, evaluation & support to improve student achievement

Implementing phase•implementing development•adapting & modifying (praxis)

Concluding phase•internal reporting•external reviewing•external reporting

identify purposes & goals, & plan what & how to evaluate

systematically implement planning & use data to self review, adapt & improve

Building school capacity to:

account for progress & achievement of goals & generate further priorities for development & review

3232

3333


ERO’s updated evaluation ERO’s updated evaluation indicatorsindicators

• Six dimensions:– Student learning: engagement, achievement

and progress– Effective teaching– Leading and managing the school– Governing the school– Safe and inclusive school culture– Engaging parents, whānau and communities

Focus on success for MāoriFocus on success for Māori

• Govt – MoE and ERO priority

• Based on clear evidence of disparity

• Expectations outlined in NAGs

• How well is your school promoting / achieving successful outcomes for Māori?

• What does your self review tell you about this evaluative question?



Support availableSupport available• From ERO

• Framework for reviews• Evaluation Indicators• Good practice case studies (not yet available)• Evaluation at a Glance: Effective Schools (not yet available)• Board/Whānau Assurance Statement Checklist• National Evaluations• Getting the most out of your ERO review booklets• Workshops and cluster meetings

• From other sources:• MoE self review tools for National Standards• MoE Kiwi Leadership material• MoE governance resource• Commercial tools• Local advisors/consultants• Each other!


To concludeTo conclude

• Be assured that self review is not new and much of what you already do fits under that umbrella

• Make on-going self review, reflection and action a “state of mind” in your school

• Ensure smaller review cycles feed data into strategic self review in a coherent way

• ERO are not looking for particular ways of doing things but at the quality of the evidence and the robustness of the processes that you use for your decision making

• Ask yourself : “How do you know that you make a difference for every student in your school?”

Documents

Making self review work for you Julie Foley & Dave Shepherd Education Review Office October 2010 [email protected]