Making reform happen and evaluating reform in education

Making reform happen and evaluating reform in education:A ‘knowledge management’ approach

Dirk Van DammeHead of the Centre for Educational Research and Innovation - OECD

2

Outline

A. Knowledge management in education

B. Knowledge as part of systemic innovation

C. Making reform happen in education

D. The ‘GPS’ approach

E. Governance and knowledge

F. Conclusions and questions

3

KNOWLEDGE MANAGEMENT IN EDUCATION

A.

Educational R&D

• Striking findings that education, as a knowledge sector, has a very weak knowledge base itself…– Low levels of educational R&D (but

difficulties in finding a methodology for comparable data collection)

– Much lower than related public policy sectors such as health or social policy

• A weak empirical research capacity…– Especially for quantitative research

• And a weak link between research and policy 4

Educational R&D

5

Educational R&D

6

Educational R&D

7

Evidence in Education

• The emergence of evidence in education…– Methodologically sound solutions found for

measurement issues in education– Comparative education indicators

• Moving from inputs to outputs and outcomes

– The PISA shock– The development of feedback systems, at

student, school and system levels• …dramatically changed the policy climate

8


• Important methodological issues and debates on what counts as (scientific research) evidence– (quasi-)experimental design, randomised

control trials, …– Lack of large-scale longitudinal studies– Scientific ideal versus pragmatically

feasible– Cost and capacity problems– Ethical issues about educational

experimenting, privacy issues related to data collection 9


• Research – Policy interaction– Not a simple model of direct impact of

evidence on policy, but mediation by all stakeholders and actors in a complex system• The influence of knowledge on policy making

may in fact be strongest not when it comes directly from the educational research community in direct advice to policy makers but when it is filtered through actors such as print or broadcast media, lobbyists, popularisers, etc. (EU, 2007, p.5)

– Mediation seems to be the weakest link in the knowledge chain

10


11

12

13

KNOWLEDGE AS PART OF SYSTEMIC INNOVATION

B.

14

Systemic Innovation in Education

• Objectives– Investigate how systems go about change – Processes and stakeholder relationships– Knowledge Management perspective

• 14 case studies: Australia, Denmark, Germany, Hungary, Mexico, and Switzerland

• Factors that facilitate/impede the use of evidence

• Lessons learned about the use of evidence

Enablers of innovation

• Consensus building• Political vision• Research evidence• Brokerage: generation and dissemination

of knowledge– Legitimating rigour/quality– Developing cooperation/trust

Barriers to innovation

• Change fatigue• Competing policy agendas• Accountability mechanisms and public

policy agendas:– Restricted risk management– Short-term planning

Model of Change in Education

Identification of needs

Development of innovation

Implementation

Evaluation & Monitoring

Outcomes

Output

Knowledge base

Implementation process

Without piloting: large-scale implementation With piloting:

1. Small-scale implementation2. Monitoring/evaluation3. Scaling-up

Knowledge base

What types of knowledge? Tacit knowledge

Explicit knowledge

What knowledge sources?

Evaluation & Monitoring = surveillance/ judgement of outcomes

How and when?

What criteria are used?

Summative or formative purpose?

What are the findings?

Identification of needs

What are the drivers of change?

Which stakeholders are involved?

Systemic Innovation in Education

• Research evidence is interacting with various other forms of knowledge in innovation processes in education:– Professional/Practitioner knowledge

• Teachers’ core pedagogical knowledge and beliefs as part of their professional identity

– Tacit knowledge• Establishing a ‘culture of evaluation’ is

critically important for the success of reform and planning of next reforms

19

Conclusions

Systemic innovation as useful analytical framework – Targeted strategy to

induce system-wide change

SI as guiding principle for innovation policy

Need for formalised knowledge base– Losing innovation

opportunities

– Not cost effective

Establish a formalised knowledge base– Monitoring and evaluation

– Support link between systems research and innovation

– Evidence-informed dialogue with stakeholders

Policy Implications

21

MAKING REFORM HAPPEN IN EDUCATION

C.

22

23

Key lessons on education reform

• Making reform happen needs to:– Actively engage stakeholders in

formulating and implementing policy responses

– Make effective use of evidence to shape policies

– Explain clearly underlying principles and aims of reforms

(from: OECD Education Policy Committee meeting at CEO level Seoul, Korea, 2008)

24


• More specific lessons from reform experience:– Policy makers need to build consensus on the aims of

education reform and actively engage stakeholders, especially teachers, in formulating and implementing policy responses

– Reforms can capitalise on external pressures as part of building a compelling case for change

– All political players and stakeholders need to develop more realistic expectations about the pace and nature of reforms to improve outcomes

– Reforms need to be backed by sustainable financing– There is some shift away from reform initiatives per se

towards building self-adjusting systems with rich feedback at all levels, incentives to react and tools to strengthen capacities to deliver better outcomes

25


• More specific lessons from reform experience:– Investment is needed in change-management skills in

the education system– Teachers need reassurance that they will be given the

tools to change and recognition of their professional motivation to improve outcomes for their students

– Evidence can be used more effectively to guide policy making, combining international benchmarks with national surveys and with inspectorates to provide a better diagnosis

– Evidence is most helpful when it is fed back to institutions along with information and tools about how they can use the information to improve outcomes

– “Whole-of-government” approaches can include education in more comprehensive reforms. These need effective co-ordination and overall leadership across all the relevant ministries

26

Making reform happen

• Case studies:– Denmark: introducing a culture of

evaluation– Finland: introducing polytechnics into the

tertiary education sector– Portugal: tertiary education reform and

teacher education reform– Norway: improving lower secondary

education– Mexico: improving schools

(EDUCATION AND TRAINING POLICY division)

27


Mexico project video

http://www.youtube.com/watch?v=MN5nWa8N9TM

28


• Key elements relevant to Making Reform Happen in Mexico:– greater understanding among stakeholders of each

others’ roles and perspectives– greater commitment to work together on lifting

performance– stronger capacity for undertaking education policy

reform– strong Mexican ownership of recommendations– obstacles to reform identified and recommendations

realistic– long-term vision as well as immediate steps to take– OECD in facilitating role, bringing international

expertise to support Mexico’s reforms

29

THE ‘GPS’ APPROACHD.

30

Leveraging the impact of knowledge

• The intent is to mobilise and integrate evidence from data, analyses and policy advice generated over the years by the Education Directorate and to make this knowledge accessible in a systematic way

• This service to member countries is an analogue of a ‘GPS navigation system’ which provides real-time evidence to guide the choices and trade-offs when they need it for policy decisions related to education

• Like the GPS system, the search by any user at any time should provide selected, focused and actionable information on specific issues for policy decision making by countries. (INDICATORS AND ANALYSIS Division)

31

Leveraging the impact of knowledge

• Overall objectives of this service to countries is:– to provide a real time comprehensive

access EDU’s rich evidence base on educational outcomes and policies experience so that they can used for policy decisions by governments.

– to offer a means by which countries can assess the contribution of educational reform to the growth and progress of their country, recognizing that the full impact of educational reform is cumulative over the trajectory of policy decisions

Strand 1: Managing what we know

• A digital tool to ensure OECD evidence is at your disposal at your desk when you need it and in a form that is suitable and adapted to the specific policy context in which you need it.

• Three types of evidence – data, analyses and policy experience – will be available.

• Work will continue to create the digital tool and to move evidence from the last three years into the data base. It is estimated that the complete data base will contain over 100,000 individual pieces of evidence which can be searched.

• The tool will be available in 2013. Its utility is dependent on having a large enough data base for it to be useful to users. Further additions to the data base will continue adding to its value.

Strand 1: Managing what we know

• The Evidence Navigator for Education will be available on the OECD Education web site and on OECD iLibrary in early 2013.

• All evidence can be viewed at once, or evidence that is data, analysis or policy separately.

• The data base will have evidence extracted from all products generated by the Education Directorate (CERI, ETP, INES, IA, CELE, IMHE) and all the surveys (PISA, PIAAC, TALIS, etc) which can be searched.

• The extracted evidence includes both published and non-published information.

• The search results show text, tables, figures, powerpoint slide or video.

• If you want more information, the evidence will be linked to the original source at the extracted location. For instance, if you are interested in a figure, clicking on it will take you to the page in the book or report from which it is drawn.

• Here is a short “trailer”.

http://publications.oecd.org:8080/gps/gps-all-teacher.html

Strand 2: Peer-learning from policy experiences

• Peer-learning from evaluated policies in other countries that have achieved successful outcomes

• Policy experience analysis will be incorporated into the GPS data base

• First volume of Strong Performers, Successful Reformers was published December 2010.

• OECD – Japan Seminar June 28-29 2011 will include a second volume.

Strand 3: Building tools for successful policy implementation

• Successful policy implementation is essential for reforms that achieve targets, within budget and on schedule. Implementation tools and delivery capacity are useful in policy planning.

• When available, such tools for managing implementation and assessing delivery capacity will be incorporated into the GPS data base for use by countries.

• A pilot of the tool developed by McKinsey and Michael Barber in the UK has been proposed to member countries in order to test its appropriateness and value.

40

GOVERNANCE AND KNOWLEDGE

E.

41

Governance and Knowledge

• Governing Complex Education Systems is a new CERI project dealing with the governance challenges of increasingly complex education systems– Increasing number of actors and

stakeholders– Multilevel governance issues– Decentralisation and re-centralisation

• What is the role of knowledge in governance?

• What is the role of governance in knowledge creation, dissemination and utilisation?


• General research questions:

• What models of governance are effective in complex education systems?

• What knowledge systems are necessary to support this?


• More specifically:– How creating the capacity at central levels

to handle complex flows of knowledge– What knowledge options do policy makers

have in making decisions and involving stakeholders

– How to provide the local levels in complex systems with sufficient knowledge to perform

– How to ensure that levels do share relevant knowledge

Implementation

Knowledge production

AccountabilityPriority settingPolicy Design

Knowledge use

Steering

Governance model

45


46

CONCLUSIONS AND QUESTIONS

F.

47

Conclusions and Questions

• As most policy areas, educational policy has become more evidence-based in most countries– What kind of evidence counts in educational

policy and practice?– What is the role of empirical scientific

research?– What is the role of comparative education

indicators?– Is evidence shared and discussed with all

stakeholders before leading to policy decisions?

– Which channels do mediate between research, stakeholders and policy makers?

– How can we improve knowledge mediation?

48


• Innovations in education often lack a systemic approach, with a clear and knowledge-driven vision on implementation, scaling, monitoring and evaluation– What role do various kinds of knowledge

play in educational innovations?– Is knowledge resulting from evaluating

innovations used in designing new innovations?

– Is evidence from evaluations used to enrich the dialogue with practitioners and stakeholders?

49


• Educational systems have become more complex, partly as a result of decentralisation, partly as a result of multiplication of stakeholders. The governance of complex systems becomes a real challenge– Can research evidence and other knowledge

help in ‘binding’ the educational system? What are the functions knowledge can play in terms of governance?

– Under which conditions and in what forms should knowledge be developed, shared and discussed in order to have a productive impact on governance?

50

Thank you !

[email protected]/edu/ceri

http://www.oecd.org/edu/ceri

http://www.oecd.org/edu/ceri