The Quality Schools Program, Mexico

1

Evidence-based Policy and Evidence-based Politics: State Structures of Implementation

and its Insertion in the Political Sphere

The Quality Schools Program, Mexico

Dra. Teresa Bracho, Mtra. Giulianna Mendieta

Center for Research and Teaching in Economics, CIDE. Mexico.

2

1. The context

What is PEC (Quality Schools Program) What is the role of evaluation and evidence

within the program. PEC as an Evidence-based Policy

3

The Quality Schools Program

PEC (for its name in Spanish Programa de Escuelas de Calidad), is a national strategy started in 2001 to reform management of public primary education in order to improve the teaching and learning conditions in public schools.

Started with c.2,200 schools, it is implemented now c. 29,000 of the c.250,000 basic public schools in the country.

4

The Quality Schools Program

It seeks to overcome the following difficulties: – The reduced margin of decision making at schools– Insufficient planning and evaluation at the States level– Inefficient administrative requirements– Scarce communication between school agents– Absenteeism (of children and teachers)– Inefficient use of resources– Low social participation – Deficiencies in infrastructure and equipment– And, essentially, to improve the learning conditions and

opportunities to learn in public schools.

5

Collectively assumes responsibility of learning and results.

Committed with school achievement Integrates all the school agents’ efforts toward a

school-project Has the basic resources to achieve its goals Guarantees that students acquire fundamental

knowledge and develop basic intellectual skills.

What is a “Quality School”?

6

The external evaluation of PEC

Relations between research and policy came from the policy officers interest, and not from the academic team. Therefore, research priorities were mainly product of a negotiated dialog between policy demand and knowledge demands.

This beginning is very important to take into account, since the research project has to build up from actual information and had to design the information necessities involved in the monitoring of the outcomes.

From an informed dialogue between research and policy makers, we moved to a productive relation in which evidence provided from the evaluation produced changes in the program, and results of the evaluation also produced changes in our conception of the capabilities of the policy

7

The external evaluation of PEC

The evaluation didn’t go from an ideal research model designed for an academic proposal, but from the immediate needs detected in the first evaluation and its analytical reconstruction of the model involved in the program, and encircled by the availability of information.

From there, a major evaluation program was developed for a six year scheme, and it was proposed to evaluate the impact of the program in the long run, not in the immediate years. Hopefully the impact of the evaluation will take place soon.

The evidence provided by the external evaluation was used not only for accountability purposes (as it is presented to the National Congress) but mainly to cement the program

8

EBP requirements within PEC

On the “P” part of the equation: political will to use evidence in order to

improve policy-making process Information the acceptance to be submitted to external

evaluation.

9

EBP requirements within PEC

On the “E” part of the equation: An academic sector willing to do applied

research (and to take the risk on lower “strict academic” production in order to produce evidence to be used in the decision-making process)

Access to information Acceptance of policy timing needs.

10

Information within PEC

At the beginning: There were no systematic sources of information

In consequence, PEC produced abundant information from all levels: Schools, State, Regional and National Coordinators.

But, this information was (and is) not always used, for reasons such as:

– Difficulties to manage and process information– Lagged / untimely for decision making purposes – Little experience in using information for implementation

11

The Quality Schools Program System of Information (SIPEC)

One of the most valuable achievements in terms of evidence,

its construction has resulted from a long and hard process of finding information, including it as pertinent to the program, accessing and mixing different sources of national information systems and, mainly, organizing its own demands of information from and to the schools

since these schools are in marginalized conditions, infrastructure and qualification had part of the generalization of SIPEC.

12

The Quality Schools Program System of Information (SIPEC)

starts to fully operate in this school cycle, it is a promising tool for the systemic and organized generation of evidence,

will have to be followed by a solid and rigorous analysis and research work that allows making sense, and coherence to the large amount of information to be produced.

13

PEC as EBP

It is a mixed model that had its origin in experience as well as theory,

It has confidence in the idea of using sound evidence in the policy-making process and a clear respect to the academic criteria of quality, relevance, and independence of the research team.

It has proposed, since its beginning, the use of evaluations as way to adjust the policy to the evidence of its effective results.

14

PEC as EBP

Strictu sensu the Program was not designed as an EBP, nor it is carried out abiding every requisite of what is called EBP

EBP, and its main proposals were constructed within the program as a need “in the road” of its implementation and induced by the evaluators conceptions of “sound evidence”

What here is presented intends to establish some of the elements of this type of orientations to relate them to PEC in terms of generation and use of information for educational planning, and assure the Programs continuity as a public policy.

This will lead to a better use of information and evaluation as evidence to be exploited as the basis for its bettering and for its scaling up and sustainability in the educational system.

15

Basic scheme for the development of an indicators system

Resources

InputsProcess Results

Objectives

Needs

RelevancePertinence

Impact

Internal efficiencyExternal efficiency

Equality

C o n t e x t o

Sufficiency/efficiency

C o n t e x t

Based on INEE, 2006.

16

An example of the use of evidence

The main example that can be used from the program is its experience in building a system of information for bettering off the results of the targeting strategy.

From the evidence provided by the evaluation, and the will to target the schools placed in marginalized communities, the short term of this process is shown in the next graphic:

17

Targeting process % of schools by marginality index of the

schools’ locality

41.4%

34.3%34.0%

30.8% 29.6%

24.1%23.5% 23.8%

15.0%

17.1%16.8%

17.4%

13.0%

22.0%23.0%

23.8%

1.0%2.5% 2.7%

4.2%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

Muy Bajo Bajo Medio Alto Muy Alto

Grado de Marginalidad

PEC 1 PEC 2 PEC 3 PEC 4

18

Part II. The use of evidence in the context of negotiations at the State level.

19

The Quality Schools State Coordination. Their experience with evidence

State Coordinators of the program have developed strategies to establish a system to provide evidence to different interest groups.

These interest groups can jeopardize the continuity or the magnitude of support to the Program.

Some of these stakeholders are political stances.

20

The Quality Schools Program in the States

State Coordinators have had to develop abilities of negotiation and political lobbying.

Negotiation and producing sound evidence are two of the main challenges to State Coordinators.

21

Evidence from where, to whom, how?

With which political stances are negotiation activities made?

In which level and what evidence is being produced?

Who produces it and how is this evidence produced?

To whom and on which topics is evidence produced?

22

Producing evidence: who produces it, how and on which topics?

In general terms, available evidence for State Coordinators comes from:

Gathered information by the planning and evaluation areas of the Ministries or Education Institutions,

results of the qualitative external evaluation, even though it is only directed to certain educational levels and to a school sample.

In some cases, by school supervisors, who are in charge of counseling schools.

23


Most information about the progress of PEC schools is through their model of self-evaluations;

from this process, a pedagogic and financial report is made,

it summarizes the level of achievement by school. It should be noted that this type of information and

evidence is particular to PEC, designed and developed by the program

24


These reports are sent to the State Coordinators at the end of each school year,

this stance is in charge of gathering information in data bases, and generating the corresponding analysis and interpretations,

Accordingly, the National Coordination has to integrate the evidence gathered into the National System of Information of PEC (named SIPEC).

SIPEC now operates on-line, so the school information is now direct from every agent in the program. And, most importantly, every agent can have access to the complete process of information and evidence generated by the program.

25

To who is evidence produced?

• The information gathered by the State Coordinators is analyzed and accounted in the formal reports to the National Coordination of the program and to the National Congress (via the external evaluation)

• But information also fulfill other purposes: • It is used to establish a complex net of negotiations within the state

ministries of education and with the state government

• These agencies will favor the Program’s continuity, or a lessened or larger presence of PEC in the educational environment.

• Evidence gathered by schools, supervisors, counselors, and academic and administrative team of the PEC, have been greatly useful to modify the operative strategy of the Program, which has been readjusted along the obtained results and the Programs growth.

26

The political stances of negotiation and their evidence requirements

In the States where less mobility of personnel of high offices at the Ministry of Education and State Coordinators, the labor of political convincement has flowed more easily.

In this framework of agreement, the accountability process, information and political convincement has achieved better results.

27


Two institutions have been decisive in the negotiation and decision making process. – The Technical Committees of State Funds (CTFEEC), – and the Executive Commissions.

These stances are formed by the representatives of the highest education authorities within the particular State (basic education, planning, financing, administration, evaluation, judiciary, social participation, compensatory programs, among others), and is presided by the highest state authority in education, which eases agreements and decision making.

28


When there are agenda issues that require the intervention of other government stances, federal or from the state (specially in terms of extra funding), it is the local Minister of Education who is in charge of these negotiations with the state or federal government.

However, PEC’s State Coordinators is in charge of generating all the relevant information for this purposes, so their role in achieving a good negotiation is central.

29


Changes in government that have affected the Program operation, have been those of Municipal Presidencies.

In these cases, it is the State Coordinators who are in charge of reestablishing agreements with these stances of government.

Negotiations have directly been with Municipal Presidents, Municipal Councils of Social Participation, Ministries of Social and Education Development, and others.

30


Support from Municipalities is owed to the work they do directly with the school principal and parents, organized through the School Councils of Social Participation.

The State Coordinators job is different by States, in some cases the relationship with municipal presidencies is led by State Coordinators, while in other cases it is mostly built by principals and parents, and finally, in others the State Coordinator serves just as a formal check-up of the relation.

31


State Coordinators have an agreement: party interest that could be present in the beginning of negotiations, is transformed during the first and second year of government, into an interest in education and in a support to the Program.

Also, an alignment between political interests and the Program’s objectives may had happen. As community is satisfied by PEC improvements in their school, municipal authorities can “claim their part” of the support, as political agreement.

32


• State Coordinators recognize that they have had to develop the following abilities: • in negotiation, agreements and dialogue, to

research and learn how the different systems and political stances work, in order to help new administrations on funding and supporting the Program.

• However, not every negotiation action has been successful.

33


Relationship of the Program with political parties has been null.

in every analyzed case, permanent interaction has been with:

– Institutions of the Executive Power of the State’s Government (Finance Ministry, State Internal Control, town halls, etc.)

– In some cases with the legislative power -with deputies, in order to inform on the PEC’s expenses, in particular municipalities, and also with the Teachers Union.

34

Conclusions

State Coordinators, in charge of the PEC’s implementation in the States, have experimented important advances and transformations, product of the organizational learning they have been exposed to.

One of them has been the construction and development of abilities of negotiation, dialogue and agreements with political stances, that, in some way, participate or influence in the decision making process of the Program.

35

Conclusions

it is possible to negotiate the Program’s continuity with political stances, from the production of evidence that is adequate to the political stances needs.

The most successful result of the negotiations has been the one obtained with municipalities that are participating in the Program through direct funding to schools and by supervising constructions, also by following-up the School-Projects agreed by schools in their jurisdiction.

36

Conclusions

The future of the Program in the medium term is seen as promising because of the World Bank’s commitment in financial and counseling support, and the growth perspectives planned.

The National and State teams of implementation, as well as schools that have experienced the Program since its beginnings, have offered important lessons and experience to start this second phase of the Program, with more certainties and clarity in relation to the processes and results expected to be achieved.

37

Conclusions

PEC is not a program that pretends that a small group of schools improve their standards of development; it is a Program that encourages change in schools, it offers lessons and experience to the whole education system.

Besides its valuable elements of design and implementation, as well as achieving better forms of action between federal government and states, we consider that the program is generating new forms of approaching well informed decision making, that improve these decisions at every level of education action, from central management to the schools.

38

Conclusions

An institutional design that articulates consistent systems of information, with a diminished complexity and with explicit channels that favor their flow,

inserted in implementation structures (State Coordinators and schools) with installed processes for building analysis and interpretation of evidence skills;

as well as skills for negotiation and dialogue with political stances under a framework of institutionalism and transparency, will favor considerably the Program’s continuity and its constant feedback that will lead to the achievement of expected results.

39

Teresa Bracho

Evidence Based Policies and Indicator Systems

University of Durham. London.

July, 2006

Documents

The Quality Schools Program, Mexico