Public sector reform – what’s the point of evaluation?

Philip KeeferInstitutions for Development DepartmentThe Inter-American Development Bank

ieGovern Impact Evaluation WorkshopIstanbul, TurkeyJanuary 27-30, 2015

: #ieGovern

PSR = pharmaceuticals?

• Our models for impact evaluation come from health.• Does a drug cure the illness?

• Problem: Precisely delineated• Treatment : Precisely delineated• Popular support for testing: Universal – nobody wants to be

poisoned, nobody wants to waste money on a drug that doesn’t work.

• Public sector reform has none of these characteristics.

Why not?• Dysfunctional public sector is a bundle of problems, not a single

illness.• Public functions of all kinds do not get performed.• Officials shirk/don’t care.• Money is diverted.• Citizens have no idea what is/is not being done on their behalf.

• Solutions are a bundle of treatments, not a single drug.• Civil service reform• Public sector financial reform• Reform of the justice sector• And each of these is, in turn, a bundle!

Our job – breaking things down

• What exactly should be different for public functions to be well-performed? • Results chain (processes and performance)

• Who needs to do what, from the moment legislation is passed all the way down to the service provider?

• Teacher needs to teach – but also needs to get paid and supervised; and teacher hiring needs to be based on merit; and school buildings need to be built and maintained.

• “Money chain” (follow the money)• The budgets to pay for these things need to be authorized; • We should be able to track the spending; • We should be able to see what was purchased.

Breaking down solutions. . .

• Break down the solution into digestible bits.• What should change along the results chain if the reform works?• How can we measure it?• Can we show ultimate effects on things that people care about?

• If we reform land administration (e.g., in Liberia!) :• Do national officials better enforce the law?• Are families more aware of what the law is?• Do land transactions increase, disputes fall, use of land shift to

higher valued crops, etc.?• Are households richer?• Do people feel that problems have been solved?

• How can we be sure that these good outcomes emerged because of new land law and administration?

It’s hard. . .why bother?

• Evaluating public sector reform MUCH harder than evaluating whether a drug works.• Molecules are easier to manage than people.• Why should we go to the effort? Who cares?• In fact:

• Politicians• Public officials• Citizens • And Reformers themselves!

• Why?

Making politicians care - 1

• Public sector reform is not politically popular ANYWHERE.• United States: who does public sector reform?

• The VICE president! • Presidents don’t bother (though not always the case!)

• Why?• Citizens don’t value PSR in and of itself.• Therefore politicians don’t, unless they know it allows them to

deliver things to citizens that they know citizens want.• No evidence that PSR does this – hence, political ambivalence.

Making politicians care - 2

• Evaluations of reform can help convince politicians that PSR is important.• Indonesia

• Politicians asked what they were getting for significantly higher wage bill.

• Created political support for impact evaluation.

• Pakistan/Punjab• Politicians cared about improving public services• Efforts to improve public official performance failed.• Needed to know what else would work.

Why might public officials welcome evaluation?

• Allow officials to convince politicians/public that they carry out the law and programs, serve citizens.• Evaluations are a sign of seriousness about reform.• They are also (depending on methodology) an opportunity for

public officials to feel like their opinions matter.

What about citizens?

• In general – they want services/responsiveness and don’t care about how government is organized to deliver.• But (like with public officials) major evaluations signal

seriousness.• And – depending on methodology – might find the evaluations

themselves to be compelling.• Don’t care about teacher hiring.• But care a lot that student test scores have risen after teacher

recruitment reformed.

Reformers and evaluation

• Development professionals (esp. economists, technocrats, “interventionists”) are still slow to grasp importance of governance/public sector.• Big problem with PSR is . . . little evidence about what works

best! • For example:

• Public sector reform is all about persuading public officials to do what’s in best interest of society.

• Hard problem. . . Intellectually, politically, operationally.• E.g.: what kind of HR system inspires greater effort by public

officials?• Lots of recommendations, none backed by rigorous evidence.

Revisiting pay for performance

• Oriana discussed yesterday . . . • She emphasized that some version of pay for performance is on

the radar screen of both economists and reformers.• Problem: few head to head comparisons of pay systems.• Sheheryar Banuri and I – use computer experiments on real

public and private sector subjects in Indonesia. • Compare pay for performance, pay for ability, flat pay.

Result: New fads not always better . . . .

• Pay for performance and pay for ability systems BOTH attract high ability and deter low ability individuals to a “public sector” task. • Pay for performance increases incumbent effort relative to both

flat pay and pay for ability. • However, in the “long run”, after selection effects have taken

hold, PFP and PFA are the same: both yield significantly higher effort than flat pay systems.• But PFA is cheaper!

And effects could be contingent

• Effects of pay systems depend strongly on • reservation wage of candidate pools (low or high relative to

public sector wage); • pro-sociality of the candidate pool (e.g., candidates for difficult

service positions). • task motivation (like the job?)

• Lab results tell us: we need to keep track of these other factors when we design reforms aimed at attracting and motivating the best and brightest.• Hard to track these in RCTs.• Hence: integrating lab experiments into RCTs in Burkina Faso

and Ecuador.

Conclusion

• Public sector reform is an area where we need to find out what works – ample scope and NEED for experimentation.• But also a potentially big payoff in terms of reform

sustainability/political support from evaluation.• Our hope for this workshop:

• Identify reforms that can be effectively evaluated.• Identify problems that reforms solve AND that we can measure

AND that politicians/citizens care about.