Pay for performance initiatives a growing trend, but are they

effective? Josephine Borghi

London School of Hygiene & Tropical Medicine

Ifakara Health Institute

Objective

• Provide an overview of the challenges in measuring the effectiveness of PBF– PBF refers to a diverse set of interventions– Implementation may not match the ideal ‘vision’– Need a better understanding of the health system

impact of PBF– What kinds of evaluation should we be doing?– What is feasible and politically acceptable?

Performance based financing – A range of applications

• Based on who is the recipient of the bonus payment– Donor to government (aid modalities)– Within the public sector (facilities or districts / regions)– Government/donor to non-state provider– Health workers

• Process or outcome indicator/target• Payment method and size• When we say PBF is effective what do we mean?

Implementation – potential constraints in low income

settings• Design constraints

– Complexity versus simplicity – designing a simple system which is effective?

• Roll out constraints– Logistics

• Timely availability of funds• Appropriate management of information• Training of managers and health workers

• Monitoring constraints– Need for external validation - costly– Lack of reliable HMIS data

Tanzanian experience – vision versus reality

Ideal Vision for P4P Actual policy design

HMIS strengthened alongside P4P

HMIS not strengthened – no baseline

Flexibility in way incentives are structured by district

No flexibility – single system applied everywhere

Financial autonomy at facility level

Funds managed at district level

Variable targets based on baseline performance

Absolute targets irrespective of baseline performance

Phased implementation to allow to learn from experience and fine tune policy

National roll out in order to ensure rapid impact

PBF: Impact on the Health System?

•Potential Risks•Understanding the processes of change as well as the impact on health outcomes

Potential risks

• Impact on health worker behaviour– Supplier induced demand – unnecessary service

delivery – Quantity over quality– Opportunity cost – focus on targeted conditions

• Equity effects– Patients: Focus on easy to reach groups– Geographic: areas where health system stronger may

have advantage– Health worker migration

Need for improved evaluation design

Key Components of an Evaluation

Process Evaluation

What is the status of implementation? How does it vary by area?

What facilitates and impedes implementation?

Impact Evaluation

What changes take place that result in the observed impacts?

Economic Evaluation

What are the ‘positive’ effects of PBF on coverage of health services?

Are there any unintended consequences of PBF?

Is PBF a good use of resources?

Need for qualitative assessm

ent to determ

ine causal pathway

What are the health benefits of PBF?

An evaluation framework based on Tanzania

Positive Effects Negative Effects

Re-allocate resources

Improve quality of care / increased patient satisfaction

Increased utilisation of targeted health services

Reduced maternal and newborn mortality

Increased motivation and trust

Coercive strategies to increase utilisation

Mis-reporting performance

Crowding out of non-targeted health services / reduced quality of care

Damage intrinsic motivators

Recommendations for improved performance

Impact Evaluation Design Options

Probability design

Plausibility design

Adequacy design

Randomised Control

Strongest evidence of impact

Political acceptability

Non-randomised Control

Interrupted time series

Need to control for inherent

differences

Need good historical data

Historical or no control

Weakest evidence of impact

Practical challenges and need for innovative thinking

• Political resistance to controls• Randomisation may not be possible– Randomisation of performance contracts

(Rwanda)

• In absence of control or historical data importance of documenting the process

Summary• Need for better understanding of which aspects

of PBF are effective and pre-conditions for their effectiveness

• What is implemented may not match what was intended, need for simple and saleable models

• Some unanswered questions on health system impact

• Need for process evaluations and more impact evaluations within realm of political feasibility