Assessing the Distributional Assessing the Distributional Impact of Social ProgramsImpact of Social Programs

The World Bank

Public Expenditure Public Expenditure Analysis and Manage Analysis and Manage Core CourseCore Course

Presented by:Presented by:

Dominique van de WalleDominique van de WalleDECRGDECRG

March 21-24, 2005March 21-24, 2005Presented to:Presented to:

The World Bank

PEAM Core Course 2

The evaluation problem1. Who gains from the programs?

Who uses public services? At what cost?Who benefits from subsidies?Who are the target groups?How should transfers be allocated?

2. How much do they gain?Is there more poverty with or without a policy?How much impact will programs have on

poverty?

The World Bank

PEAM Core Course 3

The evaluation problem• Impact is the difference between the outcome indicator with the program and that without it.

• However, we can never simultaneously observe someone in two different states of nature

• So, while a post-intervention indicator is observed, its value in the absence of the program — the counterfactual — is not.

• The essential problem in evaluation is one of missing data on the counterfactual of what would have happened in the absence of the intervention

The World Bank

PEAM Core Course 4

• To measure ‘impacts’ rigorously we need ex-post impact evaluation techniques

Econometric: generally need baseline or panel data

Experimental: require randomized assignment

• Often we must instead turn to other “quick & dirty” approaches that examine spending "incidence

The most commonly used is Benefit Incidence Analysis (BIA)

The World Bank

PEAM Core Course 5

What is benefit incidence analysis?

Step 1: rank individuals by welfare indicator

Access to services   Step 2: identify usage/participation

Utilization   Step 3: attribute "gain" or benefit identified by unit cost of providing service

Incidence of spending

The World Bank

PEAM Core Course 6

STEP 1: Access to infrastructure in rural Vietnam

All

Non-poor

Poor

Passable road

70

75

67

Passenger transport 52 56 50 Electricity 43 47 41 Pipe-borne water 5 7 4 Post office 34 36 33 Upper sec. school 10 11 9 Dispensary 33 37 30 Ag extension office 24 27 22

(% rural population with the infrastructure)

The World Bank

PEAM Core Course 7

  Step 2: Participation in public works and a means-

tested credit subsidy in Maharashtra, India

0

2

4

6

8

10

12

14

16

18

0 100 200 300

% o

f ru

ral

ho

use

ho

lds

pa

rtic

ipa

tin

g

Consumption expenditure per person

The World Bank

PEAM Core Course 8

Step 3: A typical example of a benefit incidence analysis:

Health for the rich versus the poorHealth for the rich versus the poor

0

5

10

15

20

25

30

35

40

45

50

Cote d'Ivoire Ghana Guinea Kenya Madagascar Malawi Tanzania South Africa

Sh

are

of

tota

l su

bsi

dy

Poorest 20%

Richest 20%

Health spending in Kenya, 1992

0

20

40

60

80

100

0 20 40 60 80 100

Cumulative population

Cu

mul

ativ

e su

bsid

y/in

com

e

Primary

Hospital

All health

Income

The World Bank

PEAM Core Course 10

Advantages of traditional benefit incidence analysis…

Easy to do and to present (with caveats)  

Disadvantages and limitations… 

Strong assumptions Do not explain incidence outcomes No specific policy implications

The World Bank

PEAM Core Course 11

Traditional benefit incidence analysis may...

1. Wrongly assume that the cost of provision reflects the benefit to user

2. Be sensitive to method of ranking households in the original position

  spatial prices, comprehensiveness of welfareindicator, demographics

The welfare measure matters for primary education in Ghana

0

20

40

60

80

100

0 20 40 60 80 100

Cumulative population

Cu

mu

lati

ve s

ub

sid

y

Per capita expenditure

Adult equivalentexpenditure

How quintiles are defined matters for

health in Ghana

0

20

40

60

80

100

0 20 40 60 80 100

Cumulative population

Cu

mu

lati

ve s

ub

sid

y

Household quintiles

Population quintiles

The World Bank

PEAM Core Course 14

Traditional benefit incidence analysis may...

3. Mispecify the counterfactual: BIA ignores behavioral responses

  ex: figure of transfers in Yemen

• Conclusions about targeting & incidence depend on how the counterfactual is defined

Distribution of public and private transfers in Yemen, 1998, by deciles of per capita expenditures excluding transfers (annual YR per capita)

0

5000

10000

15000

20000

25000

30000

35000

1 2 3 4 5 6 7 8 9 10

Rural

Urban

National

Distribution of public and private transfers in Yemen, 1998, by deciles of per capita expenditures including transfers (annual YR per capita) 

0

2000

4000

6000

8000

10000

12000

1 2 3 4 5 6 7 8 9 10

rural

urban

national

The World Bank

PEAM Core Course 17

Traditional benefit incidence analysis

may... 4. Give an incomplete picture of welfare effects

how did other dimensions of welfare (eg healthliteracy, nutrition) improve as a result of subsidies?

5. Be unable to assess some important public goods and services

eg safe water, sanitation, vector control, physical infrastructure

The World Bank

PEAM Core Course 18

Traditional benefit incidence analysis

may... 6. Ignore general equilibrium & indirect

effects on poor

eg indirect benefits from tertiary education

7. Confound average and marginal incidence

Distribution of gains 

The World Bank

PEAM Core Course 19

7. Average versus marginal incidence

• Standard benefit incidence estimates the distribution of average incidence at one point in time

• This can be deceptive about how changes in public spending will be distributed

• Marginal incidence is an example of a behavioral incidence analysis where one measures the incidence of actual increases or cuts in program spending

The World Bank

PEAM Core Course 20

How will gains from social program expansion be distributed across

groups? • Non-poor often capture benefits of (even targeted) social

programs

information, incentive & political problems make perfect targeting hard

• But, program capture by non-poor can differ according to how costs & benefits of participation vary with program scale

eg fees, opportunity costs of time, transport costs etc 

The World Bank

PEAM Core Course 21

How will gains from social program expansion be distributed across

groups? Model 1: Early capture by non-poor: net gains to

non-poor are positive initially, but fall with program expansion

Model 2: Late capture by non-poor: cost initially too high for the non-poor, but net gains rise over time

Average participation rates may be deceptive for inferring how gains & losses from program expansion or contraction will be distributed.

The World Bank

PEAM Core Course 22

One way to identify marginal incidence is to compare incidence across geographic areas with different program sizes

 Marginal odds of participation (MOP) =

 increment to group-specific participation rate with a change in overall participation

 

The income group specific MOP is estimated by regressing income group specific participation rate across regions on the average rate for the region.

MOP shows incidence of a change in spending

The World Bank

PEAM Core Course 23

Average and Marginal Odds of Primary School enrollment, India

1993-94

Quintile Enrollment rate %

Average odds ofenrollment(mean=1.0)

Marginal odds ofenrollment

Poorest 37.2 0.71 1.10

2nd 48.6 0.90 0.97

3rd 55.8 1.08 0.87

4th 62.6 1.21 0.67

5th 67.7 1.31 0.67

Note: odds of enrollment = ratio of quintile-specific enrollment rate to the mean rate.

The World Bank

PEAM Core Course 24

In conclusion

• Extreme care is needed when interpreting average incidence and traditional benefit incidence analysis

  • Beware of reform recommendations based solely

on BIA and concentration curves as conventionally calculated.  

The World Bank

PEAM Core Course 25

Q & A